[00:03:11] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=mogaiwiki --username-prefix=wikia:ezgender --report=1 --no-updates /home/universalomega/ezgender_pages_full.xml (START) [00:03:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:04:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.50, 7.69, 8.44 [00:05:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.45, 4.02, 3.73 [00:08:16] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 445 failures. Last run 7 minutes ago with 445 failures. Failed resources (up to 3 shown): File[/etc/powerdns/recursor.conf],File[/usr/local/bin/gen_fingerprints],File[/usr/local/bin/logsalmsg],File[/etc/vim/vimrc.local] [00:09:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.52, 9.14, 9.81 [00:10:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.11, 8.60, 9.52 [00:11:13] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=2b2twiki (END - exit=0) [00:11:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:11:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 4.87, 4.04 [00:12:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.80, 4.98, 4.26 [00:12:16] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.16, 8.93, 9.55 [00:13:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.61, 9.22, 9.57 [00:18:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.02, 3.78, 3.84 [00:18:11] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.32, 8.63, 8.43 [00:18:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.30, 9.87, 9.90 [00:19:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.52, 8.73, 9.35 [00:21:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 4.02, 3.91 [00:21:14] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.57, 7.32, 7.96 [00:21:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.70, 3.98 [00:24:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.34, 3.22, 3.61 [00:24:51] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.62, 8.77, 8.66 [00:26:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/04b79c09dcef [00:26:27] [02miraheze/mw-config] 07Universal-Omega 0304b79c0 - depool mem101 [00:26:28] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [00:26:30] [02mw-config] 07Universal-Omega opened pull request 03#4429: depool mem101 - 13https://github.com/miraheze/mw-config/pull/4429 [00:27:13] [02mw-config] 07Universal-Omega closed pull request 03#4429: depool mem101 - 13https://github.com/miraheze/mw-config/pull/4429 [00:27:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b571973ce431...3b01b79f4d92 [00:27:16] [02miraheze/mw-config] 07Universal-Omega 033b01b79 - depool mem101 (#4429) [00:27:17] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [00:27:19] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [00:27:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4429-Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/4819ecb0250c [00:27:22] [02miraheze/mw-config] 07Universal-Omega 034819ecb - Revert "depool mem101 (#4429)" [00:27:24] [02mw-config] 07Universal-Omega created branch 03revert-4429-Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [00:27:24] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [00:27:25] [02mw-config] 07Universal-Omega opened pull request 03#4430: Revert "depool mem101" - 13https://github.com/miraheze/mw-config/pull/4430 [00:27:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:33] miraheze/mw-config - Universal-Omega the build passed. [00:27:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.19, 4.12, 3.88 [00:27:42] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 18s [00:27:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.20, 9.27, 8.91 [00:27:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:58] !log restart mem101 [00:28:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:22] miraheze/mw-config - Universal-Omega the build passed. [00:28:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.43, 4.13, 4.00 [00:28:27] miraheze/mw-config - Universal-Omega the build passed. [00:28:34] [02mw-config] 07Universal-Omega closed pull request 03#4430: Revert "depool mem101" - 13https://github.com/miraheze/mw-config/pull/4430 [00:28:36] [02mw-config] 07Universal-Omega deleted branch 03revert-4429-Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [00:28:37] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4429-Universal-Omega-patch-3 [00:28:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3b01b79f4d92...552c30bdc1ad [00:28:40] [02miraheze/mw-config] 07Universal-Omega 03552c30b - Revert "depool mem101 (#4429)" (#4430) [00:28:41] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [00:28:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:58] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 16s [00:29:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/03ba9f1b03c8 [00:29:32] [02miraheze/mw-config] 07Universal-Omega 0303ba9f1 - depool mem121 [00:29:33] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [00:29:35] [02mw-config] 07Universal-Omega opened pull request 03#4431: depool mem121 - 13https://github.com/miraheze/mw-config/pull/4431 [00:29:40] miraheze/mw-config - Universal-Omega the build passed. [00:30:20] [02mw-config] 07Universal-Omega closed pull request 03#4431: depool mem121 - 13https://github.com/miraheze/mw-config/pull/4431 [00:30:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/552c30bdc1ad...9c6c72ae1290 [00:30:23] [02miraheze/mw-config] 07Universal-Omega 039c6c72a - depool mem121 (#4431) [00:30:25] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [00:30:26] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [00:30:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4431-Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/6d79e2a8a50b [00:30:28] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [00:30:29] [02miraheze/mw-config] 07Universal-Omega 036d79e2a - Revert "depool mem121 (#4431)" [00:30:31] [02mw-config] 07Universal-Omega created branch 03revert-4431-Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [00:30:32] [02mw-config] 07Universal-Omega opened pull request 03#4432: Revert "depool mem121" - 13https://github.com/miraheze/mw-config/pull/4432 [00:30:45] miraheze/mw-config - Universal-Omega the build passed. [00:30:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 4.00, 3.87 [00:30:57] !log restart mem121 [00:30:57] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 28s [00:30:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:15] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [00:31:22] [02mw-config] 07Universal-Omega closed pull request 03#4432: Revert "depool mem121" - 13https://github.com/miraheze/mw-config/pull/4432 [00:31:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9c6c72ae1290...fe38c5dca401 [00:31:24] miraheze/mw-config - Universal-Omega the build passed. [00:31:25] [02miraheze/mw-config] 07Universal-Omega 03fe38c5d - Revert "depool mem121 (#4431)" (#4432) [00:31:27] [02mw-config] 07Universal-Omega deleted branch 03revert-4431-Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [00:31:28] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4431-Universal-Omega-patch-3 [00:31:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:31] miraheze/mw-config - Universal-Omega the build passed. [00:31:32] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [00:31:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:56] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 24s [00:32:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:31] miraheze/mw-config - Universal-Omega the build passed. [00:34:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 4.08, 3.93 [00:36:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.41, 7.17, 8.27 [00:37:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.71, 3.80 [00:37:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.37, 3.52, 3.98 [00:37:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.14, 8.23, 9.64 [00:38:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.74, 3.92 [00:41:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 4.27, 6.40, 7.74 [00:41:26] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:41:58] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:42:05] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:42:18] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.254 second response time [00:42:33] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.012 second response time [00:42:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 4.83, 7.13, 9.25 [00:42:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:43:32] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:43:33] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:43:38] [02mw-config] 07AgentIsai synchronize pull request 03#4425: Add wgWikiSeoDefaultImage - 13https://github.com/miraheze/mw-config/pull/4425 [00:43:39] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:43:42] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:33] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:45] miraheze/mw-config - AgentIsai the build passed. [00:44:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.41, 2.79, 3.35 [00:45:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.02, 2.41, 3.26 [00:45:14] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.380 second response time [00:45:23] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.251 second response time [00:45:23] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 4.70, 6.07, 8.07 [00:45:39] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:45:46] [02mw-config] 07AgentIsai synchronize pull request 03#4425: Add wgWikiSeoDefaultImage - 13https://github.com/miraheze/mw-config/pull/4425 [00:45:49] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:46:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.50, 6.93, 8.20 [00:46:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.66, 2.76, 3.35 [00:46:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.49, 5.78, 8.24 [00:46:47] miraheze/mw-config - AgentIsai the build passed. [00:46:51] CosmicAlpha: ^ [00:46:52] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:47:24] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:15] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:27] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:49:10] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:49:12] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.644 second response time [00:49:40] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.728 second response time [00:49:41] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.122 second response time [00:49:42] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw102 [00:49:59] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.775 second response time [00:50:12] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.255 second response time [00:50:30] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.499 second response time [00:50:55] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.860 second response time [00:51:17] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.067 second response time [00:51:25] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.674 second response time [00:51:27] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.103 second response time [00:51:44] [02mw-config] 07Universal-Omega closed pull request 03#4425: Add wgWikiSeoDefaultImage - 13https://github.com/miraheze/mw-config/pull/4425 [00:51:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/fe38c5dca401...c6bc3ca481ac [00:51:47] [02miraheze/mw-config] 07AgentIsai 03c6bc3ca - Add wgWikiSeoDefaultImage (#4425) [00:52:21] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.747 second response time [00:52:45] miraheze/mw-config - Universal-Omega the build passed. [00:52:55] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [00:53:02] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/b974055ea353...ef1a891a7cf0 [00:53:04] [02miraheze/mediawiki] 07Universal-Omega 03ef1a891 - Update ManageWiki [00:53:29] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [00:54:35] !log [@mwtask111] starting deploy of {'config': True} to scsvg [00:54:55] Thanks! [00:55:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.11, 8.75, 8.62 [00:55:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.24, 5.07, 4.32 [00:56:02] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.066 second response time [00:56:06] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.029 second response time [00:56:13] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 98s [00:58:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 4.17, 3.74 [00:58:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:59:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.82, 7.93, 8.28 [00:59:38] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:00:11] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:00:50] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:01:02] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.341 second response time [01:01:04] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:01:16] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:01:29] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [01:01:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.21, 3.49, 3.58 [01:01:55] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:04:57] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.660 second response time [01:05:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.33, 9.01, 8.44 [01:05:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 4.05, 3.70 [01:05:37] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:05:44] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.407 second response time [01:05:49] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:06:47] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.769 second response time [01:06:56] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.799 second response time [01:07:19] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.496 second response time [01:07:57] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.795 second response time [01:07:57] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [01:08:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.29, 9.11, 8.83 [01:08:12] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.466 second response time [01:08:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:09:04] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.993 second response time [01:12:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.80, 3.75 [01:12:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.49, 7.74, 8.11 [01:12:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.43, 8.98, 7.87 [01:16:10] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:16:19] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:16:20] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:16:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.65, 8.07, 7.73 [01:16:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.62, 2.97, 3.37 [01:17:05] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:06] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:17:31] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [01:17:37] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:50] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:55] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw121 mw122 [01:17:58] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.274 second response time [01:18:01] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:18:14] !log [@test101] starting deploy of {'config': True} to skip [01:18:15] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:18:25] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:18:30] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.243 second response time [01:18:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:19:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.71, 7.52, 8.22 [01:19:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.61, 2.75, 3.34 [01:19:57] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.773 second response time [01:19:59] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.076 second response time [01:21:29] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.120 second response time [01:22:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.04, 4.00 [01:22:15] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:23:41] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.637 second response time [01:24:17] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw102 [01:24:21] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw102 [01:24:32] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.014 second response time [01:24:56] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.044 second response time [01:25:48] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.253 second response time [01:26:21] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw102 [01:27:47] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=mogaiwiki --username-prefix=wikia:ezgender --report=1 --no-updates /home/universalomega/ezgender_pages_full.xml (END - exit=0) [01:27:58] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 11 backends are healthy [01:28:15] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=mogaiwiki (START) [01:29:17] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:29:19] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:29:42] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:29:51] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:15] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 11 backends are healthy [01:30:20] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:35] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:31:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:31:38] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 11 backends are healthy [01:31:41] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.492 second response time [01:31:47] [02puppet] 07Universal-Omega opened pull request 03#2382: jobrunner: add low-priority jobs - 13https://github.com/miraheze/puppet/pull/2382 [01:31:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.90, 8.13, 8.23 [01:32:22] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [01:32:59] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.015 second response time [01:33:30] [02puppet] 07Universal-Omega synchronize pull request 03#2382: jobrunner: add low-priority jobs - 13https://github.com/miraheze/puppet/pull/2382 [01:33:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.91, 8.86, 8.21 [01:35:19] Miraheze is down? curious that I only get this error in one wiki 502... [01:35:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.68, 8.23, 8.25 [01:35:36] 502 Bad Gateway* [01:35:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:35:50] which wiki? [01:36:15] Hispano76: ^ [01:36:17] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.668 second response time [01:36:23] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.395 second response time [01:36:29] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.440 second response time [01:36:36] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.542 second response time [01:36:41] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.552 second response time [01:36:42] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.133 second response time [01:36:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.86, 2.81, 3.31 [01:36:51] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.269 second response time [01:36:54] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.576 second response time [01:36:59] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.348 second response time [01:37:12] Lots of people have been reporting issues on #general since the morning [01:37:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.39, 9.15, 8.42 [01:37:32] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.975 second response time [01:37:35] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 2.545 second response time [01:38:36] CosmicAlpha, in hispano67wiki, is intermintent [01:38:54] Agent, oh [01:39:14] I am looking into the issues [01:39:17] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.328 second response time [01:39:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:41:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.07, 10.39, 8.78 [01:41:33] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:42:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.91, 3.87, 3.20 [01:42:41] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.452 second response time [01:44:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.79, 3.42 [01:44:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 4.30, 3.91 [01:45:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.66, 3.23 [01:47:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.96, 3.33, 3.30 [01:48:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.08, 3.35, 3.18 [01:49:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+1/-0/±0] 13https://github.com/miraheze/mw-config/commit/d8938fbaaa8d [01:49:27] [02miraheze/mw-config] 07Universal-Omega 03d8938fb - Create JobRunner.php [01:49:28] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [01:49:30] [02mw-config] 07Universal-Omega opened pull request 03#4433: Create JobRunner.php - 13https://github.com/miraheze/mw-config/pull/4433 [01:50:29] miraheze/mw-config - Universal-Omega the build passed. [01:50:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.77, 9.36, 8.98 [01:51:07] [02ManageWiki] 07lens0021 commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1019393812 [01:52:21] [02ManageWiki] 07lens0021 edited a comment on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1019393812 [01:52:56] [02ManageWiki] 07lens0021 synchronize pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [01:53:24] [02ManageWiki] 07lens0021 reviewed pull request 03#329 commit - 13https://github.com/miraheze/ManageWiki/pull/329#discussion_r790203959 [01:53:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.62, 9.97, 9.24 [01:53:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.67, 3.82 [01:54:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.92, 3.58 [01:55:20] [02ManageWiki] 07lens0021 commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1019394418 [01:56:46] [02mw-config] 07Universal-Omega edited pull request 03#4433: Create JobRunner.php - 13https://github.com/miraheze/mw-config/pull/4433 [01:57:18] miraheze/ManageWiki - lens0021 the build passed. [01:57:32] [02mw-config] 07Universal-Omega closed pull request 03#4433: Create JobRunner.php - 13https://github.com/miraheze/mw-config/pull/4433 [01:57:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/mw-config/compare/c6bc3ca481ac...c36d6407edf6 [01:57:35] [02miraheze/mw-config] 07Universal-Omega 03c36d640 - Create JobRunner.php (#4433) [01:57:37] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [01:57:38] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [01:58:35] miraheze/mw-config - Universal-Omega the build passed. [01:59:00] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to skip [01:59:02] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 1s [01:59:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:59:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.51, 9.42, 9.28 [02:00:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.44, 5.13, 4.30 [02:01:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.00, 8.52, 8.01 [02:03:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.66, 10.62, 9.79 [02:03:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.57, 3.83, 3.80 [02:04:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.72, 3.53 [02:04:11] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.06, 7.28, 7.63 [02:06:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.19, 4.06, 3.94 [02:09:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.08, 8.64, 8.43 [02:10:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.86, 3.90 [02:11:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:11:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.21, 10.99, 9.47 [02:12:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.96, 8.88, 8.22 [02:12:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.97, 9.92, 8.98 [02:13:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.02, 3.95, 3.90 [02:13:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 3.60, 3.51 [02:15:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.23, 9.58, 9.02 [02:16:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.84, 3.86 [02:16:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.25, 3.04, 3.31 [02:19:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.81, 4.10, 3.94 [02:20:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4433-Universal-Omega-patch-3 [+0/-1/±0] 13https://github.com/miraheze/mw-config/commit/b22602252743 [02:20:46] [02miraheze/mw-config] 07Universal-Omega 03b226022 - Revert "Create JobRunner.php (#4433)" [02:20:48] [02mw-config] 07Universal-Omega created branch 03revert-4433-Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [02:20:49] [02mw-config] 07Universal-Omega opened pull request 03#4434: Revert "Create JobRunner.php" - 13https://github.com/miraheze/mw-config/pull/4434 [02:20:53] [02mw-config] 07Universal-Omega closed pull request 03#4434: Revert "Create JobRunner.php" - 13https://github.com/miraheze/mw-config/pull/4434 [02:20:55] [02mw-config] 07Universal-Omega deleted branch 03revert-4433-Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [02:20:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/mw-config/compare/c36d6407edf6...e4d728fe3977 [02:20:58] [02miraheze/mw-config] 07Universal-Omega 03e4d728f - Revert "Create JobRunner.php (#4433)" (#4434) [02:20:59] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4433-Universal-Omega-patch-3 [02:21:33] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.31, 8.35, 8.33 [02:22:15] miraheze/mw-config - Universal-Omega the build passed. [02:23:55] miraheze/mw-config - Universal-Omega the build passed. [02:24:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.64, 3.50 [02:25:35] [02puppet] 07Universal-Omega opened pull request 03#2383: jobrunner: remove empty statsd - 13https://github.com/miraheze/puppet/pull/2383 [02:25:57] [02puppet] 07Universal-Omega synchronize pull request 03#2383: jobrunner: remove empty statsd - 13https://github.com/miraheze/puppet/pull/2383 [02:27:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.23, 4.05, 3.66 [02:28:40] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.80, 9.59, 9.22 [02:29:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.25, 9.32, 8.79 [02:29:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.53, 3.57, 3.95 [02:31:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.95, 3.37, 3.45 [02:32:05] !log [universalomega@mwtask111] starting deploy of {'world': True} to scsvg [02:32:29] !log [universalomega@test101] starting deploy of {'world': True} to skip [02:32:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.84, 3.99 [02:32:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:35:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.55, 8.88, 9.16 [02:36:11] !log [universalomega@test101] finished deploy of {'world': True} to skip - SUCCESS in 221s [02:36:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 4.15, 3.77 [02:40:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.98, 3.78 [02:41:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.85, 10.04, 9.41 [02:41:54] !log [universalomega@mwtask111] finished deploy of {'world': True} to scsvg - SUCCESS in 588s [02:42:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.66, 3.94 [02:44:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.46, 9.56, 9.32 [02:44:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.22, 3.36, 3.89 [02:46:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 3.85, 3.73 [02:47:48] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.77, 9.99, 9.51 [02:47:54] !log [@test101] starting deploy of {'config': True} to skip [02:47:55] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:48:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:48:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:48:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.96, 4.16, 4.07 [02:49:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.58, 3.27, 3.53 [02:50:55] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.63, 9.15, 8.84 [02:51:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.80, 9.75, 9.49 [02:54:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.97, 9.05, 8.87 [02:54:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.85, 3.97 [02:55:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.02, 3.01, 3.34 [02:58:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.76, 3.90 [03:00:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.30, 9.08, 8.90 [03:04:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 4.00, 3.95 [03:05:57] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.44, 9.33, 9.07 [03:07:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.83, 3.87 [03:09:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.03, 8.76, 8.87 [03:10:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.63, 3.80 [03:12:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.76, 3.61 [03:12:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.11, 9.32, 9.09 [03:13:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.42, 3.36 [03:15:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.31, 2.97, 3.32 [03:16:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.29, 2.90, 3.16 [03:16:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.41, 4.08, 3.89 [03:17:06] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.71, 1.73, 1.99 [03:17:56] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.45, 8.99, 8.90 [03:18:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.67, 10.67, 9.69 [03:18:48] [02ManageWiki] 07lens0021 commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1019403814 [03:20:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.14, 9.46, 9.90 [03:20:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.16, 8.86, 8.88 [03:23:06] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.86, 2.19, 2.07 [03:25:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.62, 9.93, 9.69 [03:26:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.62, 9.80, 9.24 [03:28:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.50, 3.70 [03:29:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.08, 9.32, 9.15 [03:30:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.10, 10.12, 9.80 [03:31:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.82, 10.29, 9.86 [03:35:24] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.39, 1.83, 2.00 [03:38:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.32, 3.22 [03:39:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.37, 9.72, 9.45 [03:41:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.81, 3.72 [03:41:38] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.83, 3.30, 3.24 [03:42:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.99, 9.66, 9.48 [03:42:50] [02puppet] 07Universal-Omega opened pull request 03#2384: Remove vmtouch - 13https://github.com/miraheze/puppet/pull/2384 [03:43:56] [02puppet] 07Universal-Omega synchronize pull request 03#2384: Remove vmtouch - 13https://github.com/miraheze/puppet/pull/2384 [03:44:08]