[00:01:11] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.326 second response time [00:01:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.75, 3.94, 3.97 [00:02:00] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.703 second response time [00:02:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:03:53] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.982 second response time [00:04:03] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.653 second response time [00:04:04] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.915 second response time [00:06:59] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 112 failures. Last run 6 minutes ago with 112 failures. Failed resources (up to 3 shown): File[wiki.simorgh.me],File[wiki.simorgh.me_private],File[popira.info],File[popira.info_private] [00:07:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 3.99, 3.94 [00:07:44] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 102 failures. Last run 6 minutes ago with 102 failures. Failed resources (up to 3 shown): File[patternarchive.online],File[patternarchive.online_private],File[www.puzzles.wiki],File[www.puzzles.wiki_private] [00:08:23] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [00:10:09] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:02] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.324 second response time [00:15:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.78, 3.92 [00:17:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.91, 3.93 [00:19:13] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:07] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:11] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:13] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:42] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:41] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:22:29] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [00:25:23] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.206 second response time [00:26:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.96, 4.00 [00:26:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/429852b61ba5...3b67ecce1466 [00:26:55] [02miraheze/mw-config] 07Universal-Omega 033b67ecc - Update LocalSettings.php [00:27:58] miraheze/mw-config - Universal-Omega the build passed. [00:28:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 4.22, 4.10 [00:28:49] !log [@mwtask111] starting deploy of {'config': True} to scsvg [00:28:51] !log [@test101] starting deploy of {'config': True} to skip [00:28:52] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:29:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:32] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 43s [00:30:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:39] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:31:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:45] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.367 second response time [00:33:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.59, 3.89 [00:36:10] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.834 second response time [00:36:25] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.437 second response time [00:36:36] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22194 bytes in 6.625 second response time [00:36:37] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.591 second response time [00:37:23] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:38:00] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:38:13] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:38:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:38:28] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:38:33] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:38:46] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:43:16] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [00:43:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:44:16] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:33] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.882 second response time [00:44:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.81, 3.81 [00:45:10] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 2.098 second response time [00:45:21] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.464 second response time [00:45:33] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.576 second response time [00:45:49] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.710 second response time [00:46:33] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.319 second response time [00:47:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.65, 3.96 [00:49:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.49, 3.69 [00:51:49] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:57] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:59] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:12] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:22] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:22] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:30] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:36] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.40, 3.55 [00:54:11] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:55:31] [02puppet] 07Universal-Omega opened pull request 03#2339: Add more to ImageMagick policy file - 13https://github.com/miraheze/puppet/pull/2339 [00:55:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.68, 3.64 [00:56:28] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.837 second response time [00:56:32] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.918 second response time [00:56:56] [02puppet] 07Universal-Omega synchronize pull request 03#2339: Add more to ImageMagick policy file - 13https://github.com/miraheze/puppet/pull/2339 [00:57:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.55, 3.70 [00:57:21] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.400 second response time [00:58:03] paladox: ^ can you please merge that? [00:58:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.69, 4.01, 3.80 [00:58:43] [02puppet] 07paladox closed pull request 03#2339: Add more to ImageMagick policy file - 13https://github.com/miraheze/puppet/pull/2339 [00:58:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/puppet/compare/064dd8c3c8a5...381f44c1e96c [00:58:46] [02miraheze/puppet] 07Universal-Omega 03381f44c - Add more to ImageMagick policy file (#2339) [00:59:04] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.462 second response time [00:59:22] Thanks! [01:01:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.99, 3.56, 3.66 [01:02:50] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.904 second response time [01:02:54] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.218 second response time [01:02:58] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.092 second response time [01:03:03] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.694 second response time [01:03:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.87, 3.82 [01:03:11] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.585 second response time [01:06:42] [02puppet] 07Universal-Omega opened pull request 03#2340: ImageMagick: limit memory usage to 128MB - 13https://github.com/miraheze/puppet/pull/2340 [01:07:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.08, 3.05, 3.39 [01:07:13] [02puppet] 07Universal-Omega edited pull request 03#2340: ImageMagick: limit memory usage to 128MB - 13https://github.com/miraheze/puppet/pull/2340 [01:07:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:07:36] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:07:42] paladox: ^ [01:08:01] [02puppet] 07paladox closed pull request 03#2340: ImageMagick: limit memory usage to 128MB - 13https://github.com/miraheze/puppet/pull/2340 [01:08:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/381f44c1e96c...1379f9469711 [01:08:04] [02miraheze/puppet] 07Universal-Omega 031379f94 - ImageMagick: limit memory usage to 128MB (#2340) [01:08:18] Thanks again! [01:12:05] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.92, 2.95, 3.36 [01:12:46] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [01:16:17] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:16:29] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:16:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.78, 3.56, 3.50 [01:16:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.99, 3.65, 3.57 [01:17:05] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:38] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:18:27] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.547 second response time [01:18:46] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.732 second response time [01:19:18] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.317 second response time [01:19:28] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [01:19:47] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.867 second response time [01:23:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.74, 3.60 [01:24:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.16, 9.23, 9.90 [01:25:03] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:25:05] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:25:16] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:25:17] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:25:21] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:26:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.92, 3.80, 3.65 [01:26:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.72, 3.94, 3.69 [01:26:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:29:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.95, 3.75 [01:31:08] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:32:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 4.24, 3.88 [01:32:12] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.15, 4.42, 3.96 [01:32:18] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:33:39] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.670 second response time [01:33:40] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.630 second response time [01:34:05] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [01:34:10] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.326 second response time [01:34:16] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.135 second response time [01:34:20] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.023 second response time [01:38:09] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:38:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [01:38:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.32, 9.22, 9.84 [01:39:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.70, 3.79 [01:40:32] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:40:32] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.799 second response time [01:40:59] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.87, 10.12, 10.09 [01:43:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.60, 9.72, 9.94 [01:44:01] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:44:45] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.84, 4.00 [01:46:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.81, 3.72, 3.96 [01:46:51] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.616 second response time [01:48:28] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.596 second response time [01:48:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.33, 9.36, 9.35 [01:50:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 2.84, 3.32 [01:52:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.47, 3.72 [01:54:04] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:54:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.85, 9.52, 9.45 [01:54:33] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:56:51] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [01:56:58] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 3.37, 3.54 [01:57:31] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:57:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [01:58:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.27, 8.99, 9.96 [01:58:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.41, 3.59 [02:00:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 15.22, 12.54, 10.71 [02:00:14] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.42, 3.06, 3.40 [02:01:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:01:33] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.58, 9.92, 10.08 [02:01:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 3.67, 3.62 [02:04:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.25, 2.97, 3.35 [02:06:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.62, 1.73, 1.46 [02:08:56] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.18, 1.11, 1.26 [02:09:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.46, 3.27, 3.30 [02:09:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.02, 8.94, 9.04 [02:12:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.30, 3.04, 3.23 [02:12:34] looks like the only open new infra task left is gluster (not sure what is left to do for that) [02:14:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.41, 9.65, 9.38 [02:15:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.67, 9.30, 10.00 [02:16:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 3.55, 3.38 [02:18:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.12, 9.74, 9.47 [02:18:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.81, 3.83, 3.46 [02:18:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.06, 4.11, 3.60 [02:19:04] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.64, 9.37, 9.84 [02:20:00] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:21:08] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:21:09] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.63, 9.65, 9.90 [02:21:14] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:19] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:21:22] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.245 second response time [02:21:52] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:21:53] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:21:56] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 3.690 second response time [02:22:14] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.58, 4.33, 3.62 [02:22:24] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:31] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:23:00] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:13] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.692 second response time [02:23:16] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:23:26] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.743 second response time [02:23:26] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.370 second response time [02:23:31] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.192 second response time [02:23:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3b67ecce1466...6fa5745c9e6d [02:23:42] [02miraheze/mw-config] 07Universal-Omega 036fa5745 - Set luasandbox maxLangCacheSize to 30 [02:23:58] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.984 second response time [02:24:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.93, 3.57 [02:24:26] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.796 second response time [02:24:35] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.944 second response time [02:24:42] miraheze/mw-config - Universal-Omega the build passed. [02:25:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.22, 10.64, 10.20 [02:26:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.77, 4.60, 3.86 [02:28:37] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.845 second response time [02:28:41] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.347 second response time [02:29:01] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:29:18] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.893 second response time [02:29:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/0d6ea52f64bb [02:29:28] [02miraheze/mw-config] 07Universal-Omega 030d6ea52 - Set shell limits [02:29:29] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [02:29:31] [02mw-config] 07Universal-Omega opened pull request 03#4388: Set shell limits - 13https://github.com/miraheze/mw-config/pull/4388 [02:30:06] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 2.647 second response time [02:30:37] miraheze/mw-config - Universal-Omega the build passed. [02:30:40] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.318 second response time [02:31:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0d6ea52f64bb...d152c203a0ff [02:31:08] [02miraheze/mw-config] 07Universal-Omega 03d152c20 - Update Defines.php [02:31:10] [02mw-config] 07Universal-Omega synchronize pull request 03#4388: Set shell limits - 13https://github.com/miraheze/mw-config/pull/4388 [02:31:20] [02mw-config] 07Universal-Omega edited pull request 03#4388: Set shell limits (on mw101 only for now) - 13https://github.com/miraheze/mw-config/pull/4388 [02:31:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:31:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.67, 3.97, 3.83 [02:31:56] [02mw-config] 07Universal-Omega closed pull request 03#4388: Set shell limits (on mw101 only for now) - 13https://github.com/miraheze/mw-config/pull/4388 [02:31:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6fa5745c9e6d...4f440e66894b [02:31:59] [02miraheze/mw-config] 07Universal-Omega 034f440e6 - Set shell limits (on mw101 only for now) (#4388) [02:32:01] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [02:32:02] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [02:32:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.66, 9.72, 9.90 [02:32:05] miraheze/mw-config - Universal-Omega the build passed. [02:32:09] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.824 second response time [02:32:45] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:32:57] miraheze/mw-config - Universal-Omega the build passed. [02:38:11] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.45, 9.14, 9.41 [02:39:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.02, 9.51, 9.89 [02:41:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.61, 9.21, 9.46 [02:41:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.48, 3.95 [02:44:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.90, 3.10, 3.37 [02:44:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.11, 10.68, 10.00 [02:45:59] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.19, 10.09, 9.99 [02:47:48] !log [@test101] starting deploy of {'config': True} to skip [02:47:49] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:48:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:48:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.67, 9.97, 9.97 [02:49:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:49:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:49:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 3.94, 3.97 [02:49:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.35, 3.47 [02:50:31] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 9.83, 10.19, 10.06 [02:52:05] !log [@mwtask111] starting deploy of {'config': True} to scsvg [02:52:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.33, 3.99, 3.67 [02:52:42] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 36s [02:52:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:53:20] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:53:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.72, 9.42, 9.78 [02:53:41] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:53:47] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:54:10] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:54:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:54:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.66, 3.83, 3.66 [02:56:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:56:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.40, 3.70, 3.87 [02:56:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.92, 3.84, 3.66 [02:57:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.26, 9.22, 9.89 [02:59:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.90, 9.06, 9.77 [03:00:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.20, 3.35, 3.52 [03:01:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.51, 9.65, 9.92 [03:01:47] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.733 second response time [03:02:17] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.841 second response time [03:02:26] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.571 second response time [03:02:40] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.157 second response time [03:02:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.18, 3.83 [03:03:16] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.427 second response time [03:03:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.42, 2.79, 3.26 [03:03:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4f440e66894b...250a4b530b48 [03:03:55] [02miraheze/mw-config] 07Universal-Omega 03250a4b5 - All servers [03:04:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.22, 9.46, 9.79 [03:04:50] miraheze/mw-config - Universal-Omega the build passed. [03:05:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.78, 10.05, 9.73 [03:05:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.59, 9.33, 9.50 [03:08:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.36, 8.92, 9.33 [03:10:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/250a4b530b48...c459f7328559 [03:10:04] [02miraheze/mw-config] 07Universal-Omega 03c459f73 - Lower $wgMaxShellMemory to 210MB [03:10:21] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.02, 9.28, 9.40 [03:10:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.95, 2.63, 3.27 [03:11:03] miraheze/mw-config - Universal-Omega the build passed. [03:11:21] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.97, 9.31, 9.48 [03:12:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.85, 9.48, 9.47 [03:12:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.91, 2.66, 3.25 [03:13:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/c6df020c92a0 [03:13:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.64, 8.97, 9.36 [03:13:21] [02miraheze/mw-config] 07Universal-Omega 03c6df020 - Set $wgShellCgroup for use of control group [03:13:22] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [03:13:32] [02mw-config] 07Universal-Omega opened pull request 03#4389: Set $wgShellCgroup - 13https://github.com/miraheze/mw-config/pull/4389 [03:13:50] [02mw-config] 07Universal-Omega edited pull request 03#4389: Set $wgShellCgroup - 13https://github.com/miraheze/mw-config/pull/4389 [03:14:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.84, 9.79, 9.57 [03:14:34] miraheze/mw-config - Universal-Omega the build passed. [03:15:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.32, 3.38 [03:16:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.01, 9.83, 9.92 [03:16:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.01, 9.32, 9.42 [03:16:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.68, 9.74, 10.00 [03:16:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.41, 3.46 [03:16:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:17:10] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.13, 3.13, 3.30 [03:17:49] !log [@test101] starting deploy of {'config': True} to skip [03:17:50] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [03:18:24] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:18:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:35] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:19:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.38, 10.15, 10.13 [03:19:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:19:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.20, 3.97, 3.64 [03:19:19] [02puppet] 07Universal-Omega opened pull request 03#2341: Support cgroup - 13https://github.com/miraheze/puppet/pull/2341 [03:21:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.74, 3.53, 3.53 [03:21:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.88, 3.38 [03:22:03] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:06] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:27] !log [@mwtask111] starting deploy of {'config': True} to scsvg [03:22:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [03:22:51] [02puppet] 07Universal-Omega synchronize pull request 03#2341: Support cgroup - 13https://github.com/miraheze/puppet/pull/2341 [03:22:57] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 29s [03:23:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.94, 4.33, 3.81 [03:23:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:44] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.513 second response time [03:23:51] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.316 second response time [03:24:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.34, 9.52, 9.88 [03:24:29] [02puppet] 07Universal-Omega synchronize pull request 03#2341: Support cgroup - 13https://github.com/miraheze/puppet/pull/2341 [03:24:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:24:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.24, 3.12, 3.17 [03:24:53] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.109 second response time [03:24:54] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.460 second response time [03:25:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:25:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.02, 3.40, 3.53 [03:27:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.23, 9.79, 9.93 [03:27:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.04, 2.80, 3.22 [03:27:30] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:27:59] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.41, 7.88, 8.48 [03:28:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.09, 3.14, 3.40 [03:29:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.54, 9.48, 9.38 [03:30:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.62, 9.66, 9.86 [03:31:05] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:31:50] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:32:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [03:33:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.56, 8.02, 8.83 [03:33:59] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.546 second response time [03:34:46] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.039 second response time [03:35:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:35:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.76, 11.14, 10.33 [03:36:23] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [03:36:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.71, 9.15, 10.00 [03:37:47] [02puppet] 07Universal-Omega synchronize pull request 03#2341: Support cgroup - 13https://github.com/miraheze/puppet/pull/2341 [03:37:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.53, 9.12, 9.04 [03:38:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.75, 10.47, 10.37 [03:39:59] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.21, 9.20, 8.81 [03:40:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.50, 9.65, 9.50 [03:42:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.83, 9.38, 9.40 [03:43:26] [02puppet] 07Universal-Omega synchronize pull request 03#2341: Support cgroup - 13https://github.com/miraheze/puppet/pull/2341 [03:44:13] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.98, 9.50, 9.07 [03:46:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.34, 9.63, 9.16 [03:47:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.31, 9.83, 9.57 [03:49:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.19, 9.08, 9.34 [03:50:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.98, 9.38, 9.17 [03:50:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.74, 9.89, 9.63 [03:51:22] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:52] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.728 second response time [03:57:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.16, 9.29, 9.31 [03:59:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.24, 2.97 [04:00:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.46, 9.36, 9.33 [04:03:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.61, 2.90, 2.90 [04:03:28] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:06:06] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.21, 9.48, 9.35 [04:07:12] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [04:09:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.45, 8.98, 9.18 [04:09:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.67, 8.85, 9.63 [04:12:47] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:12:49] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:12:57] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:13:27] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:14:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.50, 8.73, 8.73 [04:15:19] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:15:59] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.059 second response time [04:16:01] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.272 second response time [04:16:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.89, 3.53, 3.34 [04:16:24] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.331 second response time [04:16:56] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.198 second response time [04:17:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.23, 8.58, 8.63 [04:18:48] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 2.616 second response time [04:19:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.89, 3.56 [04:19:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.98, 3.45 [04:22:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.25, 3.47, 3.33 [04:25:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.08, 3.20 [04:25:43]