[00:00:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.06, 21.21, 20.87 [00:01:13] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 28.32, 22.78, 21.12 [00:01:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.09, 3.76, 3.58 [00:02:26] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 3744s [00:02:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.58, 21.36, 20.96 [00:03:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.75, 22.28, 21.15 [00:03:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:21] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:04] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 321 failures. Last run 3 minutes ago with 321 failures. Failed resources (up to 3 shown): File[repository.archiopedia.org_private],File[sarovia.graalmilitary.com],File[sarovia.graalmilitary.com_private],File[wiki.creaturathegame.com] [00:04:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.85, 23.03, 21.62 [00:05:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.27, 7.46, 7.96 [00:05:28] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.343 second response time [00:07:14] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 27.83, 22.53, 21.34 [00:07:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.26, 8.24, 8.20 [00:07:20] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 4034s [00:08:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:08:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.31, 23.71, 22.31 [00:10:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.32, 25.54, 23.14 [00:11:04] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:11:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.98, 21.17, 21.11 [00:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.84, 3.67, 3.76 [00:12:42] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:34] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:40] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:42] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:55] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:40] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.07, 5.45, 4.98 [00:15:14] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.218 second response time [00:15:36] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.614 second response time [00:17:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.25, 19.18, 20.25 [00:18:28] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.90, 2.98, 2.25 [00:18:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.68, 6.47, 5.46 [00:20:28] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.26, 3.07, 2.37 [00:20:54] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.259 second response time [00:21:47] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.382 second response time [00:21:52] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:01] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.048 second response time [00:22:05] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.453 second response time [00:22:11] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.007 second response time [00:22:33] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:39] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:48] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:59] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 7.919 second response time [00:23:06] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:11] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.376 second response time [00:23:20] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:32] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:50] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 1.386 second response time [00:23:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.69, 3.60, 3.58 [00:24:34] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 4.195 second response time [00:24:37] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 2.663 second response time [00:24:47] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.840 second response time [00:25:23] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.345 second response time [00:25:35] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.442 second response time [00:27:13] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:13] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:16] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.771 second response time [00:27:17] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:27] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.71, 3.62 [00:27:59] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:29:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.15, 7.00, 7.90 [00:29:25] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.220 second response time [00:29:25] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.040 second response time [00:29:59] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.589 second response time [00:30:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.85, 1.28, 0.77 [00:31:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.64, 7.33, 7.99 [00:32:18] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:28] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.01, 1.18, 0.80 [00:33:50] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:52] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.55, 3.53 [00:34:25] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:35:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.31, 3.45 [00:36:01] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.208 second response time [00:36:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.50, 7.04, 7.95 [00:36:29] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 5.929 second response time [00:37:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.73, 7.52, 7.69 [00:37:32] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.536 second response time [00:37:38] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:37:40] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:37:43] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.732 second response time [00:37:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.57, 8.57, 8.23 [00:37:54] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.647 second response time [00:38:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.13, 7.95, 8.18 [00:38:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:39:36] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.505 second response time [00:39:39] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.012 second response time [00:39:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.73, 3.57 [00:40:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:41:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.78, 3.61 [00:42:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [00:44:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:45:04] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:45:10] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:53] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:48:04] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:48:04] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:14] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.327 second response time [00:51:25] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.632 second response time [00:51:59] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.635 second response time [00:52:01] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.883 second response time [00:52:15] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.677 second response time [00:53:05] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:13] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:23] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:43] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:53:56] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:58] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:54:26] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:54:34] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:54:47] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:55:17] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.812 second response time [00:56:00] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.922 second response time [00:56:21] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:56:22] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:56:38] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.798 second response time [00:56:38] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:57:14] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.326 second response time [00:57:44] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:57:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.13, 2.84, 3.27 [00:57:56] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:57:58] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.748 second response time [00:58:05] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.145 second response time [00:58:44] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.650 second response time [00:59:02] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.050 second response time [00:59:32] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.739 second response time [00:59:41] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.339 second response time [00:59:54] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.449 second response time [01:00:26] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.434 second response time [01:00:32] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.615 second response time [01:00:44] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.598 second response time [01:00:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:01:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.72, 3.83, 3.59 [01:03:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:03:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.98, 3.62, 3.55 [01:04:08] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:04:20] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:04:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [01:05:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.58, 4.44, 3.87 [01:06:05] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.323 second response time [01:06:20] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.165 second response time [01:07:22] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.48, 5.35, 5.91 [01:07:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 4.00, 3.77 [01:08:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.91, 22.40, 23.78 [01:08:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:11:26] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:11:55] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:12:01] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:12:10] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:22] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:12:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.21, 23.33, 23.84 [01:12:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:13:14] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.56, 5.21, 5.59 [01:13:26] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.341 second response time [01:13:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.08, 4.37, 3.97 [01:13:56] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.571 second response time [01:13:58] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.390 second response time [01:14:08] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.686 second response time [01:14:23] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.265 second response time [01:14:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.98, 22.37, 23.43 [01:14:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:14:53] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.79, 20.81, 20.08 [01:15:11] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.60, 5.13, 5.51 [01:15:50] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:16:03] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:16:20] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:16:59] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:17:16] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:28] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:31] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:49] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.366 second response time [01:17:52] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.87, 3.86 [01:18:00] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.014 second response time [01:18:16] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.483 second response time [01:18:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [01:19:05] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.77, 6.43, 5.91 [01:19:59] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:20:42] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.00, 20.12, 20.06 [01:21:08] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:59] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.895 second response time [01:22:03] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:07] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.81, 7.56, 7.97 [01:22:27] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:59] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.65, 5.17, 5.53 [01:23:04] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.009 second response time [01:23:15] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:23:31] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.688 second response time [01:23:48] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.336 second response time [01:23:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.60, 4.46, 4.05 [01:23:58] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.352 second response time [01:24:13] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.310 second response time [01:24:56] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.40, 6.20, 5.85 [01:25:28] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.737 second response time [01:26:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.14, 7.81, 7.97 [01:26:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.14, 24.29, 23.49 [01:26:29] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.09, 20.67, 20.32 [01:26:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:26:50] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:26:53] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.58, 5.52, 5.64 [01:27:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.20, 3.56, 3.79 [01:28:24] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.02, 20.04, 20.13 [01:28:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.57, 22.59, 22.95 [01:29:11] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.450 second response time [01:29:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.90, 3.88 [01:30:24] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.498 second response time [01:30:34] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.338 second response time [01:30:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:30:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [01:30:51] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.897 second response time [01:31:33] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.004 second response time [01:32:16] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.26, 20.69, 20.37 [01:32:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:32:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:34:14] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.57, 20.06, 20.17 [01:35:17] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:35:40] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:35:55] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:36:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.95, 4.16, 4.92 [01:37:16] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.248 second response time [01:37:39] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.591 second response time [01:37:51] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.341 second response time [01:41:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [01:41:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [01:41:58] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.57, 21.66, 20.92 [01:45:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:45:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:45:48] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.00, 19.53, 20.24 [01:50:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.02, 21.89, 21.55 [01:51:01] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:05] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:07] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:14] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:51:21] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:22] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:24] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:27] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:35] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:55] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb [01:53:00] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.683 second response time [01:53:04] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.321 second response time [01:53:09] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.006 second response time [01:53:26] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.225 second response time [01:53:28] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.218 second response time [01:53:28] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.123 second response time [01:53:33] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.273 second response time [01:53:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.60, 3.66, 4.00 [01:54:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:54:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.25, 22.42, 21.93 [01:55:17] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.198 second response time [01:55:37] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.999 second response time [01:56:04] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.956 second response time [01:56:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.34, 7.05, 7.88 [01:56:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:00:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.22, 24.24, 22.69 [02:01:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.80, 7.31, 7.90 [02:02:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.96, 7.85, 7.89 [02:02:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.08, 22.63, 22.26 [02:04:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.20, 7.61, 7.80 [02:05:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.95, 8.37, 8.18 [02:05:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.82, 3.70, 3.79 [02:06:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.03, 8.12, 7.96 [02:07:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.73, 3.79 [02:09:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.03, 7.76, 7.99 [02:11:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 7.57, 7.89, 8.03 [02:11:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.81, 3.78 [02:13:24] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [02:13:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.83, 3.80 [02:14:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.67, 7.72, 7.92 [02:15:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.62, 7.74, 7.98 [02:17:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.42, 8.24, 8.12 [02:18:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.73, 8.03, 7.99 [02:19:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.87, 3.76 [02:20:12] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:20:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.18, 7.76, 7.90 [02:20:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.31, 21.43, 21.05 [02:20:30] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:21:23] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:21:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:21:38] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:21:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.78, 3.74 [02:21:56] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:11] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.06, 20.85, 20.89 [02:22:55] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:24:15] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.964 second response time [02:24:56] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.086 second response time [02:26:50] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.097 second response time [02:27:33] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.850 second response time [02:27:50] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.544 second response time [02:28:12] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.643 second response time [02:28:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.43, 7.65, 7.74 [02:28:43] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.979 second response time [02:28:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:29:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.74, 3.04, 3.37 [02:30:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.64, 7.34, 7.63 [02:30:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/49b2a0f89748 [02:30:59] [url] Page not found · GitHub · GitHub | github.com [02:31:00] [02miraheze/mw-config] 07Universal-Omega 0349b2a0f - Sitenotice: remove unused globals [02:31:02] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [02:31:02] [url] Page not found · GitHub · GitHub | github.com [02:31:03] [02mw-config] 07Universal-Omega opened pull request 03#4371: Sitenotice: remove unused globals - 13https://github.com/miraheze/mw-config/pull/4371 [02:31:04] [url] Page not found · GitHub · GitHub | github.com [02:32:11] miraheze/mw-config - Universal-Omega the build passed. [02:32:42] [02mw-config] 07Universal-Omega closed pull request 03#4371: Sitenotice: remove unused globals - 13https://github.com/miraheze/mw-config/pull/4371 [02:32:42] [url] Page not found · GitHub · GitHub | github.com [02:32:43] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [02:32:44] [url] Page not found · GitHub · GitHub | github.com [02:32:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a05293ab2158...64edd2069245 [02:32:45] [url] Comparing a05293ab2158...64edd2069245 · miraheze/mw-config · GitHub | github.com [02:32:46] [02miraheze/mw-config] 07Universal-Omega 0364edd20 - Sitenotice: remove unused globals (#4371) [02:32:48] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [02:32:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [02:33:39] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.48, 3.48 [02:34:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.46, 7.69, 7.68 [02:34:53] miraheze/mw-config - Universal-Omega the build passed. [02:35:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.83, 7.43, 7.98 [02:35:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/64edd2069245...6e770ebad3c2 [02:35:25] [url] Comparing 64edd2069245...6e770ebad3c2 · miraheze/mw-config · GitHub | github.com [02:35:26] [02miraheze/mw-config] 07Universal-Omega 036e770eb - Move data-nosnippet for specific wiki sitenotices [02:35:36] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.747 second response time [02:35:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6e770ebad3c2...f3391d13b484 [02:35:41] [url] Comparing 6e770ebad3c2...f3391d13b484 · miraheze/mw-config · GitHub | github.com [02:35:42] [02miraheze/mw-config] 07Universal-Omega 03f3391d1 - Indentation [02:35:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 3.82, 3.60 [02:36:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.75, 7.74, 7.70 [02:36:44] miraheze/mw-config - Universal-Omega the build passed. [02:36:49] miraheze/mw-config - Universal-Omega the build passed. [02:37:12] .op [02:37:12] Attempting to OP... [02:37:29] .deop [02:37:29] Attempting to OP... [02:38:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.15, 8.24, 7.89 [02:38:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.93, 19.71, 20.33 [02:40:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f3391d13b484...713187bf958e [02:40:51] [url] Comparing f3391d13b484...713187bf958e · miraheze/mw-config · GitHub | github.com [02:40:51] [02miraheze/mw-config] 07Universal-Omega 03713187b - ?? false [02:41:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.67, 7.86, 7.91 [02:41:34] !log [@mw11] starting deploy of {'config': True} to ovlon [02:42:06] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 32s [02:42:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.58, 20.34, 20.40 [02:42:28] miraheze/mw-config - Universal-Omega the build passed. [02:42:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:43:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:43:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:44:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.19, 20.13, 20.34 [02:45:35] [02puppet] 07Universal-Omega closed pull request 03#2248: Add grant for testincidents database - 13https://github.com/miraheze/puppet/pull/2248 [02:45:36] [url] Page not found · GitHub · GitHub | github.com [02:45:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.32, 3.75, 3.79 [02:46:02] [02mw-config] 07Universal-Omega edited pull request 03#4333: Change IncidentReporting database for betaheze - 13https://github.com/miraheze/mw-config/pull/4333 [02:46:02] [url] Page not found · GitHub · GitHub | github.com [02:46:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5f2e4fd40b90...516c849c2474 [02:46:36] [url] Comparing 5f2e4fd40b90...516c849c2474 · miraheze/mw-config · GitHub | github.com [02:46:37] [02miraheze/mw-config] 07Universal-Omega 03516c849 - Update LocalSettings.php [02:46:39] [02mw-config] 07Universal-Omega synchronize pull request 03#4333: Change IncidentReporting database for betaheze - 13https://github.com/miraheze/mw-config/pull/4333 [02:46:39] [url] Page not found · GitHub · GitHub | github.com [02:46:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [02:46:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:47:03] !log [@test101] starting deploy of {'config': True} to skip [02:47:04] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:47:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.97, 3.87 [02:48:14] miraheze/mw-config - Universal-Omega the build passed. [02:48:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:18] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:50:00] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:50:24] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:50:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.89, 22.46, 21.22 [02:50:31] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:44] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:50:46] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.23, 20.87, 20.18 [02:51:23] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.220 second response time [02:51:40] !log [@mwtask111] starting deploy of {'config': True} to scsvg [02:51:47] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [02:51:47] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:56] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.008 second response time [02:52:22] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.255 second response time [02:52:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.33, 22.94, 21.57 [02:52:31] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.083 second response time [02:52:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/713187bf958e...728c2eebf62a [02:52:37] [url] Comparing 713187bf958e...728c2eebf62a · miraheze/mw-config · GitHub | github.com [02:52:37] [02miraheze/mw-config] 07Universal-Omega 03728c2ee - Remove unnecessary comment [02:52:41] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.323 second response time [02:52:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.71, 4.90, 4.18 [02:53:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:53:44] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:53:48] miraheze/mw-config - Universal-Omega the build passed. [02:54:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a01b8b2cf328 [02:54:08] [02miraheze/mw-config] 07Universal-Omega 03a01b8b2 - Change MatomoAnalytics database for betaheze [02:54:09] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [02:54:10] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:54:10] [url] Page not found · GitHub · GitHub | github.com [02:54:11] [02mw-config] 07Universal-Omega opened pull request 03#4372: Change MatomoAnalytics database for betaheze - 13https://github.com/miraheze/mw-config/pull/4372 [02:54:11] [url] Page not found · GitHub · GitHub | github.com [02:54:13] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:54:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.79, 23.57, 21.96 [02:54:38] !log [@test3] starting deploy of {'config': True} to skip [02:54:39] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s [02:54:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.25, 4.87, 4.28 [02:54:58] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:55:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:55:35] miraheze/mw-config - Universal-Omega the build passed. [02:56:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.49, 6.85, 7.68 [02:56:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.24, 23.39, 22.09 [02:56:33] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.13, 19.84, 20.04 [02:57:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.75, 3.56, 3.83 [02:59:52] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.636 second response time [02:59:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.43, 4.12, 3.99 [03:00:15] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.716 second response time [03:00:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.38, 23.36, 22.35 [03:00:33] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.453 second response time [03:00:36] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.444 second response time [03:01:05] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.343 second response time [03:02:20] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.83, 20.48, 20.25 [03:02:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.93, 21.96, 21.95 [03:02:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:02:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:04:15] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.40, 19.47, 19.89 [03:05:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.37, 6.96, 7.87 [03:08:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb [03:09:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.42, 7.79, 7.96 [03:10:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:11:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.82, 6.65, 7.78 [03:11:22] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:11:24] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:11:40] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:11:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.94, 6.96, 7.80 [03:11:49] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.69, 3.52, 3.89 [03:12:01] !log [@mw11] starting deploy of {'config': True} to ovlon [03:12:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.15, 7.54, 7.21 [03:12:40] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:12:54] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 52s [03:13:06] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:11] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:13:22] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:13:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:28] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.598 second response time [03:13:29] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.818 second response time [03:14:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:07] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:16:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.35, 7.71, 7.42 [03:17:30] !log [@test101] starting deploy of {'config': True} to skip [03:17:31] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [03:17:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.97, 8.00, 7.92 [03:17:46] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.024 second response time [03:17:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.49, 3.78, 3.84 [03:18:06] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.147 second response time [03:18:08] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.786 second response time [03:18:32] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw9 [03:18:33] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:18:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:19:03] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.20, 7.34, 7.57 [03:19:30] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.898 second response time [03:19:30] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.056 second response time [03:19:31] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.029 second response time [03:19:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.74, 3.81 [03:20:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.37, 8.33, 7.70 [03:20:41] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.716 second response time [03:21:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.48, 7.31, 7.53 [03:21:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.81, 7.72, 7.87 [03:21:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.77, 4.08, 3.92 [03:21:55] !log [@mwtask111] starting deploy of {'config': True} to scsvg [03:22:09] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 13s [03:22:32] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [03:22:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:04] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:23:05] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:12] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:23:14] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.142 second response time [03:23:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:20] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:24:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.51, 7.68, 7.58 [03:25:00] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.356 second response time [03:25:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.66, 7.84, 7.69 [03:25:04] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 0.867 second response time [03:25:07] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.326 second response time [03:25:17] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.182 second response time [03:26:12] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:26:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 39.14, 25.38, 22.71 [03:26:35] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:27:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.87, 6.87, 7.33 [03:27:39] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:27:41] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:28:40] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.531 second response time [03:29:44] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.583 second response time [03:29:45] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.947 second response time [03:30:18] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.300 second response time [03:30:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.61, 23.35, 22.51 [03:30:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:30:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:32:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.61, 7.12, 7.18 [03:35:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.39, 7.51, 7.29 [03:35:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.39, 7.39, 7.43 [03:37:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.21, 7.81, 7.46 [03:40:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.64, 7.75, 7.54 [03:41:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.67, 8.72, 7.87 [03:41:29] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/1877615e7f8a [03:41:31] [02miraheze/CreateWiki] 07Universal-Omega 031877615 - pass 'wiki' to `notifyBureaucrats()` as a separate parameter [03:41:32] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/CreateWiki [03:41:33] [url] Page not found · GitHub · GitHub | github.com [03:41:34] [02CreateWiki] 07Universal-Omega opened pull request 03#275: pass 'wiki' to `notifyBureaucrats()` as a separate parameter - 13https://github.com/miraheze/CreateWiki/pull/275 [03:41:34] ... [03:42:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.06, 8.32, 7.79 [03:42:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [03:44:12] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/1877615e7f8a...d48d2ccfd7f7 [03:44:12] [url] Comparing 1877615e7f8a...d48d2ccfd7f7 · miraheze/CreateWiki · GitHub | github.com [03:44:13] [02miraheze/CreateWiki] 07Universal-Omega 03d48d2cc - Update CreateWikiNotificationsManager.php [03:44:15] [02CreateWiki] 07Universal-Omega synchronize pull request 03#275: pass 'wiki' to `notifyBureaucrats()` as a separate parameter - 13https://github.com/miraheze/CreateWiki/pull/275 [03:44:15] [url] Page not found · GitHub · GitHub | github.com [03:45:57] MacFan4000: ^ Can MirahezeBot avoid links from `Not-*`? [03:46:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:49:08] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:49:11] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:49:27] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:49:34] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:49:35] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:50:08] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:19] miraheze/CreateWiki - Universal-Omega the build passed. [03:50:36] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.52, 4.76, 2.26 [03:51:04] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:51:15] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.499 second response time [03:51:16] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:51:21] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.975 second response time [03:51:25] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.680 second response time [03:51:32] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.453 second response time [03:51:44] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.104 second response time [03:52:12] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.344 second response time [03:53:02] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.901 second response time [03:53:16] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.319 second response time [03:56:22] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:56:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.07, 7.84, 7.92 [03:56:29] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:56:31] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.89, 1.80, 1.67 [03:56:34] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:57:40] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:57:54] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:58:28] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.678 second response time [03:58:29] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.81, 1.43, 1.54 [03:58:35] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.915 second response time [03:58:39] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.772 second response time [03:58:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:59:43] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.117 second response time [03:59:53] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.923 second response time [04:00:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.79, 7.73, 7.84 [04:00:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:02:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.69, 7.63, 7.79 [04:03:32] [02CreateWiki] 07Universal-Omega closed pull request 03#275: pass 'wiki' to `notifyBureaucrats()` as a separate parameter - 13https://github.com/miraheze/CreateWiki/pull/275 [04:03:32] [url] Page not found · GitHub · GitHub | github.com [04:03:34] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/654a71103c6f...02ad6cf38942 [04:03:34] [url] Comparing 654a71103c6f...02ad6cf38942 · miraheze/CreateWiki · GitHub | github.com [04:03:35] [02miraheze/CreateWiki] 07Universal-Omega 0302ad6cf - pass 'wiki' to `notifyBureaucrats()` as a separate parameter (#275) [04:03:42] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/CreateWiki [04:03:42] ... [04:03:43] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [04:04:12] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/CreateWiki/commit/d7ffef898aa9 [04:04:13] [02miraheze/CreateWiki] 07Universal-Omega 03d7ffef8 - Delete EchoRenameWikiPresentationModel.php [04:04:15] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/CreateWiki [04:04:15] [url] Page not found · GitHub · GitHub | github.com [04:04:22] [02CreateWiki] 07Universal-Omega opened pull request 03#276: Remove EchoRenameWikiPresentationModel - 13https://github.com/miraheze/CreateWiki/pull/276 [04:04:22] [url] Page not found · GitHub · GitHub | github.com [04:04:50] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d7ffef898aa9...be9be70f7ee8 [04:04:50] [url] Comparing d7ffef898aa9...be9be70f7ee8 · miraheze/CreateWiki · GitHub | github.com [04:04:51] [02miraheze/CreateWiki] 07Universal-Omega 03be9be70 - Update extension.json [04:04:53] [02CreateWiki] 07Universal-Omega synchronize pull request 03#276: Remove EchoRenameWikiPresentationModel - 13https://github.com/miraheze/CreateWiki/pull/276 [04:04:53] [url] Page not found · GitHub · GitHub | github.com [04:05:35] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/be9be70f7ee8...0493a8b92601 [04:05:35] [url] Comparing be9be70f7ee8...0493a8b92601 · miraheze/CreateWiki · GitHub | github.com [04:05:36] [02miraheze/CreateWiki] 07Universal-Omega 030493a8b - Update CreateWikiHooks.php [04:05:38] [02CreateWiki] 07Universal-Omega synchronize pull request 03#276: Remove EchoRenameWikiPresentationModel - 13https://github.com/miraheze/CreateWiki/pull/276 [04:05:38] [url] Page not found · GitHub · GitHub | github.com [04:06:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.27, 8.24, 7.99 [04:07:07] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/0493a8b92601...5982f33c88d7 [04:07:08] [url] Comparing 0493a8b92601...5982f33c88d7 · miraheze/CreateWiki · GitHub | github.com [04:07:09] [02miraheze/CreateWiki] 07Universal-Omega 035982f33 - Update en.json [04:07:10] [02CreateWiki] 07Universal-Omega synchronize pull request 03#276: Remove EchoRenameWikiPresentationModel - 13https://github.com/miraheze/CreateWiki/pull/276 [04:07:11] [url] Page not found · GitHub · GitHub | github.com [04:07:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.81, 4.00 [04:08:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.36, 19.40, 20.36 [04:09:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.14, 7.15, 7.89 [04:09:35] miraheze/CreateWiki - Universal-Omega the build passed. [04:09:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.67, 4.07, 4.07 [04:10:20] CosmicAlpha: no, as the nick changes, and the bot doesn’t login, and when Sar-io tried regex, the bots crashed [04:10:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.27, 6.74, 7.75 [04:11:54] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/5982f33c88d7...0ca87449baca [04:11:54] [url] Comparing 5982f33c88d7...0ca87449baca · miraheze/CreateWiki · GitHub | github.com [04:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.80, 3.97 [04:11:55] [02miraheze/CreateWiki] 07Universal-Omega 030ca8744 - Update en.json [04:11:57] [02CreateWiki] 07Universal-Omega synchronize pull request 03#276: Remove EchoRenameWikiPresentationModel - 13https://github.com/miraheze/CreateWiki/pull/276 [04:11:57] [url] Page not found · GitHub · GitHub | github.com [04:12:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.12, 7.78, 7.92 [04:12:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.83, 7.56, 7.93 [04:13:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.96, 7.18, 7.93 [04:13:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.56, 7.48, 7.80 [04:13:21] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/0ca87449baca...a020fd61daff [04:13:22] [url] Comparing 0ca87449baca...a020fd61daff · miraheze/CreateWiki · GitHub | github.com [04:13:22] [02miraheze/CreateWiki] 07Universal-Omega 03a020fd6 - Update extension.json [04:13:24] [02CreateWiki] 07Universal-Omega synchronize pull request 03#276: Remove EchoRenameWikiPresentationModel - 13https://github.com/miraheze/CreateWiki/pull/276 [04:13:25] [url] Page not found · GitHub · GitHub | github.com [04:13:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.35, 3.90, 4.01 [04:14:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.72, 20.45, 20.43 [04:15:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.70, 3.91 [04:16:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.81, 19.82, 20.20 [04:17:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.84, 7.62, 7.79 [04:17:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.77, 3.91 [04:18:16] MacFan4000: oh, so we are stuck with all this page not found then now? [04:18:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.50, 7.57, 7.70 [04:19:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.66, 8.54, 8.10 [04:19:24] miraheze/CreateWiki - Universal-Omega the build passed. [04:19:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.76, 3.89 [04:20:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.42, 6.99, 7.46 [04:20:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.85, 21.48, 20.72 [04:21:32] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [04:22:31] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:22:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [04:22:36] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:22:45] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:22:48] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:49] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:23:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.14, 6.60, 7.89 [04:23:46] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:24:07] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:24:21] [02CreateWiki] 07Universal-Omega edited pull request 03#276: Remove EchoRenameWikiPresentationModel - 13https://github.com/miraheze/CreateWiki/pull/276 [04:24:22] [url] Page not found · GitHub · GitHub | github.com [04:24:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.70, 5.49, 6.72 [04:24:27] [02CreateWiki] 07Universal-Omega closed pull request 03#276: Remove EchoRenameWikiPresentationModel - 13https://github.com/miraheze/CreateWiki/pull/276 [04:24:28] ... [04:24:29] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±3] 13https://github.com/miraheze/CreateWiki/compare/02ad6cf38942...789ef593324c [04:24:30] [url] Comparing 02ad6cf38942...789ef593324c · miraheze/CreateWiki · GitHub | github.com [04:24:30] [02miraheze/CreateWiki] 07Universal-Omega 03789ef59 - Remove EchoRenameWikiPresentationModel (#276) [04:24:31] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.457 second response time [04:24:32] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [04:24:33] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/CreateWiki [04:24:34] [url] Page not found · GitHub · GitHub | github.com [04:24:49] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.548 second response time [04:25:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:26:10] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.703 second response time [04:26:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:26:38] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.320 second response time [04:26:49] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.308 second response time [04:26:54] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.088 second response time [04:27:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.14, 7.10, 7.27 [04:27:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.49, 7.71, 7.98 [04:28:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.01, 7.56, 7.22 [04:29:15] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [04:29:53] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.021 second response time [04:30:15] miraheze/CreateWiki - Universal-Omega the build passed. [04:30:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [04:30:42] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/369e3240cb84 [04:30:43] [02miraheze/CreateWiki] 07Universal-Omega 03369e324 - Fix regex for checking the AI approval blacklist [04:30:44] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/CreateWiki [04:30:45] [url] Page not found · GitHub · GitHub | github.com [04:30:46] [02CreateWiki] 07Universal-Omega opened pull request 03#277: Fix regex for checking the AI approval blacklist - 13https://github.com/miraheze/CreateWiki/pull/277 [04:30:47] ... [04:31:35] [02CreateWiki] 07Universal-Omega closed pull request 03#277: Fix regex for checking the AI approval blacklist - 13https://github.com/miraheze/CreateWiki/pull/277 [04:31:35] [url] Page not found · GitHub · GitHub | github.com [04:31:36] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/789ef593324c...026467e75b9c [04:31:37] [url] Comparing 789ef593324c...026467e75b9c · miraheze/CreateWiki · GitHub | github.com [04:31:38] [02miraheze/CreateWiki] 07Universal-Omega 03026467e - Fix regex for checking the AI approval blacklist (#277) [04:31:39] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [04:31:41] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/CreateWiki [04:31:41] [url] Page not found · GitHub · GitHub | github.com [04:31:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.48, 3.55 [04:32:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.54, 20.06, 20.39 [04:33:05] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:33:44] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:33:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.85, 3.42, 3.53 [04:34:15] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:34:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:35:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.21, 7.98, 7.88 [04:35:03] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:35:42] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.408 second response time [04:35:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.98, 4.05, 3.74 [04:36:15] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.387 second response time [04:36:23] miraheze/CreateWiki - Universal-Omega the build passed. [04:37:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.41, 8.95, 8.24 [04:37:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.26, 7.47, 7.96 [04:37:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.92, 3.74 [04:38:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.16, 7.16, 7.50 [04:39:09] miraheze/CreateWiki - Universal-Omega the build passed. [04:40:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.14, 7.54, 7.57 [04:41:17] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.074 second response time [04:41:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.74, 7.79, 7.95 [04:42:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.26, 7.76, 7.68 [04:42:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [04:42:58] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:43:28] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [04:43:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.91, 6.85, 7.58 [04:44:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.95, 21.31, 20.62 [04:44:56] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.340 second response time [04:45:25] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:45:40] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:46:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.25, 20.40, 20.37 [04:46:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:47:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.56, 7.78, 8.00 [04:47:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:47:20] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:47:27] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:36] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.61, 4.92, 4.22 [04:48:10] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:50:18] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.077 second response time [04:50:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.77, 21.74, 20.94 [04:50:32] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:50:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb [04:51:16] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::929a/cpweb [04:51:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:51:21] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.033 second response time [04:51:25] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.629 second response time [04:51:30] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.19, 5.15, 4.44 [04:51:34] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.195 second response time [04:51:45] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:51:50] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.358 second response time [04:51:51] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:57] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:52:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:53:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.62, 7.90, 7.93 [04:53:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:53:18] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.325 second response time [04:53:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.59, 4.78, 4.39 [04:53:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.46, 8.27, 7.79 [04:53:51] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.165 second response time [04:53:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.98, 3.65, 3.58 [04:54:42] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.518 second response time [04:55:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.77, 3.64 [04:56:00] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.575 second response time [04:56:05] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.308 second response time [04:56:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.07, 7.18, 7.12 [04:56:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.81, 22.74, 21.55 [04:58:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.98, 7.43, 7.22 [04:58:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.73, 20.90, 21.03 [04:59:46] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:59:54] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:59:59] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:00:09] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:00:40] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:01:43] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.298 second response time [05:01:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.88, 7.69, 7.85 [05:01:51] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.444 second response time [05:01:57] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.430 second response time [05:02:09] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.740 second response time [05:02:36] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.575 second response time [05:03:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.47, 7.76, 7.99 [05:03:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:04:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.14, 7.18, 7.11 [05:04:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.46, 19.25, 20.28 [05:05:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.75, 7.70, 7.77 [05:05:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.27, 4.36, 3.92 [05:07:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.31, 8.48, 8.21 [05:07:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.40, 6.67, 7.77 [05:12:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [05:13:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.51, 7.50, 7.72 [05:13:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb [05:14:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.15, 21.64, 20.46 [05:14:52] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.39, 4.90, 4.35 [05:15:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:15:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.90, 7.58, 7.92 [05:15:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.92, 3.77, 3.95 [05:16:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.21, 7.67, 7.74 [05:16:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.71, 21.36, 20.50 [05:16:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:16:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.64, 4.48, 4.27 [05:17:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.47, 7.97, 8.02 [05:18:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.03, 7.73, 7.75 [05:19:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [05:20:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.66, 7.87, 7.81 [05:20:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [05:21:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.53, 7.84, 7.80 [05:22:08] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:25] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:23:29] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:24:06] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.026 second response time [05:24:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.42, 7.52, 7.59 [05:24:23] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.161 second response time [05:24:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.13, 20.39, 20.38 [05:24:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:25:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.46, 7.35, 7.54 [05:26:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.67, 7.37, 7.55 [05:29:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.95, 7.44, 7.55 [05:31:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.85, 7.68, 7.94 [05:33:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.04, 7.81, 7.95 [05:33:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.01, 3.35 [05:35:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.09, 7.78, 7.99 [05:37:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.23, 7.17, 7.66 [05:37:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.70, 7.64, 7.47 [05:37:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.35, 3.44 [05:39:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.41, 7.50, 7.71 [05:39:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.92, 7.75, 7.53 [05:39:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.33, 8.08, 8.02 [05:41:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.52, 7.29, 7.62 [05:41:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 3.73, 3.55 [05:43:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.64, 7.47, 7.62 [05:43:50] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/c8ab7913e6cd...6118b8756027 [05:43:51] [url] Comparing c8ab7913e6cd...6118b8756027 · miraheze/landing · GitHub | github.com [05:43:52] [02miraheze/landing] 07Universal-Omega 036118b87 - Update URL format [05:43:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.55, 3.51 [05:44:52] miraheze/landing - Universal-Omega the build passed. [05:46:57] [02landing] 07Universal-Omega reviewed pull request 03#58 commit - 13https://github.com/miraheze/landing/pull/58#discussion_r781774524 [05:46:58] [url] convert to json translations by Universal-Omega · Pull Request #58 · miraheze/landing · GitHub | github.com [05:47:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.96, 7.57, 7.60 [05:47:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.88, 2.97, 3.30 [05:51:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.75, 8.46, 7.94 [05:51:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.29, 3.34 [05:52:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.72, 7.79, 7.47 [05:53:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.74, 3.50 [05:55:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.47, 7.52, 7.43 [05:56:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.31, 7.41, 7.42 [05:57:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.79, 7.86, 7.96 [05:57:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.96, 6.94, 7.23 [05:57:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.85, 3.78, 3.62 [06:01:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.58, 8.24, 8.04 [06:02:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.23, 7.44, 7.32 [06:02:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.14, 6.60, 7.68 [06:04:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.05, 7.34, 7.30 [06:04:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.18, 7.52, 7.88 [06:05:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.30, 7.81, 7.48 [06:06:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.24, 7.65, 7.42 [06:07:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.72, 7.29, 7.32 [06:07:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.68, 3.82, 3.62 [06:08:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.38, 6.43, 7.41 [06:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.84, 3.70 [06:14:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.78, 7.97, 7.76 [06:14:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [06:16:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.58, 8.22, 7.87 [06:16:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.71, 5.90, 6.80 [06:16:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:17:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.65, 3.63 [06:17:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.88, 7.50, 7.96 [06:19:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.39, 20.18, 19.19 [06:19:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.15, 6.96, 7.00 [06:19:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.66, 8.28, 8.19 [06:20:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.55, 7.67, 7.30 [06:21:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.48, 7.09, 7.07 [06:22:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.56, 6.72, 6.99 [06:23:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [06:23:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.91, 7.39, 7.86 [06:25:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:25:14] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.51, 22.08, 20.22 [06:25:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.97, 21.07, 19.20 [06:26:51] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.35, 5.00, 3.79 [06:27:14] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.00, 22.61, 20.67 [06:27:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.06, 6.25, 6.68 [06:27:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.92, 8.46, 8.15 [06:28:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.58, 7.57, 7.25 [06:28:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.36, 5.15, 4.01 [06:30:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.75, 7.64, 7.31 [06:30:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.07, 4.76, 4.01 [06:31:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.33, 3.64, 3.81 [06:31:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.64, 7.82, 7.98 [06:32:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.94, 7.17, 8.00 [06:33:08] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.29, 18.57, 19.05 [06:33:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.52, 19.36, 19.95 [06:33:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:35:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.36, 4.21, 3.97 [06:36:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.22, 6.21, 6.76 [06:37:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.52, 3.56, 3.77 [06:39:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.72, 7.61, 7.67 [06:40:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.31, 5.35, 6.77 [06:40:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.73, 6.89, 6.88 [06:41:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.05, 6.40, 7.81 [06:41:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.26, 6.48, 7.73 [06:41:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.26, 7.25, 7.53 [06:44:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:44:44] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.44, 7.32, 7.02 [06:46:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.80, 6.92, 7.02 [06:46:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 8.00, 7.69, 7.20 [06:47:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.29, 3.08, 3.40 [06:49:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.90, 7.94, 7.57 [06:50:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.15, 6.21, 6.71 [06:50:25] [02puppet] 07Reception123 closed pull request 03#2259: MediaWiki: configure shellbox - 13https://github.com/miraheze/puppet/pull/2259 [06:50:25] [url] Page not found · GitHub · GitHub | github.com [06:50:26] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+3/-0/±3] 13https://github.com/miraheze/puppet/compare/8f14e1b7b3f3...b94e7f0d5fde [06:50:27] [url] Comparing 8f14e1b7b3f3...b94e7f0d5fde · miraheze/puppet · GitHub | github.com [06:50:28] [02miraheze/puppet] 07Universal-Omega 03b94e7f0 - MediaWiki: configure shellbox (#2259) [06:50:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.15, 8.53, 7.65 [06:51:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.03, 6.77, 7.26 [06:51:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.28, 7.42, 6.77 [06:51:45] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:51:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.83, 3.61 [06:51:59] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:52:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [06:52:52] [02puppet] 07Universal-Omega opened pull request 03#2295: Enable shellbox on test3 - 13https://github.com/miraheze/puppet/pull/2295 [06:52:53] [url] Page not found · GitHub · GitHub | github.com [06:53:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.66, 7.13, 6.74 [06:53:47] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.453 second response time [06:53:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.89, 3.66 [06:53:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.26, 7.61, 7.62 [06:54:03] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23577 bytes in 5.053 second response time [06:55:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.97, 3.71 [06:55:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.06, 8.43, 7.90 [06:56:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:57:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.85, 7.74, 7.59 [06:57:20] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.61, 7.64, 6.98 [06:57:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.07, 7.41, 7.31 [06:57:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.89, 3.71 [06:58:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.12, 21.17, 19.75 [06:58:49] [02MirahezeMagic] 07RhinosF1 closed pull request 03#314: Use enterprise assessment - 13https://github.com/miraheze/MirahezeMagic/pull/314 [06:58:49] [url] Page not found · GitHub · GitHub | github.com [06:58:50] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 038 commits to 03RhinosF1-patch-1 [+0/-0/±10] 13https://github.com/miraheze/MirahezeMagic/compare/32f540b450e4...90d0cffc4097 [06:58:51] [url] Comparing 32f540b450e4...90d0cffc4097 · miraheze/MirahezeMagic · GitHub | github.com [06:58:51] [02miraheze/MirahezeMagic] 07RhinosF1 0390d0cff - Merge pull request #314 from miraheze/Universal-Omega-patch-1 [06:58:53] [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#313: Recaptcha: use enterprise version - 13https://github.com/miraheze/MirahezeMagic/pull/313 [06:58:53] [url] Page not found · GitHub · GitHub | github.com [06:58:55] [02MirahezeMagic] 07RhinosF1 deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/MirahezeMagic [06:58:55] [url] Page not found · GitHub · GitHub | github.com [06:58:56] [02miraheze/MirahezeMagic] 07RhinosF1 deleted branch 03Universal-Omega-patch-1 [06:59:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.28, 7.34, 6.97 [06:59:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.35, 6.99, 7.15 [06:59:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 4.06, 3.80 [07:00:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.10, 20.26, 19.59 [07:00:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.91, 7.64, 7.76 [07:00:46] RhinosF1: we'll see if CI passes on that PR now, if it does then just needs testing and fixing appropriately as testing indicates. [07:01:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.10, 6.49, 6.70 [07:01:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.48, 3.61 [07:03:46] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.66, 6.11, 6.79 [07:03:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.43, 7.22, 7.79 [07:04:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:06:49] miraheze/MirahezeMagic - RhinosF1 the build passed. [07:07:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.92, 5.85, 6.70 [07:07:25] Yep passed [07:08:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.22, 6.86, 7.19 [07:09:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.21, 3.76, 3.61 [07:09:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.84, 7.62, 7.68 [07:10:18] CosmicAlpha: hopefully Reception123 will give us the info soon [07:10:37] Well approve the bank so I can get it [07:10:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.70, 6.25, 6.92 [07:11:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.79, 7.05, 6.91 [07:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.45, 3.51 [07:11:58] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.60, 6.59, 6.49 [07:12:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:12:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.59, 7.52, 7.31 [07:13:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.18, 7.28, 6.99 [07:13:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.37, 6.61, 6.52 [07:13:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.64, 3.88, 3.66 [07:14:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.18, 7.09, 6.47 [07:14:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.95, 7.66, 7.39 [07:14:46] CosmicAlpha: https://phabricator.miraheze.org/K21 is an api key [07:14:47] [url] Login | phabricator.miraheze.org [07:15:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.23, 6.71, 6.71 [07:15:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.65, 3.36, 3.49 [07:15:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.06, 7.91, 7.88 [07:16:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.01, 7.61, 7.39 [07:17:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.38, 7.21, 7.04 [07:17:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.16, 2.92, 3.31 [07:18:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.30, 7.52, 6.82 [07:19:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.57, 7.35, 6.95 [07:19:48] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.34, 7.65, 7.22 [07:19:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.83, 8.89, 8.23 [07:21:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.08, 8.06, 7.10 [07:21:42] RhinosF1: Thanks! So we just need the project ID then, and then you can test probably (or I can if you want)? [07:22:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.92, 7.44, 7.49 [07:23:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.99, 7.42, 7.14 [07:23:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.39, 7.33, 6.94 [07:24:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.86, 7.55, 7.02 [07:25:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.27, 8.16, 7.44 [07:26:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.61, 7.67, 7.14 [07:26:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.22, 7.78, 7.58 [07:27:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.69, 7.66, 7.35 [07:27:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.35, 7.70, 7.17 [07:27:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.37, 7.91, 7.62 [07:28:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.32, 6.96, 7.31 [07:29:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.23, 7.22, 7.04 [07:31:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.59, 7.82, 7.29 [07:32:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:33:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.31, 7.24, 7.14 [07:33:27] CosmicAlpha: project id is 'mediawikiteam' [07:33:36] We need Reception123 to authorise the card [07:33:44] Which should happen in next couple of days [07:33:55] So we probably can do on new infra [07:33:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.01, 7.13, 7.83 [07:34:15] I only added test101 to the key for use at moment [07:34:20] I will add rest later [07:34:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.12, 5.74, 6.56 [07:35:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.04, 6.01, 6.73 [07:36:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.79, 5.84, 6.74 [07:37:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.38, 7.30, 7.28 [07:38:11] RhinosF1: well you mean the bank account, whenever Owen gets the transaction [07:38:25] Reception123: yes [07:39:01] Should be today [07:39:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.35, 6.27, 6.73 [07:39:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.26, 7.26, 7.28 [07:40:25] Then wait for the mandate to clear [07:40:31] RhinosF1: alright, thanks! [07:41:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.12, 6.51, 6.70 [07:41:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.92, 6.83, 7.18 [07:43:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.40, 7.64, 7.23 [07:43:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:43:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.92, 7.18, 7.29 [07:44:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.65, 7.11, 6.90 [07:46:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.50, 6.28, 6.60 [07:47:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.03, 8.26, 7.31 [07:48:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.47, 3.25 [07:49:03] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.95, 7.54, 7.16 [07:49:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.56, 6.95, 6.71 [07:49:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.23, 7.84, 7.51 [07:49:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.06, 7.49, 7.29 [07:50:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 3.62, 3.32 [07:50:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.95, 7.43, 6.97 [07:51:16] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.67, 7.44, 6.91 [07:51:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.47, 7.47, 7.43 [07:52:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.45, 3.30 [07:53:15] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.26, 7.36, 6.95 [07:55:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.77, 8.10, 7.48 [07:55:13] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.39, 9.26, 7.69 [07:55:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.30, 8.50, 7.78 [07:56:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.75, 3.43 [07:56:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.74, 7.02, 7.00 [07:56:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb [07:57:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.26, 7.28, 7.53 [07:58:12] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:58:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.75, 3.47 [07:58:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.25, 6.37, 6.76 [07:58:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:00:12] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.730 second response time [08:01:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.58, 5.35, 6.43 [08:02:03] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.71, 3.38, 3.40 [08:03:08] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.35, 7.51, 7.81 [08:03:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:05:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.66, 7.36, 7.76 [08:05:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.50, 7.49, 7.93 [08:05:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.73, 3.45, 3.45 [08:05:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.28, 5.80, 6.74 [08:09:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 3.17, 3.36 [08:13:01] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.86, 5.87, 6.76 [08:15:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.73, 7.32, 7.31 [08:15:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.95, 7.03, 7.27 [08:16:57] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.03, 7.21, 7.12 [08:17:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.12, 7.20, 7.28 [08:17:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.24, 6.74, 7.14 [08:18:56] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.15, 6.57, 6.90 [08:19:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.50, 7.72, 6.93 [08:20:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:21:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.27, 7.70, 7.42 [08:21:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.59, 7.27, 6.34 [08:21:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.54, 7.12, 6.35 [08:21:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.92, 3.71, 3.39 [08:22:53] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.78, 6.29, 6.74 [08:23:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.12, 7.26, 7.31 [08:23:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.80, 7.46, 6.57 [08:23:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.37, 7.19, 6.42 [08:25:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.51, 7.39, 7.32 [08:25:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:25:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.27, 3.60, 3.44 [08:27:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.29, 7.13, 7.23 [08:27:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.22, 7.42, 6.74 [08:27:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.04, 6.60, 6.41 [08:28:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.40, 7.05, 6.70 [08:29:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.60, 7.24, 7.13 [08:29:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 3.98, 7.05, 7.36 [08:30:54] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.89, 5.72, 6.24 [08:31:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.99, 6.94, 7.05 [08:31:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.44, 3.12, 3.30 [08:33:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.60, 5.82, 6.70 [08:33:46] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.65, 6.30, 6.79 [08:36:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:36:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.13, 6.49, 6.38 [08:37:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.44, 5.84, 6.71 [08:38:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.89, 5.39, 5.99 [08:41:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:41:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.31, 7.02, 6.98 [08:47:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.93, 7.89, 7.34 [08:51:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.08, 7.63, 6.70 [08:52:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:53:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.79, 7.72, 6.85 [08:53:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.32, 7.62, 7.45 [08:55:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.79, 8.29, 7.15 [08:55:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.23, 8.14, 7.68 [08:56:38] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.49, 6.88, 6.58 [08:56:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.05, 7.50, 7.00 [08:58:54] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.05, 6.77, 6.79 [08:59:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.85, 6.40, 6.06 [08:59:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.94, 7.18, 7.40 [09:00:33] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.31, 6.41, 6.51 [09:01:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.00, 5.84, 5.89 [09:02:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:05:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.53, 5.31, 6.54 [09:16:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:20:10] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.14, 6.52, 5.96 [09:21:10] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.10, 6.46, 5.85 [09:21:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:21:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.37, 7.43, 6.80 [09:22:08] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.50, 6.27, 5.95 [09:22:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.05, 7.04, 6.14 [09:23:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.81, 6.06, 6.38 [09:25:07] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.81, 7.27, 6.30 [09:27:06] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.42, 6.45, 6.11 [09:27:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.12, 7.65, 6.92 [09:29:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.93, 7.59, 7.00 [09:30:08] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.47, 6.76, 5.96 [09:30:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:32:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.18, 6.21, 5.86 [09:32:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.55, 6.45, 6.39 [09:33:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.33, 6.21, 6.61 [09:35:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:50:49] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.52, 7.51, 6.67 [09:51:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.18, 7.27, 6.14 [09:51:27] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.50, 7.88, 6.49 [09:52:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.24, 6.83, 6.51 [09:53:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.64, 6.76, 6.08 [09:53:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.92, 7.21, 6.39 [09:54:45] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.40, 8.06, 6.99 [09:55:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.81, 7.62, 6.62 [09:57:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.99, 7.15, 6.43 [10:00:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:00:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.07, 7.22, 6.89 [10:01:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.10, 5.84, 6.10 [10:02:37] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.28, 7.02, 7.17 [10:02:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.03, 6.59, 6.70 [10:05:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:12:27] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.38, 7.15, 7.06 [10:13:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.95, 7.51, 6.48 [10:13:35] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.08, 6.15, 5.49 [10:14:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.53, 7.24, 7.11 [10:14:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.78, 6.74, 6.45 [10:15:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.93, 6.68, 6.32 [10:15:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.31, 7.08, 5.90 [10:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:16:27] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.31, 6.92, 5.98 [10:16:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.49, 6.14, 6.26 [10:17:26] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.88, 6.21, 5.72 [10:18:25] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.63, 6.12, 5.81 [10:20:21] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.66, 7.51, 7.17 [10:20:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.20, 8.26, 7.12 [10:21:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.44, 8.50, 7.22 [10:22:19] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.23, 7.24, 7.10 [10:24:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.06, 7.30, 7.13 [10:24:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.93, 7.64, 7.14 [10:25:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.35, 7.77, 7.21 [10:26:15] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.03, 7.21, 7.12 [10:28:13] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.45, 8.01, 7.42 [10:30:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:31:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.30, 6.05, 5.64 [10:31:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.30, 6.43, 6.74 [10:32:11] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.86, 7.99, 7.56 [10:33:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.36, 5.76, 5.58 [10:34:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.88, 6.42, 6.74 [10:42:02] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.54, 8.03, 7.49 [10:44:00] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.41, 7.35, 7.30 [10:51:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.61, 7.97, 7.42 [10:53:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.08, 6.79, 5.87 [10:53:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.11, 7.96, 7.51 [10:54:11] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.74, 6.95, 5.92 [10:54:41] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.92, 7.83, 6.62 [10:55:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.55, 7.20, 6.14 [10:56:10] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.99, 6.60, 5.92 [10:56:38] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.27, 7.46, 6.65 [10:57:28] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.94, 6.36, 5.96 [10:57:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.64, 8.20, 7.68 [10:58:03] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.61, 7.91, 7.07 [10:58:36] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.92, 6.80, 6.49 [10:59:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.29, 7.34, 6.96 [11:00:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 8.00, 7.14, 6.61 [11:01:58] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.71, 6.69, 6.51 [11:01:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.79, 7.39, 7.51 [11:05:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.91, 6.27, 6.65 [11:07:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.79, 5.96, 6.79 [11:13:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.47, 6.95, 6.68 [11:13:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:13:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.73, 6.98, 6.99 [11:15:20] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.51, 6.61, 6.57 [11:15:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.20, 7.53, 7.19 [11:18:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:19:03] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.21, 7.78, 6.93 [11:21:06] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.59, 7.70, 6.91 [11:21:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.70, 7.32, 6.86 [11:23:04] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.16, 7.11, 6.80 [11:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:24:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.97, 7.77, 7.27 [11:25:02] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.15, 7.43, 6.94 [11:25:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.04, 7.88, 7.72 [11:27:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.23, 5.84, 6.42 [11:27:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.96, 6.43, 6.65 [11:27:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.77, 8.25, 7.87 [11:28:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:30:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.99, 7.63, 7.27 [11:31:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.96, 7.33, 7.64 [11:32:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.99, 6.95, 7.07 [11:34:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.25, 5.63, 6.57 [11:35:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.09, 4.87, 6.53 [11:50:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:50:49] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.77, 7.25, 6.62 [11:51:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.92, 7.02, 6.06 [11:51:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.54, 7.43, 6.58 [11:51:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.29, 2.74 [11:52:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.70, 6.84, 6.53 [11:53:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.55, 7.69, 6.80 [11:53:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.20, 6.93, 6.17 [11:53:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.36, 2.82 [11:54:45] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.64, 6.53, 6.46 [11:55:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:55:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.31, 2.95, 2.73 [11:57:04] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.53, 7.69, 6.96 [11:59:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.50, 6.62, 6.25 [12:00:54] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.58, 7.98, 7.26 [12:01:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.59, 3.06 [12:03:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.52, 3.09 [12:07:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.78, 3.29 [12:08:28] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.17, 7.10, 6.80 [12:08:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.51, 6.51, 6.75 [12:09:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:09:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.77, 3.34 [12:10:26] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.69, 6.61, 6.64 [12:11:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.80, 3.40 [12:12:27] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.27, 7.67, 6.51 [12:12:32] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.93, 7.81, 6.62 [12:13:13] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.41, 6.45, 5.89 [12:13:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 3.28, 3.26 [12:15:12] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.61, 7.77, 6.45 [12:20:15] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.18, 8.36, 7.14 [12:21:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.37, 6.56, 6.07 [12:23:22] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.22, 6.42, 6.08 [12:28:11] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.74, 7.43, 7.35 [12:32:07] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.20, 7.75, 7.97 [12:32:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.35, 7.66, 6.91 [12:33:00] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.43, 7.01, 7.31 [12:33:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.29, 7.27, 6.50 [12:34:05] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.34, 8.22, 7.63 [12:34:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.93, 7.83, 7.07 [12:35:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.83, 7.52, 6.69 [12:36:03] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.97, 7.95, 7.63 [12:36:04] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.79, 8.11, 7.26 [12:36:57] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.43, 5.61, 6.71 [12:37:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.88, 6.15, 7.65 [12:38:01] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.51, 8.18, 7.75 [12:39:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.33, 6.96, 7.76 [12:39:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.89, 7.25, 7.13 [12:39:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.20, 7.72, 7.62 [12:41:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.32, 7.13, 7.73 [12:41:56] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.15, 6.97, 7.33 [12:41:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.23, 8.09, 7.78 [12:43:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.42, 7.73, 6.94 [12:43:54] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.36, 7.16, 7.36 [12:47:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.05, 7.83, 7.46 [12:47:50] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.83, 7.45, 7.38 [12:48:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.65, 6.96, 6.80 [12:49:15] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.73, 7.97, 7.85 [12:50:45] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.89, 7.70, 7.09 [12:52:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.88, 6.97, 6.90 [12:53:07] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.48, 7.95, 7.91 [12:55:03] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.23, 8.18, 8.00 [12:57:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.73, 7.98, 7.93 [12:58:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.77, 7.98, 7.95 [12:59:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.70, 7.31, 7.75 [12:59:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.26, 7.93, 7.91 [13:01:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.44, 7.39, 7.71 [13:04:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.43, 7.32, 7.07 [13:04:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:04:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.25, 8.08, 7.94 [13:06:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.52, 6.96, 6.97 [13:06:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.06, 7.14, 7.60 [13:07:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.55, 7.70, 7.94 [13:10:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.25, 6.02, 6.62 [13:11:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.42, 7.68, 7.85 [13:12:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.95, 8.13, 7.79 [13:13:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.31, 7.29, 7.41 [13:13:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.59, 6.87, 7.07 [13:16:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 3.51, 3.10 [13:16:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.52, 7.31, 7.05 [13:17:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:18:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.22, 3.35, 3.09 [13:18:28] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.72, 7.72, 7.22 [13:20:27] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.69, 7.38, 7.14 [13:20:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.41, 19.79, 18.54 [13:21:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [13:21:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.67, 6.90, 7.26 [13:22:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [13:23:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.43, 7.64, 7.80 [13:23:32] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:24:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:24:34] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.29, 20.04, 18.93 [13:25:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.96, 7.43, 7.34 [13:27:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.57, 3.26 [13:27:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.62, 7.66, 7.73 [13:28:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.19, 3.14, 3.14 [13:29:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.01, 7.15, 7.55 [13:33:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.33, 7.96, 7.73 [13:34:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.52, 3.28 [13:35:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.41, 8.03, 7.76 [13:35:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.61, 8.20, 7.85 [13:36:34] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 3.25, 3.22 [13:37:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.69, 7.61, 7.66 [13:39:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.61, 7.44, 7.63 [13:40:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.57, 7.41, 7.13 [13:40:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.52, 3.35 [13:41:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.92, 8.57, 8.01 [13:42:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.76, 3.46 [13:43:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.59, 7.88, 7.76 [13:44:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.25, 3.28, 3.33 [13:44:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.39, 7.99, 7.47 [13:45:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.11, 7.22, 7.63 [13:46:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.21, 7.92, 7.50 [13:48:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.46, 7.71, 7.47 [13:50:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [13:50:52] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:50:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.27, 21.12, 19.66 [13:51:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.53, 7.83, 7.75 [13:51:02] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:03] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:06] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:18] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:52:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.12, 7.73, 7.51 [13:52:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [13:52:55] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.30, 20.05, 19.44 [13:53:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.91, 4.21, 3.68 [13:54:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 7.72, 7.54 [13:54:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:56:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:56:50] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.37, 20.66, 19.86 [13:57:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.72, 7.78, 7.86 [13:57:36] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.079 second response time [13:58:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.34, 8.13, 7.73 [13:58:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.51, 20.40, 19.87 [13:59:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.52, 8.64, 8.15 [13:59:07] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.976 second response time [13:59:29] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.061 second response time [13:59:31] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.932 second response time [13:59:32] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.722 second response time [14:01:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.73, 7.18, 7.83 [14:04:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.12, 7.73, 7.71 [14:06:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.46, 8.09, 7.85 [14:10:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.43, 7.25, 7.61 [14:11:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.93, 7.69, 7.60 [14:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.73, 3.68, 3.86 [14:12:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb [14:13:18] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:13:34] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:34] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:13:53] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:14:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [14:15:14] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.956 second response time [14:15:30] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.329 second response time [14:15:32] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.254 second response time [14:15:51] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.751 second response time [14:16:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:16:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:20:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.59, 7.82, 7.71 [14:20:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [14:21:33] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:22:20] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:22:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:22:52] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:23:08] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:23:19] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:22] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:23:42] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:25:09] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:25:14] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:35] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:26:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.36, 7.35, 7.64 [14:27:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.43, 7.03, 7.88 [14:29:09] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.188 second response time [14:29:17] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.183 second response time [14:29:35] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.304 second response time [14:29:40] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.715 second response time [14:29:51] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.020 second response time [14:29:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 2.93, 3.30 [14:30:47] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.834 second response time [14:31:17] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.911 second response time [14:31:31] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.021 second response time [14:31:46] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.339 second response time [14:31:51] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.244 second response time [14:32:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:32:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:33:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.81, 7.98, 7.96 [14:36:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.98, 3.85, 3.55 [14:40:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.35, 7.84, 7.56 [14:40:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.61, 3.52 [14:44:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.74, 3.60 [14:46:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:47:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb [14:48:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.78, 3.68 [14:51:00] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:16] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:21] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:22] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:44] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:47] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:58] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:02] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:15] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:19] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:54:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:55:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 3.57, 3.53 [14:56:14] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.499 second response time [14:56:35] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.706 second response time [14:57:13] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.199 second response time [14:57:29] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.784 second response time [14:57:33] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.870 second response time [14:57:55] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.971 second response time [14:57:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.63, 3.56 [14:58:10] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.934 second response time [14:58:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb [14:58:50] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.255 second response time [14:59:51] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.554 second response time [15:01:36] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:02:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.09, 7.48, 7.95 [15:02:23] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.021 second response time [15:02:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:02:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:03:34] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.244 second response time [15:04:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.86, 8.37, 8.22 [15:05:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.44, 4.19, 3.77 [15:06:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [15:08:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:11:43] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.26, 4.80, 3.72 [15:13:11] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.32, 21.19, 20.28 [15:15:23] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:15:37] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.89, 4.66, 3.94 [15:15:44] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:16:14] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:16:24] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:17:24] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.740 second response time [15:17:40] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.011 second response time [15:18:17] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.339 second response time [15:18:26] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.120 second response time [15:19:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:21:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:21:18] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:21:21] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:43] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:21:54] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:07] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:39] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:58] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:24:05] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:24:46] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.421 second response time [15:24:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.07, 19.69, 20.04 [15:26:04] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.678 second response time [15:26:07] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.828 second response time [15:27:10] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:27:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:28:48] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.10, 22.07, 20.79 [15:30:44] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.80, 21.69, 20.81 [15:31:00] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:31:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [15:31:12] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.13, 4.27, 3.75 [15:32:26] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.735 second response time [15:32:44] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.032 second response time [15:33:10] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.52, 3.82, 3.64 [15:33:51] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.291 second response time [15:34:08] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.896 second response time [15:34:17] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.404 second response time [15:34:37] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.13, 20.08, 20.36 [15:38:14] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:38:32] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.30, 21.48, 20.88 [15:38:41] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:38:58] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:40:20] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:40:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.89, 5.19, 4.34 [15:41:12] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:41:14] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:41:36] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:45] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:42:17] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:42:46] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.743 second response time [15:42:47] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.987 second response time [15:42:52] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.34, 4.63, 4.25 [15:43:04] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.315 second response time [15:43:18] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.341 second response time [15:43:23] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.113 second response time [15:43:40] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.807 second response time [15:43:47] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.411 second response time [15:44:20] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.578 second response time [15:44:24] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.547 second response time [15:44:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.597 second response time [15:44:47] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:44:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:48:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:48:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [15:50:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.52, 22.03, 21.17 [15:51:02] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.68, 21.33, 20.99 [15:52:36] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:36] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:01] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.629 second response time [15:54:41] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.212 second response time [15:54:42] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.234 second response time [15:55:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.50, 4.00 [15:57:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.92, 4.04, 4.14 [15:59:27] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:00:00] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:00:03] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:00:11] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:00:53] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:30] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.084 second response time [16:01:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.08, 3.23, 3.79 [16:02:55] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.496 second response time [16:04:08] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.915 second response time [16:04:14] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.267 second response time [16:04:24] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.802 second response time [16:04:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:05:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.00, 4.22, 4.06 [16:06:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.26, 19.34, 20.33 [16:13:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:13:27] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:46] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:13:47] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:13:50] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:14:58] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:19:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.89, 4.00 [16:20:17] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:20:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.78, 21.00, 20.44 [16:20:59] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw9 [16:21:16] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.473 second response time [16:21:56] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.046 second response time [16:22:04] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.372 second response time [16:22:11] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.067 second response time [16:22:14] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.482 second response time [16:22:18] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:22:23] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.248 second response time [16:22:59] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [16:23:19] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:23:34] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:58] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:25:12] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:25:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.50, 3.77 [16:26:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.81, 20.26, 20.28 [16:27:31] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.412 second response time [16:27:41] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.362 second response time [16:28:24] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.277 second response time [16:28:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:29:03] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.696 second response time [16:29:24] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.580 second response time [16:29:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.99, 3.89, 3.89 [16:31:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:31:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.85, 3.87 [16:39:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.98, 3.99 [16:42:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:43:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.98, 3.98 [16:44:50] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [16:46:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:46:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:46:58] PROBLEM - test101 NTP time on test101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:47:34] PROBLEM - graylog121 NTP time on graylog121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:47:43] PROBLEM - jobchron121 NTP time on jobchron121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:48:59] RECOVERY - test101 NTP time on test101 is OK: NTP OK: Offset 0.002091050148 secs [16:49:33] RECOVERY - graylog121 NTP time on graylog121 is OK: NTP OK: Offset -0.001929312944 secs [16:49:48] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset -0.0008109807968 secs [16:50:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [16:51:11] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:22] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:36] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:42] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:51:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.55, 3.65, 3.92 [16:51:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.31, 21.74, 20.58 [16:52:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:53:41] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.446 second response time [16:53:48] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.898 second response time [16:54:00] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:55:20] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.701 second response time [16:55:26] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.794 second response time [16:55:45] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:47] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:55:51] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:56:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:56:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:57:45] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.65, 23.26, 21.62 [16:57:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.87, 4.17, 4.06 [16:58:36] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:59:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.23, 21.35, 21.12 [16:59:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.77, 3.92 [17:00:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:01:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 4.01, 3.99 [17:02:00] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.573 second response time [17:02:08] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.597 second response time [17:02:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:03:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.96, 3.97 [17:04:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [17:04:52] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.115 second response time [17:06:20] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.960 second response time [17:06:40] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.858 second response time [17:11:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.52, 19.77, 20.40 [17:11:36] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:11:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 3.64, 3.76 [17:13:08] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:13:18] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:13:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.54, 3.70 [17:15:11] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.287 second response time [17:15:19] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.062 second response time [17:16:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:17:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:17:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 3.88, 3.79 [17:19:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.59, 3.70 [17:20:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:20:53] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.67, 20.53, 20.30 [17:21:15] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:19] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.09, 6.98, 7.78 [17:22:09] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:22:18] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:58] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.527 second response time [17:23:22] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.346 second response time [17:23:36] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:23:41] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:23:54] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:24:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.73, 20.36, 20.32 [17:25:48] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:26:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:27:36] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.337 second response time [17:27:44] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.075 second response time [17:27:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 2.85, 3.32 [17:28:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.22, 6.95, 7.87 [17:29:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.53, 7.62, 7.70 [17:31:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.10, 7.43, 7.96 [17:32:02] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.351 second response time [17:32:03] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.014 second response time [17:32:14] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.877 second response time [17:32:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.58, 8.08, 8.11 [17:32:34] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.366 second response time [17:32:41] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.900 second response time [17:32:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:33:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.39, 7.76, 8.02 [17:34:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.39, 3.41 [17:36:42] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.12, 3.26, 3.36 [17:38:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [17:40:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 4.05, 3.67 [17:42:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.66, 3.64, 3.57 [17:42:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:43:45] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:44:29] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:44:41] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:44:49] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:45:18] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:45:21] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:51] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.232 second response time [17:47:17] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:35] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.605 second response time [17:48:51] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.901 second response time [17:48:57] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.104 second response time [17:49:31] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.553 second response time [17:49:36] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.943 second response time [17:50:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.76, 3.59 [17:50:16] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:51:34] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:51:36] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:51:36] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:04] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.61, 3.55 [17:52:20] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.547 second response time [17:53:24] !log added 2a10:6740::/48 to reCaptcha enterprise allowlist [17:53:33] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.797 second response time [17:53:34] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.108 second response time [17:53:36] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.699 second response time [17:53:37] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.994 second response time [17:54:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:56:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:56:44] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:57:56] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:57:56] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:58:04] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:58:07] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:58:48] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.658 second response time [17:59:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.81, 3.61 [18:00:00] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.795 second response time [18:00:03] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.570 second response time [18:00:10] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.897 second response time [18:00:13] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.284 second response time [18:00:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [18:03:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.59, 3.56 [18:09:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.74, 19.84, 18.70 [18:09:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.13, 3.37 [18:10:53] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:34] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:43] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:45] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:12:28] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:25] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.027 second response time [18:14:56] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.325 second response time [18:15:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.59, 19.81, 19.09 [18:15:48] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.086 second response time [18:15:56] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.585 second response time [18:16:02] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.889 second response time [18:16:55] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:17:43] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:18:11] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:18:39] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:18:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.70, 4.02, 3.64 [18:18:48] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.88, 3.63 [18:26:50] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.511 second response time [18:27:02] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.243 second response time [18:27:17] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.92, 20.35, 19.73 [18:27:18] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:21] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.237 second response time [18:27:23] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:27:29] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:27:39] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.080 second response time [18:27:43] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:27:54] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:28:10] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.864 second response time [18:29:29] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.842 second response time [18:29:34] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.116 second response time [18:29:47] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.516 second response time [18:29:58] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.946 second response time [18:30:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.91, 4.01, 3.76 [18:30:22] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:30:42] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:31:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.94, 20.68, 19.89 [18:31:15] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.84, 19.98, 19.77 [18:31:30] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.938 second response time [18:32:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.58, 3.49, 3.60 [18:34:07] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:35:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.58, 3.59 [18:36:39] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.187 second response time [18:37:02] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.830 second response time [18:37:04] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.26, 3.47 [18:38:19] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.683 second response time [18:38:44] PROBLEM - mail2 Puppet on mail2 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 10 minutes ago with 0 failures [18:38:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.62, 19.62, 19.86 [18:41:17] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.090 second response time [18:41:44] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:41] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:43] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:44:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:45:19] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:41] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:51:38] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:51:53] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:52:08] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:10] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:52:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:52:49] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b94e7f0d5fde...2691300ecc90 [18:52:50] [url] Comparing b94e7f0d5fde...2691300ecc90 · miraheze/puppet · GitHub | github.com [18:52:50] [02miraheze/puppet] 07Universal-Omega 032691300 - Enable shellbox on test3 (#2295) [18:52:52] [02puppet] 07Reception123 closed pull request 03#2295: Enable shellbox on test3 - 13https://github.com/miraheze/puppet/pull/2295 [18:52:52] [url] Page not found · GitHub · GitHub | github.com [18:53:36] CosmicAlpha, Reception123: shouldn't we use both test servers [18:53:50] Given in 72 hours we are switching [18:53:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.90, 2.79, 3.27 [18:53:56] RhinosF1: I think CosmicAlpha just wants to test it out for now is my understanding [18:54:16] test3 stops working in 72 hours though [18:54:59] RhinosF1: That's the idea, so if my testing does not work as planned I don't have to worry quite as much on post-cleanup on the server for it. [18:57:03] Ah ok [18:59:15] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.015 second response time [18:59:19] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.348 second response time [18:59:56] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.330 second response time [19:00:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:03:00] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.665 second response time [19:03:42] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:04:07] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.018 second response time [19:04:19] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.022 second response time [19:04:23] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:04:34] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:04:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:04:59] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.345 second response time [19:05:01] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 8.028 second response time [19:05:06] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.095 second response time [19:07:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.32, 7.29, 7.89 [19:07:03] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.716 second response time [19:07:49] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.170 second response time [19:08:28] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.046 second response time [19:08:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.43, 3.23 [19:09:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.31, 8.62, 8.29 [19:10:42] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.39, 3.36, 3.21 [19:12:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.52, 20.68, 20.43 [19:12:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:12:53] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:14:03] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.74, 19.94, 20.19 [19:14:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.59, 3.37 [19:14:44] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:16:44] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.308 second response time [19:18:23] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:18:25] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:18:37] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:19:40] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:19:56] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:19:58] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:19:59] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:20:06] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:20:26] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.19, 7.05, 7.86 [19:21:07] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:16] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:21:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.86, 20.78, 20.38 [19:21:59] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.495 second response time [19:22:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.09, 3.39, 3.37 [19:22:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.47, 7.68, 7.99 [19:22:29] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.846 second response time [19:22:38] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.386 second response time [19:22:52] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.317 second response time [19:23:16] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.578 second response time [19:27:13] [02puppet] 07Universal-Omega opened pull request 03#2296: fix shellbox nginx config - 13https://github.com/miraheze/puppet/pull/2296 [19:27:14] [url] Page not found · GitHub · GitHub | github.com [19:27:24] Reception123: ^ [19:27:42] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.00, 20.09, 20.24 [19:30:30] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.592 second response time [19:30:36] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:36] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:36] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:42] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 8.892 second response time [19:30:51] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.177 second response time [19:31:35] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.508 second response time [19:31:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/95b12b8ea3cc [19:31:42] [02miraheze/mw-config] 07Universal-Omega 0395b12b8 - set $wgShellboxUrl for test3 [19:31:44] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [19:31:44] [url] Page not found · GitHub · GitHub | github.com [19:31:47] [02mw-config] 07Universal-Omega opened pull request 03#4373: set $wgShellboxUrl for test3 - 13https://github.com/miraheze/mw-config/pull/4373 [19:31:48] ... [19:32:22] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.414 second response time [19:32:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.36, 7.10, 7.87 [19:32:34] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.268 second response time [19:32:34] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.608 second response time [19:32:35] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.860 second response time [19:33:01] miraheze/mw-config - Universal-Omega the build passed. [19:33:03] [02puppet] 07Reception123 closed pull request 03#2296: fix shellbox nginx config - 13https://github.com/miraheze/puppet/pull/2296 [19:33:04] ... [19:33:05] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2691300ecc90...91aa01b36382 [19:33:05] [url] Comparing 2691300ecc90...91aa01b36382 · miraheze/puppet · GitHub | github.com [19:33:06] [02miraheze/puppet] 07Universal-Omega 0391aa01b - fix shellbox nginx config (#2296) [19:33:49] [02mw-config] 07Universal-Omega closed pull request 03#4373: set $wgShellboxUrl for test3 - 13https://github.com/miraheze/mw-config/pull/4373 [19:33:49] [url] Page not found · GitHub · GitHub | github.com [19:33:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/728c2eebf62a...4f5e254499b4 [19:33:51] [url] Comparing 728c2eebf62a...4f5e254499b4 · miraheze/mw-config · GitHub | github.com [19:33:52] [02miraheze/mw-config] 07Universal-Omega 034f5e254 - set $wgShellboxUrl for test3 (#4373) [19:33:53] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [19:33:54] [url] Page not found · GitHub · GitHub | github.com [19:33:55] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [19:34:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.13, 8.22, 8.18 [19:34:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:34:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:35:02] miraheze/mw-config - Universal-Omega the build passed. [19:35:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.72, 3.42 [19:38:32] !log [@test3] starting deploy of {'config': True} to skip [19:38:33] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s [19:38:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.88, 3.58 [19:40:33] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:41:26] !log [@mw11] starting deploy of {'config': True} to ovlon [19:41:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 3.26, 3.39 [19:42:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:42:07] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 40s [19:42:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:44:31] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:46] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:51] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:15] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.49, 20.32, 20.09 [19:45:40] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:07] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:47:11] !log [@test101] starting deploy of {'config': True} to skip [19:47:12] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [19:47:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:47:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [19:48:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:41] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:49:55] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:50:44] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:28] !log [@mwtask111] starting deploy of {'config': True} to scsvg [19:51:34] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:34] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [19:51:42] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:57] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.986 second response time [19:57:58] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.336 second response time [19:58:03] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.425 second response time [19:58:10] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.046 second response time [19:58:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:59:05] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.904 second response time [19:59:11] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.091 second response time [19:59:15] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.529 second response time [19:59:21] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.346 second response time [19:59:38] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.339 second response time [20:00:35] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.129 second response time [20:01:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.65, 2.19, 1.43 [20:02:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:03:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.58, 1.70, 1.35 [20:09:48] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:10:10] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:10:20] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:10:29] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.21, 19.42, 20.32 [20:10:40] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:10:59] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:11:27] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:11:49] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.984 second response time [20:12:16] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.404 second response time [20:12:25] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.942 second response time [20:12:40] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:12:44] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:12:45] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.144 second response time [20:12:59] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.409 second response time [20:13:02] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:13:23] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.359 second response time [20:13:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.56, 3.33 [20:14:30] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:48] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.605 second response time [20:14:54] [02puppet] 07Universal-Omega opened pull request 03#2297: don't return 301 (redirect) for /healthz - 13https://github.com/miraheze/puppet/pull/2297 [20:14:54] [url] Page not found · GitHub · GitHub | github.com [20:15:01] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.630 second response time [20:15:32] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:44] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:18] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:16:45] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:16:53] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:17:14] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.14, 3.22 [20:18:11] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:18:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.44, 21.26, 20.73 [20:18:34] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:08] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.307 second response time [20:20:30] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.021 second response time [20:20:57] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.717 second response time [20:21:01] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.450 second response time [20:21:52] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.513 second response time [20:22:00] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.350 second response time [20:22:26] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.044 second response time [20:22:47] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.923 second response time [20:22:50] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:23:15] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:23:18] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.319 second response time [20:23:25] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:23:44] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:24:26] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.244 second response time [20:25:14] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:25:19] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:25:45] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.933 second response time [20:26:14] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:26:21] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.769 second response time [20:26:27] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:49] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:26:57] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.737 second response time [20:27:04] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:27:09] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/91aa01b36382...a396eef34004 [20:27:10] [url] Comparing 91aa01b36382...a396eef34004 · miraheze/puppet · GitHub | github.com [20:27:10] [02miraheze/puppet] 07JohnFLewis 0349c1f51 - mail: switch IPs to new infra [20:27:12] [02miraheze/puppet] 07JohnFLewis 03a396eef - Merge branch 'master' of github.com:/miraheze/puppet [20:29:32] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.343 second response time [20:29:41] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.882 second response time [20:30:25] [02mediawiki] 07Universal-Omega closed pull request 03#4934: Bump skins/Cosmos from `95efb34` to `c0cbc1a` - 13https://github.com/miraheze/mediawiki/pull/4934 [20:30:26] [url] Page not found · GitHub · GitHub | github.com [20:30:27] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Cosmos-c0cbc1a - 13https://github.com/miraheze/mediawiki [20:30:27] [url] Page not found · GitHub · GitHub | github.com [20:30:29] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Cosmos-c0cbc1a [20:30:30] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/1e4393d52558...87695d295050 [20:30:31] [url] Comparing 1e4393d52558...87695d295050 · miraheze/mediawiki · GitHub | github.com [20:30:32] [02miraheze/mediawiki] 07dependabot[bot] 0387695d2 - Bump skins/Cosmos from `95efb34` to `c0cbc1a` (#4934) [20:31:10] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.330 second response time [20:32:00] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.444 second response time [20:32:43] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:33:25] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:33:28] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:33:34] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:34:00] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:34:39] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.434 second response time [20:35:20] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.346 second response time [20:35:26] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 0.363 second response time [20:35:32] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.302 second response time [20:35:33] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.009 second response time [20:35:46] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.015 second response time [20:35:58] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.023 second response time [20:36:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:36:37] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.579 second response time [20:37:07] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23577 bytes in 6.808 second response time [20:37:14] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.201 second response time [20:37:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.27, 3.15 [20:38:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.32, 21.27, 20.93 [20:39:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.11, 3.07, 3.08 [20:39:55] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:40:08] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:40:26] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:40:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.89, 20.64, 20.74 [20:40:31] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:40:46] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:42:13] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.477 second response time [20:42:23] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.310 second response time [20:42:29] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.406 second response time [20:42:41] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.008 second response time [20:43:59] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.166 second response time [20:44:14] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:44:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.71, 19.37, 20.22 [20:44:49] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:44:50] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:45:00] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:45:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.20, 6.95, 7.89 [20:46:11] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:47:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.63, 7.50, 7.97 [20:47:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:48:39] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:48:51] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:48:53] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.139 second response time [20:48:54] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.020 second response time [20:48:55] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:48:58] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.344 second response time [20:49:09] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.323 second response time [20:49:27] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:49:46] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:50:12] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:50:14] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:50:36] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.010 second response time [20:50:48] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.325 second response time [20:50:49] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.007 second response time [20:50:51] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw12 [20:50:53] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.270 second response time [20:51:23] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.707 second response time [20:51:42] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.309 second response time [20:51:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.50, 3.18 [20:52:13] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 0.951 second response time [20:52:21] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.024 second response time [20:52:34] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.893 second response time [20:52:47] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [20:52:58] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:08] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:23] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:33] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:53:41] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:53:46] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.55, 3.24 [20:54:20] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.159 second response time [20:54:39] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:46] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:55:24] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.98, 20.91, 20.48 [20:55:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 3.30, 3.19 [20:56:40] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.282 second response time [20:57:15] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.995 second response time [20:57:20] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.94, 20.05, 20.21 [20:57:25] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.102 second response time [20:57:32] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.783 second response time [20:58:46] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:59:09] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:59:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.69, 3.35 [21:01:05] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:01:10] [02puppet] 07RhinosF1 opened pull request 03#2298: varnish: allow proxying beta - 13https://github.com/miraheze/puppet/pull/2298 [21:01:10] [url] Page not found · GitHub · GitHub | github.com [21:01:14] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.151 second response time [21:01:45] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:01:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.57, 3.35 [21:01:55] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:01:55] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.880 second response time [21:01:56] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:02:06] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.937 second response time [21:02:32] JohnLewis: I think my PR works [21:03:01] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw11 [21:04:16] [02puppet] 07paladox closed pull request 03#2297: don't return 301 (redirect) for /healthz - 13https://github.com/miraheze/puppet/pull/2297 [21:04:17] [url] Page not found · GitHub · GitHub | github.com [21:04:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a396eef34004...50da6a6543dc [21:04:18] [url] Comparing a396eef34004...50da6a6543dc · miraheze/puppet · GitHub | github.com [21:04:19] [02miraheze/puppet] 07Universal-Omega 0350da6a6 - don't return 301 (redirect) for /healthz (#2297) [21:04:59] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [21:05:41] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:06:18] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:29] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:07:21] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.557 second response time [21:08:03] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.017 second response time [21:08:05] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.319 second response time [21:08:16] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.317 second response time [21:08:25] [02puppet] 07RhinosF1 opened pull request 03#2299: varnish: don't pass cookies to backend for load.php - 13https://github.com/miraheze/puppet/pull/2299 [21:08:25] [url] Page not found · GitHub · GitHub | github.com [21:08:46] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.066 second response time [21:08:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:09:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.14, 3.11, 3.28 [21:12:03] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.777 second response time [21:12:27] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.955 second response time [21:12:34] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.016 second response time [21:12:43] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:46] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:49] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [21:13:03] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.334 second response time [21:13:23] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:13:24] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:13:33] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.355 second response time [21:14:38] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:46] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.142 second response time [21:14:46] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.775 second response time [21:14:47] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:49] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.201 second response time [21:14:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.37, 20.18, 19.68 [21:15:16] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:19] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.043 second response time [21:15:22] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.083 second response time [21:15:43] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:16:45] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:16:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.52, 19.39, 19.45 [21:17:03] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:17:04] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:17:12] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:17:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:17:45] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.263 second response time [21:17:59] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:18:45] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.930 second response time [21:19:05] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:19:09] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:19:12] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:14] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:18] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:32] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:21:51] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:58] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:07] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.359 second response time [21:22:11] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:16] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:25] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:07] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:12] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.217 second response time [21:23:12] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.893 second response time [21:23:18] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:18] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.954 second response time [21:23:19] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.147 second response time [21:23:24] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 7.844 second response time [21:23:28] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:33] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 4.090 second response time [21:23:53] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:14] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 0.265 second response time [21:24:34] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures [21:25:03] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.353 second response time [21:25:14] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.340 second response time [21:25:18] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.324 second response time [21:25:25] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.446 second response time [21:25:47] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.038 second response time [21:25:48] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:25:55] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:02] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:04] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:26:05] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.861 second response time [21:26:08] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.621 second response time [21:26:09] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:16] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:18] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.617 second response time [21:26:33] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:26:49] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:51] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:27:35] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:27:36] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:27:42] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:27:45] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:28:38] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.147 second response time [21:28:44] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.012 second response time [21:28:48] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.029 second response time [21:29:34] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.924 second response time [21:29:37] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.383 second response time [21:29:37] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.010 second response time [21:29:42] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw13 [21:29:44] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.742 second response time [21:29:47] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.599 second response time [21:30:04] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.645 second response time [21:30:16] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.820 second response time [21:30:24] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.379 second response time [21:30:36] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:30:42] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:31:38] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [21:31:51] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:31:58] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:31:59] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.047 second response time [21:32:10] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.147 second response time [21:32:14] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.003 second response time [21:32:15] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 3.081 second response time [21:32:23] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:32:33] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:32:44] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.823 second response time [21:32:58] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.479 second response time [21:33:41] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.335 second response time [21:33:57] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:34:02] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:13] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:34:33] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:34:41] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 9.974 second response time [21:34:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.32, 3.10 [21:34:50] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:35:09] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:54] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.463 second response time [21:36:04] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.108 second response time [21:36:04] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.362 second response time [21:36:06] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:36:10] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.327 second response time [21:36:10] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.936 second response time [21:36:36] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.595 second response time [21:36:39] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.583 second response time [21:36:53] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.531 second response time [21:38:12] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.852 second response time [21:38:23] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:38:47] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.109 second response time [21:38:55] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:38:59] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.811 second response time [21:39:21] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.734 second response time [21:40:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.86, 3.38 [21:40:28] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.195 second response time [21:40:43] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:40:54] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:41:02] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:42:34] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:42:42] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:43:05] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:43:07] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:43:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:44:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.99, 3.18, 3.24 [21:44:38] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.848 second response time [21:48:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.80, 3.98, 3.56 [21:48:59] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.35, 21.61, 20.33 [21:49:15] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.448 second response time [21:49:25] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.752 second response time [21:49:32] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.090 second response time [21:50:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.58, 3.41, 3.40 [21:50:25] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:50:38] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:50] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:50:55] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.337 second response time [21:50:58] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.014 second response time [21:51:05] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.347 second response time [21:51:15] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.775 second response time [21:51:17] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:18] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:18] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.594 second response time [21:51:19] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:28] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:33] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.042 second response time [21:51:37] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.753 second response time [21:51:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.87, 3.19, 3.31 [21:51:58] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:01] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:53:34] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.424 second response time [21:55:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.53, 3.45 [21:56:46] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.43, 19.87, 20.05 [21:57:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 3.75, 3.53 [21:58:00] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:58:04] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:58:21] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.168 second response time [21:58:38] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.022 second response time [21:59:14] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.678 second response time [21:59:38] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.780 second response time [21:59:50] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.867 second response time [21:59:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.37, 3.44, 3.46 [22:03:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.63, 3.53 [22:04:02] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:04:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.804 second response time [22:04:34] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.105 second response time [22:05:33] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.597 second response time [22:05:53] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.302 second response time [22:05:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.54, 3.51 [22:07:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.98, 2.94, 3.29 [22:08:19] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.849 second response time [22:10:35] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.924 second response time [22:11:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.72, 3.54 [22:13:36] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:13:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.55, 3.50 [22:15:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.75, 3.57 [22:16:15] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:25] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:16:38] [02dns] 07RhinosF1 opened pull request 03#242: search: remove my personal email and annotate - 13https://github.com/miraheze/dns/pull/242 [22:16:38] [url] Page not found · GitHub · GitHub | github.com [22:17:31] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:17:35] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:17:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.83, 3.30, 3.43 [22:18:33] paladox: can you merge my DNS [22:18:58] !log deleted gmail me from reCaptcha so only official me exists [22:19:35] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.285 second response time [22:19:37] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9ae1ac3cbe94...52ceba44c580 [22:19:38] [url] Comparing 9ae1ac3cbe94...52ceba44c580 · miraheze/dns · GitHub | github.com [22:19:38] [02miraheze/dns] 07RhinosF1 0352ceba4 - search: remove my personal email and annotate (#242) [22:19:39] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.265 second response time [22:19:40] [02dns] 07paladox closed pull request 03#242: search: remove my personal email and annotate - 13https://github.com/miraheze/dns/pull/242 [22:19:40] ... [22:19:50] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.193 second response time [22:19:50] Ty paladox [22:19:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.60, 3.19, 3.38 [22:20:22] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.366 second response time [22:20:25] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.319 second response time [22:20:45] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:20:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:21:02] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:21:34] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:21:52] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:14] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:14] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:15] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:36] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:23:57] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.107 second response time [22:24:07] !log search: move Agent to @mh.org rather than personal [22:24:17] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.915 second response time [22:24:17] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.645 second response time [22:24:18] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.911 second response time [22:24:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:24:44] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.488 second response time [22:24:55] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.658 second response time [22:25:17] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.565 second response time [22:25:42] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.551 second response time [22:26:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:26:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:29:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.48, 3.38 [22:33:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.14, 3.27 [22:39:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.47, 7.26, 7.92 [22:39:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.56, 3.42 [22:41:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.53, 8.22, 8.20 [22:43:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [22:43:37] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:43:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:43:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.25, 3.35 [22:44:35] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:44:37] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:44:38] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:44:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.43, 1.04 [22:45:43] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.438 second response time [22:45:55] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.708 second response time [22:46:35] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.886 second response time [22:46:37] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.710 second response time [22:46:39] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.310 second response time [22:46:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.93, 1.23, 1.01 [22:49:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:50:05] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:19] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:33] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:52:04] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.700 second response time [22:52:19] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:30] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:31] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:41] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:53:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.54, 3.38 [22:54:17] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:55:11] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:55:18] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:55:21] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:55:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.20, 3.36, 3.33 [22:56:23] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.591 second response time [22:56:26] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:56:27] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.979 second response time [22:56:33] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 1 minute ago with 0 failures [22:56:41] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.400 second response time [22:56:41] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.273 second response time [22:56:58] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.914 second response time [22:58:31] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.405 second response time [22:58:46] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.256 second response time [22:59:12] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.126 second response time [22:59:20] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.324 second response time [22:59:26] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.426 second response time [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:03] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:01:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.69, 6.62, 7.71 [23:01:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.64, 7.80, 8.00 [23:05:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.19, 7.35, 7.80 [23:06:15] PROBLEM - test3 HTTPS on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [23:07:15] PROBLEM - test3 MediaWiki Rendering on test3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.011 second response time [23:08:34] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:09:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.28, 7.71, 7.80 [23:09:16] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 23604 bytes in 1.128 second response time [23:10:16] RECOVERY - test3 HTTPS on test3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.024 second response time [23:13:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:14:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:15:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:18:34] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures [23:19:24] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [23:19:49] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1187s [23:21:00] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 1258s [23:21:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:21:08] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:18] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:28] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:21:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:21:51] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 7.28, 7.91 [23:30:19] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.742 second response time [23:31:27] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.290 second response time [23:31:34] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.462 second response time [23:31:34] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:36] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.529 second response time [23:32:02] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.284 second response time [23:32:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:33:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.67, 8.58, 8.21 [23:33:36] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.958 second response time [23:34:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:42:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:43:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:44:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:45:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:49:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:50:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [23:51:14] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:14] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:51:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.14, 21.11, 20.21 [23:51:24] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:30] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:57] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:52:12] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:52:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.34, 3.41, 3.32 [23:52:49] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:53:07] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:53:14] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.808 second response time [23:53:17] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.158 second response time [23:53:27] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.041 second response time [23:53:53] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.017 second response time [23:54:01] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:54:02] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:54:05] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:54:16] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.330 second response time [23:54:19] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:54:23] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:54:25] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:54:34] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:54:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.55, 3.52, 3.35 [23:56:00] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.983 second response time [23:56:03] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.909 second response time [23:56:08] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.013 second response time [23:56:22] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.141 second response time [23:56:24] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:56:26] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.412 second response time [23:56:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.24, 3.39, 3.33 [23:57:08] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.29, 19.65, 19.93 [23:58:05] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.20, 20.65, 19.64 [23:58:24] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.997 second response time