[00:00:34] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:00:49] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.822 second response time [00:00:57] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:38] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:51] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:14] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.46, 22.14, 20.68 [00:03:01] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.340 second response time [00:03:26] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 22 failures. Last run 3 minutes ago with 22 failures. Failed resources (up to 3 shown): File[/usr/share/nginx/favicons/default.ico],File[/usr/share/nginx/favicons/apple-touch-icon-default.png],File[/usr/local/bin/mediawiki-firejail-convert],File[/etc/firejail/mediawiki.local] [00:03:40] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.888 second response time [00:03:43] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:54] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.742 second response time [00:04:14] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.731 second response time [00:04:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.54, 21.79, 20.72 [00:04:45] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.325 second response time [00:05:08] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:51] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:55] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:25] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:33] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:08:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.06, 2.79 [00:08:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.68, 22.97, 21.39 [00:10:28] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.027 second response time [00:10:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.31, 21.79, 21.14 [00:10:41] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.009 second response time [00:11:11] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.896 second response time [00:11:21] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:11:53] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:11:54] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:04] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.576 second response time [00:12:14] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.026 second response time [00:12:28] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:06] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:07] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:21] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.748 second response time [00:13:21] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:52] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.465 second response time [00:13:55] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.354 second response time [00:14:01] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.02, 2.98, 2.91 [00:14:16] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:14:27] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.341 second response time [00:14:39] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:15:07] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:15:09] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.839 second response time [00:15:23] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.606 second response time [00:15:29] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:15:56] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.508 second response time [00:16:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.32, 23.77, 22.18 [00:16:44] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.756 second response time [00:17:06] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.026 second response time [00:17:21] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.892 second response time [00:17:25] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.122 second response time [00:17:41] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:17:50] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:13] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:18:21] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.375 second response time [00:18:22] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:18:31] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:54] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:29] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.554 second response time [00:20:45] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:51] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:53] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.300 second response time [00:21:45] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.017 second response time [00:21:59] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.754 second response time [00:22:20] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.481 second response time [00:22:34] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.106 second response time [00:22:46] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.370 second response time [00:22:52] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.356 second response time [00:23:05] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:19] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:22] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:30] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:39] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:49] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:51] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:54] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:24:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.61, 23.61, 22.87 [00:24:46] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:23] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.301 second response time [00:25:29] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.106 second response time [00:25:36] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.451 second response time [00:25:55] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.474 second response time [00:26:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.20, 24.03, 23.10 [00:26:40] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:26:53] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.977 second response time [00:27:01] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:11] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:15] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:20] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:31] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:49] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:52] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:55] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.372 second response time [00:28:04] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.699 second response time [00:28:36] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.76, 23.59, 23.04 [00:29:10] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.828 second response time [00:29:12] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.462 second response time [00:29:15] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.652 second response time [00:29:27] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.139 second response time [00:29:29] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.579 second response time [00:29:49] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.031 second response time [00:29:51] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.034 second response time [00:30:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:30:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.65, 7.66, 7.97 [00:31:24] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [00:31:43] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:27] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:32:53] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.076 second response time [00:32:54] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.508 second response time [00:33:22] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.928 second response time [00:33:35] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.059 second response time [00:34:24] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.032 second response time [00:34:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.21, 7.78, 7.94 [00:36:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.99, 7.82, 7.93 [00:38:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.76, 8.77, 8.26 [00:38:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:39:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.75, 3.95, 3.42 [00:45:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.87, 3.60 [00:46:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.19, 18.69, 20.40 [00:49:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 4.05, 3.73 [00:50:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.92, 20.28, 20.69 [00:50:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [00:51:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.99, 7.19, 7.97 [00:51:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.90, 3.61, 3.60 [00:51:58] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:04] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.57, 7.61, 7.99 [00:52:38] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:45] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:53:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.10, 7.33, 7.91 [00:53:37] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:53:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.62, 4.17, 3.79 [00:54:08] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.402 second response time [00:54:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.41, 8.09, 8.13 [00:56:14] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.214 second response time [00:56:19] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.010 second response time [00:56:52] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.275 second response time [00:56:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [00:57:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.94, 3.79 [00:59:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:59:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 4.24, 3.92 [01:00:41] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:00:46] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:01:13] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:01:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.75, 3.75, 3.78 [01:02:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.36, 7.62, 7.93 [01:02:37] [02puppet] 07Universal-Omega opened pull request 03#2312: check_reverse_dns: add support for `for.uz` tld - 13https://github.com/miraheze/puppet/pull/2312 [01:04:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.83, 7.42, 7.99 [01:04:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.79, 19.45, 20.12 [01:04:52] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.892 second response time [01:04:53] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.793 second response time [01:05:06] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.228 second response time [01:05:17] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.113 second response time [01:06:00] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:06:01] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:06:03] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:06:16] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:06:23] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.376 second response time [01:06:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.28, 8.16, 8.08 [01:06:41] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:08:00] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.329 second response time [01:08:02] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.870 second response time [01:08:06] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.503 second response time [01:08:19] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.565 second response time [01:08:46] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.082 second response time [01:09:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.63, 3.69 [01:10:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [01:12:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.53, 7.91, 7.88 [01:13:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.04, 3.29, 3.59 [01:14:03] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:14:09] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:14:20] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:14:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.02, 7.80, 7.87 [01:15:21] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:30] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.88, 4.78, 3.73 [01:15:48] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:30] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 5.02, 4.60, 3.78 [01:18:18] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.258 second response time [01:18:24] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.755 second response time [01:18:32] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.038 second response time [01:18:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.57, 7.23, 7.88 [01:19:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.48, 3.03, 3.38 [01:20:29] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:20:32] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:20:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.87, 8.43, 8.04 [01:20:38] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:20:45] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:18] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:31] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.20, 7.46, 7.98 [01:22:42] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:51] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:52] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.226 second response time [01:22:57] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:24:22] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.652 second response time [01:24:33] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.539 second response time [01:24:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.63, 7.31, 7.63 [01:24:39] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.311 second response time [01:24:51] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.835 second response time [01:25:00] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.312 second response time [01:25:03] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.227 second response time [01:26:31] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 7.02, 7.84, 8.05 [01:27:27] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:29:22] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.388 second response time [01:29:48] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 8.165 second response time [01:30:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.32, 21.09, 20.49 [01:30:28] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.972 second response time [01:30:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.88, 7.28, 7.61 [01:30:49] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:53] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:31:05] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:31:11] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.213 second response time [01:31:12] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:31:18] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:31:59] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.30, 20.14, 20.23 [01:32:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.61, 7.65, 7.70 [01:34:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.86, 7.20, 7.96 [01:34:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.71, 7.25, 7.54 [01:34:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:35:19] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.694 second response time [01:35:24] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.597 second response time [01:35:33] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.213 second response time [01:36:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.60, 7.74, 7.69 [01:36:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:36:58] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.963 second response time [01:37:14] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.787 second response time [01:38:03] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:38:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.54, 3.40 [01:38:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:38:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:40:08] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.100 second response time [01:40:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.59, 4.06, 3.59 [01:40:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.44, 7.51, 7.81 [01:40:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:42:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.62, 3.40, 3.41 [01:44:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.03, 7.66, 7.83 [01:46:34] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.09, 8.19, 8.00 [01:47:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 3.09, 3.30 [01:48:35] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.197 second response time [01:50:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:50:52] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:50:54] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:50:58] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:05] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:16] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:54:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:54:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.55, 21.60, 20.26 [01:56:26] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:39] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.90, 21.63, 20.43 [01:56:42] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:54] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:55] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:57:09] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.833 second response time [01:57:16] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.593 second response time [01:58:49] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.574 second response time [01:59:10] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.450 second response time [01:59:21] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.322 second response time [01:59:42] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.024 second response time [01:59:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.49, 3.61, 3.40 [02:00:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:00:37] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.026 second response time [02:00:54] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.420 second response time [02:01:01] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.264 second response time [02:01:08] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.576 second response time [02:01:36] [02puppet] 07Universal-Omega synchronize pull request 03#2312: check_reverse_dns: add support for `for.uz` tld - 13https://github.com/miraheze/puppet/pull/2312 [02:02:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.34, 20.19, 20.19 [02:04:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:07:30] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [02:07:34] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:08:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.47, 7.21, 7.86 [02:10:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.82, 7.75, 7.97 [02:11:47] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.874 second response time [02:12:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [02:13:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.96, 3.93 [02:15:05] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:15:15] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:17:03] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.513 second response time [02:17:12] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.562 second response time [02:20:05] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:20:07] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:20:33] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:21:21] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:35] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:06] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:06] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 6.104 second response time [02:22:09] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:11] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.648 second response time [02:25:27] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.837 second response time [02:25:43] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.977 second response time [02:26:17] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.564 second response time [02:26:19] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.033 second response time [02:26:51] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.015 second response time [02:26:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:27:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.71, 3.64 [02:30:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:33:36] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:33:36] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:43] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:33:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.52, 3.61 [02:35:38] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.853 second response time [02:35:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.24, 4.04, 3.78 [02:37:45] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.469 second response time [02:37:57] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.453 second response time [02:39:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:41:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.91, 3.80, 3.81 [02:42:09] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:42:12] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:42:27] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:42:29] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:43:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:43:12] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:43:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.86, 4.24, 3.97 [02:45:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.95, 3.90 [02:46:01] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:46:07] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:19] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.007 second response time [02:46:21] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.244 second response time [02:46:27] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:46:40] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.190 second response time [02:46:43] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:46:46] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.516 second response time [02:47:16] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:47:19] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.670 second response time [02:47:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.98, 4.57, 4.12 [02:50:53] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:51:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.83, 1.39, 1.07 [02:52:13] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.877 second response time [02:52:31] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.878 second response time [02:52:43] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.253 second response time [02:52:50] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.539 second response time [02:52:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:53:00] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.503 second response time [02:53:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.02, 1.22, 1.05 [02:53:41] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.798 second response time [02:56:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [02:56:59] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:57:11] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:58:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:59:17] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.767 second response time [03:00:30] [02mw-config] 07lens0021 opened pull request 03#4376: Add $wgWBQualityConstraints* to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4376 [03:01:15] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.128 second response time [03:01:27] miraheze/mw-config - lens0021 the build passed. [03:01:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.64, 3.95 [03:02:32] [02mw-config] 07lens0021 edited pull request 03#4376: Add $wgWBQualityConstraints* to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4376 [03:02:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:05:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.49, 7.19, 8.00 [03:05:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.76, 3.93 [03:07:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.72, 8.07, 8.22 [03:07:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.61, 3.42, 3.78 [03:08:41] [02mw-config] 07lens0021 synchronize pull request 03#4376: Add $wgWBQualityConstraints* to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4376 [03:09:41] miraheze/mw-config - lens0021 the build passed. [03:10:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.45, 7.46, 7.93 [03:12:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 7.58, 7.82, 8.02 [03:12:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:13:33] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [03:14:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:15:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:15:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.97, 2.84, 3.36 [03:16:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.30, 7.72, 7.95 [03:18:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.26, 7.99, 8.02 [03:20:20] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:21:43] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:21:54] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:02] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:22] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.442 second response time [03:22:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.30, 3.42 [03:23:42] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.815 second response time [03:23:55] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.029 second response time [03:24:04] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.137 second response time [03:24:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.74, 3.57 [03:30:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.40, 7.42, 7.93 [03:32:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.22, 3.69, 3.64 [03:32:31] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.76, 6.66, 7.85 [03:32:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.26, 7.82, 8.00 [03:33:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.49, 7.20, 7.82 [03:34:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.78, 3.67 [03:34:32] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.27, 7.18, 7.89 [03:36:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.64, 3.63 [03:36:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.14, 8.03, 7.97 [03:37:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.16, 7.63, 7.84 [03:39:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.75, 7.49, 7.78 [03:41:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.87, 8.02, 7.93 [03:42:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.21, 7.77, 7.65 [03:44:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.72, 7.37, 7.51 [03:45:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.67, 3.69, 3.58 [03:47:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.70, 7.65, 7.81 [03:47:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.41, 3.48 [03:49:00] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [03:49:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [03:50:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.54, 21.26, 19.36 [03:51:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.43, 7.69, 7.75 [03:51:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.82, 3.63 [03:52:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.71, 21.11, 19.56 [03:52:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:53:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.67, 7.65, 7.74 [03:53:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.90, 3.47, 3.52 [03:54:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.27, 8.21, 7.73 [03:55:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:55:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.88, 8.19, 7.93 [03:55:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.64, 3.58 [03:56:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [03:57:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.87, 7.61, 7.74 [03:57:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.36, 3.48 [03:58:08] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:58:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.74, 19.95, 19.64 [03:58:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:00:04] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.918 second response time [04:01:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.77, 3.60 [04:02:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.43, 7.49, 7.65 [04:03:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.49, 8.49, 8.06 [04:03:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.69, 3.59 [04:06:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.07, 7.95, 7.75 [04:08:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.32, 7.00, 7.77 [04:08:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.29, 7.53, 7.61 [04:12:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.61, 8.03, 8.01 [04:12:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.21, 8.41, 7.94 [04:13:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.91, 3.68 [04:14:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.08, 7.81, 7.94 [04:14:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.73, 7.72, 7.75 [04:15:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.90, 3.70 [04:16:11] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.48, 8.03, 8.00 [04:16:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.78, 8.36, 7.97 [04:21:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.90, 3.72 [04:22:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 7.90, 7.93 [04:26:31] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.10, 7.75, 7.99 [04:27:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.84, 3.86, 3.82 [04:28:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.45, 6.89, 7.77 [04:28:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:30:03] PROBLEM - wiki.scrungecraft.gg - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:31:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.78, 6.75, 7.68 [04:37:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 2.90, 3.33 [04:38:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.52, 7.54, 7.84 [04:38:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.25, 6.86, 7.12 [04:38:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:41:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.98, 7.04, 7.30 [04:41:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.21, 7.85, 7.92 [04:41:39] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.22, 7.97, 7.66 [04:41:39] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.35, 7.00, 7.14 [04:41:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.58, 3.81, 3.57 [04:42:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.88, 7.84, 7.59 [04:43:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.82, 7.39, 7.45 [04:43:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.39, 3.44 [04:44:31] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.15, 7.42, 7.53 [04:45:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.53, 3.69, 3.53 [04:46:31] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.22, 8.63, 7.97 [04:46:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.76, 7.61, 7.34 [04:47:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [04:47:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.89, 8.84, 7.94 [04:47:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.50, 3.48 [04:47:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [04:49:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:49:45] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:50:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.05, 7.76, 7.50 [04:52:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.17, 7.83, 7.79 [04:52:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.83, 8.26, 7.72 [04:53:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.00, 3.28 [04:54:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.44, 8.31, 7.96 [04:56:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.85, 7.50, 7.57 [04:57:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.36, 7.41, 7.70 [04:57:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.51, 3.43 [04:58:31] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.22, 6.83, 7.68 [04:58:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.48, 7.89, 7.87 [04:58:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:59:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.47, 3.06, 3.27 [05:00:31] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.30, 7.32, 7.75 [05:00:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.68, 8.01, 7.91 [05:02:31] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.44, 7.13, 7.61 [05:02:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.79, 7.16, 7.61 [05:04:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.60, 7.16, 7.30 [05:05:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.75, 7.76, 7.69 [05:06:31] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 12.33, 9.00, 8.18 [05:06:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.68, 7.59, 7.64 [05:08:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.47, 7.24, 7.39 [05:09:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.25, 7.37, 7.61 [05:11:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.72, 8.15, 7.88 [05:11:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:13:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.74, 7.20, 7.56 [05:14:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.17, 7.27, 7.57 [05:16:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.14, 7.38, 7.86 [05:16:31] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.47, 7.21, 7.96 [05:16:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:18:36] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.56, 5.92, 6.73 [05:20:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.14, 8.13, 7.99 [05:21:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.42, 7.05, 7.18 [05:22:31] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.86, 7.16, 7.62 [05:22:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.66, 7.46, 7.44 [05:24:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.90, 7.26, 7.00 [05:25:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:28:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.76, 7.74, 7.61 [05:28:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.30, 7.44, 7.13 [05:30:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.73, 7.90, 7.33 [05:32:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.30, 7.60, 7.56 [05:33:13] PROBLEM - wiki.sanctuary-rts.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.sanctuary-rts.com' expires in 15 day(s) (Sun 30 Jan 2022 05:28:55 GMT +0000). [05:34:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.62, 7.38, 7.49 [05:35:11] PROBLEM - data.memipedia.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.memipedia.ga' expires in 15 day(s) (Sun 30 Jan 2022 05:31:27 GMT +0000). [05:36:41] PROBLEM - wiki.sanctuary-rts.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.sanctuary-rts.com' expires in 15 day(s) (Sun 30 Jan 2022 05:28:55 GMT +0000). [05:39:36] PROBLEM - data.memipedia.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.memipedia.ga' expires in 15 day(s) (Sun 30 Jan 2022 05:31:27 GMT +0000). [05:40:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/31b867d36b8d...ecbdc8ac13b6 [05:40:48] [02miraheze/ssl] 07MirahezeSSLBot 03ecbdc8a - Bot: Update SSL cert for wiki.sanctuary-rts.com [05:42:36] PROBLEM - wiki.scrungecraft.gg - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.scrungecraft.gg could not be found [05:45:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ecbdc8ac13b6...3a8062b820e0 [05:45:47] [02miraheze/ssl] 07MirahezeSSLBot 033a8062b - Bot: Update SSL cert for data.memipedia.ga [05:46:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.18, 8.27, 7.82 [05:49:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [05:49:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 149.56.141.75/cpweb [05:50:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.18, 7.57, 7.64 [05:51:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.89, 20.43, 19.30 [05:52:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.49, 7.92, 7.76 [05:52:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.62, 3.31 [05:53:34] PROBLEM - wiki.scrungecraft.gg - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])rDNS WARNING - reverse DNS entry for wiki.scrungecraft.gg could not be found [05:53:50] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.02, 19.62, 19.15 [05:54:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 3.17, 3.18 [05:55:25] [02ssl] 07Universal-Omega opened pull request 03#472: Remove wiki.scrungecraft.gg - 13https://github.com/miraheze/ssl/pull/472 [05:55:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:55:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:56:09] [02ssl] 07Universal-Omega synchronize pull request 03#472: Remove wiki.scrungecraft.gg - 13https://github.com/miraheze/ssl/pull/472 [05:56:25] RECOVERY - wiki.scrungecraft.gg - reverse DNS on sslhost is OK: SSL OK - wiki.scrungecraft.gg reverse DNS resolves to cp21.miraheze.org - CNAME OK [06:00:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.86, 7.71, 7.82 [06:04:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.52, 3.31 [06:04:56] RECOVERY - mw11 APT on mw11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:06:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [06:06:40] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:07:22] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.02, 20.07, 18.89 [06:08:05] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [06:08:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.83, 3.48 [06:08:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.25, 6.83, 7.91 [06:08:17] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:08:38] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.519 second response time [06:09:16] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:09:27] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:09:37] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:09:49] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:10:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.86, 3.54 [06:10:25] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:10:42] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:10:44] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:11:02] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:12:01] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.89, 3.58 [06:12:09] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:12:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.42, 7.28, 7.81 [06:14:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.73, 6.74, 7.56 [06:14:15] RECOVERY - wiki.scrungecraft.gg - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/li [06:14:15] os/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - wiki.scrungecraft.gg reverse DNS resolves to cp21.miraheze.org - CNAME OK [06:14:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.50, 5.79, 6.69 [06:15:58] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.307 second response time [06:16:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.50, 7.10, 7.56 [06:16:18] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.015 second response time [06:16:36] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.878 second response time [06:16:52] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.346 second response time [06:17:36] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.246 second response time [06:17:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.49, 3.53 [06:18:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.60, 7.08, 7.52 [06:18:21] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.02, 6.96, 7.99 [06:19:54] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:20:00] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.615 second response time [06:20:02] PROBLEM - wiki.scrungecraft.gg - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.scrungecraft.gg could not be found [06:20:16] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.11, 7.74, 7.70 [06:20:23] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.86, 7.68, 8.13 [06:20:46] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:21:05] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:07] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:21:20] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:21:21] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:21:28] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:21:47] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:50] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:22:00] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:22:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.08, 6.59, 6.65 [06:22:54] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:23:02] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:04] PROBLEM - wiki.scrungecraft.gg - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])rDNS WARNING - reverse DNS entry for wiki.scrungecraft.gg could not be found [06:23:12] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.147 second response time [06:23:19] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:23:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.09, 2.79, 3.23 [06:24:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.43, 7.26, 7.54 [06:24:22] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:24:31] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:24:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.08, 7.38, 6.93 [06:25:16] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:25:23] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.52, 22.10, 20.74 [06:25:25] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.321 second response time [06:25:29] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.014 second response time [06:25:50] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.465 second response time [06:26:12] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.342 second response time [06:26:16] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:26:20] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.060 second response time [06:26:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.51, 7.27, 6.94 [06:26:35] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:26:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.50, 6.70, 7.80 [06:26:47] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:27:04] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.007 second response time [06:27:12] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.392 second response time [06:27:12] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.772 second response time [06:27:15] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:27:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.90, 21.01, 20.50 [06:27:39] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.964 second response time [06:27:42] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:05] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.077 second response time [06:28:11] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.037 second response time [06:28:34] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.313 second response time [06:28:48] RECOVERY - mw10 APT on mw10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:29:18] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.911 second response time [06:29:19] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.790 second response time [06:29:23] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.600 second response time [06:29:29] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.320 second response time [06:29:41] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.845 second response time [06:30:03] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.144 second response time [06:30:07] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.503 second response time [06:30:27] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:30:31] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.454 second response time [06:30:31] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.67, 7.09, 7.93 [06:30:58] PROBLEM - wiki.scrungecraft.gg - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.scrungecraft.gg and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:31:29] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.574 second response time [06:31:53] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.864 second response time [06:32:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:32:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.56, 7.18, 7.27 [06:32:31] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.507 second response time [06:32:43] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.592 second response time [06:33:11] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.153 second response time [06:33:22] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.17, 19.94, 20.26 [06:34:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.62, 7.11, 7.24 [06:34:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.58, 7.18, 7.55 [06:35:11] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.58, 19.27, 19.99 [06:35:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [06:36:21] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.75, 7.50, 7.96 [06:36:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.44, 6.75, 7.34 [06:37:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:37:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.59, 6.26, 7.66 [06:37:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:38:31] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.84, 7.86, 7.80 [06:40:02] RECOVERY - mw13 APT on mw13 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:40:11] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.09, 7.03, 7.13 [06:40:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.03, 7.80, 7.38 [06:42:21] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.59, 7.71, 7.84 [06:42:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.19, 7.59, 7.46 [06:44:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.83, 7.16, 7.31 [06:44:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.06, 20.65, 20.06 [06:45:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.25, 7.35, 7.62 [06:45:31] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:46:22] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.73, 7.98, 7.98 [06:46:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.76, 7.57, 7.48 [06:46:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.54, 7.56, 7.42 [06:46:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.73, 20.33, 20.02 [06:48:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.44, 7.90, 7.68 [06:48:21] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.18, 8.34, 8.10 [06:48:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.04, 8.00, 7.64 [06:48:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.54, 7.07, 7.25 [06:49:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.69, 7.58, 7.70 [06:50:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.34, 8.55, 7.95 [06:50:45] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.54, 20.83, 20.28 [06:51:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.11, 8.00, 7.83 [06:51:29] RECOVERY - wiki.scrungecraft.gg - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scrungecraft.gg' will expire on Mon 21 Feb 2022 17:47:42 GMT +0000. [06:54:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.69, 7.64, 7.66 [06:54:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.77, 8.13, 7.61 [06:55:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.16, 7.72, 7.78 [06:55:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:57:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.41, 8.54, 8.08 [06:57:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:00:18] PROBLEM - wiki.scrungecraft.gg - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.scrungecraft.gg and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:00:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.50, 7.91, 7.74 [07:02:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 7.95, 7.79 [07:02:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.06, 7.33, 7.54 [07:03:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.30, 7.69, 7.91 [07:04:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.67, 7.37, 7.93 [07:04:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.63, 7.66, 7.62 [07:05:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.99, 8.07, 8.01 [07:05:38] [02ssl] 07Reception123 closed pull request 03#468: Remove wiki.konjitownmc.cf - 13https://github.com/miraheze/ssl/pull/468 [07:05:39] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/3a8062b820e0...b440a2c65f15 [07:05:41] [02miraheze/ssl] 07Universal-Omega 03b440a2c - Remove wiki.konjitownmc.cf (#468) [07:06:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 7.57, 7.60 [07:08:22] RECOVERY - wiki.scrungecraft.gg - reverse DNS on sslhost is OK: SSL OK - wiki.scrungecraft.gg reverse DNS resolves to cp21.miraheze.org - CNAME OK [07:08:35] [02ssl] 07Reception123 closed pull request 03#472: Remove wiki.scrungecraft.gg - 13https://github.com/miraheze/ssl/pull/472 [07:08:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.84, 7.93, 7.72 [07:08:36] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/b440a2c65f15...fe81c30d0827 [07:08:38] [02miraheze/ssl] 07Universal-Omega 03fe81c30 - Remove wiki.scrungecraft.gg (#472) [07:09:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.95, 7.66, 7.88 [07:10:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.01, 7.63, 7.67 [07:11:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.52, 8.38, 8.09 [07:12:11] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.29, 7.62, 7.65 [07:12:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [07:13:59] RECOVERY - wiki.scrungecraft.gg - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scrungecraft.gg' will expire on Mon 21 Feb 2022 17:47:42 GMT +0000. [07:14:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.00, 7.01, 7.48 [07:16:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:17:12] PROBLEM - wiki.scrungecraft.gg - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.scrungecraft.gg could not be found [07:18:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.45, 23.10, 21.63 [07:20:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [07:20:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.71, 22.93, 21.75 [07:20:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [07:22:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.06, 25.11, 22.68 [07:22:48] PROBLEM - wiki.scrungecraft.gg - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.scrungecraft.gg and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:23:17] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:24:14] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:26:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:26:23] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.003 second response time [07:26:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.20, 7.26, 7.35 [07:26:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.27, 23.81, 22.69 [07:27:39] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.20, 21.23, 20.29 [07:28:22] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:28:23] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:28:29] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.39, 6.70, 7.15 [07:28:35] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:29:38] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.34, 20.20, 20.03 [07:29:43] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.324 second response time [07:30:21] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.845 second response time [07:30:22] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.273 second response time [07:30:39] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.161 second response time [07:31:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.06, 7.47, 7.95 [07:31:36] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:35:53] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.53, 6.83, 7.77 [07:38:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.50, 3.29 [07:38:17] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.39, 7.22, 7.10 [07:39:43] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.06, 8.20, 8.09 [07:39:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.64, 3.35 [07:41:17] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.56, 7.74, 7.79 [07:41:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.57, 3.36 [07:42:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.78, 22.99, 22.42 [07:43:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [07:43:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.46, 7.76, 7.80 [07:44:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.73, 22.12, 22.18 [07:45:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:45:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 3.57, 3.38 [07:46:07] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.18, 7.79, 7.60 [07:47:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.63, 3.42 [07:50:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.52, 22.76, 22.24 [07:51:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [07:51:50] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:52:00] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.18, 7.45, 7.43 [07:52:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [07:52:11] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:52:21] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:26] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:52:57] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.33, 7.56, 7.54 [07:53:24] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:53:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:54:17] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.701 second response time [07:54:23] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 5.990 second response time [07:54:29] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.739 second response time [07:54:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.11, 22.76, 22.40 [07:54:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.54, 7.65, 7.59 [07:55:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.90, 3.27, 3.38 [07:55:22] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.576 second response time [07:56:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:56:01] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.328 second response time [07:56:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.15, 23.86, 22.85 [07:58:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.09, 22.83, 22.61 [07:59:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.49, 3.45 [08:01:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.32, 7.55, 7.97 [08:02:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.55, 7.74, 7.64 [08:03:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.95, 8.19, 8.16 [08:05:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 3.22, 3.37 [08:05:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.40, 7.78, 7.91 [08:09:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.60, 3.52 [08:11:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.18, 2.98, 3.29 [08:12:23] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.02, 7.68, 7.80 [08:12:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.40, 22.06, 22.01 [08:13:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.19, 7.52, 7.70 [08:14:20] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 7.74, 8.02, 7.92 [08:14:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.63, 21.74, 21.90 [08:19:45] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [08:19:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.49, 7.64, 7.71 [08:19:45] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [08:20:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.44, 7.68, 7.89 [08:20:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.27, 23.75, 22.66 [08:21:02] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:21:06] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:21:28] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:21:37] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:21:45] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:22:01] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:22:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.36, 22.29, 22.27 [08:22:49] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:23:18] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:24:09] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.595 second response time [08:24:49] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.465 second response time [08:25:15] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.504 second response time [08:25:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:27:43] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.681 second response time [08:27:58] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.475 second response time [08:28:06] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.457 second response time [08:28:32] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:29:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.77, 7.24, 7.88 [08:29:40] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.77, 7.03, 7.92 [08:29:40] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.891 second response time [08:29:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.06, 7.17, 7.94 [08:31:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.84, 7.86, 8.03 [08:32:02] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:32:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.03, 6.96, 7.29 [08:32:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.56, 22.81, 22.12 [08:32:34] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:33:08] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteArchivedRevisions.php --wiki=smallvillewiki (END - exit=0) [08:33:20] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteArchivedRevisions.php --wiki=smallvillewiki --delete (END - exit=0) [08:33:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:30] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.428 second response time [08:33:32] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteArchivedRevisions.php --wiki=superwiki --delete (END - exit=0) [08:33:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:59] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.978 second response time [08:34:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.91, 6.22, 6.98 [08:34:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.41, 21.71, 21.79 [08:34:34] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.078 second response time [08:35:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:36:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.57, 7.42, 7.30 [08:38:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.94, 7.52, 7.36 [08:39:38] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.92, 6.79, 7.86 [08:41:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.46, 7.25, 7.17 [08:41:30] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.51, 7.08, 7.39 [08:41:37] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.12, 8.15, 8.23 [08:41:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.14, 7.37, 7.54 [08:42:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.22, 7.69, 7.45 [08:42:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:43:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.74, 7.18, 7.17 [08:44:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.90, 6.97, 7.22 [08:44:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.43, 21.64, 21.32 [08:45:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.97, 7.58, 7.55 [08:45:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.07, 7.80, 7.70 [08:46:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.27, 20.00, 20.74 [08:47:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.61, 7.79, 7.61 [08:49:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.98, 7.67, 7.61 [08:50:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.50, 7.76, 7.41 [08:51:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.83, 7.98, 7.77 [08:51:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.33, 6.74, 7.27 [08:55:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.11, 7.10, 7.28 [08:57:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.91, 7.81, 7.66 [09:01:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.05, 7.28, 7.89 [09:02:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.43, 7.56, 7.69 [09:03:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.39, 6.53, 6.80 [09:03:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.05, 7.63, 7.67 [09:05:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.06, 7.45, 7.69 [09:11:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.15, 5.81, 6.70 [09:12:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:13:21] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.82, 7.12, 7.95 [09:14:09] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.31, 5.64, 6.59 [09:15:01] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.51, 6.07, 6.76 [09:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:15:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.13, 7.08, 7.14 [09:17:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.64, 7.51, 7.30 [09:19:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.67, 7.66, 7.27 [09:19:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.96, 8.59, 7.71 [09:20:21] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.61, 7.64, 7.11 [09:21:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.83, 7.90, 7.42 [09:22:20] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.98, 7.27, 7.04 [09:24:18] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.79, 7.85, 7.28 [09:24:50] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.30, 7.69, 6.85 [09:25:01] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.58, 7.30, 6.82 [09:25:14] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.90, 7.94, 7.73 [09:26:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.11, 6.82, 6.71 [09:28:15] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.29, 7.78, 7.40 [09:28:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.61, 7.62, 7.05 [09:28:54] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.32, 6.31, 6.54 [09:30:13] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.37, 8.16, 7.57 [09:30:32] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.60, 19.51, 20.29 [09:30:43] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.52, 8.01, 7.26 [09:31:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.25, 7.72, 7.44 [09:32:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.03, 7.75, 7.26 [09:33:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.50, 7.23, 7.30 [09:36:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.11, 7.99, 7.46 [09:37:43] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.00, 8.07, 7.22 [09:38:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.46, 7.74, 7.43 [09:39:39] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.33, 7.48, 7.11 [09:40:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.61, 7.89, 7.79 [09:40:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.71, 7.97, 7.54 [09:41:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.37, 7.72, 7.46 [09:42:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.46, 7.84, 7.55 [09:43:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.35, 7.33, 7.36 [09:44:01] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.22, 8.05, 7.87 [09:47:25] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.90, 6.19, 6.74 [09:50:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.75, 21.88, 20.80 [09:53:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.92, 7.72, 7.99 [09:55:01] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.46, 6.25, 6.78 [09:56:11] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.05, 6.94, 7.04 [09:58:09] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.95, 6.56, 6.89 [09:59:48] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.65, 7.45, 7.73 [10:01:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.92, 7.34, 7.66 [10:01:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.32, 7.36, 7.92 [10:02:32] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.45, 19.82, 20.38 [10:05:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:06:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.01, 20.49, 20.53 [10:09:39] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.87, 7.54, 7.49 [10:09:55] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.24, 7.18, 7.04 [10:10:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:11:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.21, 8.47, 8.07 [10:11:52] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.11, 7.13, 7.05 [10:15:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 6.89, 6.43 [10:17:41] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.96, 6.54, 6.36 [10:17:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.04, 7.23, 7.10 [10:18:32] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.71, 19.76, 20.34 [10:19:13] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.14, 6.92, 6.79 [10:19:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.22, 7.37, 7.69 [10:19:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.49, 6.64, 6.90 [10:21:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.68, 8.42, 8.04 [10:21:41] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.91, 6.25, 6.72 [10:22:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.68, 21.41, 20.90 [10:27:01] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.58, 6.52, 6.70 [10:27:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.56, 7.20, 7.68 [10:27:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 6.81, 6.76 [10:31:03] .op [10:31:03] Attempting to OP... [10:32:32] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.10, 19.84, 20.32 [10:35:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.03, 6.19, 6.57 [10:35:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.99, 6.70, 7.76 [10:36:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.22, 20.71, 20.59 [10:37:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.20, 7.15, 7.35 [10:39:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 7.36, 7.40 [10:41:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.70, 8.17, 7.69 [10:41:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.11, 8.83, 8.27 [10:42:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.45, 22.03, 21.09 [10:43:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.23, 6.60, 6.37 [10:43:04] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.56, 7.47, 6.84 [10:43:10] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.58, 7.75, 7.08 [10:45:01] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.23, 6.02, 6.18 [10:45:01] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.68, 7.02, 6.74 [10:45:10] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.97, 7.80, 7.18 [10:45:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.01, 7.72, 7.63 [10:46:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.95, 21.60, 21.18 [10:47:08] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.21, 8.31, 7.44 [10:49:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.55, 7.49, 6.75 [10:49:05] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.62, 7.77, 7.36 [10:49:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.90, 8.00, 7.72 [10:51:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.17, 6.90, 6.62 [10:51:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.62, 7.81, 7.69 [10:53:01] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.89, 6.21, 6.40 [10:56:43] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.77, 6.49, 6.75 [10:59:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.73, 6.16, 6.70 [10:59:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.12, 6.44, 7.54 [11:01:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.25, 5.68, 6.72 [11:02:54] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.36, 6.53, 7.76 [11:03:41] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.92, 6.57, 6.42 [11:04:53] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.13, 7.15, 7.84 [11:05:35] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.06, 5.86, 6.20 [11:05:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:06:32] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.20, 19.32, 20.15 [11:06:53] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.41, 7.04, 7.73 [11:08:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:10:53] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.68, 7.75, 7.87 [11:11:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.86, 7.50, 7.33 [11:13:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.32, 7.14, 6.53 [11:13:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.12, 7.23, 6.84 [11:13:50] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.33, 7.29, 6.75 [11:15:10] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.74, 7.13, 6.59 [11:15:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.63, 6.61, 6.67 [11:15:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.83, 7.21, 7.31 [11:17:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.89, 8.30, 7.72 [11:19:01] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.41, 6.72, 6.54 [11:19:42] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.32, 6.63, 6.64 [11:20:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.97, 21.60, 20.54 [11:22:14] PROBLEM - sportsanalytics.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.89, 7.40, 6.88 [11:23:05] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.74, 6.97, 6.70 [11:25:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.89, 8.16, 7.21 [11:25:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.33, 7.53, 6.94 [11:25:13] PROBLEM - sportsanalytics.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns102.registrar-servers.com.', 'dns101.registrar-servers.com.', 'dns1.registrar-servers.com.', 'dns2.registrar-servers.com.'], 'CNAME': None} [11:26:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.32, 7.06, 6.86 [11:26:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.93, 7.45, 6.97 [11:27:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.41, 7.12, 6.86 [11:28:22] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.60, 6.79, 6.78 [11:29:26] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.68, 7.78, 7.14 [11:30:51] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.24, 7.92, 7.27 [11:34:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.97, 7.73, 7.41 [11:35:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.90, 7.23, 7.17 [11:35:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.85, 7.39, 7.87 [11:37:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.11, 7.47, 7.60 [11:38:05] PROBLEM - sportsanalytics.wiki - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])rDNS WARNING - reverse DNS entry for sportsanalytics.wiki could not be found [11:41:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.07, 8.00, 7.88 [11:42:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.41, 6.02, 6.72 [11:43:23] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.25, 19.42, 20.23 [11:43:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.96, 7.60, 7.76 [11:45:07] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.54, 6.43, 6.76 [11:47:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.65, 7.49, 7.62 [11:48:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:49:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.81, 8.36, 7.36 [11:50:20] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.74, 7.15, 6.86 [11:50:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [11:50:27] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:51:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.13, 7.78, 7.34 [11:51:05] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.48, 22.53, 21.08 [11:52:25] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:52:27] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.395 second response time [11:52:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:53:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.83, 7.41, 7.26 [11:53:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.32, 22.06, 21.07 [11:53:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.55, 7.83, 7.42 [11:54:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.48, 7.63, 7.17 [11:54:29] PROBLEM - sportsanalytics.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address sportsanalytics.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [11:55:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.04, 7.59, 7.34 [11:55:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.44, 8.13, 7.58 [11:58:59] RECOVERY - sportsanalytics.wiki - reverse DNS on sslhost is OK: SSL OK - sportsanalytics.wiki reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [11:59:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.22, 7.18, 7.26 [12:01:54] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.65, 7.00, 6.87 [12:02:11] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.27, 6.52, 6.77 [12:03:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.25, 7.75, 7.45 [12:03:45] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.52, 7.19, 7.50 [12:03:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.94, 7.65, 7.98 [12:05:49] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.37, 7.89, 7.21 [12:06:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.36, 7.50, 7.10 [12:06:32] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.96, 19.18, 20.21 [12:07:38] PROBLEM - sportsanalytics.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sportsanalytics.wiki could not be found [12:07:47] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.94, 7.80, 7.26 [12:08:24] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.37, 20.13, 19.06 [12:09:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [12:09:45] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.06, 8.84, 7.70 [12:10:53] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.94, 6.93, 7.96 [12:11:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [12:11:55] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:12:53] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.96, 7.72, 8.12 [12:15:12] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:16:08] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.540 second response time [12:16:53] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.29, 7.11, 7.85 [12:17:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:19:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [12:19:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.17, 7.67, 7.34 [12:20:53] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.74, 8.03, 8.01 [12:20:58] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:21:10] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.04, 20.22, 20.00 [12:21:15] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:21:17] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:24] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:21:36] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:21:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:21:54] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:24] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:22:38] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:22:40] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:23:06] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.56, 19.99, 19.95 [12:23:34] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.335 second response time [12:23:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.81, 7.69, 7.43 [12:23:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:23:52] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.029 second response time [12:24:22] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.021 second response time [12:24:36] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.382 second response time [12:24:37] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.531 second response time [12:25:02] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.091 second response time [12:25:06] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.355 second response time [12:25:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:25:25] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.744 second response time [12:25:26] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.251 second response time [12:25:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.27, 7.52, 7.21 [12:25:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.08, 7.94, 7.57 [12:26:17] PROBLEM - rangpurpedia.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.71, 7.32, 7.16 [12:29:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [12:31:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:32:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:33:18] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.76, 7.22, 7.95 [12:34:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.51, 19.65, 20.32 [12:34:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.74, 20.61, 20.19 [12:35:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.09, 7.97, 7.84 [12:37:25] PROBLEM - www.rangpurpedia.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:39:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.52, 6.78, 7.81 [12:39:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.25, 6.45, 6.79 [12:39:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:40:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.78, 7.33, 7.95 [12:41:48] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.00, 7.69, 7.72 [12:43:05] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.44, 7.23, 7.44 [12:44:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.36, 7.85, 8.01 [12:45:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.52, 8.31, 7.47 [12:47:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.34, 8.03, 7.85 [12:48:06] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.93, 20.45, 20.17 [12:48:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.31, 22.69, 21.49 [12:49:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.87, 7.43, 7.64 [12:50:40] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [12:50:44] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:50:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:50:55] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:51:02] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:51:03] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:51:24] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:56] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:52:29] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:52:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.91, 23.08, 21.94 [12:52:56] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.447 second response time [12:52:56] PROBLEM - www.rangpurpedia.xyz - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns102.registrar-servers.com.', 'dns2.registrar-servers.com.', 'dns1.registrar-servers.com.', 'dns101.registrar-servers.com.'], 'CNAME': None} [12:53:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 7.93, 8.04, 7.88 [12:53:02] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.302 second response time [12:55:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.80, 7.82, 7.80 [12:56:20] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.33, 5.35, 4.66 [12:56:31] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:56:41] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:24] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:57:29] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:58:01] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.58, 19.81, 20.10 [12:58:06] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.494 second response time [12:58:17] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.81, 5.32, 4.74 [12:58:35] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.005 second response time [12:58:39] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.435 second response time [12:58:42] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.917 second response time [12:58:53] PROBLEM - rangpurpedia.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rangpurpedia.xyz could not be found [12:59:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.73, 7.81, 7.76 [12:59:15] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.769 second response time [12:59:24] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.560 second response time [12:59:28] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.322 second response time [12:59:34] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.266 second response time [12:59:54] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.839 second response time [13:00:14] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:00:15] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.89, 4.67, 4.58 [13:00:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:01:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.99, 7.62, 7.67 [13:01:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.37, 7.91, 7.94 [13:01:30] PROBLEM - www.rangpurpedia.xyz - LetsEncrypt on sslhost is CRITICAL: connect to address www.rangpurpedia.xyz and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:03:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.32, 7.93, 7.93 [13:06:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.45, 7.27, 7.96 [13:07:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.08, 8.02, 7.77 [13:07:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.26, 3.01 [13:08:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.71, 5.07, 4.75 [13:08:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.98, 7.94, 8.12 [13:08:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.32, 22.68, 22.04 [13:08:56] PROBLEM - rangpurpedia.xyz - LetsEncrypt on sslhost is CRITICAL: connect to address rangpurpedia.xyz and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:09:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.48, 7.75, 7.70 [13:09:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.24, 2.84, 2.88 [13:09:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.07, 7.81, 7.91 [13:09:59] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.64, 4.53, 4.59 [13:10:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.38, 22.54, 22.07 [13:10:38] PROBLEM - rangpurpedia.xyz - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])rDNS WARNING - reverse DNS entry for rangpurpedia.xyz could not be found [13:11:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.60, 7.88, 7.74 [13:11:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.88, 8.26, 8.06 [13:11:30] PROBLEM - www.rangpurpedia.xyz - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])rDNS WARNING - reverse DNS entry for www.rangpurpedia.xyz could not be found [13:12:31] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.01, 23.17, 22.36 [13:13:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.77, 7.34, 7.58 [13:14:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.46, 6.97, 7.73 [13:14:27] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.68, 22.64, 22.27 [13:15:53] RECOVERY - rangpurpedia.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'rangpurpedia.xyz' will expire on Mon 21 Feb 2022 17:55:55 GMT +0000. [13:16:34] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.568 second response time [13:17:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.93, 7.77, 7.66 [13:17:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.93, 7.22, 7.89 [13:17:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb [13:18:51] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:18:56] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:19:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.62, 7.25, 7.48 [13:20:09] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:20:10] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:20:11] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:20:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.98, 23.58, 22.77 [13:21:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.31, 7.32, 7.74 [13:21:34] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:21:50] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:22:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.13, 7.17, 7.33 [13:22:13] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.696 second response time [13:22:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.38, 22.77, 22.57 [13:23:03] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.69, 8.05, 7.78 [13:23:09] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:13] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:23:13] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.904 second response time [13:23:22] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:23:23] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:24:28] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.361 second response time [13:24:30] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.257 second response time [13:24:47] PROBLEM - rangpurpedia.xyz - LetsEncrypt on sslhost is CRITICAL: connect to address rangpurpedia.xyz and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:25:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.60, 7.57, 7.77 [13:25:23] Reception123: expired ^ [13:25:40] Can you make pr for after switch [13:26:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.36, 7.36, 7.38 [13:26:10] RhinosF1: probably easier to just do the whole thing after as if it's expired before the bot I'll need to do private keys too [13:26:32] Reception123: no the domain is [13:26:44] Like it's not registered anymore [13:26:49] RhinosF1: oh, I thought you meant the cert was [13:26:50] It's showing a namecheap page [13:26:53] in that case ok I'll make the removal PR [13:28:32] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:28:35] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/27c33b622f67 [13:28:37] [02miraheze/ssl] 07Reception123 0327c33b6 - remove rangpurpedia.xyz cert (expired domain) [13:28:38] [02ssl] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/ssl [13:28:40] [02ssl] 07Reception123 opened pull request 03#473: remove rangpurpedia.xyz cert (expired domain) - 13https://github.com/miraheze/ssl/pull/473 [13:29:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.16, 7.87, 7.83 [13:29:36] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/27c33b622f67...73c6e22bc323 [13:29:37] [02miraheze/ssl] 07Reception123 0373c6e22 - Update certs.yaml [13:29:38] [02ssl] 07Reception123 synchronize pull request 03#473: remove rangpurpedia.xyz cert (expired domain) - 13https://github.com/miraheze/ssl/pull/473 [13:29:49] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/73c6e22bc323...6824253388bb [13:29:51] [02miraheze/ssl] 07Reception123 036824253 - Update redirects.yaml [13:29:52] [02ssl] 07Reception123 synchronize pull request 03#473: remove rangpurpedia.xyz cert (expired domain) - 13https://github.com/miraheze/ssl/pull/473 [13:30:26] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.017 second response time [13:30:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.49, 23.28, 22.80 [13:31:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.10, 8.29, 7.98 [13:31:02] [02ssl] 07Reception123 deleted branch 03JohnFLewis-patch-2 - 13https://github.com/miraheze/ssl [13:31:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:31:03] [02miraheze/ssl] 07Reception123 deleted branch 03JohnFLewis-patch-2 [13:31:05] [02ssl] 07Reception123 deleted branch 03JohnFLewis-patch-1 - 13https://github.com/miraheze/ssl [13:31:06] [02miraheze/ssl] 07Reception123 deleted branch 03JohnFLewis-patch-1 [13:31:21] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.505 second response time [13:31:47] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.431 second response time [13:31:58] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.701 second response time [13:32:18] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.289 second response time [13:32:18] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.257 second response time [13:33:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.49, 7.51, 7.73 [13:33:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:33:31] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.009 second response time [13:34:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.35, 23.92, 23.18 [13:34:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:39:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [13:39:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:41:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.73, 7.93, 7.76 [13:41:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.74, 7.71, 7.38 [13:41:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:45:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.66, 7.74, 7.78 [13:45:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.67, 7.24, 7.33 [13:45:08] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:45:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [13:45:35] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:20] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:46:24] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:46:34] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:46:42] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:46:43] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:47:02] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:48:18] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [13:49:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.69, 8.24, 7.93 [13:50:09] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.69, 6.20, 6.77 [13:50:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.90, 23.14, 22.75 [13:50:41] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.057 second response time [13:50:50] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.275 second response time [13:51:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.74, 7.62, 7.74 [13:51:21] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.008 second response time [13:51:45] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.072 second response time [13:51:50] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:14] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:52:23] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.138 second response time [13:53:07] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:54:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.10, 7.53, 7.17 [13:54:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.92, 23.50, 23.03 [13:55:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.42, 7.86, 7.80 [13:55:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.92, 7.83, 7.54 [13:56:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.98, 7.69, 7.27 [13:56:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.29, 24.35, 23.41 [13:56:32] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.85, 21.32, 19.77 [13:57:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:57:19] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:58:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.99, 8.19, 7.51 [13:58:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.63, 5.21, 4.54 [13:59:26] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.800 second response time [13:59:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.05, 7.43, 7.94 [13:59:42] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.06, 7.74, 7.43 [14:01:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:01:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.69, 7.83, 7.71 [14:01:42] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.543 second response time [14:01:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:02:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.01, 8.08, 7.58 [14:02:35] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.827 second response time [14:02:36] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.722 second response time [14:02:46] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.44, 4.86, 4.55 [14:03:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.48, 7.54, 7.75 [14:04:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.64, 7.25, 7.34 [14:04:17] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.774 second response time [14:04:29] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.90, 19.99, 19.97 [14:04:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.54, 23.25, 23.30 [14:05:30] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.799 second response time [14:05:47] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.028 second response time [14:06:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.19, 7.61, 7.46 [14:07:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.75, 7.65, 7.72 [14:08:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.41, 23.74, 23.46 [14:09:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.97, 7.98, 7.66 [14:10:34] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.60, 5.56, 5.01 [14:11:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.91, 7.86, 7.65 [14:11:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.31, 7.65, 7.59 [14:12:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.49, 23.88, 23.58 [14:12:40] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:12:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:12:55] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:13:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.28, 8.11, 7.75 [14:13:18] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:23] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.53, 20.64, 20.36 [14:13:52] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:14:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.63, 7.67, 7.70 [14:14:17] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:14:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.48, 24.12, 23.70 [14:16:34] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.66, 5.08, 4.96 [14:17:15] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:17:23] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.49, 20.33, 20.31 [14:19:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.23, 7.80, 7.73 [14:20:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.88, 5.85, 5.28 [14:20:47] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:05] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.94, 7.79, 7.84 [14:21:06] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:21:23] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.46, 21.29, 20.69 [14:21:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.12, 7.97, 7.81 [14:22:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.01, 8.03, 7.84 [14:22:34] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.73, 5.65, 5.27 [14:22:44] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.790 second response time [14:23:08] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.999 second response time [14:23:46] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:23:59] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:07] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:24:11] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:24:31] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:25:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.17, 8.06, 7.95 [14:25:37] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:25:53] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.902 second response time [14:26:03] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.629 second response time [14:26:10] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.151 second response time [14:26:14] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.412 second response time [14:26:29] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.422 second response time [14:28:34] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.07, 4.66, 4.94 [14:29:19] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.463 second response time [14:29:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.773 second response time [14:29:37] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:30:10] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.562 second response time [14:30:35] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.886 second response time [14:30:38] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.091 second response time [14:31:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 7.60, 7.83 [14:31:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:33:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.70, 8.40, 8.09 [14:33:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.47, 7.95, 7.93 [14:33:46] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:35:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.18, 8.04, 7.96 [14:35:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [14:35:55] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.560 second response time [14:36:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.92, 23.07, 23.86 [14:38:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.23, 23.22, 23.79 [14:38:58] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:38:59] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:39:02] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:39:08] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:39:10] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.029 second response time [14:40:22] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:41:07] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:41:24] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:41:49] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:42:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.81, 23.21, 23.64 [14:42:33] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:46] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:05] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:43:16] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.83, 5.36, 5.12 [14:43:16] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:43:26] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:43:29] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:45:22] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.592 second response time [14:45:31] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.768 second response time [14:45:32] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.912 second response time [14:46:37] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.541 second response time [14:47:11] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.14, 5.80, 5.32 [14:47:23] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.008 second response time [14:47:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.85, 7.27, 7.82 [14:47:35] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.320 second response time [14:47:45] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.329 second response time [14:47:51] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.319 second response time [14:48:02] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:48:19] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:48:27] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.84, 23.55, 23.62 [14:49:08] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.53, 5.68, 5.34 [14:49:30] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.54, 8.20, 8.07 [14:49:45] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:49:52] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:50:00] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:50:01] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.428 second response time [14:50:16] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.048 second response time [14:50:24] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.140 second response time [14:50:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.10, 23.70, 23.70 [14:51:17] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.705 second response time [14:51:22] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.256 second response time [14:51:44] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.344 second response time [14:51:50] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.891 second response time [14:51:50] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:57] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.608 second response time [14:52:02] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:11] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:13] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.89, 24.27, 23.91 [14:53:02] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.83, 4.64, 5.03 [14:53:23] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.06, 19.21, 20.22 [14:55:53] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.112 second response time [14:56:03] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.600 second response time [14:56:10] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.031 second response time [14:56:14] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.973 second response time [14:56:14] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.387 second response time [14:56:22] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.476 second response time [14:56:29] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.533 second response time [14:56:43] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:56:49] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:06] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.177 second response time [14:57:28] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.296 second response time [14:57:33] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:57:40] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:57:43] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.256 second response time [14:57:43] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:11] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:58:26] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:58:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.12, 23.75, 23.84 [14:59:38] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.856 second response time [14:59:43] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.877 second response time [15:00:08] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.871 second response time [15:00:26] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.569 second response time [15:02:20] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.73, 20.61, 20.48 [15:02:27] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:02:59] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.172 second response time [15:03:07] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.379 second response time [15:03:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.73, 7.22, 7.87 [15:04:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:04:31] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.288 second response time [15:05:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:07:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.06, 7.47, 7.78 [15:07:40] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.55, 4.90, 4.81 [15:07:57] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.159 second response time [15:08:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [15:09:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.05, 7.05, 7.58 [15:09:40] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.51, 5.17, 4.90 [15:10:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.83, 23.91, 23.38 [15:11:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.29, 7.85, 7.80 [15:11:37] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.60, 4.95, 4.86 [15:11:56] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:12:06] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:12:16] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.81, 23.70, 23.37 [15:12:40] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:13:01] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:15] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.97, 20.08, 20.33 [15:14:28] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.44, 24.09, 23.56 [15:16:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.38, 23.34, 23.32 [15:18:27] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.678 second response time [15:19:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.61, 5.51, 5.09 [15:19:37] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:20:12] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:20:14] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.18, 21.16, 20.58 [15:20:27] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:20:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 23.72, 24.04, 23.64 [15:20:34] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:20:36] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:44] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:48] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:20:48] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:21:10] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:21:14] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:21:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.02, 5.06, 4.99 [15:21:32] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:21:52] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:05] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:14] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.37, 21.51, 20.78 [15:22:15] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.65, 23.29, 23.42 [15:22:39] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.114 second response time [15:22:42] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.797 second response time [15:22:54] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.302 second response time [15:22:54] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.961 second response time [15:22:57] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:05] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:09] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:18] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:21] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.511 second response time [15:23:28] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:30] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 2.882 second response time [15:23:32] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:35] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:43] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.701 second response time [15:24:47] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.318 second response time [15:24:56] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.349 second response time [15:25:05] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.327 second response time [15:25:48] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.349 second response time [15:26:16] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.972 second response time [15:26:22] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.242 second response time [15:26:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.51, 23.22, 23.32 [15:26:39] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:26:54] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw11 [15:26:56] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:27:01] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:27:04] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:27:08] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.168 second response time [15:27:10] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:27:17] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:27:20] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.037 second response time [15:27:21] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.371 second response time [15:27:32] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.291 second response time [15:27:43] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.617 second response time [15:27:47] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:28:15] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.245 second response time [15:28:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.43, 23.26, 23.32 [15:28:33] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.439 second response time [15:28:35] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.427 second response time [15:28:39] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.381 second response time [15:28:54] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [15:29:06] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.66, 7.62, 7.97 [15:30:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.48, 23.89, 23.50 [15:30:32] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.339 second response time [15:30:41] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:31:05] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.76, 7.75, 7.96 [15:31:06] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:31:12] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.595 second response time [15:31:14] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.783 second response time [15:31:26] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:32:00] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.592 second response time [15:32:03] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.661 second response time [15:32:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.45, 23.71, 23.50 [15:33:06] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.797 second response time [15:33:36] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.263 second response time [15:33:37] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.252 second response time [15:33:39] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.596 second response time [15:33:57] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.634 second response time [15:34:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.97, 24.60, 23.85 [15:35:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.90, 7.85, 7.96 [15:35:44] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:35:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:36:08] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.04, 19.92, 20.35 [15:36:09] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:36:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:37:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.04, 8.06, 8.03 [15:37:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.47, 3.18 [15:38:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.95, 23.75, 23.69 [15:39:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.22, 3.12 [15:40:06] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.96, 20.41, 20.49 [15:40:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:41:38] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb [15:41:52] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.81, 5.09, 4.87 [15:41:58] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:42:05] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.65, 20.14, 20.38 [15:42:05] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:42:29] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:42:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.40, 24.14, 23.81 [15:42:39] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:43:49] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.21, 4.70, 4.76 [15:46:02] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.14, 20.65, 20.53 [15:46:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:46:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.04, 22.97, 23.45 [15:47:42] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.08, 5.89, 5.24 [15:48:01] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.53, 20.19, 20.37 [15:48:22] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.499 second response time [15:48:33] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.957 second response time [15:48:46] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.236 second response time [15:48:52] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.917 second response time [15:49:39] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.59, 5.40, 5.14 [15:50:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [15:50:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.28, 24.13, 23.71 [15:51:36] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.57, 5.09, 5.05 [15:52:47] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:01] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:04] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:18] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:56:10] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.502 second response time [15:56:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.65, 23.49, 23.62 [15:56:45] Hi all. I'm about to fix the issue with the Cloudflare and turn off proxying for my site, however it's sort of a dealbreaker for me to lose all of these stats and we're considering migrating off of Miraheze because of this. Do you guys provide these stats at all? [15:56:46] https://cdn.discordapp.com/attachments/808001911868489748/931577587417903144/unknown.png [15:57:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.93, 7.68, 7.97 [15:57:26] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.43, 5.17, 5.09 [15:57:44] @jmk: I can get the top 2 [15:57:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.42, 3.11 [15:58:08] is there like a page for that? [15:58:18] the latter aren't as relevant but cache hit rate is on grafana for all requests overall [15:58:46] Special:Analytics should let you calculate it on a 30 day rolling basis [15:58:59] i'm happy to grab it on request from matomo though [15:59:11] I see 🤔 [15:59:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.19, 5.74, 5.29 [15:59:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.06, 3.01 [16:00:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.48, 24.02, 23.78 [16:00:32] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:01:16] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.504 second response time [16:01:18] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.092 second response time [16:01:36] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.052 second response time [16:01:47] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.294 second response time [16:02:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:02:33] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.097 second response time [16:02:49] @jmk which wiki [16:03:20] robloxapi.wiki [16:03:23] they are closing us down tomorrow [16:03:29] because we're proxying through cf [16:03:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:04:51] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.64, 21.27, 20.62 [16:05:17] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.22, 5.59, 5.38 [16:05:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.61, 4.00, 3.37 [16:06:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [16:06:49] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.572 second response time [16:07:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.24, 7.78, 7.82 [16:07:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:09:12] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:09:12] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.62, 6.63, 5.80 [16:09:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.39, 3.32, 3.25 [16:09:44] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:09:46] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:29] !log [rhinos@mwtask1] sudo -u www-data php /srv/mediawiki/w/extensions/MatomoAnalytics/maintenance/addMissingMatomos.php --wiki=loginwiki (END - exit=0) [16:10:35] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:10:44] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:10:48] [02puppet] 07RhinosF1 opened pull request 03#2313: ssl-renew: connect to mwtask111 - 13https://github.com/miraheze/puppet/pull/2313 [16:11:02] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.245 second response time [16:11:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.13, 7.29, 7.62 [16:11:12] paladox: ^ [16:11:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.37, 7.06, 7.98 [16:11:29] [02puppet] 07paladox closed pull request 03#2313: ssl-renew: connect to mwtask111 - 13https://github.com/miraheze/puppet/pull/2313 [16:11:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/34361d2c2d75...9a23c1b5f055 [16:11:32] [02miraheze/puppet] 07RhinosF1 039a23c1b - ssl-renew: connect to mwtask111 (#2313) [16:11:38] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:12:04] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:12:25] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:12:48] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.95, 20.17, 20.40 [16:13:01] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:06] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.86, 7.91, 7.81 [16:13:08] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.80, 5.39, 5.49 [16:13:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:23] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.614 second response time [16:13:28] !log [rhinos@mwtask1] sudo -u www-data php /srv/mediawiki/w/extensions/MatomoAnalytics/maintenance/modifyMatomo.php --wiki=roapiwiki (END - exit=0) [16:13:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.31, 7.69, 8.08 [16:13:33] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:13:43] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.986 second response time [16:13:52] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:14:08] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.500 second response time [16:14:26] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:14:33] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fe81c30d0827...f8875e5b39c5 [16:14:35] [02miraheze/ssl] 07paladox 03f8875e5 - Update data.memipedia.ga.crt [16:15:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.66, 7.75, 7.79 [16:15:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:15:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.99, 7.47, 7.93 [16:15:35] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.210 second response time [16:15:41] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/ [16:15:41] b/nagios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query return self.resolve(qname, rdtype, rdclass, tcp, source, File "/usr/lib/python3/dist-packages/dns/resolver.py", line 10 [16:15:41] resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikipedia.yogfront.ooo. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:15:58] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.277 second response time [16:16:34] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.956 second response time [16:16:40] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.772 second response time [16:16:41] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:16:46] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.08, 20.97, 20.71 [16:17:01] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.921 second response time [16:17:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.42, 7.87, 7.82 [16:17:05] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.65, 6.62, 5.94 [16:17:09] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:17:09] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.532 second response time [16:17:27] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:17:33] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:17:38] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:17:46] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:18:21] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.535 second response time [16:19:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.39, 5.34, 5.55 [16:19:39] I will get you our stats but apparently i don't have matomo access [16:19:53] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:20:30] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:20:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.76, 3.43 [16:21:07] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.13, 2.34, 1.31 [16:21:28] I see [16:21:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.19, 7.92, 7.93 [16:22:32] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:22:38] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.09, 3.15, 3.25 [16:22:44] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:22:46] RECOVERY - wikipedia.yogfront.ooo - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/ [16:22:46] gios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - wikipedia.yogfront.ooo reverse DNS resolves to cp20.miraheze.org - CNAME OK [16:23:07] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.52, 1.64, 1.17 [16:23:28] I can get the stats if needed? [16:23:41] i got it [16:24:26] Alright [16:24:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.02, 6.30, 5.81 [16:25:49] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:21] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.759 second response time [16:26:41] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.276 second response time [16:26:45] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.657 second response time [16:26:52] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.423 second response time [16:26:56] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.753 second response time [16:27:16] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.176 second response time [16:27:16] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.837 second response time [16:27:38] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.676 second response time [16:27:45] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.308 second response time [16:28:01] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.998 second response time [16:28:07] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.318 second response time [16:28:12] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.673 second response time [16:28:15] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.773 second response time [16:28:16] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.142 second response time [16:28:22] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:28:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 4.12, 3.59 [16:28:43] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:28:50] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:28:59] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:29:10] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.239 second response time [16:29:24] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.017 second response time [16:30:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.89, 23.32, 24.00 [16:30:47] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.93, 5.95, 5.88 [16:30:50] RECOVERY - data.memipedia.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'data.memipedia.ga' will expire on Thu 14 Apr 2022 15:13:58 GMT +0000. [16:31:12] RECOVERY - data.memipedia.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'data.memipedia.ga' will expire on Thu 14 Apr 2022 15:13:58 GMT +0000. [16:31:39] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.778 second response time [16:31:40] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.222 second response time [16:32:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 23.72, 24.01, 24.20 [16:33:00] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.795 second response time [16:33:00] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.164 second response time [16:33:08] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.359 second response time [16:33:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.64, 7.57, 7.94 [16:34:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.86, 3.64 [16:34:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.16, 23.45, 23.95 [16:36:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.41, 25.50, 24.64 [16:37:22] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:37:27] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:37:29] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:37:46] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr [16:37:47] on3/dist-packages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query yogfront.ooo. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:38:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.23, 2.88, 3.28 [16:38:14] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:38:15] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:38:22] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:30] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:41:30] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.38, 8.21, 7.92 [16:42:38] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.27, 19.84, 20.25 [16:42:58] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.240 second response time [16:43:15] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.324 second response time [16:43:38] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.052 second response time [16:43:44] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.004 second response time [16:44:31] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.315 second response time [16:44:32] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.315 second response time [16:44:33] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.312 second response time [16:44:34] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.046 second response time [16:45:36] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:46:04] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:46:09] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:46:19] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.28, 5.52, 5.44 [16:46:43] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:46:55] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:47:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.26, 6.69, 7.61 [16:47:24] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:47:27] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:41] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:48:15] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.073 second response time [16:48:33] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.17, 5.27, 5.35 [16:48:40] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 2.530 second response time [16:48:52] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:48:53] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 1.796 second response time [16:48:56] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:58] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:49:02] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:49:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.59, 7.69, 7.87 [16:49:34] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.636 second response time [16:50:46] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.028 second response time [16:50:52] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.321 second response time [16:50:57] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.315 second response time [16:51:02] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.321 second response time [16:51:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.13, 7.65, 7.77 [16:51:15] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:25] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:51:26] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.76, 8.34, 8.10 [16:51:38] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:50] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:57] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [16:52:23] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:52:24] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:52:45] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:53:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.81, 7.40, 7.64 [16:53:22] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.881 second response time [16:53:32] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.326 second response time [16:53:33] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.680 second response time [16:53:52] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.241 second response time [16:54:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.41, 7.69, 8.00 [16:54:20] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.603 second response time [16:54:20] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.826 second response time [16:54:29] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:54:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.27, 5.92, 5.59 [16:54:44] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.006 second response time [16:55:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.94, 7.92, 7.80 [16:55:05] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:55:17] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:24] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:55:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.08, 7.32, 7.78 [16:55:30] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:55:30] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.40, 20.56, 20.23 [16:55:45] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.452 second response time [16:55:54] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.521 second response time [16:56:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.27, 8.44, 8.25 [16:56:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.18, 2.99 [16:57:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.84, 7.74, 7.86 [16:57:40] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.007 second response time [16:58:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.11, 3.07, 2.98 [16:58:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.22, 7.58, 7.96 [16:58:11] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.356 second response time [16:58:34] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.52, 5.58, 5.58 [16:58:34] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/ [16:58:34] b/nagios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query return self.resolve(qname, rdtype, rdclass, tcp, source, File "/usr/lib/python3/dist-packages/dns/resolver.py", line 10 [16:58:35] resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikipedia.yogfront.ooo. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:58:35] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.834 second response time [16:58:53] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.007 second response time [16:58:56] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.327 second response time [16:58:59] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/ [16:58:59] n3/dist-packages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikipedia.yogfront.ooo. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:59:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.86, 7.48, 7.67 [16:59:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.87, 7.76, 7.87 [16:59:29] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.50, 19.98, 20.08 [17:00:18] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:00:34] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:00:38] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:01:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.34, 7.76, 7.74 [17:01:14] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:01:29] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:17] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:04:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.95, 6.12, 5.78 [17:04:59] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.010 second response time [17:05:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.49, 7.58, 7.66 [17:05:15] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.007 second response time [17:05:27] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.117 second response time [17:05:41] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.726 second response time [17:05:44] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 5.897 second response time [17:05:56] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:02] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.431 second response time [17:06:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.80, 7.57, 7.71 [17:06:10] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.719 second response time [17:06:17] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.378 second response time [17:06:38] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.654 second response time [17:06:40] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.275 second response time [17:06:49] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.852 second response time [17:08:11] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.37, 7.04, 7.50 [17:09:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.44, 7.38, 7.98 [17:09:59] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:10:29] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:10:40] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:10:43] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:11:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.33, 7.80, 7.71 [17:11:07] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.02, 1.59, 1.05 [17:11:12] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:11:20] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:11:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.23, 7.36, 7.45 [17:11:29] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:01] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.15, 7.52, 7.55 [17:12:11] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:16] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:23] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.98, 20.50, 20.23 [17:12:35] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:37] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:43] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:48] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [17:12:59] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [17:13:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.88, 7.39, 7.57 [17:13:04] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:13:07] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.01, 1.29, 1.00 [17:13:11] no longer pointing [17:13:15] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:13:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.15, 7.66, 7.92 [17:14:00] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:14:01] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:14:05] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:14:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.94, 7.18, 7.42 [17:14:22] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.67, 19.59, 19.95 [17:14:34] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.73, 5.33, 5.69 [17:14:39] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.300 second response time [17:14:41] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.349 second response time [17:14:47] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.354 second response time [17:14:52] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.810 second response time [17:15:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.08, 7.13, 7.37 [17:15:35] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.790 second response time [17:15:36] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.786 second response time [17:15:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.38, 7.63, 7.89 [17:16:04] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.542 second response time [17:16:08] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.491 second response time [17:16:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.91, 7.79, 7.61 [17:16:10] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.793 second response time [17:16:11] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.513 second response time [17:16:17] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.477 second response time [17:16:18] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.893 second response time [17:16:19] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.047 second response time [17:16:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.95, 5.62, 5.73 [17:16:34] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.304 second response time [17:16:51] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.812 second response time [17:17:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.02, 7.61, 7.61 [17:17:06] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.780 second response time [17:17:10] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.889 second response time [17:17:26] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.509 second response time [17:17:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.17, 7.64, 7.53 [17:17:32] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.320 second response time [17:18:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.80, 7.80, 7.64 [17:19:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.83, 7.05, 7.41 [17:19:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.57, 7.56, 7.51 [17:19:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.23, 8.34, 8.11 [17:19:59] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.15, 8.13, 7.79 [17:20:35] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.81, 5.19, 5.53 [17:20:43] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:49] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:03] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:21:18] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.74, 8.47, 7.86 [17:21:30] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:35] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:48] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:22:00] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:22:01] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.985 second response time [17:22:41] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.383 second response time [17:23:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.02, 7.34, 7.43 [17:23:26] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.118 second response time [17:23:27] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.600 second response time [17:23:43] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.036 second response time [17:23:44] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.322 second response time [17:24:03] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.458 second response time [17:24:59] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.221 second response time [17:25:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.14, 2.96 [17:25:15] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.982 second response time [17:26:55] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rd [17:26:55] tname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query return self.resolve(qname, rdtype, rdclass, tcp, source, File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(st [17:26:55] ifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404021739959717 seconds [17:27:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.49, 7.57, 7.51 [17:27:06] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.22, 3.16, 2.99 [17:27:41] RhinosF1: will do PR to remove in a bit if you want. [17:28:11] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.30, 7.82, 7.84 [17:28:11] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:28:17] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:28:22] CosmicAlpha: k [17:28:32] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:28:32] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:28:38] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.01, 7.63, 7.54 [17:30:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.02, 8.19, 7.98 [17:30:31] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.745 second response time [17:30:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.11, 5.78, 5.65 [17:30:37] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.263 second response time [17:31:07] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:31:18] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.372 second response time [17:31:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:32:15] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:32:18] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.525 second response time [17:32:22] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.213 second response time [17:32:34] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.58, 5.52, 5.59 [17:32:43] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:32:43] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.420 second response time [17:33:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.87, 7.99, 7.73 [17:33:12] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.995 second response time [17:33:31] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:33:54] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:56] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:33:58] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [17:34:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.14, 7.99, 7.97 [17:35:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.09, 7.93, 7.73 [17:35:35] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.921 second response time [17:35:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [17:35:51] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.805 second response time [17:35:55] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.460 second response time [17:36:21] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.637 second response time [17:37:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.31, 7.82, 7.72 [17:38:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.50, 8.06, 7.99 [17:38:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.30, 7.03, 6.10 [17:39:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.47, 7.85, 7.72 [17:39:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:40:11] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:40:23] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:41:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.52, 7.67, 7.67 [17:42:44] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:42:47] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:43:01] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.25, 7.74, 7.68 [17:43:10] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:43:21] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:43:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9a23c1b5f055...d1916c271418 [17:43:58] [02miraheze/puppet] 07paladox 03d1916c2 - matomo: Set noreply email [17:44:01] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:45:05] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.068 second response time [17:45:17] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.495 second response time [17:45:24] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.188 second response time [17:46:08] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.811 second response time [17:46:46] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.878 second response time [17:46:48] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.509 second response time [17:46:50] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.551 second response time [17:47:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.17, 7.77, 7.75 [17:47:38] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.610 second response time [17:48:23] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.481 second response time [17:48:54] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a576b4d9cbdb [17:48:55] [02miraheze/mw-config] 07RhinosF1 03a576b4d - parsercache: up from 7->10 days [17:48:57] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-5 - 13https://github.com/miraheze/mw-config [17:48:58] [02mw-config] 07RhinosF1 opened pull request 03#4377: parsercache: up from 7->10 days - 13https://github.com/miraheze/mw-config/pull/4377 [17:49:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.73, 8.03, 7.86 [17:49:14] CosmicAlpha: if we want it ^ [17:49:20] For probably tommorow [17:49:58] miraheze/mw-config - RhinosF1 the build passed. [17:50:36] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:51:06] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:51:33] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:53] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:51:54] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:05] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:45] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.26, 7.73, 7.84 [17:53:14] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:14] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.050 second response time [17:53:18] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:54:34] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.82, 5.65, 5.94 [17:55:06] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rd [17:55:06] tname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query return self.resolve(qname, rdtype, rdclass, tcp, source, File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(st [17:55:06] ifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405766725540161 seconds [17:55:09] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.75, 1.56, 1.32 [17:56:13] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.308 second response time [17:56:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.68, 6.39, 6.18 [17:56:34] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.010 second response time [17:57:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.01, 8.10, 7.93 [17:57:09] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.54, 1.15, 1.20 [17:57:19] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.449 second response time [17:57:22] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.010 second response time [17:57:23] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.460 second response time [17:58:32] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:00:00] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.403 second response time [18:00:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:00:33] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.830 second response time [18:02:04] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [18:02:31] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.663 second response time [18:02:33] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.285 second response time [18:03:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.55, 7.65, 7.95 [18:03:31] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.696 second response time [18:04:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.11, 6.95, 7.75 [18:04:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [18:05:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.18, 7.93, 8.02 [18:05:32] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.994 second response time [18:06:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.19, 7.18, 7.71 [18:06:34] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.95, 5.77, 6.00 [18:07:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.96, 7.44, 7.82 [18:08:35] JohnLewis: paladox: if you have a chance, NDK needs to be removed from stewards@ [18:08:41] Per the resignation [18:09:09] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rd [18:09:09] tname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query return self.resolve(qname, rdtype, rdclass, tcp, source, File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(st [18:09:09] ifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407543897628784 seconds [18:09:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.13, 7.59, 7.82 [18:09:29] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/puppet/compare/d1916c271418...be382a912b7d [18:09:30] [02miraheze/puppet] 07JohnFLewis 03be382a9 - monitoring: ipmi sensor monitoring [18:10:23] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:25] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:10:34] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:10:42] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:10:45] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:10:54] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:23] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:24] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:26] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:28] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.75, 7.51, 7.76 [18:11:33] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:38] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:39] Thanks RhinosF1! [18:12:57] MacFan4000 With NDKilla no longer a Steward, shouldn't there be an upgrade to the entirety of the team? [18:13:08] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:15] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:15] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:13:22] There are still 3 remaining stewards [18:13:22] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:02] PROBLEM - ldap111 Puppet on ldap111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:14:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.80, 6.84, 6.30 [18:14:58] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:15:43] PROBLEM - es121 Puppet on es121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:16:06] MacFan4000, yes. dmehus for example has been doing quite well. [18:16:21] And we still have John and Void [18:16:48] Ah yes, they are both exceptionally great. Nobody will ever mess with them, or even @dmehus. [18:16:49] PROBLEM - gluster101 Puppet on gluster101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:16:54] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.734 second response time [18:17:19] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.747 second response time [18:17:46] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.008 second response time [18:17:50] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.414 second response time [18:19:53] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is UNKNOWN: CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected). [18:19:53] PROBLEM - cp30 Stunnel Http for phab121 on cp30 is UNKNOWN: CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected). [18:19:53] PROBLEM - es101 Puppet on es101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:19:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.62, 3.50, 3.13 [18:20:00] JohnLewis: could you remove PuppyKun from stewards alias pls [18:20:02] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:20:05] Per MacFan4000 comment above [18:20:30] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:33] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:54] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:02] PROBLEM - db121 Puppet on db121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:21:15] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:16] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:17] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:21:22] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:30] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:33] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:33] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:41] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:21:43] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:44] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:21:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.14, 3.04 [18:21:55] PROBLEM - cp30 Stunnel Http for phab121 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 5456 bytes in 2.287 second response time [18:21:56] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:59] PROBLEM - mem101 Puppet on mem101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:22:00] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:04] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:10] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:13] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:32] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:22:59] PROBLEM - mon2 Puppet on mon2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:22:59] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.694 second response time [18:23:05] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.112 second response time [18:23:06] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.382 second response time [18:23:13] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:23:18] PROBLEM - mem2 Puppet on mem2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:23:18] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.028 second response time [18:23:21] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:23:32] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.461 second response time [18:23:34] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.859 second response time [18:23:39] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 8.664 second response time [18:23:41] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:23:50] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.021 second response time [18:23:59] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.449 second response time [18:24:00] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.274 second response time [18:24:07] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.164 second response time [18:24:34] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:24:44] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:24:49] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.332 second response time [18:24:56] [02puppet] 07Universal-Omega opened pull request 03#2314: Fix check_ipmi_sensors filename - 13https://github.com/miraheze/puppet/pull/2314 [18:25:05] PROBLEM - es111 Puppet on es111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:25:14] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:25:20] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.560 second response time [18:25:20] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.300 second response time [18:25:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.94, 8.38, 7.89 [18:25:33] If JohnLewis or somebody else could also remove their flags in #miraheze, that would be great [18:25:37] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.200 second response time [18:25:38] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:25:45] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.232 second response time [18:25:46] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:25:51] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.371 second response time [18:25:54] JohnLewis: please merge the fix filename PR [18:26:04] RhinosF1: can you please remove NDK’s flags in -overflow and -offtopic? [18:26:15] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.029 second response time [18:26:18] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.485 second response time [18:26:21] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.442 second response time [18:26:25] [02puppet] 07JohnFLewis closed pull request 03#2314: Fix check_ipmi_sensors filename - 13https://github.com/miraheze/puppet/pull/2314 [18:26:26] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/be382a912b7d...395848c28336 [18:26:28] [02miraheze/puppet] 07Universal-Omega 03395848c - Fix check_ipmi_sensors filename (#2314) [18:26:32] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.074 second response time [18:26:49] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.611 second response time [18:27:05] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:27:19] (-offtopic can only be done by you as they have +F and you are the only other person with that flag) [18:27:20] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:27:38] PROBLEM - gluster4 Puppet on gluster4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:27:52] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:27:54] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:28:06] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:19] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:28:32] !log removed ndkilla from stewards@ [18:28:34] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.82, 5.49, 5.96 [18:28:40] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:28:47] PROBLEM - cloud5 Puppet on cloud5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:28:53] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:29:03] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.199 second response time [18:29:22] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.308 second response time [18:29:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.32, 3.12 [18:29:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:29:43] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.329 second response time [18:29:47] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.198 second response time [18:29:49] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.651 second response time [18:29:51] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.210 second response time [18:30:02] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.322 second response time [18:30:43] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [18:31:23] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_ipmi_sensors] [18:31:24] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.65, 3.26 [18:31:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.74, 7.67, 7.73 [18:31:38] PROBLEM - cloud11 IPMI Sensors on cloud11 is UNKNOWN: NRPE: Unable to read output [18:32:41] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.66, 1.93, 1.20 [18:32:42] JohnLewis: ^ [18:33:03] ? [18:33:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.30, 7.35, 7.87 [18:33:18] Check failing [18:33:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.94, 3.42, 3.23 [18:34:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 7.58, 7.98 [18:34:18] potentially because the file doesn't exist [18:34:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.09, 6.12, 6.04 [18:34:39] [02ssl] 07Universal-Omega opened pull request 03#474: Remove wikipedia.yogfront.ooo - 13https://github.com/miraheze/ssl/pull/474 [18:34:41] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.72, 1.50, 1.13 [18:34:51] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:35:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.33, 7.81, 7.98 [18:35:08] [02ssl] 07Universal-Omega synchronize pull request 03#474: Remove wikipedia.yogfront.ooo - 13https://github.com/miraheze/ssl/pull/474 [18:35:12] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.05, 2.98, 3.09 [18:35:16] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:36:03] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:36:04] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:36:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.39, 7.91, 8.05 [18:36:33] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.11, 5.84, 5.95 [18:36:44] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [18:36:58] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.518 second response time [18:36:59] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:37:14] paladox: ^^ puppet run fixes it [18:37:14] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.423 second response time [18:37:21] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.501 second response time [18:37:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.33, 8.02, 7.88 [18:37:58] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.148 second response time [18:38:03] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.345 second response time [18:38:06] Ohhh [18:38:27] MacFan4000: 18:38:00 Flags -AFRVefiorstv were set on Nibbly in #miraheze-offtopic. [18:39:05] 18:38:51 Flags -AORVefiorstv were set on Nibbly in #miraheze-overflow. [18:41:01] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:42:14] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:18] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:21] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:30] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:30] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:34] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:42] RECOVERY - ldap111 Puppet on ldap111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:42:49] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:52] RECOVERY - es121 Puppet on es121 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:42:52] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:57] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:58] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:43:06] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:43:44] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:44:24] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.715 second response time [18:44:27] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.573 second response time [18:44:29] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.612 second response time [18:44:29] RECOVERY - gluster101 Puppet on gluster101 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:44:33] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.806 second response time [18:44:37] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:44:48] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.131 second response time [18:44:48] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.854 second response time [18:46:24] 1 hour to upload your files [18:46:50] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:46:58] RECOVERY - es101 Puppet on es101 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:48:08] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:48:42] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:48:50] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:48:57] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:48:58] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:03] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:49:03] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [18:49:30] RECOVERY - mem101 Puppet on mem101 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:50:20] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:50:31] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.306 second response time [18:50:38] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.013 second response time [18:50:45] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.396 second response time [18:50:57] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:51:18] RECOVERY - mem2 Puppet on mem2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:51:20] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:51:32] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.099 second response time [18:51:33] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.154 second response time [18:51:44] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:44] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:52:28] RECOVERY - es111 Puppet on es111 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:52:33] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:52:38] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:52:44] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [18:52:51] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:53:13] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:53:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.36, 7.27, 7.88 [18:53:32] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:53:40] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:54:00] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:54:20] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:54:49] RECOVERY - cloud5 Puppet on cloud5 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:55:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.87, 8.46, 8.25 [18:55:37] RECOVERY - gluster4 Puppet on gluster4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:55:55] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.277 second response time [18:55:59] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.602 second response time [18:56:01] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:01] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:56:09] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:56:17] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:56:24] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.493 second response time [18:56:54] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:56:59] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:57:03] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:57:06] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:57:16] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:57:37] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.313 second response time [18:58:14] RECOVERY - wikipedia.yogfront.ooo - reverse DNS on sslhost is OK: SSL OK - wikipedia.yogfront.ooo reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:59:22] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:59:34] RECOVERY - wikipedia.yogfront.ooo - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/ [18:59:34] gios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - wikipedia.yogfront.ooo reverse DNS resolves to cp21.miraheze.org - CNAME OK [19:00:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.53, 3.69, 3.25 [19:00:33] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.79, 6.46, 5.75 [19:00:52] [02ssl] 07Universal-Omega commented on pull request 03#474: Remove wikipedia.yogfront.ooo - 13https://github.com/miraheze/ssl/pull/474#issuecomment-1013379296 [19:00:54] [02ssl] 07Universal-Omega closed pull request 03#474: Remove wikipedia.yogfront.ooo - 13https://github.com/miraheze/ssl/pull/474 [19:01:04] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.247 second response time [19:01:07] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.190 second response time [19:01:19] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.889 second response time [19:02:34] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.46, 5.37, 5.44 [19:04:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.86, 3.29, 3.18 [19:04:38] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.415 second response time [19:04:39] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.183 second response time [19:05:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.00, 7.61, 7.99 [19:06:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.56, 5.81, 5.56 [19:08:07] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.268 second response time [19:08:18] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.861 second response time [19:08:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/47a8839a0c24 [19:08:55] [02miraheze/puppet] 07paladox 0347a8839 - Add new key for paladox [19:08:56] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [19:08:58] [02puppet] 07paladox opened pull request 03#2315: Add new key for paladox - 13https://github.com/miraheze/puppet/pull/2315 [19:09:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.66, 8.03, 8.05 [19:09:45] [02puppet] 07JohnFLewis closed pull request 03#2315: Add new key for paladox - 13https://github.com/miraheze/puppet/pull/2315 [19:09:47] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/395848c28336...b35500a3c375 [19:09:48] [02miraheze/puppet] 07paladox 03b35500a - Add new key for paladox (#2315) [19:09:50] [02puppet] 07JohnFLewis deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [19:09:51] [02miraheze/puppet] 07JohnFLewis deleted branch 03paladox-patch-5 [19:11:20] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:03] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:04] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:06] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:13] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.094 second response time [19:12:19] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:12:24] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:36] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:40] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:42] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:50] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.758 second response time [19:13:00] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:14:05] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.174 second response time [19:14:07] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.917 second response time [19:14:19] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.126 second response time [19:14:23] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.411 second response time [19:15:55] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:16:08] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.079 second response time [19:16:28] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:16:39] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.551 second response time [19:16:48] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.869 second response time [19:17:09] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.538 second response time [19:17:11] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:17:17] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:17:23] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.028 second response time [19:17:25] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:17:41] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:17:43] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.581 second response time [19:17:46] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:17:46] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:53] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:18:33] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:18:48] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:19:16] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:19:21] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.536 second response time [19:19:27] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw9 [19:19:38] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw9 [19:19:39] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.016 second response time [19:19:48] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 6.457 second response time [19:19:53] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 4.608 second response time [19:20:06] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 6.214 second response time [19:20:26] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.786 second response time [19:21:09] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.110 second response time [19:21:15] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.835 second response time [19:21:24] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [19:21:28] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 6.043 second response time [19:21:33] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [19:21:33] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 5.922 second response time [19:21:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:12] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:36] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:47] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.162 second response time [19:22:50] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.972 second response time [19:23:26] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.229 second response time [19:23:59] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.051 second response time [19:24:13] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.105 second response time [19:25:35] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:25:35] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:25:52] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:26:38] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:27:01] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:27:01] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.376 second response time [19:27:02] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.285 second response time [19:27:31] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.937 second response time [19:27:31] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:27:34] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.330 second response time [19:27:35] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:27:49] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:27:52] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.317 second response time [19:28:27] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:32] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.370 second response time [19:28:41] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.209 second response time [19:28:42] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:29:01] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.320 second response time [19:29:07] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:29:08] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:29:22] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:29:30] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.989 second response time [19:29:36] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.810 second response time [19:31:17] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d6ccc565739d...3d45cb207a66 [19:31:18] [02miraheze/mw-config] 07RhinosF1 033d45cb2 - Update LocalSettings.php [19:31:20] [02mw-config] 07RhinosF1 synchronize pull request 03#4353: [19:45] DC-Switch: Stop uploads - 13https://github.com/miraheze/mw-config/pull/4353 [19:31:23] CosmicAlpha: ^ [19:31:41] Thanks! [19:31:56] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.527 second response time [19:31:58] Last 14 mins for uploads!@ [19:32:03] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3d45cb207a66...5d3ae66d53ac [19:32:04] [02miraheze/mw-config] 07github-actions 035d3ae66 - CI: lint code to MediaWiki standards [19:32:06] [02mw-config] 07github-actions[bot] synchronize pull request 03#4353: [19:45] DC-Switch: Stop uploads - 13https://github.com/miraheze/mw-config/pull/4353 [19:32:11] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.323 second response time [19:32:39] miraheze/mw-config - RhinosF1 the build passed. [19:32:43] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.622 second response time [19:32:46] [02mw-config] 07Universal-Omega reviewed pull request 03#4353 commit - 13https://github.com/miraheze/mw-config/pull/4353#discussion_r785111741 [19:32:53] CosmicAlpha: you able to clean up tabbing [19:33:10] [02mw-config] 07Universal-Omega reviewed pull request 03#4353 commit - 13https://github.com/miraheze/mw-config/pull/4353#discussion_r785113391 [19:33:28] RhinosF1: oh yes can do. [19:34:01] CosmicAlpha: I'm on my phone at moment. Laptop is next room but [19:34:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5d3ae66d53ac...a6346720e7f5 [19:34:53] [02miraheze/mw-config] 07Universal-Omega 03a634672 - Tabbing + global [19:34:55] [02mw-config] 07Universal-Omega synchronize pull request 03#4353: [19:45] DC-Switch: Stop uploads - 13https://github.com/miraheze/mw-config/pull/4353 [19:36:14] miraheze/mw-config - Universal-Omega the build passed. [19:36:20] Ty CosmicAlpha [19:36:57] No problem! [19:39:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.13, 7.67, 7.97 [19:41:44] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:41:44] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:41:58] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:18] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.011 second response time [19:43:41] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.019 second response time [19:43:41] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.332 second response time [19:43:55] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.570 second response time [19:44:04] paladox: ready? [19:44:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:31] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:43] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:44] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:20] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.177 second response time [19:46:19] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.343 second response time [19:46:27] I'm around [19:46:30] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.274 second response time [19:46:45] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.980 second response time [19:46:46] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.328 second response time [19:46:51] paladox: merging the [19:46:55] Just run puppet [19:47:01] [02mw-config] 07paladox closed pull request 03#4353: [19:45] DC-Switch: Stop uploads - 13https://github.com/miraheze/mw-config/pull/4353 [19:47:03] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/592deee5d844...f00294646592 [19:47:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.80, 8.14, 7.98 [19:47:04] [02miraheze/mw-config] 07RhinosF1 03f002946 - [19:45] DC-Switch: Stop uploads (#4353) [19:47:06] [02mw-config] 07paladox deleted branch 03RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [19:47:07] [02miraheze/mw-config] 07paladox deleted branch 03RhinosF1-patch-2 [19:47:23] Thanks! [19:47:28] !log [@test101] starting deploy of {'config': True} to skip [19:47:29] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [19:47:45] Just puppet on mw11/test3/mwtask111 [19:47:51] test101 has picked it up [19:48:19] ok doing [19:48:19] miraheze/mw-config - paladox the build passed. [19:48:21] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:26] puppet running on both [19:48:33] !log [@mw11] starting deploy of {'config': True} to ovlon [19:48:52] !log [@mwtask111] starting deploy of {'config': True} to scsvg [19:48:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-2315-paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b868eee88d12 [19:48:54] [02miraheze/puppet] 07paladox 03b868eee - Revert "Add new key for paladox (#2315)" [19:48:55] [02puppet] 07paladox created branch 03revert-2315-paladox-patch-5 - 13https://github.com/miraheze/puppet [19:48:57] [02puppet] 07paladox opened pull request 03#2316: Revert "Add new key for paladox" - 13https://github.com/miraheze/puppet/pull/2316 [19:48:57] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 5s [19:48:59] [02puppet] 07paladox closed pull request 03#2316: Revert "Add new key for paladox" - 13https://github.com/miraheze/puppet/pull/2316 [19:49:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b35500a3c375...8408e5f1e52f [19:49:02] [02miraheze/puppet] 07paladox 038408e5f - Revert "Add new key for paladox (#2315)" (#2316) [19:49:04] [02miraheze/puppet] 07paladox deleted branch 03revert-2315-paladox-patch-5 [19:49:05] [02puppet] 07paladox deleted branch 03revert-2315-paladox-patch-5 - 13https://github.com/miraheze/puppet [19:49:09] paladox: test3? [19:49:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:30] test3 running [19:49:35] K [19:49:48] RhinosF1: can you check it's disabled [19:49:49] pls [19:49:55] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.994 second response time [19:49:57] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.743 second response time [19:50:00] Has mw11 finished paladox ? [19:50:01] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.842 second response time [19:50:05] yes [19:50:09] oh [19:50:11] still running [19:50:13] Weird, no finish log [19:50:14] Yeah [19:50:17] it's on [19:50:18] Info: Git::Clone[MediaWiki config]: Scheduling refresh of Exec[MediaWiki Config Sync] [19:50:18] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.829 second response time [19:50:20] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.044 second response time [19:50:22] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 109s [19:50:22] !log [@test3] starting deploy of {'config': True} to skip [19:50:24] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [19:50:35] Now [19:50:39] That was slow [19:50:42] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:57] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:09] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:14] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:20] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:33] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:34] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:39] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8408e5f1e52f...d75b9fe1611e [19:51:41] [02miraheze/puppet] 07paladox 03d75b9fe - Revert "mediawiki: Increase opcache.revalidate_freq to 30 for php (#2283)" [19:51:54] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:08] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:12] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:19] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:38] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.171 second response time [19:52:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:54] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:53:22] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.755 second response time [19:53:31] !log starting final sync from old gluster to new gluster [19:53:33] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.581 second response time [19:53:43] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 8.535 second response time [19:53:49] !log FILE UPLOADS ARE DISABLED [19:53:59] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 7.990 second response time [19:54:14] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 8.643 second response time [19:54:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:23] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:26] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:48] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:55] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.421 second response time [19:55:02] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.907 second response time [19:55:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.42, 7.74, 7.98 [19:55:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:16] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.471 second response time [19:55:27] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.114 second response time [19:55:33] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.169 second response time [19:56:31] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:56:41] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:51] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:57:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.36, 8.26, 8.14 [19:57:14] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:57:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:21] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.736 second response time [19:58:25] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:58:27] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.856 second response time [19:58:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:27] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:59:40] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:59:51] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:59:55] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:00:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:31] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.688 second response time [20:00:39] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.341 second response time [20:00:42] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.114 second response time [20:00:51] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.960 second response time [20:00:53] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.515 second response time [20:01:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:07] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.445 second response time [20:01:15] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.390 second response time [20:01:29] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.025 second response time [20:01:29] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:01:34] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.307 second response time [20:02:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.429 second response time [20:02:34] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.50, 5.17, 5.78 [20:02:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:06:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.44, 21.97, 23.81 [20:07:51] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.934 second response time [20:07:54] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.621 second response time [20:08:05] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.409 second response time [20:08:09] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.413 second response time [20:08:31] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.001 second response time [20:09:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:10:57] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:11:32] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:11:53] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:12:10] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:12:42] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:12:58] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:13:04] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:13:28] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:13:49] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:34] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.47, 3.88, 4.86 [20:15:03] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.412 second response time [20:15:08] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.291 second response time [20:15:28] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.051 second response time [20:15:50] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.924 second response time [20:16:49] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.196 second response time [20:19:08] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:19:28] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:19:31] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:19:35] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:19:44] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:19:51] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:40] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:49] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:08] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:21:14] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.699 second response time [20:22:00] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:12] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.684 second response time [20:22:37] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.303 second response time [20:22:54] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.581 second response time [20:23:10] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.298 second response time [20:23:25] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.193 second response time [20:25:34] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:26:41] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:27:38] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.761 second response time [20:27:49] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.463 second response time [20:27:49] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.642 second response time [20:28:18] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.950 second response time [20:28:39] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.328 second response time [20:29:48] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:30:10] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.871 second response time [20:30:15] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.292 second response time [20:30:24] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.383 second response time [20:30:53] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:30:56] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.236 second response time [20:31:13] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.808 second response time [20:31:51] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.031 second response time [20:32:54] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.256 second response time [20:34:24] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:34:36] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:34:50] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:34:52] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:26] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.40, 5.40, 4.93 [20:36:24] [02ssl] 07Universal-Omega opened pull request 03#475: Remove sportsanalytics.wiki - 13https://github.com/miraheze/ssl/pull/475 [20:36:30] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.294 second response time [20:36:44] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.856 second response time [20:36:47] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 1.229 second response time [20:36:50] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 0.550 second response time [20:36:54] [02ssl] 07Universal-Omega synchronize pull request 03#475: Remove sportsanalytics.wiki - 13https://github.com/miraheze/ssl/pull/475 [20:37:17] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.06, 2.66, 1.58 [20:37:23] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:39:00] [02dns] 07Universal-Omega opened pull request 03#244: Remove rangpurpedia.xyz zone - 13https://github.com/miraheze/dns/pull/244 [20:39:20] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.832 second response time [20:39:21] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.39, 4.34, 4.64 [20:39:44] [02dns] 07Universal-Omega opened pull request 03#245: Remove sportsanalytics.wiki zone - 13https://github.com/miraheze/dns/pull/245 [20:41:05] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:41:16] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.80, 1.65, 1.40 [20:42:24] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:42:56] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:00] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:09] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:16] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:50] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:12] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:45:07] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:45:12] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:45:50] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.320 second response time [20:46:27] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.538 second response time [20:46:56] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.846 second response time [20:46:59] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:47:11] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:47:12] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.530 second response time [20:47:14] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:47:25] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.692 second response time [20:47:27] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:47:46] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:47:49] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:47:53] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:47:57] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:48:32] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.46, 19.43, 20.20 [20:48:40] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:49:17] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.426 second response time [20:49:19] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.874 second response time [20:50:26] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.008 second response time [20:51:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:36] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23577 bytes in 9.221 second response time [20:52:54] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:15] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.644 second response time [20:53:19] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:41] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:41] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:52] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.070 second response time [20:54:01] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:09] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.503 second response time [20:54:23] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:54:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.84, 21.09, 20.63 [20:54:58] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:55:15] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.436 second response time [20:55:26] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.692 second response time [20:55:28] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.523 second response time [20:55:39] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.031 second response time [20:56:04] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.306 second response time [20:56:19] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.758 second response time [20:56:32] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.70, 20.20, 20.37 [20:56:52] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.197 second response time [20:58:09] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.054 second response time [20:58:52] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:59:37] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:59:58] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.234 second response time [21:00:01] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.546 second response time [21:00:12] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.296 second response time [21:00:15] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:00:19] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.039 second response time [21:00:20] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.218 second response time [21:00:23] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:00:29] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:00:35] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.609 second response time [21:03:13] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:03:30] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.312 second response time [21:03:36] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.228 second response time [21:05:13] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.614 second response time [21:06:41] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:08:31] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:09:44] What is MirahezeLSBot [21:09:44] MH-Discord: Miraheze is a non-profit wikifarm running MediaWiki. If you would like more information please see https://meta.miraheze.org/ or #miraheze. [21:09:54] Unhelpful [21:10:58] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:11:07] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:28] It lets people add log entries directly from the shell or autologs when puppet runs and finishes [21:12:33] NDKilla: direct logging from server using logsalmsg. [21:12:55] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.757 second response time [21:13:00] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.313 second response time [21:13:09] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.079 second response time [21:13:32] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:10] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:24] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:24] Sounds nice :O [21:14:30] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:14:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.59, 21.82, 20.80 [21:14:49] Now all we need is wmfs command line scripts to automatically pool and unspool servers. Then get as many servers as wmf [21:15:27] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.451 second response time [21:15:28] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.63, 4.96, 4.39 [21:15:30] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.398 second response time [21:16:01] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.499 second response time [21:16:05] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.303 second response time [21:16:24] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.839 second response time [21:16:28] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.392 second response time [21:16:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:16:54] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.281 second response time [21:16:57] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.353 second response time [21:17:26] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.42, 4.90, 4.45 [21:17:39] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.830 second response time [21:18:23] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.274 second response time [21:19:41] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.98, 20.56, 19.36 [21:19:50] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:19:57] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:20:02] NDKilla: If you can donate about £4000, then we could potentially get about another 15 or so :P [21:20:08] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:20:17] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:20:34] It would be much appreciated [21:20:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [21:20:49] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:21:23] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:03] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:13] -has about… 1/10th of that in the bank- [21:22:42] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:52] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:59] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:23:33] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.997 second response time [21:24:09] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.435 second response time [21:25:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.96, 7.47, 7.96 [21:26:14] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.969 second response time [21:26:15] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.691 second response time [21:26:16] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.390 second response time [21:26:39] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.698 second response time [21:26:47] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.314 second response time [21:26:57] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.508 second response time [21:27:09] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.157 second response time [21:27:10] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 8.155 second response time [21:27:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.52, 7.81, 8.02 [21:28:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:28:58] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 1.549 second response time [21:30:01] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:30:01] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:30:08] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:30:45] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:31:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.34, 7.68, 7.94 [21:31:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.50, 19.75, 19.87 [21:33:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.16, 7.94, 7.98 [21:33:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:34:31] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.68, 19.95, 20.36 [21:36:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:37:57] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:38:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.32, 20.68, 20.55 [21:39:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:40:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:44:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:48:13] <30 mins before wiki creations are halted I believe [21:48:38] MacFan4000, it's actually 27 minutes now. [21:48:52] 27 is still <30 [21:48:59] ^ [21:49:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.76, 7.31, 7.98 [21:49:11] I'm usually very consistent with time. [21:50:22] At any rate, there's now about 25 minutes, but who's counting anyway? [21:50:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.27, 23.54, 21.96 [21:50:51] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.186 second response time [21:51:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.35, 7.52, 7.96 [21:51:11] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:14] You really don't need to say it every 2 minutes. [21:51:22] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.329 second response time [21:51:24] Okay, I'll stop then. [21:51:26] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.021 second response time [21:51:34] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:51] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:55] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.511 second response time [21:51:57] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:52:00] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:52:02] Thanks! I mean no objections to periodic updates but that's like every 10-15 minutes maybe. [21:52:15] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.500 second response time [21:52:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.20, 22.04, 21.59 [21:53:28] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:53:46] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:19] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:23] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:24] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:48] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.780 second response time [21:56:02] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.866 second response time [21:56:09] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.947 second response time [21:56:09] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.208 second response time [21:57:17] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.855 second response time [21:57:25] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.565 second response time [21:57:31] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.539 second response time [21:57:33] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.468 second response time [21:58:16] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.45, 1.51, 1.00 [21:59:46] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.712 second response time [22:00:06] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.414 second response time [22:00:16] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.89, 1.21, 0.95 [22:01:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:04:53] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:04:56] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:06:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:06:53] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.167 second response time [22:06:53] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.566 second response time [22:07:15] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03revert-4353-RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/aa680738451d [22:07:17] [02miraheze/mw-config] 07RhinosF1 03aa68073 - Revert "[19:45] DC-Switch: Stop uploads (#4353)" [22:07:18] [02mw-config] 07RhinosF1 created branch 03revert-4353-RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [22:07:20] [02mw-config] 07RhinosF1 opened pull request 03#4378: Revert "[19:45] DC-Switch: Stop uploads" - 13https://github.com/miraheze/mw-config/pull/4378 [22:07:44] [02mw-config] 07RhinosF1 edited pull request 03#4378: [22:53] Revert "DC-Switch: Stop uploads" - 13https://github.com/miraheze/mw-config/pull/4378 [22:08:26] miraheze/mw-config - RhinosF1 the build passed. [22:09:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:10:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [22:10:48] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:12:24] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:12:24] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:12:33] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:12:48] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:12:53] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:12:58] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:13:18] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:13:28] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:13:31] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:13:47] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:13:54] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:03] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:07] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:14:23] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.035 second response time [22:14:27] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.196 second response time [22:14:50] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.903 second response time [22:14:53] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.172 second response time [22:15:00] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.286 second response time [22:15:01] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.491 second response time [22:15:13] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.062 second response time [22:15:32] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.635 second response time [22:15:33] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.747 second response time [22:16:08] [02mw-config] 07RhinosF1 closed pull request 03#4354: [22:15] DC-Switch: Disable wiki creation (trust Stews for rename) - 13https://github.com/miraheze/mw-config/pull/4354 [22:16:10] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 - 13https://github.com/miraheze/mw-config [22:16:11] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 [22:16:13] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f00294646592...1a9bd6f8a165 [22:16:14] [02miraheze/mw-config] 07RhinosF1 031a9bd6f - [22:15] DC-Switch: Disable wiki creation (#4354) [22:16:18] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03revert-4354-RhinosF1-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/e72f4c4ffd58 [22:16:19] [02miraheze/mw-config] 07RhinosF1 03e72f4c4 - Revert "[22:15] DC-Switch: Disable wiki creation (#4354)" [22:16:21] [02mw-config] 07RhinosF1 created branch 03revert-4354-RhinosF1-patch-3 - 13https://github.com/miraheze/mw-config [22:16:32] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.85, 20.06, 20.38 [22:16:40] [02mw-config] 07RhinosF1 opened pull request 03#4379: [22:53] Revert "DC-Switch: Disable wiki creation" - 13https://github.com/miraheze/mw-config/pull/4379 [22:17:09] miraheze/mw-config - RhinosF1 the build passed. [22:17:13] !log [rhinos@test3] starting deploy of {} to skip [22:17:14] !log [rhinos@test3] finished deploy of {} to skip - SUCCESS in 0s [22:17:19] !log [rhinos@test101] starting deploy of {} to skip [22:17:21] !log [rhinos@test101] finished deploy of {} to skip - SUCCESS in 0s [22:17:24] !log [rhinos@mw11] starting deploy of {} to ovlon [22:17:25] !log [rhinos@mw11] finished deploy of {} to ovlon - SUCCESS in 1s [22:17:30] !log [rhinos@mwtask111] starting deploy of {} to scsvg [22:17:31] !log [rhinos@mwtask111] finished deploy of {} to scsvg - SUCCESS in 0s [22:17:45] miraheze/mw-config - RhinosF1 the build passed. [22:17:57] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.261 second response time [22:18:03] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.288 second response time [22:18:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:18:15] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.110 second response time [22:18:18] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: DC Switchover - RhinosF1, last run 23 minutes ago with 0 failures [22:18:18] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.193 second response time [22:18:21] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet is currently disabled, message: DC Switchover - RhinosF1, last run 26 minutes ago with 0 failures [22:18:32] PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: DC Switchover - RhinosF1, last run 31 minutes ago with 0 failures [22:18:39] !log Wiki Creations Disabled [22:18:44] PROBLEM - mw11 Puppet on mw11 is WARNING: WARNING: Puppet is currently disabled, message: DC Switchover - RhinosF1, last run 7 minutes ago with 0 failures [22:18:47] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.015 second response time [22:18:47] RhinosF1: you didn't deploy config? "deploy of {}" [22:19:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:19:12] !log [rhinos@mwtask111] starting deploy of {'config': True} to scsvg [22:19:16] !log [rhinos@test101] starting deploy of {'config': True} to skip [22:19:17] !log [rhinos@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:19:18] !log [rhinos@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 5s [22:19:27] guess it's a bit behind [22:19:43] !log [rhinos@test101] starting deploy of {'config': True} to skip [22:19:44] !log [rhinos@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:20:18] !log [rhinos@mwtask111] starting deploy of {'config': True} to scsvg [22:20:24] !log [rhinos@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 5s [22:20:24] !log [rhinos@mw11] starting deploy of {'config': True} to ovlon [22:20:29] !log [rhinos@test3] starting deploy of {'config': True} to skip [22:20:30] !log [rhinos@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:20:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.83, 20.75, 20.59 [22:20:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:20:57] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:21:04] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:21:27] !log [rhinos@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 63s [22:21:31] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:21:33] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:21:52] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:07] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:24] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:22:39] CosmicAlpha: CW is still open for me [22:23:34] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.445 second response time [22:23:35] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.102 second response time [22:23:53] RhinosF1: Did you manually pull? (As --pull=config) wasn't used. [22:24:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:24:09] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.032 second response time [22:24:13] CosmicAlpha: yeah [22:24:21] [02dns] 07paladox closed pull request 03#241: [22:38] betaheze: reduce TTL to 10 seconds - 13https://github.com/miraheze/dns/pull/241 [22:24:22] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/5f3c39749340...eb3068707536 [22:24:24] [02miraheze/dns] 07RhinosF1 03eb30687 - betaheze: reduce TTL to 10 seconds (#241) [22:24:32] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.985 second response time [22:24:41] !log [rhinos@mw11] starting deploy of {'config': True} to ovlon [22:24:57] https://www.irccloud.com/pastebin/pYEv3ow4/ [22:25:35] ty paladox [22:25:37] RhinosF1: oh, use revoke permissions, managewiki is overriding maybe, somehow, if it had ever been originally added to MW tables it would. Also testing either personal account right? [22:25:38] !log [rhinos@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 57s [22:25:58] CosmicAlpha: yes [22:25:59] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:26:16] CosmicAlpha: MW shouldn't have that access [22:26:19] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:26:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:26:45] RhinosF1: I think it would if it was added through managewiki before blacklisted. [22:27:02] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1a9bd6f8a165...5d27819357d1 [22:27:04] [02miraheze/mw-config] 07RhinosF1 035d27819 - Update LocalSettings.php [22:27:07] CosmicAlpha: right [22:27:43] RhinosF1: doesn't work [22:27:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:27:47] https://www.irccloud.com/pastebin/znXUMVwm/ [22:28:05] miraheze/mw-config - RhinosF1 the build passed. [22:28:07] paladox: can you change the config for it [22:28:12] !log [rhinos@mwtask111] starting deploy of {'config': True} to scsvg [22:28:16] !log [rhinos@test101] starting deploy of {'config': True} to skip [22:28:17] !log [rhinos@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:28:18] !log [rhinos@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 5s [22:28:19] !log [rhinos@test3] starting deploy of {'config': True} to skip [22:28:20] !log [rhinos@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:28:25] !log [rhinos@mw11] starting deploy of {'config': True} to ovlon [22:28:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:28:37] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/eb3068707536...6fbe501831d0 [22:28:39] [02miraheze/dns] 07paladox 036fbe501 - set acme_challenge_dns_ttl to 10 [22:28:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:28:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:28:48] !log [rhinos@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 23s [22:29:16] !log wiki creations actually stopped [22:29:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:29:56] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 53 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [22:30:11] paladox: that ok? [22:30:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:30:31] yesno [22:30:33] no [22:30:36] i'm just gonna revert [22:31:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:31:36] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/6fbe501831d0...fbc945ecea30 [22:31:38] [02miraheze/dns] 07paladox 03fbc945e - Set acme_challenge_dns_ttl to 0 [22:31:47] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [22:31:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:31:50] paladox: if not you're going to have to merge the beta flip before 22:39 [22:32:23] that doesn't work either [22:32:25] so reverting [22:32:36] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.624 second response time [22:32:36] Paladox: 300 is the standard TTL [22:32:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:32:47] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.640 second response time [22:32:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:31] [02miraheze/dns] 07paladox pushed 033 commits to 03master [+0/-0/±3] 13https://github.com/miraheze/dns/compare/fbc945ecea30...cd184d76cb5b [22:33:33] [02miraheze/dns] 07paladox 0367927b4 - Revert "Set acme_challenge_dns_ttl to 0" [22:33:34] [02miraheze/dns] 07paladox 039ae8f22 - Revert "set acme_challenge_dns_ttl to 10" [22:33:36] [02miraheze/dns] 07paladox 03cd184d7 - Revert "betaheze: reduce TTL to 10 seconds (#241)" [22:33:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:56] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [22:34:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:34:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:35:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:35:35] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.973 second response time [22:35:46] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:35:46] JohnLewis: we want less so we can switchover quick [22:35:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:36:00] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.473 second response time [22:36:10] paladox: merge the beta flip then now [22:36:12] You can’t have less than 300 for acme records in GDNSD3 [22:36:17] ok [22:36:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:36:27] [02dns] 07paladox closed pull request 03#238: [22:48] betaheze: use new test server - 13https://github.com/miraheze/dns/pull/238 [22:36:27] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.193 second response time [22:36:28] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/cd184d76cb5b...79cb88f7acd0 [22:36:30] [02miraheze/dns] 07RhinosF1 0379cb88f - betaheze: use new test server (#238) [22:36:30] Without making every other acme record that number [22:36:32] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.96, 19.87, 20.40 [22:36:40] JohnLewis: interesting [22:37:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:37:44] CosmicAlpha: beta.betaheze.org should now use test101 [22:37:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:38:06] RhinosF1: OK, great! [22:38:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:39:12] JohnLewis: you are replicating phab right? [22:39:27] Yes [22:39:30] in which case i'll stop php on it to switch it after the migration [22:39:35] oh [22:39:38] i can set it to read only [22:39:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:39:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [22:40:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d75b9fe1611e...00777267d5ba [22:40:23] [02miraheze/puppet] 07paladox 030077726 - phabricator: set read only [22:40:30] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:40:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:40:49] PROBLEM - betaheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for betaheze.org could not be found [22:40:50] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:41:17] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:41:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:41:35] JohnLewis: do the new appservers have rDNS? [22:41:47] PROBLEM - betaheze.org - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')rDNS WARNIN [22:41:47] verse DNS entry for betaheze.org could not be found [22:41:57] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:42:04] Paladox was supposed to be doing the PTR records [22:42:15] i'm guessing by the warning not [22:42:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:42:34] i'm gonna pull read only [22:42:53] [02mw-config] 07RhinosF1 closed pull request 03#4355: [22:45] DC-Switch: All wikis read only - 13https://github.com/miraheze/mw-config/pull/4355 [22:42:55] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-4 - 13https://github.com/miraheze/mw-config [22:42:56] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-4 [22:42:58] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5d27819357d1...792134fba9ab [22:42:59] [02miraheze/mw-config] 07RhinosF1 03792134f - [22:45] DC-Switch: All wikis read only (#4355) [22:43:01] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03revert-4355-RhinosF1-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/68c5f02e16c4 [22:43:02] [02miraheze/mw-config] 07RhinosF1 0368c5f02 - Revert "[22:45] DC-Switch: All wikis read only (#4355)" [22:43:04] [02mw-config] 07RhinosF1 created branch 03revert-4355-RhinosF1-patch-4 - 13https://github.com/miraheze/mw-config [22:43:08] [02mw-config] 07RhinosF1 opened pull request 03#4380: [22:53] Revert "DC-Switch: All wikis read only" - 13https://github.com/miraheze/mw-config/pull/4380 [22:43:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:43:38] !log [rhinos@test3] starting deploy of {'config': True} to skip [22:43:39] !log [rhinos@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:44:00] miraheze/mw-config - RhinosF1 the build passed. [22:44:09] PROBLEM - phab2 phd on phab2 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd' [22:44:10] miraheze/mw-config - RhinosF1 the build passed. [22:44:20] !log [rhinos@test101] starting deploy of {'config': True} to skip [22:44:21] !log [rhinos@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:44:26] PROBLEM - phab2 Puppet on phab2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[phd] [22:44:43] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:44:45] !log [rhinos@mwtask111] starting deploy of {'config': True} to scsvg [22:44:51] !log [rhinos@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 5s [22:44:51] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:44:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:45:06] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:45:16] paladox: going read only [22:45:21] ok [22:45:22] !log [rhinos@mw11] starting deploy of {'config': True} to ovlon [22:45:24] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:45:25] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:45:26] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.327 second response time [22:45:59] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.411 second response time [22:46:12] !log [rhinos@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 49s [22:46:30] paladox: checking it's everywhere [22:46:35] ok [22:46:42] tell me when and i'll run STOP SLAVE; RESET SLAVE ALL;) [22:46:57] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.659 second response time [22:47:11] !log Read Only Confirmed [22:47:24] paladox: please stop replication once caught up [22:47:49] RhinosF1: no logs if readonly. Will have to be added after. [22:47:49] i see no sql delay [22:47:51] so stopping [22:47:58] !log stopping replication [22:48:03] CosmicAlpha: it's for visibility [22:48:14] done [22:48:27] RhinosF1: can i do https://github.com/miraheze/puppet/pull/2241 [22:48:28] [url] [22:50] MediaWiki: Switchover to new infra by RhinosF1 · Pull Request #2241 · miraheze/puppet · GitHub | github.com [22:48:28] ? [22:48:36] paladox: https://github.com/miraheze/puppet/pull/2241 [22:48:37] [url] [22:50] MediaWiki: Switchover to new infra by RhinosF1 · Pull Request #2241 · miraheze/puppet · GitHub | github.com [22:48:38] let's go [22:48:47] ok [22:48:52] [02puppet] 07paladox closed pull request 03#2241: [22:50] MediaWiki: Switchover to new infra - 13https://github.com/miraheze/puppet/pull/2241 [22:48:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/00777267d5ba...34656300a653 [22:48:55] [02miraheze/puppet] 07RhinosF1 033465630 - [DO NOT MERGE] MediaWiki: Switchover to new infra (#2241) [22:49:43] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.243 second response time [22:50:09] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:19] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:03] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.225 second response time [22:51:09] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.323 second response time [22:51:21] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.032 second response time [22:51:39] CosmicAlpha: i'm gonna merge the RC feed switch [22:51:41] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.330 second response time [22:51:43] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.319 second response time [22:51:44] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.322 second response time [22:51:53] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:54] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:06] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.008 second response time [22:52:09] [02mw-config] 07RhinosF1 closed pull request 03#4366: [22:53] DC-Switch: Switch to mon111 for IRC RecentChanges feed - 13https://github.com/miraheze/mw-config/pull/4366 [22:52:10] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/792134fba9ab...a5814e888fda [22:52:12] [02miraheze/mw-config] 07Universal-Omega 03a5814e8 - DC-Switch: Switch to mon111 for IRC RecentChanges feed (#4366) [22:52:13] [02mw-config] 07RhinosF1 deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [22:52:15] RhinosF1: think we should unset read only [22:52:15] [02miraheze/mw-config] 07RhinosF1 deleted branch 03Universal-Omega-patch-3 [22:52:16] switched now [22:52:19] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.136 second response time [22:52:54] paladox: rebasing to do so [22:52:54] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.006 second response time [22:52:59] ok [22:53:09] miraheze/mw-config - RhinosF1 the build passed. [22:53:48] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.302 second response time [22:53:49] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.013 second response time [22:53:51] [02miraheze/mw-config] 07RhinosF1 pushed 0333 commits to 03all-wikis-rw [+0/-0/±42] 13https://github.com/miraheze/mw-config/compare/209c4b05d736...297dffadbfa4 [22:53:52] [02miraheze/mw-config] 07RhinosF1 03297dffa - Merge branch 'master' into all-wikis-rw [22:53:54] [02mw-config] 07RhinosF1 synchronize pull request 03#4356: [22:53] DC-Switch: Make SCSVG RW - 13https://github.com/miraheze/mw-config/pull/4356 [22:54:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 9.06, 4.15, 1.80 [22:54:13] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03all-wikis-rw [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/297dffadbfa4...13419af46deb [22:54:14] [02miraheze/mw-config] 07github-actions 0313419af - CI: lint code to MediaWiki standards [22:54:16] [02mw-config] 07github-actions[bot] synchronize pull request 03#4356: [22:53] DC-Switch: Make SCSVG RW - 13https://github.com/miraheze/mw-config/pull/4356 [22:54:28] [02mw-config] 07RhinosF1 closed pull request 03#4356: [22:53] DC-Switch: Make SCSVG RW - 13https://github.com/miraheze/mw-config/pull/4356 [22:54:28] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:54:29] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.316 second response time [22:54:29] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a5814e888fda...6aea5b8441a5 [22:54:31] [02miraheze/mw-config] 07RhinosF1 036aea5b8 - [22:53] DC-Switch: Make SCSVG RW (#4356) [22:54:32] [02miraheze/mw-config] 07RhinosF1 deleted branch 03all-wikis-rw [22:54:34] [02mw-config] 07RhinosF1 deleted branch 03all-wikis-rw - 13https://github.com/miraheze/mw-config [22:54:34] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:54:35] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.030 second response time [22:54:38] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:54:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:54:45] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:54:48] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:54:48] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6aea5b8441a5...515de759afe9 [22:54:48] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:54:49] [02miraheze/mw-config] 07RhinosF1 03515de75 - Revert "[22:15] DC-Switch: Disable wiki creation (#4354)" [22:54:52] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:54:55] miraheze/mw-config - RhinosF1 the build passed. [22:54:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.69, 5.72, 2.47 [22:54:57] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:55:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 1.36, 5.50, 7.39 [22:55:05] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.769 second response time [22:55:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.86, 5.74, 2.46 [22:55:06] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:55:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.08, 5.65, 2.36 [22:55:13] !log [rhinos@test3] starting deploy of {'config': True} to skip [22:55:14] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.089 second response time [22:55:14] !log [rhinos@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:55:19] !log [rhinos@mw11] starting deploy of {'config': True} to ovlon [22:55:20] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:22] !log [rhinos@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 3s [22:55:29] Consistent 503s [22:55:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 1.34, 5.32, 7.84 [22:55:30] miraheze/mw-config - RhinosF1 the build passed. [22:55:37] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [22:55:42] miraheze/mw-config - RhinosF1 the build passed. [22:55:46] CosmicAlpha: probably low parser cache [22:55:47] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:55:48] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:55:56] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:01] CosmicAlpha: you want me to hold or? [22:56:01] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [22:56:03] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:56:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.03, 6.25, 2.78 [22:56:08] RhinosF1: not slow, immediately giving 503s. [22:56:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 1.09, 5.22, 7.54 [22:56:15] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:56:24] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw102 mw121 [22:56:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.52, 6.22, 2.90 [22:56:25] CosmicAlpha: can you get errors [22:56:26] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw102 mw112 mw121 [22:56:33] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.227 second response time [22:56:33] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.342 second response time [22:56:33] like logs [22:56:35] cc paladox [22:56:37] icinga saying basically all backends are down [22:56:50] Voidwalker: aware [22:56:53] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 1.56, 4.36, 7.58 [22:56:55] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.871 second response time [22:57:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 0.95, 4.41, 7.26 [22:57:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 0.83, 3.96, 6.60 [22:57:10] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:57:10] RhinosF1: can you deploy the change pls [22:57:14] no cache = slowness [22:57:15] Voidwalker: locally looks fine [22:57:23] !log [rhinos@test101] starting deploy of {'config': True} to skip [22:57:24] !log [rhinos@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:57:27] going ahead [22:57:31] we're up on server [22:57:38] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 4 backends are down. mw101 mw111 mw112 mw122 [22:57:44] !log [rhinos@mwtask111] starting deploy of {'config': True} to scsvg [22:57:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 0.93, 4.05, 7.49 [22:57:49] !log [rhinos@mwtask111] DEPLOY ABORTED: Canary check failed for mw101 [22:57:57] meta loads for me, but with no stylization [22:57:58] !log [rhinos@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [22:58:04] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.470 second response time [22:58:09] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 0.80, 3.74, 6.71 [22:58:09] paladox: ok to RW files? [22:58:17] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.615 second response time [22:58:34] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.006 second response time [22:58:34] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.023 second response time [22:58:40] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.318 second response time [22:58:42] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.219 second response time [22:58:48] paladox: it isn't slow for me, it gives instantaneous 503s. [22:59:01] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 0.87, 3.22, 6.47 [22:59:03] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.299 second response time [22:59:08] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.313 second response time [22:59:09] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.328 second response time [22:59:12] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.025 second response time [22:59:15] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [22:59:17] It’s slow for me and there aren’t any skin styles being loaded [22:59:22] 502 bad gateway here, no lag [22:59:24] !log [rhinos@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 86s [22:59:29] CosmicAlpha: it's just upset [22:59:29] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 0.69, 2.81, 6.23 [22:59:37] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.015 second response time [22:59:41] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.312 second response time [22:59:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 0.80, 2.95, 6.67 [22:59:55] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 3.574 second response time [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:01] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:11] That's all cache issues. Sario, MacFan4000. RhinosF1 do we need to set $wgCacheEpoch or not? [23:00:19] CosmicAlpha: why [23:00:28] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:00:28] To force a refresh of cache. [23:00:49] I'm not sure if it's necessary if none exist though. [23:00:49] CosmicAlpha: it'll refresh itself [23:00:53] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 0.96, 2.60, 6.12 [23:00:54] there's none there [23:00:55] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.392 second response time [23:01:20] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.291 second response time [23:01:29] RhinosF1: ok [23:01:36] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [23:02:01] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:02:22] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.325 second response time [23:02:32] Streambot won’t work correctly until the old Infra rc bot is stopped and the new infra bot gets the original nick [23:02:39] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:02:44] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:02] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:07] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:32] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:35] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:21] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:23] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:25] we cannot concentrate on the bots right now, things aren't loading [23:04:41] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.444 second response time [23:04:56] It’s still slow but styles are loading now [23:05:34] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.905 second response time [23:05:35] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw101 mw102 mw121 [23:05:52] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.010 second response time [23:06:03] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:24] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.111 second response time [23:06:46] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:50] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:53] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [23:07:04] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [23:07:07] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.335 second response time [23:08:10] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [23:08:12] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/rebuildtextindex.php (START) [23:08:15] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:23] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:36] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:45] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:09:04] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.639 second response time [23:09:13] Pageloads are definitely getting quicker [23:09:40] MacFan4000: they will [23:09:58] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:10:01] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:10:01] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:10:02] RhinosF1: I was saying they are [23:10:05] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:10:46] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:11:58] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.116 second response time [23:11:58] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.337 second response time [23:12:12] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.862 second response time [23:12:17] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:12:31] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [23:12:39] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.427 second response time [23:12:42] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.321 second response time [23:13:22] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 800s [23:14:00] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.547 second response time [23:14:01] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:30] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.332 second response time [23:15:08] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.779 second response time [23:15:09] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.247 second response time [23:15:10] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:16:06] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.317 second response time [23:16:08] [02mw-config] 07RhinosF1 closed pull request 03#4380: [22:53] Revert "DC-Switch: All wikis read only" - 13https://github.com/miraheze/mw-config/pull/4380 [23:16:16] [02mw-config] 07RhinosF1 closed pull request 03#4379: [22:53] Revert "DC-Switch: Disable wiki creation" - 13https://github.com/miraheze/mw-config/pull/4379 [23:16:19] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.021 second response time [23:16:19] [02miraheze/mw-config] 07RhinosF1 deleted branch 03revert-4354-RhinosF1-patch-3 [23:16:21] [02mw-config] 07RhinosF1 deleted branch 03revert-4354-RhinosF1-patch-3 - 13https://github.com/miraheze/mw-config [23:16:21] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.924 second response time [23:16:23] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.209 second response time [23:16:32] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.236 second response time [23:16:38] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:16:42] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:16:48] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.024 second response time [23:16:52] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.293 second response time [23:16:59] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:17:03] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:06] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:17:45] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:18:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 0.40, 0.08, 0.03 [23:18:07] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.019 second response time [23:18:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 0.25, 0.06, 0.02 [23:18:34] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for mw121 [23:18:35] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.316 second response time [23:18:41] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.018 second response time [23:18:57] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 1.13, 0.30, 0.10 [23:18:59] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.053 second response time [23:18:59] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.229 second response time [23:19:05] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.305 second response time [23:19:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 2.10, 0.49, 0.16 [23:19:13] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.317 second response time [23:19:15] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.525 second response time [23:19:21] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.018 second response time [23:19:40] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:19:47] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/rebuildrecentchanges.php (START) [23:19:59] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/updateSpecialPages.php (START) [23:20:00] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:07] PROBLEM - mw102 HTTPS on mw102 is CRITICAL: connect to address 2a10:6740::6:108 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:20:30] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [23:20:30] !log all 3 migration scripts are running [23:20:50] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:54] PROBLEM - mw102 Puppet on mw102 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:21:47] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.051 second response time [23:21:55] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.054 second response time [23:22:02] RECOVERY - mw102 HTTPS on mw102 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.016 second response time [23:22:07] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23573 bytes in 2.482 second response time [23:22:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 0.47, 0.14, 0.05 [23:23:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.95, 1.68, 0.58 [23:23:20] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:23:27] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:29] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:34] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:23:35] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [23:23:36] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 1.536 second response time [23:23:39] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.806 second response time [23:23:42] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:24:03] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 9.77, 5.37, 2.18 [23:24:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.13, 6.61, 2.89 [23:24:26] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 18 backends are healthy [23:24:46] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:55] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.156 second response time [23:24:57] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [23:25:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 9.53, 5.18, 2.07 [23:25:09] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.315 second response time [23:25:17] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.894 second response time [23:25:28] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.517 second response time [23:25:29] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.687 second response time [23:25:33] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 1.386 second response time [23:25:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.11, 5.30, 3.52 [23:25:40] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:25:42] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.063 second response time [23:25:51] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:00] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.77, 6.99, 2.91 [23:26:10] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.252 second response time [23:26:14] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:26:15] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:16] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:25] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.64, 5.97, 2.50 [23:26:27] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:51] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [23:27:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 9.93, 6.70, 2.93 [23:27:17] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.020 second response time [23:27:27] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.19, 2.24 [23:27:43] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 2.96, 2.04 [23:27:58] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:04] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:28:29] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.478 second response time [23:29:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.92, 5.88, 4.20 [23:29:55] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:30:02] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.49, 2.26, 1.57 [23:30:38] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.019 second response time [23:30:50] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:30:57] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:30:58] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.06, 4.98, 3.64 [23:30:59] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:23] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:25] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.47, 2.59 [23:31:34] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:35] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 4 backends are down. mw9 mw13 mw102 mw111 [23:31:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.26, 7.11, 4.85 [23:31:39] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:31:42] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1900s [23:31:44] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.230 second response time [23:31:46] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [23:31:48] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [23:31:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.16, 2.32 [23:31:59] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.615 second response time [23:32:08] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.669 second response time [23:32:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.62, 2.81 [23:32:10] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.456 second response time [23:32:18] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.815 second response time [23:32:19] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.714 second response time [23:32:34] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.621 second response time [23:32:41] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.266 second response time [23:32:50] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:33:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/515de759afe9...c2ed613c5671 [23:33:06] [02miraheze/mw-config] 07paladox 03c2ed613 - Use memcached servers directly for new infra [23:33:18] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 1996s [23:33:25] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [23:33:26] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.889 second response time [23:33:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.54, 3.96, 2.88 [23:33:42] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.805 second response time [23:33:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.33, 3.65, 2.59 [23:34:01] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23573 bytes in 1.647 second response time [23:34:09] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 43s [23:34:09] miraheze/mw-config - paladox the build passed. [23:34:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.81, 2.98 [23:34:19] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 2.672 second response time [23:34:20] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.385 second response time [23:34:24] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.219 second response time [23:34:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:35] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [23:35:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.60, 2.70 [23:35:59] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.608 second response time [23:36:02] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.517 second response time [23:36:04] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.844 second response time [23:36:17] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:36:38] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c2ed613c5671...f92136715010 [23:36:40] [02miraheze/mw-config] 07paladox 03f921367 - Another test [23:36:46] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:36:58] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.75, 5.97, 4.60 [23:37:08] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:37:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:37:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.79, 3.08 [23:37:32] paladox: no IPV6 on that. MW is not updated for your patch I don't think. [23:37:40] miraheze/mw-config - paladox the build passed. [23:37:44] yes i'm trying to deploy that [23:37:51] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw102 [23:37:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.03, 3.36, 2.73 [23:38:19] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:38:27] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:38:33] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:38:40] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:38:58] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:39:12] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/758018e344c6...f4f9db49242f [23:39:13] [02miraheze/mediawiki] 07paladox 03f4f9db4 - MemcachedClient: Add support for ipv6 [23:39:16] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [23:39:26] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:39:27] paladox: it is getting better (it seems to me) and cache is still being populated it seems, according to Grafana. [23:39:46] CosmicAlpha: it's getting there [23:39:47] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [23:39:50] just far too slow [23:40:00] CosmicAlpha: is it? [23:40:08] becuase i cannot load anything [23:40:25] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:40:30] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [23:40:31] seems to me it is. I just browsing around meta with no 502/503 now. [23:40:50] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:40:53] i can edit on meta [23:40:58] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.39, 5.92, 4.87 [23:41:04] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:41:21] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.438 second response time [23:41:41] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:41:41] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.319 second response time [23:41:48] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:41:52] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:41:55] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 84s [23:41:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.30, 3.49, 2.95 [23:41:58] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.019 second response time [23:42:02] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:42:20] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:29] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [23:42:37] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:39] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:44] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:42:58] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.56, 5.71, 4.92 [23:43:00] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.132 second response time [23:43:12] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.450 second response time [23:43:12] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.686 second response time [23:43:14] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.086 second response time [23:43:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.35, 4.01, 3.36 [23:43:32] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [23:43:48] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 78s [23:43:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 4.09, 3.26 [23:44:13] !log [paladox@mwtask111] starting deploy of {'world': True, 'force': True} to scsvg [23:44:16] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:44:32] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:44:48] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.011 second response time [23:45:24] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.077 second response time [23:45:25] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.950 second response time [23:45:26] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [23:45:48] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.392 second response time [23:45:48] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:45:52] !log [paladox@mwtask111] starting deploy of {'world': True, 'force': True} to scsvg [23:46:01] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.230 second response time [23:46:14] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.587 second response time [23:46:22] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:46:30] !log [@mw11] starting deploy of {'config': True} to ovlon [23:46:34] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 4s [23:46:47] !log [@test3] starting deploy of {'config': True} to skip [23:46:48] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [23:46:58] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.77, 5.02, 4.81 [23:47:05] !log [@test101] starting deploy of {'config': True} to skip [23:47:06] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [23:47:17] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:47:20] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [23:47:33] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 2.792 second response time [23:47:56] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.680 second response time [23:48:32] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.321 second response time [23:48:55] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.831 second response time [23:49:03] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [23:49:14] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:49:20] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.378 second response time [23:49:29] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.33, 1.70, 1.06 [23:49:36] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw112 mw122 [23:50:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:50:10] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.658 second response time [23:50:28] [02miraheze/mw-config] 07RhinosF1 pushed 033 commits to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/f92136715010...e8d74a721b9a [23:50:29] [02miraheze/mw-config] 07RhinosF1 033d7fac2 - Revert "Update LocalSettings.php" [23:50:31] [02miraheze/mw-config] 07RhinosF1 031ce5af4 - Revert "[19:45] DC-Switch: Stop uploads (#4353)" [23:50:32] [02miraheze/mw-config] 07RhinosF1 03e8d74a7 - Merge branch 'master' of https://github.com/miraheze/mw-config [23:50:37] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.77, 5.88, 5.83 [23:52:25] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.252 second response time [23:52:25] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.338 second response time [23:52:31] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.957 second response time [23:52:37] miraheze/mw-config - RhinosF1 the build passed. [23:52:56] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:52:57] IP addresses don't appear correct, as I'm seeing 2607:5300:201:3100:0:0:0:5EBC (cp31) as my IP when I'm logged out [23:52:58] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.07, 5.66, 5.15 [23:53:00] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:53:05] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [23:53:07] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:53:07] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.304 second response time [23:53:11] !log [@mwtask111] starting deploy of {'config': True} to scsvg [23:53:14] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for mw101 [23:53:16] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:53:17] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.069 second response time [23:53:24] CosmicAlpha: can you check the ip whitelist includes IPv6 [23:53:37] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.322 second response time [23:53:40] !log [rhinos@mwtask111] starting deploy of {'config': True} to ovlon [23:53:43] Ahh [23:53:44] !log [rhinos@mwtask111] starting deploy of {'config': True, 'force': True} to ovlon [23:53:45] i see [23:53:53] i need to hook cp* up to the new puppetserver [23:53:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 3.67, 3.59 [23:54:03] !log [rhinos@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [23:54:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 3.49, 7.19, 7.93 [23:54:20] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.313 second response time [23:54:20] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.263 second response time [23:54:22] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [23:54:24] !log [@test3] starting deploy of {'config': True} to skip [23:54:25] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [23:54:30] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 8 backends are down. mw8 mw10 mw11 mw101 mw102 mw111 mw112 mw121 [23:54:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/701e94db451b [23:54:34] [02miraheze/puppet] 07paladox 03701e94d - Switch cp* puppetserver to puppet111 [23:54:36] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [23:54:37] [02puppet] 07paladox opened pull request 03#2317: Switch cp* puppetserver to puppet111 - 13https://github.com/miraheze/puppet/pull/2317 [23:54:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/701e94db451b...66eaa12615ef [23:54:51] [02miraheze/puppet] 07paladox 0366eaa12 - Update cp21.yaml [23:54:53] [02puppet] 07paladox synchronize pull request 03#2317: Switch cp* puppetserver to puppet111 - 13https://github.com/miraheze/puppet/pull/2317 [23:54:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/66eaa12615ef...9caff97b3b1d [23:54:58] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.38, 5.62, 5.20 [23:54:59] [02miraheze/puppet] 07paladox 039caff97 - Update cp30.yaml [23:55:01] [02puppet] 07paladox synchronize pull request 03#2317: Switch cp* puppetserver to puppet111 - 13https://github.com/miraheze/puppet/pull/2317 [23:55:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9caff97b3b1d...895bd80f69e2 [23:55:08] [02miraheze/puppet] 07paladox 03895bd80 - Update cp31.yaml [23:55:09] [02puppet] 07paladox synchronize pull request 03#2317: Switch cp* puppetserver to puppet111 - 13https://github.com/miraheze/puppet/pull/2317 [23:55:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:16] [02puppet] 07paladox closed pull request 03#2317: Switch cp* puppetserver to puppet111 - 13https://github.com/miraheze/puppet/pull/2317 [23:55:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/34656300a653...7cb0a8dddb2c [23:55:19] [02miraheze/puppet] 07paladox 037cb0a8d - Switch cp* puppetserver to puppet111 (#2317) [23:55:21] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [23:55:22] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [23:55:27] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.973 second response time [23:55:29] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.95, 1.85, 1.39 [23:55:36] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:55:57] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:56:27] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [23:56:31] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw101 mw102 mw111 [23:56:39] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [23:56:40] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:56:46] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:52] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:56:58] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.16, 4.73, 4.92 [23:57:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.00, 3.40, 3.75 [23:57:29] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.41, 1.35, 1.26 [23:57:39] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 1.22, 3.58, 4.91 [23:57:44] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:57:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.20, 2.62, 3.20 [23:58:02] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:58:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 0.52, 5.18, 7.77 [23:58:06] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:58:07] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:58:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 0.27, 3.41, 6.20 [23:58:48] !log [rhinos@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 285s [23:58:57] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.321 second response time [23:59:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 0.44, 4.97, 7.34 [23:59:11] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:59:20] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.702 second response time [23:59:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 0.63, 2.43, 3.35 [23:59:28] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:59:38] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [23:59:48] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:59:52] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.