[00:02:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.80, 23.14, 21.35 [00:03:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.66, 3.30 [00:04:03] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 319 failures. Last run 3 minutes ago with 319 failures. Failed resources (up to 3 shown): File[sarovia.graalmilitary.com_private],File[wiki.creaturathegame.com],File[wiki.creaturathegame.com_private],File[tot.wiki] [00:04:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.34, 22.71, 21.40 [00:05:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 4.02, 3.47 [00:06:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.22, 23.56, 21.84 [00:07:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 4.00, 3.54 [00:08:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.25, 7.05, 7.99 [00:08:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.76, 23.45, 22.01 [00:09:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 4.07, 3.62 [00:10:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb [00:11:29] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.65, 3.63, 3.52 [00:12:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.26, 7.83, 8.10 [00:12:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.87, 23.45, 22.27 [00:13:48] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:21] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:21] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:22] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.29, 7.41, 7.92 [00:14:24] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:14:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.28, 23.39, 22.40 [00:14:57] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:15:39] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.911 second response time [00:15:44] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.013 second response time [00:15:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.90, 3.29, 3.40 [00:16:26] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.516 second response time [00:16:27] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.251 second response time [00:16:27] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.07, 8.63, 8.30 [00:16:29] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.471 second response time [00:17:04] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.691 second response time [00:19:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.71, 3.54 [00:20:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.81, 26.44, 23.81 [00:20:49] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:50] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:58] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:28] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:40] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:47] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.78, 3.60 [00:22:09] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:10] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:24] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:46] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.013 second response time [00:22:48] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.005 second response time [00:22:58] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.409 second response time [00:23:22] [02puppet] 07Universal-Omega opened pull request 03#2305: Use composer debian package - 13https://github.com/miraheze/puppet/pull/2305 [00:23:22] [url] Page not found · GitHub · GitHub | github.com [00:23:26] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.568 second response time [00:24:02] [02puppet] 07Universal-Omega synchronize pull request 03#2305: Use composer debian package - 13https://github.com/miraheze/puppet/pull/2305 [00:24:03] [url] Page not found · GitHub · GitHub | github.com [00:25:45] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.325 second response time [00:25:54] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.535 second response time [00:26:12] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.317 second response time [00:26:16] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.141 second response time [00:26:28] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.955 second response time [00:26:30] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:26:39] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:02] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:21] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:38] [02puppet] 07Universal-Omega synchronize pull request 03#2305: Use composer debian package - 13https://github.com/miraheze/puppet/pull/2305 [00:27:38] [url] Page not found · GitHub · GitHub | github.com [00:27:47] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:54] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.82, 3.66 [00:28:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.91, 4.29, 2.49 [00:28:45] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.082 second response time [00:29:01] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.924 second response time [00:29:20] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.033 second response time [00:29:53] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.243 second response time [00:29:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.92, 3.47, 3.55 [00:30:20] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:30:39] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:30:40] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.307 second response time [00:30:50] [02puppet] 07Universal-Omega synchronize pull request 03#2305: Use composer debian package - 13https://github.com/miraheze/puppet/pull/2305 [00:30:50] [url] Page not found · GitHub · GitHub | github.com [00:31:11] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:31:27] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:31:32] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:31:52] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:31:54] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.587 second response time [00:31:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.64, 3.59 [00:32:04] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.19, 2.18, 1.29 [00:32:13] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:32:17] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.11, 6.76, 7.91 [00:33:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.61, 3.60 [00:34:04] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.43, 1.57, 1.18 [00:34:12] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.525 second response time [00:34:14] [02puppet] 07Universal-Omega edited pull request 03#2305: MediaWiki: Use Debian composer package - 13https://github.com/miraheze/puppet/pull/2305 [00:34:15] [url] Page not found · GitHub · GitHub | github.com [00:34:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.22, 22.96, 23.78 [00:34:30] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.364 second response time [00:34:31] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.31, 1.65, 1.86 [00:34:38] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:34:48] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.334 second response time [00:35:03] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:35:09] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:35:22] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 4.988 second response time [00:35:33] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.913 second response time [00:35:45] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.494 second response time [00:36:43] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.509 second response time [00:37:05] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.833 second response time [00:37:08] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.457 second response time [00:37:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.68, 4.33, 3.85 [00:38:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.87, 7.04, 7.99 [00:38:32] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.47, 1.07, 1.57 [00:39:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.22, 7.71, 7.93 [00:39:28] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:39:46] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:10] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:40:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.28, 8.36, 8.37 [00:41:44] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:41:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.94, 3.80 [00:41:56] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:42:50] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:43:07] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:43:08] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:02] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:06] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:18] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:20] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:24] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:26] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:45:07] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.696 second response time [00:45:07] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.505 second response time [00:45:36] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.310 second response time [00:45:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.83, 3.77 [00:46:02] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.756 second response time [00:46:16] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.699 second response time [00:46:16] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.618 second response time [00:46:34] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.669 second response time [00:46:37] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw11 [00:46:41] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.329 second response time [00:46:43] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw11 [00:47:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.15, 3.19, 3.54 [00:48:02] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.019 second response time [00:48:13] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.176 second response time [00:48:20] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.314 second response time [00:48:21] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.505 second response time [00:48:27] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.265 second response time [00:48:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.20, 23.03, 23.14 [00:48:29] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.025 second response time [00:48:33] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.087 second response time [00:48:34] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [00:48:37] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.88, 5.88, 4.76 [00:48:41] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [00:49:06] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:59] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:50:36] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.16, 5.77, 4.88 [00:51:06] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.401 second response time [00:51:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.62, 3.60, 3.61 [00:51:55] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.394 second response time [00:52:35] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:37] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:42] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:54:33] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.402 second response time [00:54:35] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.67, 4.83, 4.70 [00:54:36] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:54:37] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.224 second response time [00:54:44] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.527 second response time [00:54:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:55:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 4.00, 3.78 [00:56:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.71, 22.85, 23.20 [00:58:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb [01:00:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:01:49] [02RemovePII] 07Universal-Omega closed pull request 03#43: Create GeneratePII maintenance script - 13https://github.com/miraheze/RemovePII/pull/43 [01:01:49] [url] Page not found · GitHub · GitHub | github.com [01:01:50] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/RemovePII/compare/8c799ed1eed6...1b47a35d3c5e [01:01:51] [url] Comparing 8c799ed1eed6...1b47a35d3c5e · miraheze/RemovePII · GitHub | github.com [01:01:52] [02miraheze/RemovePII] 07Universal-Omega 031b47a35 - Create GeneratePII maintenance script (#43) [01:01:53] [02RemovePII] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/RemovePII [01:01:54] ... [01:01:55] [02miraheze/RemovePII] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:04:36] [02mediawiki] 07Universal-Omega closed pull request 03#4872: Bump skins/Evelution from `3d61d87` to `833020a` - 13https://github.com/miraheze/mediawiki/pull/4872 [01:04:37] [url] Page not found · GitHub · GitHub | github.com [01:04:38] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-833020a - 13https://github.com/miraheze/mediawiki [01:04:39] ... [01:04:39] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-833020a [01:04:41] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/7fb06abb3a70...5c29eb5dfaeb [01:04:42] [url] Comparing 7fb06abb3a70...5c29eb5dfaeb · miraheze/mediawiki · GitHub | github.com [01:04:42] [02miraheze/mediawiki] 07dependabot[bot] 035c29eb5 - Bump skins/Evelution from `3d61d87` to `833020a` (#4872) [01:07:00] miraheze/RemovePII - Universal-Omega the build passed. [01:07:48] [02mediawiki] 07Universal-Omega commented on pull request 03#4916: Bump extensions/PageForms from `7c19d92` to `cc6d16f` - 13https://github.com/miraheze/mediawiki/pull/4916#issuecomment-1011615475 [01:07:49] [url] Bump extensions/PageForms from `7c19d92` to `cc6d16f` by dependabot[bot] · Pull Request #4916 · miraheze/mediawiki · GitHub | github.com [01:07:51] [02mediawiki] 07dependabot[bot] edited pull request 03#4916: Bump extensions/PageForms from `7c19d92` to `cc6d16f` - 13https://github.com/miraheze/mediawiki/pull/4916 [01:07:51] [url] Page not found · GitHub · GitHub | github.com [01:07:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.26, 4.16, 3.85 [01:08:54] [02mediawiki] 07dependabot[bot] edited pull request 03#4916: Bump extensions/PageForms from `7c19d92` to `cc6d16f` - 13https://github.com/miraheze/mediawiki/pull/4916 [01:08:55] [url] Page not found · GitHub · GitHub | github.com [01:08:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageForms-3df1b78 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2dcbeebdfa1a [01:08:58] [02miraheze/mediawiki] 07dependabot[bot] 032dcbeeb - Bump extensions/PageForms from `7c19d92` to `3df1b78` [01:09:00] [02mediawiki] 07dependabot[bot] opened pull request 03#4940: Bump extensions/PageForms from `7c19d92` to `3df1b78` - 13https://github.com/miraheze/mediawiki/pull/4940 [01:09:00] [url] Page not found · GitHub · GitHub | github.com [01:09:01] [02mediawiki] 07dependabot[bot] commented on pull request 03#4916: Bump extensions/PageForms from `7c19d92` to `cc6d16f` - 13https://github.com/miraheze/mediawiki/pull/4916#issuecomment-1011616757 [01:09:02] [url] Bump extensions/PageForms from `7c19d92` to `cc6d16f` by dependabot[bot] · Pull Request #4916 · miraheze/mediawiki · GitHub | github.com [01:09:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageForms-3df1b78 - 13https://github.com/miraheze/mediawiki [01:09:03] [url] Page not found · GitHub · GitHub | github.com [01:09:04] [02mediawiki] 07dependabot[bot] closed pull request 03#4916: Bump extensions/PageForms from `7c19d92` to `cc6d16f` - 13https://github.com/miraheze/mediawiki/pull/4916 [01:09:05] [url] Page not found · GitHub · GitHub | github.com [01:09:06] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-cc6d16f [01:09:07] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-cc6d16f - 13https://github.com/miraheze/mediawiki [01:09:07] ... [01:09:26] [02mediawiki] 07github-actions[bot] labeled pull request 03#4940: Bump extensions/PageForms from `7c19d92` to `3df1b78` - 13https://github.com/miraheze/mediawiki/pull/4940 [01:09:26] ... [01:09:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#4940: Bump extensions/PageForms from `7c19d92` to `3df1b78` - 13https://github.com/miraheze/mediawiki/pull/4940 [01:09:28] ... [01:09:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#4940: Bump extensions/PageForms from `7c19d92` to `3df1b78` - 13https://github.com/miraheze/mediawiki/pull/4940 [01:09:29] [url] Page not found · GitHub · GitHub | github.com [01:10:45] [02mediawiki] 07Universal-Omega closed pull request 03#4940: Bump extensions/PageForms from `7c19d92` to `3df1b78` - 13https://github.com/miraheze/mediawiki/pull/4940 [01:10:46] [url] Page not found · GitHub · GitHub | github.com [01:10:47] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-3df1b78 - 13https://github.com/miraheze/mediawiki [01:10:47] [url] Page not found · GitHub · GitHub | github.com [01:10:48] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-3df1b78 [01:10:50] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/5c29eb5dfaeb...758018e344c6 [01:10:51] [url] Comparing 5c29eb5dfaeb...758018e344c6 · miraheze/mediawiki · GitHub | github.com [01:10:51] [02miraheze/mediawiki] 07dependabot[bot] 03758018e - Bump extensions/PageForms from `7c19d92` to `3df1b78` (#4940) [01:13:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [01:13:26] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:13:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [01:13:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.85, 3.74, 3.79 [01:14:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.84, 22.24, 21.80 [01:15:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:15:27] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.090 second response time [01:15:38] [02mediawiki] 07Universal-Omega opened pull request 03#4941: Rel1 37 - 13https://github.com/miraheze/mediawiki/pull/4941 [01:15:39] [url] Page not found · GitHub · GitHub | github.com [01:16:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.51, 20.82, 21.34 [01:16:56] [02mediawiki] 07Universal-Omega closed pull request 03#4941: Rel1 37 - 13https://github.com/miraheze/mediawiki/pull/4941 [01:16:56] [url] Page not found · GitHub · GitHub | github.com [01:17:15] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:17:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 4.17, 3.95 [01:19:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:19:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.92, 3.77, 3.84 [01:20:09] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:20:38] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:20:40] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:20:51] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:21:05] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:07] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:24:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.42, 4.74, 4.27 [01:26:51] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.22, 5.58, 4.61 [01:27:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.65, 3.74 [01:28:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.36, 4.52, 4.33 [01:29:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.74, 3.76 [01:30:40] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.146 second response time [01:30:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:31:03] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.023 second response time [01:31:13] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.979 second response time [01:31:36] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.943 second response time [01:32:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:33:31] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.520 second response time [01:33:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.80, 4.14, 3.91 [01:35:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.98, 3.62, 3.74 [01:37:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.96, 3.85 [01:38:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.02, 19.07, 20.33 [01:39:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.88, 3.85 [01:41:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [01:42:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.99, 21.22, 20.98 [01:43:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.89, 4.03, 3.89 [01:44:48] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:12] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:32] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.85, 3.84 [01:45:59] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:01] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:47:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:49:45] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.661 second response time [01:50:05] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.317 second response time [01:50:06] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.465 second response time [01:50:53] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.237 second response time [01:51:24] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:44] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:47] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:54] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:04] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:06] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:28] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:45] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:21] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.483 second response time [01:53:45] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.118 second response time [01:53:49] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.773 second response time [01:53:54] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.132 second response time [01:53:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 4.28, 4.01 [01:54:03] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.493 second response time [01:54:08] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:54:11] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.485 second response time [01:54:27] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.772 second response time [01:54:31] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:54:32] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:47] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.837 second response time [01:55:18] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:55:37] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:55:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.90, 3.91, 3.92 [01:56:35] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.719 second response time [01:56:38] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.736 second response time [01:57:33] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.881 second response time [01:57:43] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.532 second response time [01:57:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 4.14, 4.00 [01:59:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:01:01] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:01:02] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:01:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.67, 3.56, 3.83 [02:02:33] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.391 second response time [02:03:01] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.749 second response time [02:03:02] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.583 second response time [02:03:36] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.302 second response time [02:03:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.77, 3.87 [02:04:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:08:27] [02puppet] 07Universal-Omega opened pull request 03#2306: Fix `useformat=mobile` if `title` is not the first url parameter - 13https://github.com/miraheze/puppet/pull/2306 [02:08:27] [url] Page not found · GitHub · GitHub | github.com [02:09:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.83, 20.53, 19.82 [02:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.95, 3.74, 3.88 [02:12:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:13:26] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:13:29] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:13:36] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [02:13:50] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:13:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.86, 4.07, 4.00 [02:14:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.33, 19.50, 20.33 [02:15:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.59, 20.28, 19.96 [02:15:30] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.705 second response time [02:15:30] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.991 second response time [02:15:49] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.235 second response time [02:18:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.55, 20.35, 20.50 [02:19:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.70, 3.90 [02:21:01] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:21:10] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:21:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.96, 3.97 [02:21:57] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:58] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:01] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:11] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:35] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:59] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.198 second response time [02:23:10] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.313 second response time [02:23:31] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:55] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.880 second response time [02:23:56] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.422 second response time [02:23:57] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.030 second response time [02:24:10] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.349 second response time [02:24:20] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:24:41] RECOVERY - dc-multiverse.dcwikis.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/u [02:24:41] /nagios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - dc-multiverse.dcwikis.com reverse DNS resolves to test3.miraheze.org - CNAME OK [02:25:57] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.94, 20.23, 20.00 [02:26:06] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:26:16] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:27:53] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.93, 19.90, 19.92 [02:28:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:29:41] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.160 second response time [02:30:09] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.511 second response time [02:30:20] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.410 second response time [02:30:27] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.705 second response time [02:30:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:31:02] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.164 second response time [02:31:12] RECOVERY - dc-multiverse.dcwikis.com - reverse DNS on sslhost is OK: SSL OK - dc-multiverse.dcwikis.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [02:33:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.67, 6.80, 7.73 [02:34:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.64, 19.84, 20.36 [02:39:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.40, 7.27, 7.57 [02:41:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.49, 7.29, 7.54 [02:41:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.56, 3.64, 4.00 [02:42:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb [02:43:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.49, 8.06, 7.81 [02:43:29] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:44:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:45:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.09, 7.80, 7.75 [02:45:24] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:45:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.59, 4.12, 4.10 [02:47:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 3.76, 3.97 [02:49:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.07, 7.99, 7.82 [02:51:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [02:51:50] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 4.13, 4.08 [02:52:01] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:52:08] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:52:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:52:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:52:34] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:52:35] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:52:36] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:53:23] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:53:55] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.579 second response time [02:53:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.83, 3.97 [02:54:17] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:54:24] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.190 second response time [02:55:36] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:55:39] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:55:54] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:56:01] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:56:04] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.291 second response time [02:56:16] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:56:16] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.173 second response time [02:57:32] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.016 second response time [02:57:35] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.010 second response time [02:57:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 4.14, 4.05 [02:57:55] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.331 second response time [02:58:01] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.337 second response time [02:58:16] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.030 second response time [02:59:43] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.587 second response time [03:00:34] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.047 second response time [03:00:55] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.446 second response time [03:00:57] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.850 second response time [03:01:06] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.140 second response time [03:01:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.94, 4.00 [03:02:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:02:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:04:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.39, 7.18, 7.95 [03:05:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.72, 4.24, 4.09 [03:06:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.80, 7.84, 8.10 [03:12:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:13:24] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [03:15:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:15:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.83, 3.56, 3.87 [03:16:52] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:18:10] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:19:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:20:31] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:20:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:20:50] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.012 second response time [03:20:57] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:21:15] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.268 second response time [03:21:16] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:21:35] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:13] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:24] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:29] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:33] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.467 second response time [03:22:36] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:23:39] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.047 second response time [03:24:14] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.191 second response time [03:24:20] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.578 second response time [03:24:26] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.929 second response time [03:24:31] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.770 second response time [03:24:36] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.775 second response time [03:24:53] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.006 second response time [03:24:57] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.028 second response time [03:25:26] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.320 second response time [03:25:27] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.495 second response time [03:26:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:27:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:35:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:35:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 3.81, 3.64 [03:37:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.72, 3.64 [03:43:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.88, 3.72 [03:44:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.22, 4.69, 3.76 [03:45:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.62, 3.66 [03:46:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.05, 4.45, 3.78 [03:50:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.85, 20.66, 19.85 [03:51:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.50, 3.53 [03:51:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [03:52:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.74, 7.14, 7.93 [03:52:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [03:52:42] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:52:51] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:52:57] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:53:05] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:53:22] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:53:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.52, 3.53 [03:54:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.52, 8.18, 8.19 [03:54:40] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.593 second response time [03:54:54] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.530 second response time [03:55:00] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.320 second response time [03:55:08] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.051 second response time [03:55:30] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.590 second response time [03:55:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:56:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:57:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.47, 7.31, 7.97 [03:57:48] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.60, 19.29, 19.77 [03:57:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.67, 3.59 [03:59:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.14, 7.69, 8.00 [03:59:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [03:59:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.71, 3.62 [04:01:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.45, 7.30, 7.83 [04:01:31] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:05:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.79, 8.10, 8.02 [04:05:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.16, 6.57, 7.89 [04:05:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.64, 3.97, 3.74 [04:06:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.71, 6.87, 7.70 [04:08:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.47, 7.64, 7.88 [04:10:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.06, 7.27, 7.70 [04:11:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.78, 7.59, 7.88 [04:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.93, 3.83 [04:15:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.71, 7.87, 7.82 [04:15:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 4.03, 3.87 [04:17:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.25, 7.44, 7.55 [04:17:38] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.23, 1.57, 1.12 [04:17:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.96, 3.87 [04:18:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.17, 8.02, 7.78 [04:19:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.57, 1.63, 1.20 [04:19:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 4.11, 3.93 [04:20:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.60, 7.30, 7.56 [04:21:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.54, 7.27, 7.47 [04:21:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.97, 3.90 [04:23:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.90, 3.87 [04:24:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.49, 7.72, 7.65 [04:25:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.60, 7.36, 7.82 [04:27:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 7.82, 8.07, 8.05 [04:27:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.95, 7.72, 7.49 [04:27:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.92, 3.69, 3.80 [04:28:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.10, 7.04, 7.46 [04:28:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [04:29:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.40, 7.64, 7.88 [04:30:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:31:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.72, 9.25, 8.45 [04:31:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.66, 7.40, 7.42 [04:35:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.83, 3.81 [04:37:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.34, 7.62, 7.98 [04:37:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.85, 7.49, 7.39 [04:37:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.65, 3.49, 3.69 [04:39:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.41, 7.03, 7.23 [04:40:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.43, 7.55, 7.47 [04:41:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.50, 7.98, 8.03 [04:41:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.75, 8.51, 7.73 [04:42:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.18, 7.65, 7.54 [04:42:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [04:43:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [04:44:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:45:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:45:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.87, 7.71, 7.93 [04:45:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.43, 7.98, 7.71 [04:46:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.17, 7.61, 7.53 [04:47:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.20, 8.62, 8.24 [04:47:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 3.07, 3.39 [04:48:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.85, 6.90, 7.28 [04:48:47] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:49:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.37, 8.76, 8.06 [04:50:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [04:50:50] RECOVERY - mw11 APT on mw11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:50:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb [04:50:58] RECOVERY - mw13 APT on mw13 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:51:20] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:52:21] RECOVERY - mw8 APT on mw8 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:52:25] RECOVERY - mw10 APT on mw10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:52:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:52:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:53:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.13, 7.62, 7.83 [04:53:29] RECOVERY - mwtask1 APT on mwtask1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:54:03] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:54:17] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:55:07] RECOVERY - test101 APT on test101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:55:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.41, 8.07, 7.97 [04:55:23] RECOVERY - mw102 APT on mw102 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:55:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:55:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.37, 3.38 [04:55:56] RECOVERY - mw121 APT on mw121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:55:59] RECOVERY - mw101 APT on mw101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:56:05] RECOVERY - mw122 APT on mw122 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:56:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.57, 7.49, 7.33 [04:56:44] RECOVERY - mw111 APT on mw111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:57:10] !log apt-get upgrade on mw* (both SCSVG and Ovlon), mwtask1, mwtask111, test3, and test101 [04:57:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:58:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.83, 7.38, 7.32 [04:59:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.59, 7.80, 7.91 [04:59:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 3.15, 3.30 [05:00:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:01:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.37, 7.49, 7.93 [05:01:41] RECOVERY - mw112 APT on mw112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:02:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.96, 6.27, 7.90 [05:03:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.50, 6.98, 7.96 [05:04:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.29, 6.11, 6.79 [05:05:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.56, 6.83, 7.30 [05:05:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:07:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.44, 7.82, 7.78 [05:07:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.17, 6.62, 7.18 [05:11:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.37, 3.41, 3.35 [05:11:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.91, 8.08, 7.96 [05:12:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.25, 7.31, 7.01 [05:12:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.39, 6.99, 7.33 [05:13:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.61, 7.07, 7.10 [05:13:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 3.71, 3.46 [05:13:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.72, 7.34, 7.70 [05:15:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.20, 7.09, 7.11 [05:15:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.70, 3.49 [05:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:16:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.89, 7.14, 7.34 [05:18:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.10, 7.79, 7.55 [05:21:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.75, 3.55 [05:21:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.43, 7.79, 7.35 [05:23:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.00, 3.54, 3.50 [05:23:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.94, 7.51, 7.30 [05:23:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.08, 8.38, 7.90 [05:24:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.31, 7.72, 7.72 [05:26:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.38, 8.32, 7.91 [05:26:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.13, 3.24, 3.39 [05:27:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.50, 7.77, 7.81 [05:28:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.96, 6.10, 6.64 [05:30:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:30:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.15, 6.77, 7.44 [05:31:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.18, 7.34, 7.95 [05:31:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.32, 5.77, 6.62 [05:31:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.40, 7.24, 7.88 [05:34:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.75, 3.70, 3.44 [05:36:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:40:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.64, 6.87, 7.04 [05:41:18] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.24, 7.10, 6.83 [05:41:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:41:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.62, 7.22, 7.40 [05:42:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.95, 3.76 [05:43:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.22, 6.93, 7.28 [05:44:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.62, 6.87, 7.07 [05:45:15] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.47, 7.71, 7.10 [05:46:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 3.85, 3.73 [05:47:13] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.02, 6.89, 6.87 [05:47:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.66, 3.68 [05:49:12] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.47, 6.61, 6.75 [05:49:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.64, 7.38, 7.30 [05:50:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.22, 8.26, 7.48 [05:51:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.01, 7.30, 6.65 [05:51:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.91, 3.77 [05:53:08] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 7.08, 6.93 [05:53:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 7.47, 6.79 [05:53:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:54:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.74, 7.67, 7.45 [05:55:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.27, 7.52, 7.17 [05:55:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.81, 7.66, 6.93 [05:55:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.67, 3.46, 3.64 [05:56:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.16, 8.02, 7.61 [05:57:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.51, 7.21, 7.08 [05:57:05] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.99, 5.87, 6.52 [05:57:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.64, 7.46, 6.93 [05:57:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [05:58:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.25, 7.91, 7.63 [05:59:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:00:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.21, 8.10, 7.74 [06:01:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.58, 7.39, 7.19 [06:02:06] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 10 packages available for upgrade (0 critical updates). [06:02:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.27, 7.85, 7.69 [06:03:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.78, 6.32, 6.71 [06:03:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.84, 7.75, 7.36 [06:03:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.12, 3.37 [06:04:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.25, 8.49, 7.93 [06:05:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.88, 7.90, 7.45 [06:07:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.76, 7.75, 7.34 [06:07:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.04, 7.27, 7.28 [06:08:56] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.58, 7.78, 7.05 [06:09:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.09, 8.78, 7.59 [06:09:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.87, 6.86, 7.06 [06:11:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.28, 8.18, 7.68 [06:13:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.74, 6.37, 6.80 [06:13:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.48, 3.46 [06:15:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:17:29] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:17:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.24, 3.24, 3.36 [06:18:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.76, 7.12, 7.67 [06:20:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.99, 7.75, 7.83 [06:21:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.12, 7.25, 6.97 [06:21:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.68, 3.52 [06:22:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.79, 7.76, 7.82 [06:23:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [06:23:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.02, 7.09, 6.94 [06:24:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [06:25:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.44, 7.25, 7.94 [06:26:17] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.89, 20.52, 19.10 [06:27:09] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:27:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.50, 6.53, 6.71 [06:27:37] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:07] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 10 packages available for upgrade (0 critical updates). [06:28:12] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.91, 20.29, 19.18 [06:28:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:29:34] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.574 second response time [06:30:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:31:14] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.901 second response time [06:31:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 4.01, 3.67 [06:31:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.03, 7.81, 7.97 [06:32:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.10, 7.03, 7.13 [06:33:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.65, 7.48, 6.98 [06:34:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.06, 6.40, 6.90 [06:35:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.07, 6.26, 6.60 [06:35:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.07, 7.39, 7.81 [06:37:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.42, 3.79, 3.75 [06:38:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:38:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.53, 6.25, 6.75 [06:41:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.12, 7.85, 7.75 [06:42:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.98, 7.12, 7.87 [06:42:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:42:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.56, 7.37, 7.08 [06:43:03] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 10 packages available for upgrade (0 critical updates). [06:45:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.23, 6.99, 6.70 [06:45:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.60, 7.99, 7.88 [06:46:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.53, 7.84, 7.93 [06:46:35] RECOVERY - db11 APT on db11 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [06:46:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.99, 8.00, 7.45 [06:47:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.43, 7.32, 6.84 [06:47:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.49, 8.31, 8.02 [06:48:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.38, 7.48, 7.80 [06:48:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.87, 8.32, 7.61 [06:52:29] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.25, 7.69, 7.79 [06:53:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.34, 2.91, 3.37 [06:54:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.30, 7.39, 7.68 [06:55:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.79, 7.85, 7.36 [06:58:26] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.85, 7.87, 7.77 [06:59:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.60, 3.55 [07:00:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.19, 6.97, 7.46 [07:01:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.29, 3.17, 3.40 [07:02:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.81, 7.96, 7.98 [07:03:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.97, 7.47, 7.97 [07:03:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.03, 7.27, 7.93 [07:04:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.86, 8.11, 8.02 [07:05:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.73, 7.86, 8.06 [07:07:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.48, 7.43, 7.86 [07:08:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.94, 7.72, 7.89 [07:08:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.43, 3.44 [07:09:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.66, 8.18, 8.11 [07:09:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.81, 7.80, 7.93 [07:10:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.20, 7.51, 7.76 [07:13:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.65, 7.46, 7.20 [07:13:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.69, 7.28, 7.75 [07:15:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.31, 7.30, 7.79 [07:16:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.24, 6.23, 6.75 [07:16:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.75, 7.64, 7.88 [07:17:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.03, 7.96, 7.97 [07:17:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.50, 7.15, 7.19 [07:19:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.64, 7.70, 7.87 [07:19:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.04, 7.56, 7.62 [07:20:14] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.47, 4.21, 3.77 [07:20:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.50, 7.95, 7.92 [07:21:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.43, 7.67, 7.82 [07:21:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.71, 20.10, 18.81 [07:22:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.54, 3.51, 3.56 [07:22:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.62, 7.74, 7.30 [07:23:34] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.88, 19.66, 18.81 [07:24:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.66, 8.18, 7.51 [07:25:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.97, 7.19, 7.62 [07:25:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.50, 8.11, 7.57 [07:25:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.99, 7.26, 7.53 [07:26:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.70, 7.42, 7.33 [07:27:05] [02puppet] 07Reception123 closed pull request 03#2306: Fix `useformat=mobile` if `title` is not the first url parameter - 13https://github.com/miraheze/puppet/pull/2306 [07:27:06] [url] Page not found · GitHub · GitHub | github.com [07:27:07] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6828040a8538...3b9bb933f457 [07:27:08] [url] Comparing 6828040a8538...3b9bb933f457 · miraheze/puppet · GitHub | github.com [07:27:08] [02miraheze/puppet] 07Universal-Omega 033b9bb93 - Fix `useformat=mobile` if `title` is not the first url parameter (#2306) [07:27:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.62, 7.66, 7.63 [07:27:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.60, 3.54 [07:28:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.74, 8.17, 7.63 [07:29:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [07:29:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.74, 3.60 [07:31:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.37, 7.34, 7.46 [07:31:29] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:31:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.84, 3.65 [07:32:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.86, 1.74, 1.17 [07:33:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.91, 7.92, 7.88 [07:33:34] RhinosF1: FYI, as you asked me to earlier, I did apt a couple hours ago. [07:33:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.64, 3.60 [07:34:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.76, 1.24 [07:34:43] CosmicAlpha: thanks [07:35:06] No problem! [07:35:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.35, 7.70, 7.77 [07:35:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.67, 3.61 [07:36:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.74, 1.32, 1.14 [07:37:12] CosmicAlpha: can you give https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DisplayTitle/+/753609 a review [07:37:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.85, 7.36, 7.64 [07:38:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.95, 7.14, 7.85 [07:39:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.89, 7.85, 7.89 [07:41:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.21, 7.83, 7.87 [07:41:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.27, 6.96, 7.36 [07:41:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.94, 8.43, 8.11 [07:41:47] RhinosF1: looking. [07:42:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.20, 8.33, 8.09 [07:43:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.23, 8.08, 7.96 [07:43:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.76, 3.80 [07:43:57] I'll give it a +1 tomorrow after testing if it works, but from what I can tell it looks like it should work. [07:44:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.29, 7.53, 7.86 [07:45:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.67, 7.49, 7.75 [07:45:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.74, 7.74, 7.64 [07:45:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.58, 7.53, 7.83 [07:46:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.64, 8.05, 8.01 [07:46:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.77, 7.72, 7.91 [07:48:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.74, 7.80, 7.94 [07:49:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.20, 8.35, 7.85 [07:50:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.08, 8.42, 8.13 [07:50:51] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.34, 4.72, 3.56 [07:51:03] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.01, 7.19, 7.48 [07:51:47] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.87, 21.87, 20.13 [07:52:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.14, 8.09, 7.98 [07:53:03] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.64, 7.29, 7.50 [07:54:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.07, 7.57, 7.89 [07:54:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.60, 5.34, 4.09 [07:55:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.14, 7.49, 7.54 [07:55:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.16, 7.96, 7.95 [07:55:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.61, 3.65 [07:56:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.58, 7.76, 7.86 [07:56:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.88, 4.99, 4.10 [07:57:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.07, 7.31, 7.47 [07:57:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.58, 22.51, 21.06 [07:57:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.31, 3.53 [07:58:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.70, 8.23, 8.04 [07:59:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 2.93, 3.36 [08:00:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.94, 7.62, 7.83 [08:01:28] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.42, 20.04, 20.37 [08:03:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.67, 6.91, 7.16 [08:03:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.78, 7.54, 7.37 [08:04:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.57, 7.84, 7.80 [08:05:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.41, 6.85, 7.12 [08:05:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.81, 6.17, 7.66 [08:06:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.60, 7.21, 7.56 [08:07:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.08, 5.77, 6.76 [08:07:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:07:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.57, 6.86, 7.12 [08:07:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.19, 3.30 [08:09:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.38, 6.98, 7.06 [08:09:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.96, 3.18, 3.28 [08:11:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:11:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.54, 8.39, 7.65 [08:11:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.65, 7.53, 7.72 [08:12:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.09, 7.55, 7.55 [08:12:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.03, 7.78, 7.37 [08:13:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.30, 7.14, 6.97 [08:13:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.36, 7.72, 7.51 [08:14:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.60, 7.26, 7.45 [08:14:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.17, 7.66, 7.39 [08:15:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.80, 6.72, 7.02 [08:15:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.55, 7.23, 7.02 [08:15:36] paladox: around? [08:16:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.25, 7.82, 7.47 [08:17:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.08, 7.78, 7.56 [08:19:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.03, 7.50, 7.48 [08:19:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.42, 3.31 [08:20:28] Semi Reception123 [08:20:31] Mobile only [08:20:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb [08:20:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.02, 21.14, 19.55 [08:21:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.29, 7.87, 7.35 [08:21:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.94, 7.74, 7.55 [08:22:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:23:54] paladox: I'm trying to generate a wildcard for the gyannipedia wikis as there's a lot of them but getting an error [08:23:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.15, 2.97, 3.19 [08:23:56] doing reception@mwtask111:~$ sudo /root/ssl-certificate -d gyaanipedia.com -s *.gyaanipedia.com -g -p -o [08:24:02] https://www.irccloud.com/pastebin/ve7mLjr9/ [08:24:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.60, 7.94, 7.56 [08:25:08] You have to do -w [08:25:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.97, 8.07, 7.54 [08:28:25] paladox: oh, thanks. there's a typo in the docs then, I'll fix that [08:28:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.85, 20.38, 19.93 [08:29:13] paladox: oh though I guess we can't do wildcards unless we control the NS due to the challenges [08:29:22] (or I mean we can but it would be complicated to coordinate) [08:29:30] Yeh :/ [08:29:42] It should be pointing at us [08:29:42] :( [08:32:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::5ebc/cpweb [08:32:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.80, 7.61, 7.68 [08:32:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.29, 21.92, 20.59 [08:34:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:34:30] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.33, 21.10, 20.45 [08:35:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.91, 3.51 [08:37:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.59, 3.45 [08:40:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.52, 20.36, 20.35 [08:41:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.13, 7.82, 7.90 [08:43:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.42, 7.92, 7.99 [08:44:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.52, 21.08, 20.69 [08:45:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.06, 7.97, 7.99 [08:45:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.44, 7.33, 7.85 [08:47:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.61, 7.65, 7.86 [08:47:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.21, 3.25, 3.38 [08:49:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.23, 7.80, 7.78 [08:49:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.70, 7.57, 7.80 [08:51:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.70, 7.66, 7.76 [08:51:34] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±12] 13https://github.com/miraheze/ssl/compare/8f277410ace8...31b867d36b8d [08:51:35] [url] Comparing 8f277410ace8...31b867d36b8d · miraheze/ssl · GitHub | github.com [08:51:36] [02miraheze/ssl] 07Reception123 0331b867d - renew *.gyaanipedia.com certs [08:51:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.42, 7.50, 7.74 [08:51:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.58, 3.50 [08:53:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.64, 7.54, 7.70 [08:54:14] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:54:30] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:54:54] !log manually deleted page on benpediawiki using deleteBatch.php [08:55:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:55:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.89, 7.82, 7.80 [08:55:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.85, 3.16, 3.34 [08:56:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.94, 19.64, 20.27 [08:56:34] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:56:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.77, 6.89, 7.96 [08:56:59] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:57:42] Reception123: you can't do wildcards for them [08:57:49] You'd have to issue manually for each [08:57:57] Or one big with all languages listed [08:58:02] RhinosF1: yeah I know, I manually did them now [08:58:11] because we can't do WC unless we host the DNS since we need the challenges [08:58:15] Reception123: puppet is failing due to nginx [08:58:19] So can you check [08:58:44] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.49, 7.42, 8.01 [08:58:49] RhinosF1: it's always the same thing when I replace keys, I just need to wait until it does its thing and re-run puppet [08:59:03] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:59:16] Reception123: make sure ssl is pulled on the puppet server [09:01:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.62, 7.43, 7.54 [09:01:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.50, 7.21, 7.38 [09:01:29] yeah there's nothing surprising about the error, it's always happened in these months when I did manual renewals [09:01:32] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:01:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.08, 7.79, 7.74 [09:02:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.98, 7.78, 7.31 [09:03:08] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::5ebc/cpweb [09:03:31] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [09:03:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [09:03:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.98, 7.19, 7.53 [09:04:14] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:04:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.19, 7.15, 7.15 [09:05:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.52, 7.07, 7.44 [09:05:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:05:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.37, 7.91, 7.69 [09:05:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:06:05] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.25, 5.12, 2.86 [09:07:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.16, 7.69, 7.63 [09:07:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.69, 21.45, 20.64 [09:09:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.08, 7.56, 7.58 [09:11:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.18, 7.30, 7.49 [09:11:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.94, 7.57, 7.42 [09:12:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.54, 8.05, 7.46 [09:13:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.99, 8.19, 7.80 [09:13:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [09:14:04] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.61, 1.44, 1.88 [09:14:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.20, 7.63, 7.37 [09:15:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.82, 7.74, 7.69 [09:15:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:15:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.25, 7.54, 7.46 [09:18:04] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.57, 0.95, 1.57 [09:18:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.50, 3.33 [09:20:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.28, 3.27 [09:20:57] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.45, 21.86, 21.24 [09:21:15] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:21:38] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:49] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:22:20] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:22:28] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [09:22:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.84, 21.51, 21.19 [09:23:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [09:23:18] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.386 second response time [09:23:46] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.615 second response time [09:23:55] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.777 second response time [09:24:01] RECOVERY - hi.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:38 GMT +0000. [09:24:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.97, 6.07, 6.73 [09:24:25] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.461 second response time [09:24:33] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [09:25:46] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.70, 6.37, 6.80 [09:26:08] RECOVERY - en.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:27 GMT +0000. [09:26:33] RECOVERY - es.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'es.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:32 GMT +0000. [09:26:34] RECOVERY - zh.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'zh.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:46:48 GMT +0000. [09:26:47] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.80, 23.03, 21.84 [09:26:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [09:26:50] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:26:55] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:26:56] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:26:57] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:26:59] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:27:02] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [09:27:17] RECOVERY - de.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'de.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:17 GMT +0000. [09:27:22] RECOVERY - pa.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pa.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:46:39 GMT +0000. [09:27:24] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:03] RECOVERY - mr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mr.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:22 GMT +0000. [09:28:20] RECOVERY - meta.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:46:58 GMT +0000. [09:28:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.91, 7.25, 7.08 [09:28:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.56, 22.12, 21.65 [09:28:45] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fr.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:46:53 GMT +0000. [09:28:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:28:50] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.712 second response time [09:28:55] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.226 second response time [09:28:55] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.097 second response time [09:28:57] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.521 second response time [09:29:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:29:26] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.090 second response time [09:29:30] RECOVERY - zh.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'zh.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:46:48 GMT +0000. [09:29:31] RECOVERY - en.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:27 GMT +0000. [09:29:33] RECOVERY - meta.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:46:58 GMT +0000. [09:29:33] RECOVERY - es.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'es.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:32 GMT +0000. [09:29:53] RECOVERY - lg.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lg.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:46:44 GMT +0000. [09:29:53] RECOVERY - commons.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:43 GMT +0000. [09:30:14] RECOVERY - hi.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:38 GMT +0000. [09:30:47] RECOVERY - bn.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'bn.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:12 GMT +0000. [09:30:52] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fr.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:46:53 GMT +0000. [09:31:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.02, 5.96, 6.69 [09:31:03] RECOVERY - commons.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:43 GMT +0000. [09:31:15] RECOVERY - de.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'de.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:17 GMT +0000. [09:31:23] RECOVERY - bn.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'bn.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:12 GMT +0000. [09:31:36] RECOVERY - pa.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pa.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:46:39 GMT +0000. [09:31:36] RECOVERY - mr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mr.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:40:22 GMT +0000. [09:33:09] RECOVERY - lg.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lg.gyaanipedia.com' will expire on Wed 13 Apr 2022 07:46:44 GMT +0000. [09:33:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.87, 7.12, 6.98 [09:34:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.34, 5.83, 6.54 [09:39:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.57, 8.02, 7.40 [09:41:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.05, 6.52, 6.51 [09:41:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.54, 7.63, 7.34 [09:42:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.83, 7.56, 6.98 [09:43:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.61, 5.82, 6.25 [09:44:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.96, 6.27, 6.56 [09:45:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.23, 7.74, 7.42 [09:49:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.19, 7.18, 6.66 [09:51:01] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:51:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.37, 6.59, 6.50 [09:51:34] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:51:35] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:51:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.74, 7.68, 7.63 [09:51:48] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:58] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.864 second response time [09:53:35] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.667 second response time [09:53:38] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.283 second response time [09:53:53] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 5.969 second response time [09:55:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.31, 7.29, 6.83 [09:57:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.26, 6.64, 6.65 [10:01:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.67, 8.62, 7.92 [10:03:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.44, 8.00, 7.21 [10:04:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.85, 19.18, 20.20 [10:05:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.33, 7.42, 7.11 [10:05:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.10, 7.60, 7.72 [10:06:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:07:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.67, 8.39, 7.48 [10:07:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/bb1a148ae15d [10:07:56] [url] Page not found · GitHub · GitHub | github.com [10:07:57] [02miraheze/puppet] 07paladox 03bb1a148 - Switch ns2 puppet server [10:07:58] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [10:07:59] ... [10:08:00] [02puppet] 07paladox opened pull request 03#2307: Switch ns2 puppet server - 13https://github.com/miraheze/puppet/pull/2307 [10:08:00] ... [10:08:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bb1a148ae15d...020e724831be [10:08:49] [url] Comparing bb1a148ae15d...020e724831be · miraheze/puppet · GitHub | github.com [10:08:50] [02miraheze/puppet] 07paladox 03020e724 - Update ns2.yaml [10:08:52] [02puppet] 07paladox synchronize pull request 03#2307: Switch ns2 puppet server - 13https://github.com/miraheze/puppet/pull/2307 [10:08:52] [url] Page not found · GitHub · GitHub | github.com [10:09:28] [02puppet] 07paladox closed pull request 03#2307: Switch ns2 puppet server - 13https://github.com/miraheze/puppet/pull/2307 [10:09:29] [url] Page not found · GitHub · GitHub | github.com [10:09:29] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [10:09:31] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [10:09:31] [url] Page not found · GitHub · GitHub | github.com [10:09:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3b9bb933f457...cca92d3147cc [10:09:33] [url] Comparing 3b9bb933f457...cca92d3147cc · miraheze/puppet · GitHub | github.com [10:09:34] [02miraheze/puppet] 07paladox 03cca92d3 - Switch ns2 puppet server (#2307) [10:11:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.84, 7.53, 7.34 [10:11:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.02, 6.86, 7.83 [10:11:46] [02puppet] 07paladox closed pull request 03#2304: dns: update ns2 IPs in recursor - 13https://github.com/miraheze/puppet/pull/2304 [10:11:46] [url] Page not found · GitHub · GitHub | github.com [10:11:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cca92d3147cc...cd56646eb782 [10:11:48] [url] Comparing cca92d3147cc...cd56646eb782 · miraheze/puppet · GitHub | github.com [10:11:49] [02miraheze/puppet] 07Universal-Omega 03cd56646 - dns: update ns2 IPs in recursor (#2304) [10:12:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:13:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.07, 7.35, 7.89 [10:14:00] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.06, 6.95, 6.80 [10:14:20] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [10:15:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.11, 7.87, 7.47 [10:15:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.06, 8.41, 7.99 [10:17:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.41, 7.83, 7.52 [10:17:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.09, 7.31, 7.77 [10:17:37] PROBLEM - puppet111 Current Load on puppet111 is CRITICAL: CRITICAL - load average: 7.92, 4.11, 2.27 [10:17:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:17:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.39, 7.25, 7.61 [10:17:57] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.41, 5.67, 6.36 [10:19:37] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 2.40, 3.41, 2.25 [10:19:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.45, 8.15, 7.87 [10:21:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.52, 7.53, 7.36 [10:21:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.79, 8.13, 8.01 [10:21:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.92, 7.85, 7.81 [10:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:23:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.33, 8.01, 7.87 [10:27:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.56, 7.92, 7.67 [10:27:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.97, 7.61, 7.80 [10:35:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.65, 6.86, 7.78 [10:36:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.81, 6.52, 7.76 [10:37:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.16, 7.16, 7.31 [10:38:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:39:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.55, 6.77, 7.17 [10:41:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.95, 7.20, 7.52 [10:42:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:42:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.40, 7.44, 7.65 [10:43:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.14, 7.09, 7.05 [10:43:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.60, 7.21, 7.48 [10:44:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.71, 7.18, 7.54 [10:45:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.62, 7.35, 7.15 [10:47:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.51, 6.77, 6.35 [10:48:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.05, 7.42, 7.54 [10:49:32] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.85, 5.75, 6.03 [10:50:28] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:53:06] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.62, 19.92, 19.14 [10:55:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.76, 7.61, 7.24 [10:55:03] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.26, 19.18, 18.95 [10:55:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.39, 8.12, 7.45 [10:57:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.21, 7.20, 7.15 [10:57:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.72, 8.07, 7.60 [10:57:26] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.12, 6.98, 6.56 [10:57:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.75, 7.78, 7.41 [11:03:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.67, 7.47, 7.18 [11:03:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.57, 6.34, 6.43 [11:03:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.39, 7.55, 7.35 [11:05:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 7.86, 7.83 [11:05:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.36, 6.93, 7.16 [11:07:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.62, 7.82, 7.44 [11:08:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.53, 7.09, 7.88 [11:09:18] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.67, 7.85, 7.05 [11:11:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.86, 7.51, 7.42 [11:11:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.25, 7.52, 7.02 [11:11:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.50, 7.49, 7.60 [11:11:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.59, 7.63, 7.49 [11:12:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.75, 7.52, 7.82 [11:13:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.27, 7.66, 7.68 [11:14:44] !log reimaging ns1 (bullseye) [11:14:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:15:14] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.12, 7.47, 7.08 [11:15:36] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:17:02] PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:17:28] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cd56646eb782...3cf372c7b4f0 [11:17:29] [url] Comparing cd56646eb782...3cf372c7b4f0 · miraheze/puppet · GitHub | github.com [11:17:29] [02miraheze/puppet] 07Reception123 033cf372c - switch ns1 hiera to use puppet111 (SCSVG) [11:17:48] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:17:50] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:18:06] PROBLEM - ns1 APT on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:18:13] PROBLEM - ns1 conntrack_table_size on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:18:26] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:18:29] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 100% [11:18:40] PROBLEM - ns1 ferm_active on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:18:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: connect to address 2607:f1c0:1800:1f2::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:1f2::1 port 5666: Connection refused [11:18:54] PROBLEM - ns1 NTP time on ns1 is CRITICAL: connect to address 2607:f1c0:1800:1f2::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:1f2::1 port 5666: Connection refused [11:18:57] RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [11:19:11] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.66, 6.46, 6.80 [11:19:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.92, 8.11, 7.83 [11:20:33] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 111.03 ms [11:21:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.55, 6.11, 6.74 [11:21:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.57, 7.54, 7.66 [11:22:43] paladox: dpkg: error: cannot access archive 'puppet7-release-bullseye.deb': No such file or directory [11:22:54] do buster for that one [11:23:19] ok [11:23:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.64, 8.21, 7.87 [11:25:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.16, 7.18, 7.06 [11:26:46] paladox: Notice: Did not receive certificate and puppetserver ca list empty :( [11:26:52] I did sudo puppet agent -tv --server puppet111.miraheze.org --waitforcert 60 [11:27:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.28, 7.52, 7.20 [11:27:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.69, 7.65, 7.14 [11:27:09] hmm [11:27:43] do it without waitforcert [11:29:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.99, 7.37, 7.17 [11:29:03] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.95, 6.93, 6.94 [11:29:09] paladox: interesting, worked after rm -rf /var/lib/puppet/ssl but the issue is it now appears as localhost.localdomain instead of ns`1 [11:29:18] oh [11:29:30] guess I'd just edit /etc/hostname? [11:29:33] i guess it uses cloud-init [11:29:39] and yeh [11:33:00] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.99, 7.55, 7.16 [11:33:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.89, 7.54, 7.24 [11:33:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.48, 7.75, 7.31 [11:35:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.70, 7.96, 7.46 [11:35:57] https://www.irccloud.com/pastebin/qYJ8deKx/ [11:35:59] ^ paladox hmm [11:36:08] that's puppet3 [11:36:12] not puppet111 [11:36:21] oh nevermind yeah :( [11:37:33] paladox: but the same error for 111 [11:37:41] https://www.irccloud.com/pastebin/2aZlW9eR/ [11:37:55] i guess delete the cert and try again [11:39:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 12.05, 9.15, 7.97 [11:41:42] paladox: same error :( [11:41:53] i don't understand how? [11:42:14] you also have to delete it puppet111 end [11:42:27] me neither, I just did root@ns1:~# puppet agent -tv --server puppet111.miraheze.org and then puppetserver ca sign --certname ns1.miraheze.org [11:42:40] paladox: yeah, I did ca clean before trying again [11:43:22] hmm, can you send me the pass, i can try [11:43:43] ok, will PM [11:43:47] thanks! [11:46:50] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.47, 7.11, 7.55 [11:52:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [11:54:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:01:03] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.87, 5.85, 6.60 [12:01:16] PROBLEM - ns1 GDNSD Datacenters on ns1 is UNKNOWN: NRPE: Unable to read output [12:01:22] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoLite2-Country.mmdb] [12:01:28] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:03:22] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.249 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [12:03:22] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.90, 7.48, 7.92 [12:03:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3cf372c7b4f0...bf2fdb9a501c [12:03:58] [url] Comparing 3cf372c7b4f0...bf2fdb9a501c · miraheze/puppet · GitHub | github.com [12:03:59] [02miraheze/puppet] 07paladox 03bf2fdb9 - Ensure /usr/share/GeoIP directory exists [12:05:15] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [12:05:18] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.29, 8.47, 8.23 [12:06:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.91, 7.40, 7.93 [12:07:37] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/landing/compare/6118b8756027...d24c6290e502 [12:07:37] [url] Comparing 6118b8756027...d24c6290e502 · miraheze/landing · GitHub | github.com [12:07:38] [02miraheze/landing] 07translatewiki 03d24c629 - Localisation updates from https://translatewiki.net. [12:07:39] [url] Main page - translatewiki.net | translatewiki.net [12:07:40] [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/f97664c111ce...e9282bef4932 [12:07:40] [url] Comparing f97664c111ce...e9282bef4932 · miraheze/GlobalNewFiles · GitHub | github.com [12:07:41] [02miraheze/GlobalNewFiles] 07translatewiki 03e9282be - Localisation updates from https://translatewiki.net. [12:07:42] [url] Main page - translatewiki.net | translatewiki.net [12:07:43] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±31] 13https://github.com/miraheze/CreateWiki/compare/026467e75b9c...2b0f27deab37 [12:07:44] [url] Comparing 026467e75b9c...2b0f27deab37 · miraheze/CreateWiki · GitHub | github.com [12:07:44] [02miraheze/CreateWiki] 07translatewiki 032b0f27d - Localisation updates from https://translatewiki.net. [12:07:45] [url] Main page - translatewiki.net | translatewiki.net [12:07:46] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±11] 13https://github.com/miraheze/MirahezeMagic/compare/324fad04fa32...55a7813f642b [12:07:46] [url] Comparing 324fad04fa32...55a7813f642b · miraheze/MirahezeMagic · GitHub | github.com [12:07:47] [02miraheze/MirahezeMagic] 07translatewiki 0355a7813 - Localisation updates from https://translatewiki.net. [12:07:48] [url] Main page - translatewiki.net | translatewiki.net [12:08:31] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.18, 7.43, 7.86 [12:08:58] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.88, 8.20, 7.34 [12:09:28] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:09:30] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:55] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [12:10:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.29, 6.99, 7.65 [12:10:31] miraheze/landing - translatewiki the build passed. [12:10:42] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:11:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.53, 3.04 [12:12:35] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:12:54] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:13:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.97, 2.94, 2.88 [12:13:38] miraheze/CreateWiki - translatewiki the build passed. [12:14:41] [02miraheze/puppet] 07paladox created branch 03paladox-patch-4 13https://github.com/miraheze/puppet/commit/66abbc40c97c95790c2633a8b57fbca20c76e0c8 [12:14:42] [02miraheze/puppet] 07paladox created branch 03paladox-patch-4-1 13https://github.com/miraheze/puppet/commit/66abbc40c97c95790c2633a8b57fbca20c76e0c8 [12:14:44] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [12:14:44] [url] Page not found · GitHub · GitHub | github.com [12:14:45] [02puppet] 07paladox created branch 03paladox-patch-4-1 - 13https://github.com/miraheze/puppet [12:14:45] [url] Page not found · GitHub · GitHub | github.com [12:14:47] [02puppet] 07paladox opened pull request 03#2308: Update prometheus-gdnsd-stats.py - 13https://github.com/miraheze/puppet/pull/2308 [12:14:47] [url] Page not found · GitHub · GitHub | github.com [12:18:09] miraheze/GlobalNewFiles - translatewiki the build passed. [12:18:56] miraheze/MirahezeMagic - translatewiki the build passed. [12:18:56] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.16, 20.62, 18.77 [12:19:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a9579ad5a256 [12:19:56] [url] Page not found · GitHub · GitHub | github.com [12:19:57] [02miraheze/puppet] 07paladox 03a9579ad - check_gdnsd_datacenters: Add support for gdnsd 3.x [12:19:58] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [12:19:58] [url] Page not found · GitHub · GitHub | github.com [12:20:00] [02puppet] 07paladox opened pull request 03#2309: check_gdnsd_datacenters: Add support for gdnsd 3.x - 13https://github.com/miraheze/puppet/pull/2309 [12:20:00] ... [12:20:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a9579ad5a256...287cf93e5827 [12:20:20] [url] Comparing a9579ad5a256...287cf93e5827 · miraheze/puppet · GitHub | github.com [12:20:20] [02miraheze/puppet] 07paladox 03287cf93 - Update check_gdnsd_datacenters.py [12:20:22] [02puppet] 07paladox synchronize pull request 03#2309: check_gdnsd_datacenters: Add support for gdnsd 3.x - 13https://github.com/miraheze/puppet/pull/2309 [12:20:23] ... [12:23:43] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:25:41] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.141 second response time [12:26:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.81, 6.81, 7.88 [12:26:15] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.356 second response time [12:26:17] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.635 second response time [12:26:53] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.432 second response time [12:27:19] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.593 second response time [12:27:32] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.934 second response time [12:28:01] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.76, 7.86, 7.57 [12:28:06] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.15, 8.03, 8.19 [12:29:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.95, 7.38, 7.43 [12:30:54] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.13, 20.27, 19.76 [12:31:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.59, 7.52, 7.45 [12:32:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.39, 3.15 [12:34:42] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.31, 3.33, 3.15 [12:44:47] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:45:06] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:46:50] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.705 second response time [12:50:08] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:31] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:51:17] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:51:46] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:52:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.76, 20.36, 19.25 [12:52:42] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:53:05] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:53:08] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:09] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:54:26] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.360 second response time [12:54:38] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.498 second response time [12:54:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.13, 20.17, 19.31 [12:55:28] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.053 second response time [12:55:37] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.367 second response time [12:55:57] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.732 second response time [12:56:47] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.259 second response time [12:57:13] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.480 second response time [12:57:18] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.302 second response time [12:57:18] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.884 second response time [12:58:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/287cf93e5827...564ad568dbed [12:58:26] [url] Comparing 287cf93e5827...564ad568dbed · miraheze/puppet · GitHub | github.com [12:58:27] [02miraheze/puppet] 07paladox 03564ad56 - Update check_gdnsd_datacenters.py [12:58:28] [02puppet] 07paladox synchronize pull request 03#2309: check_gdnsd_datacenters: Add support for gdnsd 3.x - 13https://github.com/miraheze/puppet/pull/2309 [12:58:29] [url] Page not found · GitHub · GitHub | github.com [13:09:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.56, 3.21 [13:09:58] [02puppet] 07paladox closed pull request 03#2309: check_gdnsd_datacenters: Add support for gdnsd 3.x - 13https://github.com/miraheze/puppet/pull/2309 [13:09:59] [url] Page not found · GitHub · GitHub | github.com [13:10:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bf2fdb9a501c...563bb98c6f19 [13:10:01] [url] Comparing bf2fdb9a501c...563bb98c6f19 · miraheze/puppet · GitHub | github.com [13:10:01] [02miraheze/puppet] 07paladox 03563bb98 - check_gdnsd_datacenters: Add support for gdnsd 3.x (#2309) [13:10:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [13:10:05] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [13:10:05] [url] Page not found · GitHub · GitHub | github.com [13:10:18] [02puppet] 07paladox closed pull request 03#2308: Update prometheus-gdnsd-stats.py - 13https://github.com/miraheze/puppet/pull/2308 [13:10:19] [url] Page not found · GitHub · GitHub | github.com [13:10:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/563bb98c6f19...8ae27c294a3d [13:10:21] [url] Comparing 563bb98c6f19...8ae27c294a3d · miraheze/puppet · GitHub | github.com [13:10:21] [02miraheze/puppet] 07paladox 038ae27c2 - Update prometheus-gdnsd-stats.py (#2308) [13:10:23] [02puppet] 07paladox deleted branch 03paladox-patch-4-1 - 13https://github.com/miraheze/puppet [13:10:23] [url] Page not found · GitHub · GitHub | github.com [13:10:24] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4-1 [13:11:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 3.10, 3.08 [13:13:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/aeff100f8ce1 [13:13:39] [02miraheze/puppet] 07paladox 03aeff100 - Fix check_gdnsd_datacenters [13:13:41] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [13:13:41] [url] Page not found · GitHub · GitHub | github.com [13:13:42] [02puppet] 07paladox opened pull request 03#2310: Fix check_gdnsd_datacenters - 13https://github.com/miraheze/puppet/pull/2310 [13:13:43] ... [13:14:19] [02puppet] 07paladox closed pull request 03#2310: Fix check_gdnsd_datacenters - 13https://github.com/miraheze/puppet/pull/2310 [13:14:19] ... [13:14:20] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [13:14:22] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [13:14:22] [url] Page not found · GitHub · GitHub | github.com [13:14:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8ae27c294a3d...565808137558 [13:14:24] [url] Comparing 8ae27c294a3d...565808137558 · miraheze/puppet · GitHub | github.com [13:14:25] [02miraheze/puppet] 07paladox 035658081 - Fix check_gdnsd_datacenters (#2310) [13:15:46] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:17:41] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.323 second response time [13:21:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.65, 7.11, 7.90 [13:22:01] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:22:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:37] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:22:44] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:23:02] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:23:05] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:23:56] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.330 second response time [13:23:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/18c28dcd010e [13:24:00] [02miraheze/puppet] 07paladox 0318c28dc - Allow nagios to use sudo for gdnsdctl states [13:24:01] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [13:24:02] [url] Page not found · GitHub · GitHub | github.com [13:24:03] [02puppet] 07paladox opened pull request 03#2311: Allow nagios to use sudo for gdnsdctl states - 13https://github.com/miraheze/puppet/pull/2311 [13:24:03] [url] Page not found · GitHub · GitHub | github.com [13:24:15] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 8.876 second response time [13:24:40] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.029 second response time [13:24:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/18c28dcd010e...9927d41216ee [13:24:45] [url] Comparing 18c28dcd010e...9927d41216ee · miraheze/puppet · GitHub | github.com [13:24:46] [02miraheze/puppet] 07paladox 039927d41 - Update nrpe.cfg.erb [13:24:47] [02puppet] 07paladox synchronize pull request 03#2311: Allow nagios to use sudo for gdnsdctl states - 13https://github.com/miraheze/puppet/pull/2311 [13:24:48] [url] Page not found · GitHub · GitHub | github.com [13:24:56] [02puppet] 07paladox closed pull request 03#2311: Allow nagios to use sudo for gdnsdctl states - 13https://github.com/miraheze/puppet/pull/2311 [13:24:56] [url] Page not found · GitHub · GitHub | github.com [13:24:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/565808137558...0a7f0d5d0be4 [13:24:58] [url] Comparing 565808137558...0a7f0d5d0be4 · miraheze/puppet · GitHub | github.com [13:24:59] [02miraheze/puppet] 07paladox 030a7f0d5 - Allow nagios to use sudo for gdnsdctl states (#2311) [13:25:01] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [13:25:02] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [13:25:03] ... [13:25:06] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.423 second response time [13:25:13] CosmicAlpha: the reason mhbot is reacting to notifico links at all right now is because, we had git.io ignored in the url module, but now the git.io service is discontinued, and you can’t shorten new links anymore and thus notifico posts full links and GitHub.com isn’t ignored [13:25:20] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:25:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.78, 7.82, 7.99 [13:27:18] .blocks add Not-*!*@* [13:27:18] RhinosF1: I could not figure out what you wanted to do. [13:27:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.19, 7.71, 7.94 [13:27:30] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:27:30] .help blocks [13:27:30] RhinosF1: Manage Sopel's blocking features. See [ignore system documentation]({% link _usage/ignoring-people.md %}). [13:27:45] .blocks nick add Not-9244 [13:27:45] RhinosF1: I could not figure out what you wanted to do. [13:27:54] .blocks add nick Not-9244 [13:27:55] RhinosF1: Successfully added block: Not-9244 [13:28:06] .blocks add nick Not-*!*@* [13:28:06] RhinosF1: Successfully added block: Not-*!*@* [13:28:11] Hopefully [13:28:16] RhinosF1: regex wont work since when Sario tried it it killed the bots [13:28:33] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.03, 20.99, 19.92 [13:28:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [13:29:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0a7f0d5d0be4...99e32d7a4776 [13:29:15] [02miraheze/puppet] 07paladox 0399e32d7 - Fix nagios sudo perms [13:29:16] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.317 second response time [13:29:22] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.020 second response time [13:29:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.28, 8.89, 8.34 [13:29:29] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.019 second response time [13:29:43] MacFan4000: they still alive [13:30:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:30:31] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.189 second response time [13:31:01] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.322 second response time [13:31:12] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:31:21] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:33] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:31:36] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:31:59] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:32:21] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:32:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.10, 20.10, 19.77 [13:33:14] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.113 second response time [13:33:22] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.490 second response time [13:33:31] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.021 second response time [13:33:37] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.090 second response time [13:33:57] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.307 second response time [13:34:22] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.824 second response time [13:35:42] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:35:57] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:36:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.70, 21.23, 20.39 [13:36:45] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:39:07] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:40:01] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:40:14] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:40:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.66, 19.68, 19.99 [13:40:50] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.405 second response time [13:40:52] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:40:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:00] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.229 second response time [13:42:26] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.944 second response time [13:42:52] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:44:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.981 second response time [13:44:53] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.484 second response time [13:44:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:45:07] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.025 second response time [13:45:12] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.980 second response time [13:45:23] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.121 second response time [13:46:14] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.856 second response time [13:48:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:49:39] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:44] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is UNKNOWN: CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected). [13:50:33] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:50:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.58, 20.54, 20.21 [13:51:43] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.738 second response time [13:51:46] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.254 second response time [13:52:32] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.282 second response time [13:54:20] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:41] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:55:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 3.78, 3.20 [13:57:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.55, 2.98, 2.98 [13:58:05] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 3.101 second response time [13:58:05] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:58:12] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:58:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.46, 22.90, 21.49 [13:58:55] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:59:01] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:00:00] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:00:05] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.447 second response time [14:00:05] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.023 second response time [14:00:15] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.429 second response time [14:00:37] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:00:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.05, 22.16, 21.39 [14:00:57] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.810 second response time [14:01:02] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.460 second response time [14:01:03] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.570 second response time [14:01:59] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.599 second response time [14:02:29] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.797 second response time [14:02:36] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.028 second response time [14:02:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:03:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:06:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [14:07:26] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:07:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [14:08:29] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:08:49] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:10] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:09:27] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:10:53] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.908 second response time [14:12:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.22, 3.64, 3.23 [14:12:50] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:13:20] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:13:27] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:13:48] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.992 second response time [14:13:55] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:13:57] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.50, 3.24, 3.13 [14:14:39] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.651 second response time [14:14:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:15:33] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.855 second response time [14:15:40] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.057 second response time [14:17:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:20:12] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.731 second response time [14:20:18] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.570 second response time [14:21:20] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.957 second response time [14:21:34] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.746 second response time [14:21:51] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.717 second response time [14:22:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [14:23:28] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:23:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [14:24:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.96, 19.63, 20.28 [14:25:28] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.307 second response time [14:26:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.84, 3.81, 3.32 [14:26:46] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:26:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:27:47] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:27:48] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:27:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.67, 3.32 [14:28:12] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:28:36] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:29:53] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.514 second response time [14:29:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.85, 3.42 [14:29:56] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.830 second response time [14:30:43] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.627 second response time [14:30:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [14:30:58] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.581 second response time [14:31:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.62, 3.38 [14:32:22] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.101 second response time [14:33:38] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:33:48] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:34:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.23, 20.64, 20.21 [14:34:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:35:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:35:35] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.137 second response time [14:35:44] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.311 second response time [14:37:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.72, 3.49 [14:38:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.61, 19.88, 20.02 [14:39:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 3.22, 3.33 [14:41:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [14:41:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [14:42:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:43:54] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:44:14] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:44:31] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:44:42] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:44:51] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:43] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.333 second response time [14:51:00] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.716 second response time [14:51:05] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:13] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.352 second response time [14:51:28] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.63, 20.62, 20.35 [14:51:46] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:55] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:16] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:17] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.491 second response time [14:52:26] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:28] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.827 second response time [14:52:46] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:11] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:21] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:28] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:29] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.23, 19.88, 20.12 [14:53:35] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 8.622 second response time [14:53:49] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 7.508 second response time [14:53:51] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:10] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.736 second response time [14:55:22] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.912 second response time [14:55:24] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.482 second response time [14:55:48] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.088 second response time [14:56:04] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.584 second response time [14:56:18] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.641 second response time [14:56:33] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.787 second response time [14:56:38] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:56:48] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:56:58] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.379 second response time [14:57:13] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:57:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.60, 21.00, 20.53 [14:57:27] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:57:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:37] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.572 second response time [15:00:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.36, 6.95, 7.80 [15:01:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:01:16] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.64, 20.03, 20.30 [15:01:23] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.888 second response time [15:01:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.39, 6.76, 7.90 [15:01:44] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.652 second response time [15:02:02] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.228 second response time [15:02:31] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.70, 7.04, 7.92 [15:02:50] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.576 second response time [15:02:55] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.310 second response time [15:03:03] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:03:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.79, 7.85, 8.16 [15:04:31] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.13, 8.05, 8.17 [15:04:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.37, 7.71, 7.87 [15:06:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.42, 7.47, 7.75 [15:06:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:08:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.10, 8.10, 7.97 [15:08:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:10:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.26, 7.87, 7.91 [15:12:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.09, 8.92, 8.30 [15:12:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:12:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:12:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.94, 20.53, 20.25 [15:13:30] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.24, 4.49, 3.73 [15:13:46] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:49] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.53, 20.31, 20.22 [15:14:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:15:22] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:15:28] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:15:41] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:15:49] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:54] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:15:54] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:17:24] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.034 second response time [15:17:27] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:17:30] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.57, 3.95, 3.72 [15:17:41] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.121 second response time [15:17:52] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.508 second response time [15:17:55] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.898 second response time [15:17:57] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.670 second response time [15:18:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:21:48] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:07] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:10] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:20] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:21] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.18, 7.32, 7.88 [15:22:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:23:18] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.462 second response time [15:23:34] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.097 second response time [15:23:39] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.217 second response time [15:24:13] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.753 second response time [15:24:18] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.037 second response time [15:24:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.73, 8.01, 8.07 [15:25:57] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.620 second response time [15:26:11] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.394 second response time [15:26:15] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.310 second response time [15:26:16] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.009 second response time [15:26:27] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.324 second response time [15:26:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:30:47] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:50] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:31:07] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:31:58] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:31:59] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.91, 7.62, 8.00 [15:34:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:34:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.75, 8.22, 8.18 [15:34:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.47, 20.28, 20.08 [15:36:08] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.687 second response time [15:37:08] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.263 second response time [15:37:12] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.379 second response time [15:37:24] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.889 second response time [15:38:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:44:46] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:44:53] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:44:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:44:56] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:44:58] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:48:39] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [15:49:25] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:49:43] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:49:43] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:50:57] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.490 second response time [15:51:05] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.922 second response time [15:51:06] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.656 second response time [15:51:12] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.389 second response time [15:51:15] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.929 second response time [15:51:50] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:53:28] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.961 second response time [15:53:51] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.056 second response time [15:53:54] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.118 second response time [15:53:56] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.631 second response time [15:54:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.42, 3.04 [15:54:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:54:45] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.901 second response time [15:56:30] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 3.05, 2.94 [15:57:21] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:57:31] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:57:32] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:57:34] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:41] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:58:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:58:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [16:02:22] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:02:30] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:04:25] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.746 second response time [16:04:36] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.909 second response time [16:06:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:06:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:10:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:10:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:11:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.52, 3.18 [16:13:48] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.348 second response time [16:14:18] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.020 second response time [16:14:23] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.832 second response time [16:14:37] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.403 second response time [16:14:44] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.053 second response time [16:14:44] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:15:05] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:15:06] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:11] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:15:35] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:16:18] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.69, 2.25, 1.20 [16:16:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.65, 19.39, 20.27 [16:18:18] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.88, 1.72, 1.13 [16:18:34] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/eec138bc6367...81381af8fdba [16:18:36] [02miraheze/dns] 07JohnFLewis 0381381af - add csw1.mh.o [16:19:46] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.02, 3.28, 3.22 [16:20:05] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:20:18] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.50, 1.30, 1.04 [16:20:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.58, 6.87, 7.76 [16:20:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.34, 20.74, 20.65 [16:20:42] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:20:53] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:21:10] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:18] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:22:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.41, 19.97, 20.39 [16:22:56] JohnLewis: should we blacklist? [16:23:14] csw[num]wiki [16:23:57] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.790 second response time [16:24:04] If you want to, it is a service name [16:25:23] Cool [16:25:33] Generally we do anything that can't exist as a subdomain [16:25:48] Because we don't want people to do a Reception123 and create a wiki people can't access [16:28:21] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:29:53] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.909 second response time [16:30:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.71, 7.02, 7.27 [16:32:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.93, 6.78, 7.15 [16:32:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:33:15] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.905 second response time [16:33:18] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.950 second response time [16:33:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.67, 7.28, 7.95 [16:33:50] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.337 second response time [16:34:03] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.266 second response time [16:34:08] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.052 second response time [16:34:13] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.284 second response time [16:34:33] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.196 second response time [16:34:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.55, 7.59, 7.38 [16:34:36] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.024 second response time [16:35:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.94, 8.49, 8.31 [16:35:39] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.703 second response time [16:36:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [16:38:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:40:39] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:41] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:40:48] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:40:57] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:41:01] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:42:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:43:37] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:45:39] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.829 second response time [16:45:52] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:22] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:47:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cdbe43e324e3...83060c9728f9 [16:47:48] [02miraheze/mw-config] 07Universal-Omega 0383060c9 - Add csw to CreateWiki blacklist [16:48:47] miraheze/mw-config - Universal-Omega the build passed. [16:49:05] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.013 second response time [16:49:07] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.012 second response time [16:50:03] Ty CosmicAlpha [16:50:06] Saves me a job [16:50:35] No problem RhinosF1! [16:50:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.82, 20.68, 20.03 [16:50:58] Oh did we fix the links from MirahezeBot to notifico? [16:50:59] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:10] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.162 second response time [16:51:14] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.870 second response time [16:51:18] !log [@mwtask111] starting deploy of {'config': True} to scsvg [16:51:24] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 5s [16:52:00] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.136 second response time [16:52:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:52:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:52:59] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.134 second response time [16:53:06] CosmicAlpha: I banned Not-9244 from it [16:53:24] RhinosF1: oh MacFan said it wouldn't work. [16:53:25] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.998 second response time [16:53:36] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.912 second response time [16:53:43] CosmicAlpha: I did specific and wildcard [16:54:28] !log [@test3] starting deploy of {'config': True} to skip [16:54:29] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [16:54:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.41, 20.07, 19.95 [16:55:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:55:41] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.977 second response time [16:55:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:19] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:25] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:57:45] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:59:05] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.342 second response time [16:59:25] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.758 second response time [16:59:45] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.548 second response time [17:00:22] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.969 second response time [17:01:37] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.321 second response time [17:01:39] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.023 second response time [17:03:16] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:03:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.56, 3.23 [17:04:37] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:04:48] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:05:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.56, 3.27 [17:06:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:06:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:07:31] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.549 second response time [17:07:46] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.32, 3.22 [17:08:34] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [17:10:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:11:16] !log [@mw11] starting deploy of {'config': True} to ovlon [17:11:55] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:11:59] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:16] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 59s [17:12:26] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:28] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:41] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:48] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:57] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:13:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:14:42] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:15:00] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:15:08] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:15:18] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:15:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:16:25] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:37] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.788 second response time [17:16:42] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.870 second response time [17:16:47] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.463 second response time [17:17:00] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.804 second response time [17:17:06] !log [@test101] starting deploy of {'config': True} to skip [17:17:06] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [17:17:22] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.827 second response time [17:18:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:18:27] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.010 second response time [17:18:50] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.290 second response time [17:19:10] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.425 second response time [17:19:23] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.053 second response time [17:19:25] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.168 second response time [17:19:41] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:19:47] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:55] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:19:57] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:22] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.575 second response time [17:20:59] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:10] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:19] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:22:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:15] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:22:44] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.124 second response time [17:22:56] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.807 second response time [17:23:05] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.861 second response time [17:23:09] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.060 second response time [17:23:10] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.404 second response time [17:23:15] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.325 second response time [17:24:11] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.401 second response time [17:24:18] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.862 second response time [17:25:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.14, 20.94, 19.83 [17:26:57] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.62, 20.39, 19.76 [17:27:39] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.247 second response time [17:27:55] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:28:01] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:28:29] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:45] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:29:37] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.420 second response time [17:29:51] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.010 second response time [17:30:00] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.599 second response time [17:31:51] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.288 second response time [17:32:11] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.734 second response time [17:32:30] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.560 second response time [17:32:44] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.215 second response time [17:34:00] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:34:15] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:34:25] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.850 second response time [17:34:25] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:35:07] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.975 second response time [17:35:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.48, 20.50, 20.13 [17:36:02] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.205 second response time [17:36:21] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.330 second response time [17:36:29] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.643 second response time [17:36:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:36:40] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 5.173 second response time [17:37:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.04, 19.67, 19.87 [17:38:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:41:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.51, 3.18 [17:42:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [17:43:28] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.27, 21.16, 20.28 [17:43:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.74, 3.30 [17:44:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [17:45:25] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.95, 20.55, 20.16 [17:45:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.89, 3.42 [17:46:08] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:46:16] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:46:23] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:46:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:47:16] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:22] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.59, 19.54, 19.83 [17:48:09] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:49:12] [02ssl] 07Universal-Omega opened pull request 03#468: Remove wiki.konjitownmc.cf - 13https://github.com/miraheze/ssl/pull/468 [17:49:41] [02ssl] 07Universal-Omega synchronize pull request 03#468: Remove wiki.konjitownmc.cf - 13https://github.com/miraheze/ssl/pull/468 [17:49:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.59, 3.28, 3.29 [17:50:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:50:41] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.047 second response time [17:52:41] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.639 second response time [17:53:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.11, 4.08, 3.59 [17:54:13] [02ssl] 07Universal-Omega edited pull request 03#468: Remove wiki.konjitownmc.cf - 13https://github.com/miraheze/ssl/pull/468 [17:54:49] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.91, 1.40, 1.02 [17:55:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.99, 3.62 [17:55:49] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:56:49] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.02, 1.20, 0.99 [17:57:47] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.011 second response time [18:00:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:01:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.92, 3.96, 3.67 [18:04:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:05:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 3.54, 3.56 [18:06:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.52, 20.30, 19.96 [18:06:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [18:07:01] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.064 second response time [18:07:10] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.680 second response time [18:07:19] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.082 second response time [18:08:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:08:39] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.559 second response time [18:08:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.51, 19.58, 19.73 [18:09:14] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:09:15] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:09:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.01, 3.04, 3.34 [18:09:47] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:09:55] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.301 second response time [18:10:22] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:10:45] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:39] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:56] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:12:07] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:12:08] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:12:11] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:03] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:07] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:20] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:36] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:38] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:54] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.985 second response time [18:14:22] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:12] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.464 second response time [18:15:18] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.262 second response time [18:15:36] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.536 second response time [18:15:36] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.326 second response time [18:16:18] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.034 second response time [18:16:23] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.230 second response time [18:16:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.01, 7.07, 7.82 [18:17:35] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:18:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.16, 8.06, 8.08 [18:18:41] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:18:45] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:22] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:31] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.00, 20.09, 19.64 [18:20:50] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:27] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:35] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:42] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:58] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:04] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:35] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.871 second response time [18:22:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.84, 20.09, 19.72 [18:23:29] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.978 second response time [18:24:29] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.526 second response time [18:24:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.43, 7.69, 7.97 [18:24:45] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.207 second response time [18:24:54] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:24:58] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.013 second response time [18:25:35] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.306 second response time [18:25:50] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.505 second response time [18:25:56] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.642 second response time [18:26:02] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.667 second response time [18:26:12] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.757 second response time [18:26:20] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.892 second response time [18:26:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.83, 8.17, 8.11 [18:28:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.49, 7.45, 7.85 [18:29:37] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.673 second response time [18:29:44] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.921 second response time [18:29:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.62, 3.34 [18:30:03] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.019 second response time [18:30:13] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.313 second response time [18:30:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:30:50] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.389 second response time [18:31:17] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.036 second response time [18:31:17] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.840 second response time [18:32:39] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.293 second response time [18:32:55] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.514 second response time [18:32:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:32:59] [02miraheze/dns] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/81381af8fdba...5f3c39749340 [18:33:01] [02miraheze/dns] 07JohnFLewis 03564433e - add mgmt address [18:33:02] [02miraheze/dns] 07JohnFLewis 035f3c397 - move mail from mail2 to mail121 [18:33:14] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.997 second response time [18:36:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [18:36:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [18:37:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.06, 3.27, 3.27 [18:39:18] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:32] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:39:36] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:39:49] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:40:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.17, 7.47, 7.60 [18:41:31] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:03] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/puppet/compare/99e32d7a4776...0662d2ea12d0 [18:42:05] [02miraheze/puppet] 07JohnFLewis 030662d2e - postfix: change ldap server [18:42:13] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:34] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:56] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:43:31] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.444 second response time [18:43:40] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:44:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 7.95, 7.79 [18:44:36] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.669 second response time [18:44:53] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:23] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:32] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:41] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:46:05] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:46:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:17] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:46:26] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:46:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.70, 8.35, 7.97 [18:46:53] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.169 second response time [18:47:24] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 2.501 second response time [18:47:26] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.022 second response time [18:47:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:47:53] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:07] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.823 second response time [18:48:10] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.304 second response time [18:48:13] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.949 second response time [18:48:27] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.116 second response time [18:48:40] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:49:49] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.059 second response time [18:50:46] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.800 second response time [18:51:32] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:52:03] PROBLEM - mw13 APT on mw13 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:52:15] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.830 second response time [18:53:06] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:54:03] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.373 second response time [18:54:12] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.122 second response time [18:54:27] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.983 second response time [18:54:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.36, 3.83, 3.31 [18:54:38] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.624 second response time [18:54:40] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.804 second response time [18:54:56] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.870 second response time [18:56:18] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.06, 2.35, 1.38 [18:56:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.75, 3.64, 3.30 [18:56:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:56:42] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:57:10] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:57:32] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:57:34] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:09] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:58:18] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.23, 1.62, 1.23 [19:00:27] PROBLEM - mw12 APT on mw12 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:00:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.01, 3.26, 3.21 [19:00:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [19:00:57] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.212 second response time [19:01:14] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.021 second response time [19:01:41] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.233 second response time [19:01:43] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.118 second response time [19:02:09] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.908 second response time [19:02:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:03:10] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:03:13] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:03:17] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:03:36] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:03:47] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.85, 20.68, 19.55 [19:05:17] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.586 second response time [19:05:21] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.345 second response time [19:05:23] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.389 second response time [19:05:37] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.651 second response time [19:05:48] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.363 second response time [19:06:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:08:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.18, 20.33, 19.65 [19:10:06] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:10:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.15, 7.08, 7.95 [19:10:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [19:11:28] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:11:33] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:11:49] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:11:57] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:11:59] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:07] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:08] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.054 second response time [19:12:11] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:24] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.27, 7.50, 8.00 [19:12:45] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.13, 20.97, 20.14 [19:14:05] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.320 second response time [19:14:06] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.929 second response time [19:14:12] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.299 second response time [19:14:14] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.938 second response time [19:14:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.71, 7.53, 7.95 [19:15:59] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.855 second response time [19:16:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.60, 19.82, 19.90 [19:20:04] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.904 second response time [19:20:25] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:20:31] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.609 second response time [19:20:32] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.436 second response time [19:20:33] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:20:35] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.415 second response time [19:20:35] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:20:35] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:21:59] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.025 second response time [19:22:20] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:22] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.333 second response time [19:22:31] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.538 second response time [19:22:33] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.244 second response time [19:22:35] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:35] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.811 second response time [19:22:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.46, 7.71, 7.77 [19:22:44] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:23:16] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:24:26] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:24:48] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.641 second response time [19:26:25] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.531 second response time [19:26:29] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.096 second response time [19:26:39] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.738 second response time [19:27:28] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.527 second response time [19:28:19] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:28:32] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:28:44] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:28:55] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:29:04] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.67, 7.52, 7.73 [19:32:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.63, 8.21, 7.95 [19:33:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb [19:36:43] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.849 second response time [19:37:01] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.045 second response time [19:37:05] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.613 second response time [19:37:22] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.052 second response time [19:37:36] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.637 second response time [19:37:41] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:38:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:41:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.59, 3.26 [19:41:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:41:49] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:42:04] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [19:43:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.78, 3.32, 3.21 [19:43:06] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:11] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:22] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:24] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:32] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:33] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:39] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:35] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.80, 20.49, 19.52 [19:44:56] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:03] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:10] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.198 second response time [19:45:17] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:18] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.322 second response time [19:45:26] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.035 second response time [19:45:27] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:32] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.322 second response time [19:49:40] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.990 second response time [19:50:42] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.908 second response time [19:52:00] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:05] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:19] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:22] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:46] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:57] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:59] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:53:55] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.012 second response time [19:54:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.99, 1.42, 0.95 [19:54:05] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.323 second response time [19:54:11] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:11] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:18] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 0.646 second response time [19:54:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.80, 20.32, 20.11 [19:55:00] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.42, 1.37, 0.99 [19:56:14] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.381 second response time [19:56:22] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.718 second response time [19:56:52] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.604 second response time [19:56:59] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.329 second response time [19:57:04] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.274 second response time [20:01:45] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.225 second response time [20:01:48] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.033 second response time [20:02:15] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.905 second response time [20:02:38] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.720 second response time [20:02:53] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.396 second response time [20:02:54] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.583 second response time [20:03:07] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.743 second response time [20:03:20] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.751 second response time [20:04:07] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.414 second response time [20:04:18] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.922 second response time [20:04:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:08:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:12:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.58, 21.95, 20.77 [20:13:13] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:13:19] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:13:32] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:13:34] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:13:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:14:12] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:15] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:23] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:41] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:01] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:07] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.07, 21.83, 20.99 [20:20:25] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.083 second response time [20:20:29] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.687 second response time [20:20:58] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.840 second response time [20:21:05] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.491 second response time [20:21:30] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.228 second response time [20:21:34] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:21:43] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:21:45] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.785 second response time [20:21:53] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.512 second response time [20:21:54] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.83, 20.96, 20.77 [20:22:24] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:26] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:34] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.517 second response time [20:22:41] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.732 second response time [20:24:19] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.897 second response time [20:27:41] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 3.582 second response time [20:27:57] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.74, 19.91, 20.35 [20:27:59] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.307 second response time [20:28:01] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.319 second response time [20:28:32] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 2.428 second response time [20:28:43] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.029 second response time [20:28:46] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:30:46] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0662d2ea12d0...34361d2c2d75 [20:30:47] [02miraheze/puppet] 07JohnFLewis 0334361d2 - postfix: strict ownership is not wanted [20:30:50] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.183 second response time [20:31:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.14, 7.28, 7.94 [20:33:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.37, 7.62, 7.97 [20:33:38] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:33:41] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:41] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 5.983 second response time [20:35:41] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.151 second response time [20:36:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a3a5fb1b9555 [20:36:29] [02miraheze/mw-config] 07Universal-Omega 03a3a5fb1 - Remove CategoryBrowser site notice [20:36:31] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [20:36:32] [02mw-config] 07Universal-Omega opened pull request 03#4375: Remove CategoryBrowser site notice - 13https://github.com/miraheze/mw-config/pull/4375 [20:37:34] miraheze/mw-config - Universal-Omega the build passed. [20:38:12] [02mw-config] 07Universal-Omega closed pull request 03#4375: Remove CategoryBrowser site notice - 13https://github.com/miraheze/mw-config/pull/4375 [20:38:13] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [20:38:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/83060c9728f9...592deee5d844 [20:38:16] [02miraheze/mw-config] 07Universal-Omega 03592deee - Remove CategoryBrowser site notice (#4375) [20:38:18] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [20:39:18] miraheze/mw-config - Universal-Omega the build passed. [20:41:43] !log [@mw11] starting deploy of {'config': True} to ovlon [20:42:01] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:42:37] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 53s [20:42:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:43:27] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:31] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:43:51] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:56] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.311 second response time [20:45:06] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:45:11] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:45:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:45:43] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:46:00] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:47:15] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.282 second response time [20:47:38] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.989 second response time [20:48:00] !log [@test101] starting deploy of {'config': True} to skip [20:48:01] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [20:48:06] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.422 second response time [20:48:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:50:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.53, 19.72, 19.52 [20:51:05] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:09] !log [@mwtask111] starting deploy of {'config': True} to scsvg [20:51:14] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 5s [20:51:23] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.797 second response time [20:51:24] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:29] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.808 second response time [20:51:33] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.090 second response time [20:51:42] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:58] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.018 second response time [20:51:58] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:10] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.014 second response time [20:52:29] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:52:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:02] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:13] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:53:27] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.992 second response time [20:53:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:34] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:42] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.142 second response time [20:53:43] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:57] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 5.105 second response time [20:54:04] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:26] !log [@test3] starting deploy of {'config': True} to skip [20:54:27] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [20:54:36] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.553 second response time [20:54:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.26, 19.87, 19.62 [20:55:19] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.473 second response time [20:55:35] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.688 second response time [20:55:44] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.012 second response time [20:55:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:55:59] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.347 second response time [20:56:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.52, 3.24 [20:56:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:56:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:57:10] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.399 second response time [20:57:57] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:58:09] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:58:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.16, 3.33, 3.20 [20:58:15] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.53, 21.37, 20.32 [20:59:00] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:03:30] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [21:04:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.95, 19.95, 20.04 [21:06:33] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.863 second response time [21:06:35] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.324 second response time [21:07:22] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.339 second response time [21:07:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:08:08] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.943 second response time [21:08:21] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.896 second response time [21:08:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:11:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:11:28] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:11:34] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:00] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:05] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:05] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:06] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:08] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:10] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:31] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:13:30] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.633 second response time [21:13:37] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.236 second response time [21:14:01] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:02] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.471 second response time [21:14:06] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.319 second response time [21:14:07] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.355 second response time [21:14:07] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.205 second response time [21:14:11] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.649 second response time [21:14:16] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.388 second response time [21:14:16] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:34] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.968 second response time [21:15:29] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:15:29] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:35] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:18:31] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.663 second response time [21:19:33] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.387 second response time [21:19:34] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.425 second response time [21:19:41] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.073 second response time [21:20:04] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.340 second response time [21:20:27] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:20:34] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:20:36] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:20:40] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:20:56] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:04] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:49] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:21:53] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:03] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:15] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:24:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.29, 3.10 [21:30:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.85, 3.40, 3.25 [21:31:39] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.003 second response time [21:32:09] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.950 second response time [21:32:17] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.676 second response time [21:32:33] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.850 second response time [21:32:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:32:41] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.571 second response time [21:32:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:33:15] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.235 second response time [21:33:34] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.819 second response time [21:34:58] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.018 second response time [21:35:36] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.940 second response time [21:35:48] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.660 second response time [21:38:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:39:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:39:35] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:40:04] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:40:05] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:40:15] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:41:24] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:42:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:42:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:44:15] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:44] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:46:14] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.740 second response time [21:46:36] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:46:46] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.545 second response time [21:49:37] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:50:09] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.999 second response time [21:50:29] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:50:39] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.324 second response time [21:50:41] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.010 second response time [21:50:42] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.316 second response time [21:51:36] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.026 second response time [21:51:37] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.547 second response time [22:00:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:00:36] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:01:13] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:01:19] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:54] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:02:41] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.897 second response time [22:03:17] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.213 second response time [22:03:25] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.637 second response time [22:05:57] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [22:06:00] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.036 second response time [22:08:16] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:08:17] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:09:05] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:09:20] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:09:26] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:11:15] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:11:29] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:13:11] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.006 second response time [22:13:26] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.353 second response time [22:14:28] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.136 second response time [22:14:39] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.253 second response time [22:15:00] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:13] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.012 second response time [22:15:30] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.480 second response time [22:15:36] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:15:39] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.336 second response time [22:16:15] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:16:19] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:16:21] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:26] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:21:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:21:39] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:21:56] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:02] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:36] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:10] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:25:08] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:25:19] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.438 second response time [22:25:31] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.045 second response time [22:25:44] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:25:58] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.337 second response time [22:26:00] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.323 second response time [22:26:37] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.114 second response time [22:27:07] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.651 second response time [22:27:41] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.959 second response time [22:28:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:29:45] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.033 second response time [22:30:43] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.218 second response time [22:31:12] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.550 second response time [22:31:19] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.926 second response time [22:31:22] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.590 second response time [22:31:52] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:31:59] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:32:13] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:32:17] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:32:26] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:32:28] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:32:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:32:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.82, 20.97, 20.00 [22:32:58] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:14] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.557 second response time [22:36:26] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.306 second response time [22:36:58] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:37:04] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.753 second response time [22:38:13] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.399 second response time [22:38:15] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.754 second response time [22:38:45] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.376 second response time [22:38:47] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.527 second response time [22:39:03] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.717 second response time [22:41:53] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:42:12] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:42:18] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:42:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.10, 20.21, 20.25 [22:42:40] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:43:04] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:43:14] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:43:23] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:44:36] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:44:43] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:44:47] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:45:04] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.010 second response time [22:45:09] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:45:10] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.125 second response time [22:45:15] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:45:19] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:46:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.13, 20.82, 20.59 [22:46:43] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 0.733 second response time [22:50:18] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.053 second response time [22:50:38] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.294 second response time [22:50:48] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.125 second response time [22:50:52] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.602 second response time [22:51:42] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.507 second response time [22:54:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.46, 19.74, 20.23 [22:55:36] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.452 second response time [22:55:38] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.360 second response time [22:55:47] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.683 second response time [22:57:08] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.871 second response time [22:57:35] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.078 second response time [22:58:00] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:58:49] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:20] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:31] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:31] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:00:01] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:01:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [23:05:44] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.111 second response time [23:05:48] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.753 second response time [23:05:51] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.653 second response time [23:06:09] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.684 second response time [23:07:11] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.937 second response time [23:07:19] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.54, 3.26 [23:08:46] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:09:16] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:09:22] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.66, 19.56, 18.66 [23:09:25] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:09:34] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:09:41] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:09:45] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:02] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:10:19] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:10:33] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:11:09] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:11:14] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 672s [23:11:23] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.41, 19.44, 18.73 [23:12:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.99, 3.23, 3.19 [23:12:38] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 756s [23:13:28] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:29] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:37] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:39] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.983 second response time [23:13:40] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:52] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:14:01] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.339 second response time [23:14:12] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:14] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:15] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:16] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:24] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:14:24] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.381 second response time [23:14:26] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:34] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.022 second response time [23:14:56] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.072 second response time [23:15:25] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.681 second response time [23:15:31] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.026 second response time [23:15:39] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.619 second response time [23:15:43] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.469 second response time [23:15:45] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.824 second response time [23:15:53] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 5.332 second response time [23:15:54] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.871 second response time [23:15:55] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.967 second response time [23:16:21] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.810 second response time [23:19:55] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:02] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:20:15] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:20:18] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:48] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.88, 20.55, 19.87 [23:22:14] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:22:15] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.040 second response time [23:22:15] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:38] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.783 second response time [23:22:48] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.156 second response time [23:23:23] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.36, 19.26, 19.49 [23:23:27] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:35] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:46] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:59] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.423 second response time [23:24:09] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.931 second response time [23:24:22] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.465 second response time [23:24:26] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.456 second response time [23:25:23] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.76, 20.41, 19.17 [23:25:24] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:43] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:28:50] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:28:58] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.477 second response time [23:29:00] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.133 second response time [23:29:12] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.221 second response time [23:29:18] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.514 second response time [23:29:27] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:29:45] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.375 second response time [23:30:18] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.510 second response time [23:30:52] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.539 second response time [23:33:09] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:33:38] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2015s [23:33:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.23, 3.06 [23:34:18] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:34:33] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:34:39] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:34:55] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.221 second response time [23:35:01] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:35:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:10] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.142 second response time [23:35:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 2.76, 2.90 [23:36:02] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.574 second response time [23:36:42] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.290 second response time [23:36:48] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.913 second response time [23:37:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.93, 20.07, 19.55 [23:37:05] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.555 second response time [23:37:20] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 2239s [23:37:21] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.574 second response time [23:37:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:38:11] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.065 second response time [23:38:18] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.312 second response time [23:38:20] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.834 second response time [23:38:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:38:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:38:59] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.82, 19.13, 19.26 [23:42:11] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:42:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [23:42:52] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.61, 20.71, 19.91 [23:42:55] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:43:15] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:44:45] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:44:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.90, 20.20, 19.83 [23:46:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb [23:47:22] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.09, 19.80, 20.09 [23:48:04] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:56] [02ssl] 07Universal-Omega opened pull request 03#469: Remove robloxapi.wiki - 13https://github.com/miraheze/ssl/pull/469 [23:49:59] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.168 second response time [23:50:30] [02ssl] 07Universal-Omega synchronize pull request 03#469: Remove robloxapi.wiki - 13https://github.com/miraheze/ssl/pull/469 [23:50:34] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.036 second response time [23:50:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:50:59] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.323 second response time [23:51:11] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.322 second response time [23:51:16] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:20] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:36] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:39] [02ssl] 07Universal-Omega opened pull request 03#470: Remove wiki.hausos.co - 13https://github.com/miraheze/ssl/pull/470 [23:51:40] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.225 second response time [23:51:59] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:52:11] [02ssl] 07Universal-Omega synchronize pull request 03#470: Remove wiki.hausos.co - 13https://github.com/miraheze/ssl/pull/470 [23:52:31] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.781 second response time [23:52:32] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:53:15] [02ssl] 07Universal-Omega opened pull request 03#471: Remove wiki.3805.co.uk - 13https://github.com/miraheze/ssl/pull/471 [23:53:19] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.781 second response time [23:53:20] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.635 second response time [23:53:38] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.030 second response time [23:53:46] [02ssl] 07Universal-Omega synchronize pull request 03#471: Remove wiki.3805.co.uk - 13https://github.com/miraheze/ssl/pull/471 [23:54:01] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.278 second response time [23:54:31] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.591 second response time [23:54:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [23:58:50] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds