[00:00:15] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 8.56, 6.54, 5.98 [00:01:13] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 5.578 second response time [00:01:31] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.235 second response time [00:01:59] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20514 bytes in 4.117 second response time [00:02:05] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.334 second response time [00:02:22] dmehus: it looks to be getting more stable [00:02:31] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:35] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.40, 6.32, 6.56 [00:02:47] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.316 second response time [00:02:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.69, 26.43, 21.24 [00:03:25] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:03:30] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 7.169 second response time [00:03:36] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:03:38] RhinosF1, I'm not seeing that [00:04:14] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.43, 6.49, 6.80 [00:04:15] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.10, 5.73, 5.82 [00:04:25] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:32] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.815 second response time [00:04:37] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:42] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:45] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:50] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw11 mw12 [00:04:58] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:03] dmehus: it did for like a minute [00:05:14] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw10 mw11 mw12 [00:05:17] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.321 second response time [00:05:20] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.321 second response time [00:05:29] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.007 second response time [00:05:47] RhinosF1, okay, fair. I'm getting persistent ` [00:05:47] Error contacting the Parsoid/RESTBase server: (curl error: 28) Timeout was reached` errors on `metawiki` (and other wikis) [00:06:00] dmehus: don't bother with parsoid [00:06:12] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:15] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.30, 6.03, 5.92 [00:06:31] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.390 second response time [00:06:31] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:40] RhinosF1, nothing to do with my configuration; it's the way it's set on the wiki/wiki farm. It's DiscussionTools [00:06:42] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw8 mw9 [00:06:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.04, 21.19, 20.45 [00:07:09] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:07:11] dmehus: use wikitext reply [00:07:19] like the normal editor [00:07:30] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.022 second response time [00:07:38] parsoid will perform worse until it recovers [00:07:39] RhinosF1, oh, you mean temporarily? [00:07:40] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.238 second response time [00:07:51] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:54] dmehus: yes [00:07:57] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 3.905 second response time [00:08:03] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:08:15] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.59, 5.70, 5.81 [00:08:15] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:08:19] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20514 bytes in 9.871 second response time [00:08:26] RhinosF1, oh ok, fair enough. To be clear, I was using the 2010 wikitext source editor, not the wiki editor source editor [00:08:28] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:08:33] but I could switching that yeah [00:08:42] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [00:08:50] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [00:08:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.10, 22.67, 21.07 [00:09:01] Is it just me, or is Miraheze a bit slower than usual? [00:09:05] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.922 second response time [00:09:14] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [00:09:20] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:09:38] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:09:39] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:09:41] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:09:45] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 1.945 second response time [00:09:46] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:09:47] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:10:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.29, 22.20, 21.09 [00:11:56] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 0.649 second response time [00:12:00] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.872 second response time [00:12:06] Can't even change my preferences [00:12:09] I'm timing out [00:12:16] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.039 second response time [00:12:36] DarkMatterMan4500, no it's not just you [00:12:36] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.505 second response time [00:12:39] dmehus: it's bad but i don't get why workers have suddenly stopped working [00:12:42] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:48] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 3.200 second response time [00:12:51] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:12:54] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.70, 23.30, 21.63 [00:13:01] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:14] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw11 [00:13:15] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.899 second response time [00:13:16] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.752 second response time [00:13:17] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.178 second response time [00:13:18] What do you mean by "workers"? You mean the automated bots that are tasked to keep the site up and running smoothly? [00:13:21] RhinosF1, my guess would be very high demand...just wan out of php workers or whatever [00:13:27] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:29] dmehus: we're running at about 20-50% of our normal capacity [00:13:41] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.598 second response time [00:13:47] they aren't spawing though according to grafana [00:14:10] Maybe create a Phabricator task for SRE Infrastructure to look into? [00:14:15] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.62, 5.76, 5.72 [00:14:43] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.398 second response time [00:14:49] dmehus: it's MW team [00:14:56] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 9.175 second response time [00:14:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.44, 23.23, 21.80 [00:15:01] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.965 second response time [00:15:32] RhinosF1, true [00:15:42] workers = processes which handle rendering the page and doing other backend stuff [00:15:48] (reply to DMM) [00:15:51] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.137 second response time [00:15:53] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.202 second response time [00:15:55] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 7.036 second response time [00:16:02] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.775 second response time [00:16:12] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.641 second response time [00:16:33] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:16:40] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:17:14] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [00:17:15] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:17:20] Ah. [00:17:31] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:17:36] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:17:38] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.250 second response time [00:17:58] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:02] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:18:58] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.192 second response time [00:18:59] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.31, 23.60, 22.18 [00:19:03] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:19:15] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.029 second response time [00:19:20] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:19:26] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:19:32] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.703 second response time [00:20:08] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:11] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:12] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:13] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:15] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.01, 5.23, 5.54 [00:20:15] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:15] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:40] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 2.157 second response time [00:20:43] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw8 mw10 mw13 [00:20:48] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 3 backends are down. mw8 mw10 mw13 [00:20:49] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.006 second response time [00:21:14] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw11 [00:21:23] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.605 second response time [00:21:34] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 7.379 second response time [00:22:12] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.967 second response time [00:22:45] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [00:23:11] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.863 second response time [00:23:14] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [00:23:26] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 4.560 second response time [00:23:37] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.873 second response time [00:23:40] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.063 second response time [00:24:01] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.193 second response time [00:24:03] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.665 second response time [00:24:18] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 9.122 second response time [00:24:19] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20514 bytes in 8.842 second response time [00:24:42] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [00:24:57] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.004 second response time [00:24:58] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:01] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:25:05] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:26:23] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.616 second response time [00:26:30] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.108 second response time [00:26:44] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:26:47] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:05] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.264 second response time [00:27:07] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.462 second response time [00:27:22] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:36] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:50] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:15] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.55, 5.43, 5.43 [00:28:25] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:28:28] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.009 second response time [00:28:42] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.320 second response time [00:28:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.29, 21.35, 22.26 [00:29:04] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:29:06] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.927 second response time [00:29:06] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 2.723 second response time [00:29:09] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.013 second response time [00:29:10] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.591 second response time [00:29:23] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.830 second response time [00:30:15] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.28, 5.04, 5.30 [00:30:32] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.877 second response time [00:30:51] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:31:08] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:31:10] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:31:14] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.004 second response time [00:31:15] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:31:18] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:31:34] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:31:42] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:31:52] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.193 second response time [00:33:05] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.930 second response time [00:33:08] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.170 second response time [00:33:10] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:11] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.012 second response time [00:33:32] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:36] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.169 second response time [00:33:44] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:45] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.496 second response time [00:34:47] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.005 second response time [00:34:52] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.785 second response time [00:35:00] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:35:01] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:35:05] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.345 second response time [00:35:09] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.007 second response time [00:35:14] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.352 second response time [00:35:28] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.310 second response time [00:35:38] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.013 second response time [00:35:55] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 0.256 second response time [00:36:16] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:49] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:36:52] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:37:04] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20514 bytes in 5.877 second response time [00:37:05] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 3.812 second response time [00:37:18] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 2.790 second response time [00:37:25] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 5.253 second response time [00:37:28] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:30] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:37:37] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:38:12] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.281 second response time [00:38:15] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.37, 4.44, 4.98 [00:38:47] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.967 second response time [00:38:52] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.492 second response time [00:38:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.45, 17.33, 19.79 [00:39:31] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:39:31] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:39:48] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.012 second response time [00:39:51] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:40:16] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:18] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:41:23] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:41:26] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.394 second response time [00:41:38] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.366 second response time [00:41:42] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.289 second response time [00:41:48] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:41:48] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:53] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.718 second response time [00:41:53] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:42:01] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:42:13] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:42:23] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 9.702 second response time [00:42:44] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw11 [00:42:45] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw11 [00:43:19] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.432 second response time [00:43:26] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.432 second response time [00:43:36] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.631 second response time [00:43:43] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.148 second response time [00:43:50] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.712 second response time [00:43:50] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 4.707 second response time [00:43:51] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.010 second response time [00:43:52] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:43:53] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.692 second response time [00:43:55] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.606 second response time [00:43:56] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.244 second response time [00:43:56] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.007 second response time [00:44:01] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.316 second response time [00:44:18] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.697 second response time [00:44:19] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:23] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:41] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [00:45:29] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:43] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:46] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:45:46] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.189 second response time [00:45:51] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.344 second response time [00:45:55] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.011 second response time [00:46:03] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:46:07] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:46:14] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.339 second response time [00:46:15] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:46:22] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.867 second response time [00:47:35] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:37] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.186 second response time [00:47:46] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.324 second response time [00:48:01] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.412 second response time [00:48:04] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.338 second response time [00:48:05] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:48:15] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.743 second response time [00:48:27] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:48:43] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [00:49:34] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20514 bytes in 0.187 second response time [00:51:16] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:29] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw12 [00:51:47] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:54] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:06] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:07] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:22] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:23] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:27] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:33] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:37] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:38] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:40] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.244 second response time [00:53:23] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [00:53:23] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 9.362 second response time [00:53:50] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 5.075 second response time [00:54:10] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:54:12] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:54:19] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.071 second response time [00:54:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.96, 5.31, 5.91 [00:54:37] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:54:49] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:54:59] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:55:49] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:55:51] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.214 second response time [00:55:59] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 7.280 second response time [00:56:00] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:08] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:56:09] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:56:09] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:56:19] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.575 second response time [00:56:32] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:56:34] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.348 second response time [00:56:35] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.008 second response time [00:56:39] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.319 second response time [00:56:42] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw8 [00:57:45] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:59:42] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.416 second response time [00:59:59] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.304 second response time [01:00:11] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20514 bytes in 6.245 second response time [01:00:13] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 3.969 second response time [01:00:14] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.012 second response time [01:00:15] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.496 second response time [01:00:18] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.070 second response time [01:00:38] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.886 second response time [01:00:41] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.168 second response time [01:00:42] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.098 second response time [01:00:42] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [01:00:47] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.109 second response time [01:00:49] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:00:59] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:01:01] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:01:01] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:02:08] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.43, 5.96, 5.91 [01:02:50] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.346 second response time [01:02:56] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.332 second response time [01:02:57] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.061 second response time [01:02:57] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.011 second response time [01:04:24] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 1.602 second response time [01:04:28] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.814 second response time [01:05:05] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.328 second response time [01:05:11] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.072 second response time [01:05:22] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.007 second response time [01:06:08] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 2.908 second response time [01:06:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.05, 5.27, 5.69 [01:09:22] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:09:23] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:09:25] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:09:26] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:09:51] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:10] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:10:19] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:10:30] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:10:34] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:11:23] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:11:30] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.426 second response time [01:11:32] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.534 second response time [01:11:37] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.239 second response time [01:11:56] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:12:12] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:25] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.009 second response time [01:12:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.33, 3.87, 4.92 [01:12:32] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:12:36] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:12:42] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:12:43] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:58] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.267 second response time [01:13:14] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:13:23] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.320 second response time [01:14:17] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 7.727 second response time [01:14:41] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.757 second response time [01:14:41] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:43] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.769 second response time [01:14:48] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.128 second response time [01:15:03] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.742 second response time [01:15:12] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.322 second response time [01:15:55] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:15:56] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.476 second response time [01:15:58] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:16:07] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:16:30] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.124 second response time [01:16:51] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 2.731 second response time [01:17:36] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.609 second response time [01:17:52] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.011 second response time [01:17:56] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.036 second response time [01:18:07] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.340 second response time [01:18:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.45, 6.03, 5.50 [01:18:39] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.264 second response time [01:18:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JyaaQ [01:18:41] [02miraheze/mw-config] 07Universal-Omega 03ac309eb - Update Defines.php [01:18:42] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [01:18:45] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:19:37] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:19:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jyaa5 [01:19:47] [02miraheze/mw-config] 07Universal-Omega 0331f6f2c - Update LocalSettings.php [01:20:20] [02mw-config] 07Universal-Omega opened pull request 03#4307: Configure `$wgUrlShortenerAllowedDomains` for betaheze - 13https://git.io/JyaaF [01:20:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.16, 5.73, 5.46 [01:20:51] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:20:52] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.204 second response time [01:20:57] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.319 second response time [01:21:07] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.308 second response time [01:21:14] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:21:20] miraheze/mw-config - Universal-Omega the build passed. [01:21:21] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:29] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw8 mw10 mw13 [01:21:34] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:07] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:17] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:17] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:22] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw12 [01:22:22] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:37] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 4.656 second response time [01:23:06] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:23:11] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 0.132 second response time [01:23:17] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw9 mw13 [01:23:17] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.323 second response time [01:23:17] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:23:21] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:23:32] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.307 second response time [01:24:22] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.173 second response time [01:24:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.87, 4.46, 4.99 [01:24:40] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw12 [01:25:33] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.308 second response time [01:26:10] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.080 second response time [01:26:17] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [01:26:22] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.129 second response time [01:26:31] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.961 second response time [01:26:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JyaDO [01:26:40] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [01:26:40] [02miraheze/mw-config] 07Universal-Omega 03c671d2e - fix wgLanguageConverterCacheType for CLI [01:26:42] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [01:26:43] [02mw-config] 07Universal-Omega opened pull request 03#4308: fix wgLanguageConverterCacheType for CLI - 13https://git.io/JyaDn [01:27:01] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.730 second response time [01:27:05] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [01:27:26] [02mw-config] 07Universal-Omega closed pull request 03#4308: fix wgLanguageConverterCacheType for CLI - 13https://git.io/JyaDn [01:27:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyaSN [01:27:29] [02miraheze/mw-config] 07Universal-Omega 033599253 - fix wgLanguageConverterCacheType for CLI (#4308) [01:27:31] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [01:27:32] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [01:27:39] [02mw-config] 07Universal-Omega closed pull request 03#4307: Configure `$wgUrlShortenerAllowedDomains` for betaheze - 13https://git.io/JyaaF [01:27:40] miraheze/mw-config - Universal-Omega the build passed. [01:27:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jya9K [01:27:42] [02miraheze/mw-config] 07Universal-Omega 03589b2eb - Configure `$wgUrlShortenerAllowedDomains` for betaheze (#4307) [01:27:43] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [01:27:45] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:28:27] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.363 second response time [01:28:28] miraheze/mw-config - Universal-Omega the build passed. [01:28:46] !log [universalomega@test3] starting deploy of {'pull': 'config', 'config': True} to skip [01:28:46] miraheze/mw-config - Universal-Omega the build passed. [01:28:47] !log [universalomega@test3] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 0s [01:29:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.12, 6.64, 6.07 [01:29:08] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.716 second response time [01:29:14] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [01:29:20] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.018 second response time [01:29:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:29:37] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.318 second response time [01:29:37] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.009 second response time [01:29:42] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [01:30:57] !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 75s [01:31:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:31:21] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:31:31] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:31:34] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:31:53] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:31:58] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:33:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.57, 6.58, 6.22 [01:33:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:05] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:08] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:36:00] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 1.072 second response time [01:36:07] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.010 second response time [01:36:08] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 8.682 second response time [01:36:08] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.522 second response time [01:37:27] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:37:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:11] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:38:11] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:39:25] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.108 second response time [01:39:54] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 2.279 second response time [01:40:07] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.302 second response time [01:40:09] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.014 second response time [01:40:16] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:40:32] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:40:34] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:41:21] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:41:27] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:41:28] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:41:55] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw13 [01:42:11] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:42:15] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.299 second response time [01:43:46] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:43:47] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:43:53] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [01:43:57] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:44:00] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:44:00] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:11] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:44:17] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:44:25] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:28] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:44:34] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:00] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:00] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:42] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.319 second response time [01:45:45] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.136 second response time [01:45:45] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.693 second response time [01:46:11] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.920 second response time [01:46:23] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.828 second response time [01:46:27] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.367 second response time [01:46:28] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 9.072 second response time [01:46:37] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.582 second response time [01:46:52] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw9 mw11 [01:47:06] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.557 second response time [01:47:45] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.010 second response time [01:47:52] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.366 second response time [01:47:56] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.016 second response time [01:47:58] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.545 second response time [01:48:11] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 6.996 second response time [01:48:28] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.047 second response time [01:48:29] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.468 second response time [01:48:30] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.318 second response time [01:48:48] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [01:49:29] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:49:39] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:49:54] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:59] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:50:01] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:50:07] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:50:08] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:14] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:50:14] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:50:50] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:50:53] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.017 second response time [01:51:24] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.897 second response time [01:51:24] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.343 second response time [01:51:38] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 1.588 second response time [01:51:48] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.482 second response time [01:51:52] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20514 bytes in 0.317 second response time [01:51:56] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.175 second response time [01:52:01] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.068 second response time [01:52:03] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 1.854 second response time [01:52:05] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.048 second response time [01:52:14] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.577 second response time [01:52:15] !log restart php-fpm on mw* [01:52:16] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.657 second response time [01:52:46] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.345 second response time [01:52:52] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.187 second response time [01:53:08] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 9.082 second response time [01:53:12] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:53:16] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:53:25] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.350 second response time [01:53:43] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:54:32] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:56] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:55:08] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.584 second response time [01:55:12] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.314 second response time [01:55:40] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.289 second response time [01:55:49] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:55:53] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:03] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:12] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:13] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:21] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:27] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:28] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.004 second response time [01:56:29] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:43] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:45] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:46] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.004 second response time [01:56:47] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.235 second response time [01:57:08] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:57:12] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.004 second response time [01:57:19] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:57:32] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.236 second response time [01:57:32] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw11 [01:57:46] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:58:07] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.632 second response time [01:58:09] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:58:16] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.338 second response time [01:58:18] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.867 second response time [01:58:30] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.950 second response time [01:58:33] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 3.606 second response time [01:58:52] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.342 second response time [01:58:54] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.484 second response time [01:59:23] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw11 [01:59:26] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw11 [01:59:29] PROBLEM - mw11 php-fpm on mw11 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.3' [01:59:35] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw11 mw12 [01:59:47] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 6.234 second response time [02:00:10] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 3.958 second response time [02:00:41] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.00, 6.63, 6.17 [02:02:14] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.820 second response time [02:02:17] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:02:26] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.007 second response time [02:02:26] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.252 second response time [02:02:39] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.29, 6.13, 6.05 [02:02:40] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:03:05] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.022 second response time [02:03:35] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.844 second response time [02:03:44] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:04:08] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:04:12] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.006 second response time [02:04:13] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.880 second response time [02:04:20] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.015 second response time [02:04:26] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:05:21] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:05:37] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:06:29] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:06:31] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 7.662 second response time [02:06:43] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:47] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.041 second response time [02:06:55] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 0.161 second response time [02:07:02] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:07:27] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:07:32] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.327 second response time [02:07:49] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.012 second response time [02:07:58] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:08:08] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.317 second response time [02:08:38] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:08:44] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.47, 6.72, 6.22 [02:08:51] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:09:19] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.61, 5.13, 4.91 [02:09:27] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.511 second response time [02:10:05] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.006 second response time [02:10:25] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:10:31] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.008 second response time [02:10:33] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.331 second response time [02:10:44] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.193 second response time [02:10:46] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [02:11:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 2.93, 2.65 [02:11:14] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [02:11:22] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [02:11:26] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.342 second response time [02:11:29] RECOVERY - mw11 php-fpm on mw11 is OK: PROCS OK: 13 processes with command name 'php-fpm7.3' [02:11:34] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [02:12:05] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.331 second response time [02:12:22] !log mw11: sudo service php7.3-fpm restart [02:12:28] !log mw12: sudo service php7.3-fpm restart [02:12:30] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.361 second response time [02:12:39] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.444 second response time [02:12:43] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.73, 6.95, 6.26 [02:12:46] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.071 second response time [02:12:49] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 5.047 second response time [02:12:58] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:13:09] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 5.869 second response time [02:13:10] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.789 second response time [02:13:12] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.18, 6.24, 5.37 [02:13:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:13:29] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:13:38] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.636 second response time [02:13:51] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:13:54] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.755 second response time [02:14:19] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:14:34] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:14:42] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.97, 6.61, 6.22 [02:15:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:32] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 3.437 second response time [02:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:15:49] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20514 bytes in 1.161 second response time [02:16:17] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.288 second response time [02:16:30] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.846 second response time [02:16:31] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:16:34] !log: mw8: sudo service php7.4-fpm restart [02:16:44] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.25, 6.34, 6.31 [02:16:46] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:16:46] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:16:48] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:17:00] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:17:01] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:17:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.02, 2.81 [02:17:06] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:17:07] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 5.802 second response time [02:17:11] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:17:14] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:17:26] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:17:30] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:17:40] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:17:47] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:18:00] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:18:07] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:18:57] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:18:57] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:19:54] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:20:38] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.96, 6.91, 6.47 [02:20:56] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.244 second response time [02:20:57] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.511 second response time [02:21:02] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.318 second response time [02:21:14] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw9 [02:21:20] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 0.419 second response time [02:21:25] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.038 second response time [02:21:31] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.281 second response time [02:21:53] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.319 second response time [02:22:06] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.340 second response time [02:22:37] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.56, 6.40, 6.34 [02:23:06] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.009 second response time [02:23:07] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.010 second response time [02:23:12] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.239 second response time [02:23:14] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [02:23:15] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 7.325 second response time [02:23:21] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.012 second response time [02:23:55] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:23:56] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.861 second response time [02:23:58] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.127 second response time [02:24:42] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw8 mw10 [02:24:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.91, 5.31, 5.59 [02:25:03] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 1.136 second response time [02:25:09] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.682 second response time [02:25:12] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.917 second response time [02:25:22] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw8 [02:25:33] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw8 mw12 [02:25:46] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:26:00] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.130 second response time [02:26:09] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.387 second response time [02:26:29] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:26:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.27, 6.35, 4.89 [02:26:42] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:26:42] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [02:27:04] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.850 second response time [02:27:22] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:27:26] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.477 second response time [02:27:29] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.312 second response time [02:27:34] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.006 second response time [02:27:51] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:56] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:28:09] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.482 second response time [02:28:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.59, 6.50, 5.12 [02:28:34] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.012 second response time [02:28:40] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.044 second response time [02:28:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.14, 5.25, 5.46 [02:29:06] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.05, 5.59, 4.91 [02:29:20] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.354 second response time [02:29:22] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [02:29:32] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [02:30:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.91, 6.84, 6.48 [02:30:46] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.17, 4.95, 5.33 [02:31:02] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.71, 5.40, 4.91 [02:32:23] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.79, 5.91, 6.16 [02:32:40] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:32:41] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:32:42] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.70, 4.44, 5.09 [02:32:51] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.422 second response time [02:33:14] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:33:16] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.23, 6.73, 6.33 [02:33:28] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:33:28] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.571 second response time [02:33:55] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.016 second response time [02:33:58] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:34:00] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.014 second response time [02:34:02] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 0.235 second response time [02:34:24] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:34:32] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:34:41] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:34:43] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.558 second response time [02:34:53] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:35:13] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.600 second response time [02:35:57] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.965 second response time [02:36:30] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 8.393 second response time [02:36:36] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.590 second response time [02:37:03] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:37:42] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.130 second response time [02:37:46] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:09] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:38:10] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:38:10] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:38:16] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.313 second response time [02:38:39] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.298 second response time [02:38:55] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.024 second response time [02:39:00] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.502 second response time [02:39:33] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:40:23] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:40:47] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:40:59] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:41:08] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:41:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.05, 6.77, 6.23 [02:41:11] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.27, 6.66, 6.52 [02:41:51] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.211 second response time [02:42:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jyw04 [02:42:09] [02miraheze/mw-config] 07Universal-Omega 031235775 - Disable the commentlist api [02:42:10] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [02:42:12] [02mw-config] 07Universal-Omega opened pull request 03#4309: Disable the commentlist api - 13https://git.io/Jyw0g [02:42:16] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.007 second response time [02:42:19] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.401 second response time [02:42:21] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.326 second response time [02:42:22] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.017 second response time [02:42:41] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.475 second response time [02:42:43] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.38, 5.05, 5.00 [02:42:56] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.385 second response time [02:43:06] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.341 second response time [02:43:08] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.07, 6.43, 6.17 [02:43:13] miraheze/mw-config - Universal-Omega the build passed. [02:43:16] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.125 second response time [02:43:16] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:43:16] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:43:20] been reading the slowlogs? [02:43:32] yes. [02:43:37] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.126 second response time [02:43:49] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:44:00] yeah, looks to be an optimization problem there [02:44:06] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:44:08] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:44:12] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:44:42] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:44:42] not sure why the commentlist api seems to be fetching SocialProfile Avatar files, but it seems to be really slow [02:44:57] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:15] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 1.413 second response time [02:45:23] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:45:55] @Void: Yeah I reproduced php-fpm usage increase on test3 also by proxying to https://....miraheze.org/w/api.php?action=commentlist&format=json&pageID=35313&order=1&pagerPage=1&showForm=1&_=1640744980816 (a rise of about 20% usage just loading that from test3 it seemed), so it seems to have some issues there also. Any objections to merging that PR to disable it and see if we have things getting any better? [02:47:05] in terms of an emergency solution to load problems, not really. In terms of a permanent solution, well it isn't one, so we do still need to figure that out [02:47:20] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:47:22] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.323 second response time [02:47:25] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.745 second response time [02:47:26] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:47:50] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.320 second response time [02:48:02] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.032 second response time [02:48:02] @Void: fair enough, I will do it for now and see if anything gets better. Another slow API also seems to be discussiontools though that one I'm not sure on other effects of disabling. [02:48:06] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.034 second response time [02:48:36] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:48:44] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:48:51] I might see about linking slowlogs to pages to try and see if there is any sort of pattern [02:48:58] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:49:22] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw8 mw13 [02:49:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.63, 6.75, 6.23 [02:49:23] [02mw-config] 07Universal-Omega closed pull request 03#4309: Disable the commentlist api - 13https://git.io/Jyw0g [02:49:25] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [02:49:26] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [02:49:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyw6g [02:49:29] [02miraheze/mw-config] 07Universal-Omega 035a18cd9 - Disable the commentlist api (#4309) [02:49:40] !log [universalomega@test3] starting deploy of {'pull': 'config', 'config': True} to skip [02:49:43] !log [universalomega@test3] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 2s [02:50:19] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [02:50:23] miraheze/mw-config - Universal-Omega the build passed. [02:50:29] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw9 [02:50:37] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.344 second response time [02:50:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:50:42] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw9 [02:50:44] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.363 second response time [02:50:58] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.865 second response time [02:51:15] !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 55s [02:51:22] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [02:51:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:29] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [02:52:31] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.007 second response time [02:52:42] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [02:52:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:52:55] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 3.408 second response time [02:52:59] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.510 second response time [02:53:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.60, 6.39, 6.19 [02:53:24] Voidwalker: disabling it seems to have helped quite a bit unless it's a coincidence that php-fpm usage is low on almost all servers now. (only 2 with high usage stilll) [02:53:30] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.587 second response time [02:53:33] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.938 second response time [02:53:47] I somewhat doubt a coincidence [02:53:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:54:06] maybe it was a coincidence since they went up again. though still a little better [02:54:23] Meta loads much faster than before [02:54:57] We'll see then. They aren't all constant 100% usage anymore. [02:55:28] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.30, 6.65, 6.10 [02:56:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.95, 6.96, 6.27 [02:56:35] hmm, it's also not the only source of slow php-fpm requests either [02:57:26] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.16, 5.75, 5.84 [02:57:39] Voidwalker: maybe disable discussiontools api also? It seems to be the other slow api. [02:58:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.53, 6.04, 6.01 [02:59:22] hard to say tbh, as that is from slow curl requests, and that's an entirely different problem [03:00:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:00:54] Voidwalker: Could it hurt to try? Or do you think it would not help at all? [03:01:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.82, 3.30 [03:01:48] I'm not sure, I think we should try and look at requests that not only trigger slowlog but also wind up not completing at all [03:03:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.63, 3.29 [03:03:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:05:49] stuff like this: [NOTICE] [pool www] child 19307 exited with code 0 after 1035.170480 seconds from start [03:06:04] is probably more worth investigating than just turning off features [03:09:24] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.72, 3.41 [03:09:54] Voidwalker: for the first time in 5 hours php-fpm usage is in acceptable ranges for all mw servers. [03:11:05] well was it went back up, but not 100% anymore. [03:11:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.40, 3.34 [03:12:18] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.35, 7.95, 6.71 [03:12:53] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.96, 6.54, 6.02 [03:13:04] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.32, 6.64, 5.91 [03:14:17] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.12, 7.17, 6.55 [03:14:52] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.40, 5.91, 5.85 [03:15:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.56, 6.21, 5.85 [03:16:16] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.19, 6.34, 6.31 [03:17:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.96, 3.45, 3.37 [03:19:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.06, 3.38, 3.37 [03:19:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb [03:21:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:23:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 3.93, 3.59 [03:27:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.71, 3.59 [03:34:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.03, 6.55, 5.94 [03:35:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.67, 3.66, 3.58 [03:38:11] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.31, 6.11, 5.93 [03:44:41] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Fri 14 Jan 2022 03:39:45 GMT +0000). [03:44:48] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Fri 14 Jan 2022 03:39:45 GMT +0000). [03:44:49] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Fri 14 Jan 2022 03:39:45 GMT +0000). [03:45:16] PROBLEM - wiki.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Fri 14 Jan 2022 03:39:45 GMT +0000). [03:46:15] PROBLEM - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 15 day(s) (Fri 14 Jan 2022 03:43:20 GMT +0000). [03:46:33] PROBLEM - www.opendatascot.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'opendatascot.org' expires in 15 day(s) (Fri 14 Jan 2022 03:42:07 GMT +0000). [03:47:21] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 15 day(s) (Fri 14 Jan 2022 03:43:20 GMT +0000). [03:50:40] PROBLEM - opendatascot.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'opendatascot.org' expires in 15 day(s) (Fri 14 Jan 2022 03:42:07 GMT +0000). [03:51:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.70, 6.86, 6.31 [03:51:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyrYj [03:51:15] [02miraheze/ssl] 07MirahezeSSLBot 037afaa0f - Bot: Update SSL cert for www.thesimswiki.com [03:51:18] PROBLEM - wikipariksha.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikipariksha.com' expires in 15 day(s) (Fri 14 Jan 2022 03:47:14 GMT +0000). [03:51:37] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zymonic.com' expires in 15 day(s) (Fri 14 Jan 2022 03:49:14 GMT +0000). [03:52:46] PROBLEM - wikien.wildterra2.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikien.wildterra2.com' expires in 15 day(s) (Fri 14 Jan 2022 03:48:05 GMT +0000). [03:53:01] PROBLEM - www.journeytheword.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.journeytheword.wiki' expires in 15 day(s) (Fri 14 Jan 2022 03:44:27 GMT +0000). [03:53:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.04, 6.78, 6.35 [03:53:18] PROBLEM - journeytheword.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.journeytheword.wiki' expires in 15 day(s) (Fri 14 Jan 2022 03:44:27 GMT +0000). [03:53:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyrOU [03:53:25] [02miraheze/ssl] 07MirahezeSSLBot 03d76579a - Bot: Update SSL cert for marinebiodiversitymatrix.org [03:53:25] PROBLEM - with.cpt-ra.bid - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'with.cpt-ra.bid' expires in 15 day(s) (Fri 14 Jan 2022 03:45:14 GMT +0000). [03:54:20] PROBLEM - wikiru.wildterra2.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiru.wildterra2.com' expires in 15 day(s) (Fri 14 Jan 2022 03:46:13 GMT +0000). [03:54:39] PROBLEM - wiki.xysspon.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.xysspon.com' expires in 15 day(s) (Fri 14 Jan 2022 03:50:51 GMT +0000). [03:57:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.82, 4.00 [03:57:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyrOq [03:57:27] [02miraheze/ssl] 07MirahezeSSLBot 0357ed2a5 - Bot: Update SSL cert for www.journeytheword.wiki [03:58:16] PROBLEM - wiki.whentheycry.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.whentheycry.org' expires in 15 day(s) (Fri 14 Jan 2022 03:51:48 GMT +0000). [03:58:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyrOY [03:58:26] [02miraheze/ssl] 07MirahezeSSLBot 03c3a632b - Bot: Update SSL cert for wiki.xysspon.com [03:58:36] PROBLEM - wiki.valkyrienskies.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.valkyrienskies.org' expires in 15 day(s) (Fri 14 Jan 2022 03:55:49 GMT +0000). [03:59:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 4.03, 4.06 [03:59:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyrOs [03:59:34] [02miraheze/ssl] 07MirahezeSSLBot 033c9f09d - Bot: Update SSL cert for wiki.zymonic.com [03:59:46] PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.warframestat.us' expires in 15 day(s) (Fri 14 Jan 2022 03:52:46 GMT +0000). [04:02:45] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.villagecollaborative.net' expires in 15 day(s) (Fri 14 Jan 2022 03:53:59 GMT +0000). [04:03:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyrOC [04:03:25] [02miraheze/ssl] 07MirahezeSSLBot 0309ec41d - Bot: Update SSL cert for wikien.wildterra2.com [04:04:43] PROBLEM - wiki.triplescripts.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.triplescripts.org' expires in 15 day(s) (Fri 14 Jan 2022 03:56:31 GMT +0000). [04:08:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyrOE [04:08:43] [02miraheze/ssl] 07MirahezeSSLBot 0385adb6a - Bot: Update SSL cert for wiki.warframestat.us [04:15:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyrO6 [04:15:37] [02miraheze/ssl] 07MirahezeSSLBot 03bcf439b - Bot: Update SSL cert for wikiru.wildterra2.com [04:19:21] PROBLEM - wiki.starship.digital - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.starship.digital' expires in 15 day(s) (Fri 14 Jan 2022 04:17:12 GMT +0000). [04:20:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyr01 [04:20:25] [02miraheze/ssl] 07MirahezeSSLBot 03a2a7d47 - Bot: Update SSL cert for with.cpt-ra.bid [04:22:12] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyrVk [04:22:13] [02miraheze/ssl] 07MirahezeSSLBot 03fb9488e - Bot: Update SSL cert for wiki.villagecollaborative.net [04:23:25] PROBLEM - wiki.s23.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.s23.org' expires in 15 day(s) (Fri 14 Jan 2022 04:19:04 GMT +0000). [04:23:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.87, 6.77, 6.10 [04:23:47] PROBLEM - wiki.ripto.gq - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ripto.gq' expires in 15 day(s) (Fri 14 Jan 2022 04:19:41 GMT +0000). [04:24:44] PROBLEM - wiki.seredrau.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.seredrau.xyz' expires in 15 day(s) (Fri 14 Jan 2022 04:18:13 GMT +0000). [04:25:05] PROBLEM - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.rebirthofthenight.com' expires in 15 day(s) (Fri 14 Jan 2022 04:21:08 GMT +0000). [04:25:20] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Tue 29 Mar 2022 02:51:09 GMT +0000. [04:25:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyryi [04:25:46] [02miraheze/ssl] 07MirahezeSSLBot 0359f91d1 - Bot: Update SSL cert for opendatascot.org [04:26:04] RECOVERY - wikien.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikien.wildterra2.com' will expire on Tue 29 Mar 2022 03:03:18 GMT +0000. [04:26:10] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Tue 29 Mar 2022 02:59:27 GMT +0000. [04:26:23] RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.warframestat.us' will expire on Tue 29 Mar 2022 03:08:37 GMT +0000. [04:26:26] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Tue 29 Mar 2022 02:51:09 GMT +0000. [04:26:32] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Tue 29 Mar 2022 02:51:09 GMT +0000. [04:26:59] RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Tue 29 Mar 2022 02:53:19 GMT +0000. [04:27:11] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.nowchess.org' expires in 15 day(s) (Fri 14 Jan 2022 04:22:56 GMT +0000). [04:27:30] RECOVERY - journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Tue 29 Mar 2022 02:57:20 GMT +0000. [04:27:33] RECOVERY - www.journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Tue 29 Mar 2022 02:57:20 GMT +0000. [04:27:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.57, 6.70, 6.24 [04:28:27] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Tue 29 Mar 2022 02:53:19 GMT +0000. [04:28:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyrhl [04:28:31] [02miraheze/ssl] 07MirahezeSSLBot 03db45954 - Bot: Update SSL cert for wiki.seredrau.xyz [04:28:38] RECOVERY - wiki.xysspon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.xysspon.com' will expire on Tue 29 Mar 2022 02:58:20 GMT +0000. [04:28:39] RECOVERY - wikiru.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiru.wildterra2.com' will expire on Tue 29 Mar 2022 03:15:30 GMT +0000. [04:29:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.95, 3.74, 3.98 [04:29:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyovn [04:29:11] [02miraheze/ssl] 07MirahezeSSLBot 030239b4c - Bot: Update SSL cert for wiki.nowchess.org [04:30:01] PROBLEM - wiki.nevillepedia.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.nevillepedia.eu' expires in 15 day(s) (Fri 14 Jan 2022 04:23:56 GMT +0000). [04:30:43] PROBLEM - wiki.raghuveer.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.raghuveer.net' expires in 15 day(s) (Fri 14 Jan 2022 04:21:50 GMT +0000). [04:31:14] PROBLEM - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mxlinuxusers.de' expires in 15 day(s) (Fri 14 Jan 2022 04:24:59 GMT +0000). [04:32:21] PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.minkyu.kim' expires in 15 day(s) (Fri 14 Jan 2022 04:26:43 GMT +0000). [04:32:21] PROBLEM - wiki.mobilityengineer.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mobilityengineer.com' expires in 15 day(s) (Fri 14 Jan 2022 04:26:02 GMT +0000). [04:33:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.61, 3.85 [04:33:49] PROBLEM - wiki.minecraftathome.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.minecraftathome.com' expires in 15 day(s) (Fri 14 Jan 2022 04:27:31 GMT +0000). [04:35:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.50, 3.79 [04:37:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyoIr [04:37:10] [02miraheze/ssl] 07MirahezeSSLBot 039f743fd - Bot: Update SSL cert for wiki.starship.digital [04:39:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.47, 3.66 [04:41:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.60, 3.68 [04:43:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyomP [04:43:12] [02miraheze/ssl] 07MirahezeSSLBot 03ae77600 - Bot: Update SSL cert for wiki.ripto.gq [04:45:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 3.95, 3.77 [04:47:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.82, 3.76 [04:48:05] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyo3p [04:48:07] [02miraheze/ssl] 07MirahezeSSLBot 036caa4ba - Bot: Update SSL cert for wiki.mobilityengineer.com [04:48:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:51:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.51, 3.61 [04:52:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:53:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.50, 3.34, 3.56 [04:54:33] RECOVERY - www.opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Tue 29 Mar 2022 03:25:39 GMT +0000. [04:55:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyocA [04:55:25] [02miraheze/ssl] 07MirahezeSSLBot 032eb5fd0 - Bot: Update SSL cert for wiki.minecraftathome.com [04:56:08] RECOVERY - with.cpt-ra.bid - LetsEncrypt on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Tue 29 Mar 2022 03:20:18 GMT +0000. [04:56:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyoCx [04:56:52] [02miraheze/ssl] 07MirahezeSSLBot 030b4f193 - Bot: Update SSL cert for wiki.whentheycry.org [04:57:33] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Tue 29 Mar 2022 03:22:06 GMT +0000. [04:58:14] RECOVERY - wiki.ripto.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ripto.gq' will expire on Tue 29 Mar 2022 03:43:04 GMT +0000. [04:58:36] RECOVERY - wiki.seredrau.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seredrau.xyz' will expire on Tue 29 Mar 2022 03:28:24 GMT +0000. [04:58:40] RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Tue 29 Mar 2022 03:25:39 GMT +0000. [04:59:09] RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Tue 29 Mar 2022 03:48:00 GMT +0000. [04:59:21] RECOVERY - wiki.starship.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.starship.digital' will expire on Tue 29 Mar 2022 03:37:03 GMT +0000. [04:59:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyol9 [04:59:36] [02miraheze/ssl] 07MirahezeSSLBot 0377a954b - Bot: Update SSL cert for wiki.rebirthofthenight.com [05:02:07] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Tue 29 Mar 2022 03:29:04 GMT +0000. [05:03:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.52, 2.96, 3.30 [05:03:33] PROBLEM - spiral.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'spiral.wiki' expires in 15 day(s) (Fri 14 Jan 2022 04:59:51 GMT +0000). [05:04:39] PROBLEM - vise.dayid.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vise.dayid.org' expires in 15 day(s) (Fri 14 Jan 2022 05:01:48 GMT +0000). [05:05:45] PROBLEM - wiki.anglish.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.anglish.info' expires in 15 day(s) (Fri 14 Jan 2022 05:00:51 GMT +0000). [05:06:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyo0Q [05:06:51] [02miraheze/ssl] 07MirahezeSSLBot 03bec9d4f - Bot: Update SSL cert for wiki.raghuveer.net [05:06:57] PROBLEM - kunwok.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kunwok.org' expires in 15 day(s) (Fri 14 Jan 2022 05:03:30 GMT +0000). [05:07:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:09:54] PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thelonsdalebattalion.co.uk' expires in 15 day(s) (Fri 14 Jan 2022 05:04:27 GMT +0000). [05:10:46] PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vedopedia.witches-empire.com' expires in 15 day(s) (Fri 14 Jan 2022 05:02:37 GMT +0000). [05:11:29] PROBLEM - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lefrenchmelee.fr' expires in 15 day(s) (Fri 14 Jan 2022 05:09:01 GMT +0000). [05:12:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyogC [05:12:20] [02miraheze/ssl] 07MirahezeSSLBot 033087711 - Bot: Update SSL cert for wiki.s23.org [05:12:41] PROBLEM - wiki.climatechange.ai - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.climatechange.ai' expires in 15 day(s) (Fri 14 Jan 2022 05:05:51 GMT +0000). [05:14:02] PROBLEM - savagepedia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'savagepedia.wiki' expires in 15 day(s) (Fri 14 Jan 2022 05:11:33 GMT +0000). [05:14:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:16:34] PROBLEM - spcodex.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'spcodex.wiki' expires in 15 day(s) (Fri 14 Jan 2022 05:10:40 GMT +0000). [05:16:56] PROBLEM - wiki.mcpirevival.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mcpirevival.tk' expires in 15 day(s) (Fri 14 Jan 2022 05:14:08 GMT +0000). [05:17:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.87, 4.76, 3.91 [05:18:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyowC [05:18:11] [02miraheze/ssl] 07MirahezeSSLBot 0347c6194 - Bot: Update SSL cert for wiki.mxlinuxusers.de [05:19:28] PROBLEM - wiki.kourouklides.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kourouklides.com' expires in 15 day(s) (Fri 14 Jan 2022 05:12:55 GMT +0000). [05:20:15] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.03, 6.37, 6.02 [05:24:02] PROBLEM - runzeppelin.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'runzeppelin.ru' expires in 15 day(s) (Fri 14 Jan 2022 05:15:23 GMT +0000). [05:24:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyo6Z [05:24:24] [02miraheze/ssl] 07MirahezeSSLBot 030156df0 - Bot: Update SSL cert for thelonsdalebattalion.co.uk [05:24:44] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.88, 6.75, 5.98 [05:26:12] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.57, 6.35, 6.19 [05:26:44] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.55, 6.72, 6.07 [05:26:52] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonciclopedia.org' expires in 15 day(s) (Fri 14 Jan 2022 05:19:09 GMT +0000). [05:26:57] PROBLEM - wiki.mastodon.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mastodon.kr' expires in 15 day(s) (Fri 14 Jan 2022 05:18:13 GMT +0000). [05:27:01] RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Tue 29 Mar 2022 04:18:03 GMT +0000. [05:27:26] PROBLEM - icclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'icclopedia.org' expires in 15 day(s) (Fri 14 Jan 2022 05:20:22 GMT +0000). [05:27:45] PROBLEM - miraheze.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.ga' expires in 15 day(s) (Fri 14 Jan 2022 05:21:35 GMT +0000). [05:28:00] RECOVERY - wiki.minecraftathome.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minecraftathome.com' will expire on Tue 29 Mar 2022 03:55:17 GMT +0000. [05:28:26] PROBLEM - bebaskanpengetahuan.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bebaskanpengetahuan.id' expires in 15 day(s) (Fri 14 Jan 2022 05:25:05 GMT +0000). [05:28:33] RECOVERY - wiki.whentheycry.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.whentheycry.org' will expire on Tue 29 Mar 2022 03:56:46 GMT +0000. [05:29:36] PROBLEM - history.estill.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'history.estill.org' expires in 15 day(s) (Fri 14 Jan 2022 05:23:22 GMT +0000). [05:29:38] PROBLEM - vmcodex.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vmcodex.net' expires in 15 day(s) (Fri 14 Jan 2022 05:24:18 GMT +0000). [05:30:07] PROBLEM - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dc-multiverse.dcwikis.com' expires in 15 day(s) (Fri 14 Jan 2022 05:22:09 GMT +0000). [05:32:03] RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Tue 29 Mar 2022 04:12:14 GMT +0000. [05:32:05] RECOVERY - wiki.raghuveer.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.raghuveer.net' will expire on Tue 29 Mar 2022 04:06:44 GMT +0000. [05:32:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyoMn [05:32:09] [02miraheze/ssl] 07MirahezeSSLBot 032ee8b94 - Bot: Update SSL cert for wiki.mcpirevival.tk [05:33:05] RECOVERY - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rebirthofthenight.com' will expire on Tue 29 Mar 2022 03:59:29 GMT +0000. [05:34:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyoDy [05:34:24] [02miraheze/ssl] 07MirahezeSSLBot 03ad98387 - Bot: Update SSL cert for wiki.lefrenchmelee.fr [05:34:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:34:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyoym [05:35:00] [02miraheze/ssl] 07MirahezeSSLBot 03d04a7ef - Bot: Update SSL cert for icclopedia.org [05:42:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyKLH [05:42:31] [02miraheze/ssl] 07MirahezeSSLBot 03c850e0d - Bot: Update SSL cert for vmcodex.net [05:45:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:49:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyKoO [05:49:56] [02miraheze/ssl] 07MirahezeSSLBot 038875bab - Bot: Update SSL cert for wiki.anglish.info [05:56:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy6U8 [05:56:21] [02miraheze/ssl] 07MirahezeSSLBot 0383f029e - Bot: Update SSL cert for history.estill.org [05:57:20] RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Tue 29 Mar 2022 04:42:23 GMT +0000. [05:57:28] RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Tue 29 Mar 2022 04:24:17 GMT +0000. [05:57:52] RECOVERY - wiki.mcpirevival.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcpirevival.tk' will expire on Tue 29 Mar 2022 04:32:02 GMT +0000. [05:59:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy6nJ [05:59:30] [02miraheze/ssl] 07MirahezeSSLBot 03a984f2c - Bot: Update SSL cert for kunwok.org [06:00:22] RECOVERY - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lefrenchmelee.fr' will expire on Tue 29 Mar 2022 04:34:17 GMT +0000. [06:01:37] RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Tue 29 Mar 2022 04:34:53 GMT +0000. [06:03:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.69, 3.59, 3.94 [06:03:28] RECOVERY - mw13 APT on mw13 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:03:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy6K0 [06:04:01] [02miraheze/ssl] 07MirahezeSSLBot 03d9a84f6 - Bot: Update SSL cert for wiki.climatechange.ai [06:04:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy6ip [06:04:33] [02miraheze/ssl] 07MirahezeSSLBot 035ecf0dd - Bot: Update SSL cert for spiral.wiki [06:05:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.76, 3.96 [06:06:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.85, 6.78, 4.47 [06:06:20] RECOVERY - mw10 APT on mw10 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:07:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.54, 3.85 [06:08:13] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.73, 7.43, 4.99 [06:10:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyifZ [06:10:04] [02miraheze/ssl] 07MirahezeSSLBot 03ac967fc - Bot: Update SSL cert for wiki.kourouklides.com [06:10:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [06:12:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:13:02] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:13:07] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:11] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:14] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:15:00] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.334 second response time [06:15:06] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.011 second response time [06:15:11] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.333 second response time [06:15:12] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.026 second response time [06:16:00] PROBLEM - db12 Current Load on db12 is CRITICAL: CRITICAL - load average: 4.58, 9.60, 6.60 [06:16:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.55, 7.96, 6.40 [06:17:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyiuE [06:17:47] [02miraheze/ssl] 07MirahezeSSLBot 033c3f722 - Bot: Update SSL cert for vise.dayid.org [06:17:56] PROBLEM - db12 Current Load on db12 is WARNING: WARNING - load average: 3.91, 7.39, 6.12 [06:19:53] RECOVERY - db12 Current Load on db12 is OK: OK - load average: 1.43, 5.35, 5.52 [06:23:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyiuS [06:23:22] [02miraheze/ssl] 07MirahezeSSLBot 0331d5cac - Bot: Update SSL cert for vedopedia.witches-empire.com [06:24:13] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.23, 6.56, 6.39 [06:25:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.13, 2.76, 3.32 [06:26:40] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Tue 29 Mar 2022 05:17:40 GMT +0000. [06:27:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyiu5 [06:27:36] [02miraheze/ssl] 07MirahezeSSLBot 032d5d16e - Bot: Update SSL cert for bebaskanpengetahuan.id [06:27:42] RECOVERY - wiki.climatechange.ai - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.climatechange.ai' will expire on Tue 29 Mar 2022 05:03:53 GMT +0000. [06:27:50] RECOVERY - wiki.anglish.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.anglish.info' will expire on Tue 29 Mar 2022 04:49:49 GMT +0000. [06:28:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyiuF [06:28:02] [02miraheze/ssl] 07MirahezeSSLBot 03e504f3a - Bot: Update SSL cert for spcodex.wiki [06:28:29] RECOVERY - wiki.kourouklides.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kourouklides.com' will expire on Tue 29 Mar 2022 05:09:56 GMT +0000. [06:28:57] RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Tue 29 Mar 2022 04:59:23 GMT +0000. [06:31:37] RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Tue 29 Mar 2022 04:56:14 GMT +0000. [06:31:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyiux [06:31:52] [02miraheze/ssl] 07MirahezeSSLBot 03d2c855a - Bot: Update SSL cert for savagepedia.wiki [06:32:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyiup [06:32:24] [02miraheze/ssl] 07MirahezeSSLBot 031c82b8f - Bot: Update SSL cert for wiki.mastodon.kr [06:32:33] RECOVERY - spiral.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiral.wiki' will expire on Tue 29 Mar 2022 05:04:27 GMT +0000. [06:37:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyiuh [06:37:23] [02miraheze/ssl] 07MirahezeSSLBot 03308a314 - Bot: Update SSL cert for wiki.triplescripts.org [06:40:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:41:14] PROBLEM - wiki.cyberfurs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cyberfurs.org' expires in 15 day(s) (Fri 14 Jan 2022 06:38:38 GMT +0000). [06:42:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyizJ [06:42:23] [02miraheze/ssl] 07MirahezeSSLBot 0301df9bb - Bot: Update SSL cert for miraheze.ga [06:46:46] RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:47:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:50:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyizq [06:50:17] [02miraheze/ssl] 07MirahezeSSLBot 03f16fde9 - Bot: Update SSL cert for runzeppelin.ru [06:50:43] RECOVERY - wiki.triplescripts.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.triplescripts.org' will expire on Tue 29 Mar 2022 05:37:16 GMT +0000. [06:54:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyizn [06:54:51] [02miraheze/ssl] 07MirahezeSSLBot 031af9a5f - Bot: Update SSL cert for nonciclopedia.org [06:55:33] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Tue 29 Mar 2022 05:23:14 GMT +0000. [06:56:12] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=velorenwiki --username-prefix veloren --report 1 --no-updates /home/reception/Veloren_Wiki-20211217161710.xml (START) [06:56:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:56:45] RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Tue 29 Mar 2022 05:42:16 GMT +0000. [06:57:01] RECOVERY - savagepedia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'savagepedia.wiki' will expire on Tue 29 Mar 2022 05:31:45 GMT +0000. [06:57:26] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Tue 29 Mar 2022 05:27:29 GMT +0000. [06:57:28] RECOVERY - mw11 APT on mw11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:58:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyiz4 [06:58:31] [02miraheze/ssl] 07MirahezeSSLBot 0342ba02c - Bot: Update SSL cert for wiki.nevillepedia.eu [06:59:34] RECOVERY - spcodex.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spcodex.wiki' will expire on Tue 29 Mar 2022 05:27:55 GMT +0000. [07:00:51] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=velorenwiki --username-prefix veloren --report 1 --no-updates /home/reception/Veloren_Wiki-20211217161710.xml (END - exit=0) [07:00:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:01:09] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=velorenwiki (START) [07:01:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:01:33] RECOVERY - mw12 APT on mw12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [07:06:01] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=jamesemirzianwaldementersoftwarewiki --username-prefix wikia:jamesemirzianwaldementersoftwareonwikia --report 1 --no-updates /home/reception/jamesemirzianwaldementersoftwareonwikia_pages_full.xml (START) [07:06:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyiz5 [07:06:23] [02miraheze/ssl] 07MirahezeSSLBot 032675dbd - Bot: Update SSL cert for wikipariksha.com [07:07:29] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=velorenwiki (END - exit=0) [07:07:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyizA [07:08:45] [02miraheze/ssl] 07MirahezeSSLBot 03164c79a - Bot: Update SSL cert for dc-multiverse.dcwikis.com [07:12:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:16:05] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=sunrinwiki --username-prefix sunrin --report 1 --no-updates /home/reception/sunrinwiki.xml (START) [07:16:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:16:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyig3 [07:16:21] [02miraheze/ssl] 07MirahezeSSLBot 03c57a56d - Bot: Update SSL cert for wiki.valkyrienskies.org [07:16:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:19:13] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=jamesemirzianwaldementersoftwarewiki --username-prefix wikia:jamesemirzianwaldementersoftwareonwikia --report 1 --no-updates /home/reception/jamesemirzianwaldementersoftwareonwikia_pages_full.xml (END - exit=0) [07:19:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:20:24] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=jamesemirzianwaldementersoftwarewiki (START) [07:20:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:20:56] PROBLEM - db13 Current Load on db13 is WARNING: WARNING - load average: 6.96, 6.33, 5.18 [07:21:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyigo [07:21:16] [02miraheze/ssl] 07MirahezeSSLBot 03d6bf911 - Bot: Update SSL cert for wiki.cyberfurs.org [07:21:18] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=jamesemirzianwaldementersoftwarewiki (END - exit=256) [07:21:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:22:56] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 4.78, 5.76, 5.11 [07:25:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyigx [07:25:21] [02miraheze/ssl] 07MirahezeSSLBot 032550128 - Bot: Update SSL cert for wiki.minkyu.kim [07:26:36] RECOVERY - wiki.valkyrienskies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.valkyrienskies.org' will expire on Tue 29 Mar 2022 06:16:13 GMT +0000. [07:27:05] RECOVERY - wikipariksha.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipariksha.com' will expire on Tue 29 Mar 2022 06:06:16 GMT +0000. [07:27:07] RECOVERY - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dc-multiverse.dcwikis.com' will expire on Tue 29 Mar 2022 06:08:38 GMT +0000. [07:28:33] RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Tue 29 Mar 2022 05:50:10 GMT +0000. [07:30:01] RECOVERY - wiki.nevillepedia.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nevillepedia.eu' will expire on Tue 29 Mar 2022 05:58:24 GMT +0000. [07:30:13] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 10.02, 8.19, 6.80 [07:32:11] RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Tue 29 Mar 2022 05:54:44 GMT +0000. [07:32:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.22, 7.96, 6.91 [07:36:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:40:13] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.23, 6.67, 6.71 [07:43:34] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JyiaP [07:43:35] [02miraheze/dns] 07Reception123 0325f079c - add wikicryptos.org zone [07:43:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:44:46] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/Jyia9 [07:44:48] [02miraheze/dns] 07Reception123 038aa3720 - add kiseki.wiki zone [07:46:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.15, 6.94, 6.81 [07:48:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:52:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:54:13] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.64, 6.36, 6.65 [07:56:15] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Tue 29 Mar 2022 06:21:09 GMT +0000. [08:00:21] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Tue 29 Mar 2022 06:25:14 GMT +0000. [08:07:15] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=sunrinwiki --username-prefix sunrin --report 1 --no-updates /home/reception/sunrinwiki.xml (END - exit=0) [08:07:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:07:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:08:09] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ysmwikiwiki (START) [08:08:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:09:51] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ysmwikiwiki (END - exit=256) [08:10:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:12:13] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.36, 6.68, 6.29 [08:14:13] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.32, 6.49, 6.27 [08:14:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:18:04] !log MariaDB [ysmwikiwiki]> ALTER TABLE searchindex ADD FULLTEXT si_title (si_title); ALTER TABLE searchindex ADD FULLTEXT si_text (si_text); [08:18:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:29:48] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.65, 6.86, 6.53 [08:31:43] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.54, 6.70, 6.51 [08:38:27] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.38, 6.83, 6.60 [08:42:17] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.98, 6.38, 6.47 [08:52:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.56, 6.83, 6.52 [08:53:40] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://git.io/JyiKZ [08:53:41] [02miraheze/ssl] 07Reception123 038e63839 - add wikicryptos.org cert [08:53:43] [02miraheze/ssl] 07Reception123 03c1ecfca - add kiseki.wiki cert [08:56:13] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.96, 6.52, 6.47 [09:04:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:08:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.76, 7.01, 6.61 [09:09:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:12:13] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.12, 7.39, 6.84 [09:14:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.24, 7.40, 6.92 [09:16:13] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.70, 8.18, 7.26 [09:20:38] @Void: fpm processes don't last forever [09:21:11] They will quit at around 15m / 900s [09:24:11] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.09, 7.79, 7.49 [09:24:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:29:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:33:59] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.41, 7.24, 7.25 [09:34:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:35:57] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.96, 7.13, 7.21 [09:39:52] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 4.53, 6.12, 6.80 [09:45:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:46:48] [02puppet] 07RhinosF1 opened pull request 03#2177: mediawiki: add new servers - 13https://git.io/JyiSs [09:48:20] [02puppet] 07RhinosF1 synchronize pull request 03#2177: mediawiki: add new servers - 13https://git.io/JyiSs [09:50:49] [02puppet] 07RhinosF1 synchronize pull request 03#2177: mediawiki: add new servers - 13https://git.io/JyiSs [09:51:59] [02puppet] 07RhinosF1 edited pull request 03#2177: mediawiki: add new servers - 13https://git.io/JyiSs [09:53:38] [02puppet] 07RhinosF1 synchronize pull request 03#2177: mediawiki: add new servers - 13https://git.io/JyiSs [09:53:56] Reception123: ^ [09:58:19] [02puppet] 07RhinosF1 synchronize pull request 03#2177: mediawiki: add new servers - 13https://git.io/JyiSs [09:59:33] [02puppet] 07RhinosF1 edited pull request 03#2177: mediawiki: add new servers - 13https://git.io/JyiSs [10:00:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:01:54] [02puppet] 07RhinosF1 opened pull request 03#2178: runner: support multi DC - 13https://git.io/JyibY [10:03:04] [02puppet] 07RhinosF1 synchronize pull request 03#2178: runner: support multi DC - 13https://git.io/JyibY [10:03:30] [02puppet] 07RhinosF1 synchronize pull request 03#2178: runner: support multi DC - 13https://git.io/JyibY [10:06:34] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/Jyixj [10:06:35] [02miraheze/mw-config] 07RhinosF1 035429dda - SCSVG: setup [10:06:37] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [10:06:38] [02mw-config] 07RhinosF1 opened pull request 03#4310: SCSVG: setup - 13https://git.io/JyipJ [10:07:09] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JyipB [10:07:10] [02miraheze/mw-config] 07github-actions 036089697 - CI: lint code to MediaWiki standards [10:07:12] [02mw-config] 07github-actions[bot] synchronize pull request 03#4310: SCSVG: setup - 13https://git.io/JyipJ [10:07:42] miraheze/mw-config - RhinosF1 the build passed. [10:09:38] [02mw-config] 07RhinosF1 edited pull request 03#4310: SCSVG: setup - 13https://git.io/JyipJ [10:24:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:49:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:54:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:59:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:15:39] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48956 MB (10% inode=98%); [11:20:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:30:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:41:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:51:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:54:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:23:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.43, 3.65, 3.01 [12:25:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.01, 3.14, 2.89 [12:25:39] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49658 MB (11% inode=98%); [12:29:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:41:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:49:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.32, 2.95 [12:51:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.38, 3.38, 3.03 [12:57:53] [02puppet] 07RhinosF1 synchronize pull request 03#2177: mediawiki: add new servers - 13https://git.io/JyiSs [13:01:54] [02puppet] 07RhinosF1 opened pull request 03#2179: site: add memcache + jobchron in new DC - 13https://git.io/JyXIS [13:03:11] [02puppet] 07RhinosF1 synchronize pull request 03#2179: site: add memcache + jobchron in new DC - 13https://git.io/JyXIS [13:04:25] [02puppet] 07RhinosF1 synchronize pull request 03#2179: site: add memcache + jobchron in new DC - 13https://git.io/JyXIS [13:04:48] [02puppet] 07RhinosF1 synchronize pull request 03#2179: site: add memcache + jobchron in new DC - 13https://git.io/JyXIS [13:13:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.98, 7.06, 6.32 [13:15:00] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.24, 6.70, 6.28 [13:22:33] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.04, 6.87, 6.16 [13:22:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [13:22:57] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:23:05] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:23:15] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:51] * RhinosF1 here [13:24:15] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:24:32] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:24:56] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.527 second response time [13:25:01] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.399 second response time [13:25:12] !log restart php-fpm on mw12, locked up after DT api request [13:25:14] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.278 second response time [13:25:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:26:10] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.337 second response time [13:26:29] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.344 second response time [13:26:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:32:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.88, 7.08, 6.66 [13:32:33] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.75, 6.36, 6.32 [13:36:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.07, 6.27, 6.45 [13:44:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.05, 7.19, 6.82 [13:46:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.05, 6.60, 6.65 [13:46:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:49:43] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JyXWb [13:49:44] [02miraheze/mw-config] 07RhinosF1 0375855bb - Parsoid: timeout after 15 seconds [13:49:46] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [13:51:27] [02mw-config] 07RhinosF1 opened pull request 03#4311: Parsoid: timeout after 15 seconds - 13https://git.io/JyXlz [13:51:48] paladox, CosmicAlpha: ^ [13:52:27] miraheze/mw-config - RhinosF1 the build passed. [13:53:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:54:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.39, 6.77, 6.65 [13:56:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.74, 6.76, 6.67 [13:58:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.17, 6.48, 6.26 [14:00:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.13, 6.86, 6.41 [14:02:26] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.96, 6.00, 6.14 [14:03:43] i'm going to deploy the above [14:09:46] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.50, 7.30, 6.25 [14:10:09] [02mw-config] 07RhinosF1 closed pull request 03#4311: Parsoid: timeout after 15 seconds - 13https://git.io/JyXlz [14:10:10] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [14:10:12] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 [14:10:13] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyXEf [14:10:15] [02miraheze/mw-config] 07RhinosF1 03d1c32a8 - Parsoid: timeout after 15 seconds (#4311) [14:11:16] !log [@mw11] starting deploy of {'config': True} to ovlon [14:11:19] miraheze/mw-config - RhinosF1 the build passed. [14:11:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:11:43] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.02, 6.82, 6.21 [14:11:43] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.10, 6.95, 6.77 [14:11:45] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 28s [14:12:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.35, 3.06 [14:12:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:12:43] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.27, 7.06, 6.63 [14:13:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.20, 6.69, 6.24 [14:14:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.56, 3.17 [14:14:10] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:14:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:14:39] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.88, 6.75, 6.58 [14:15:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.42, 7.00, 6.55 [14:15:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.64, 3.25 [14:16:08] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.503 second response time [14:16:13] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.598 second response time [14:17:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.35, 6.75, 6.51 [14:17:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.36, 3.16, 3.12 [14:21:04] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [14:22:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:23:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.86, 6.59, 6.78 [14:23:56] !log [@test3] starting deploy of {'config': True} to skip [14:23:57] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [14:24:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:25:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:26:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:26:53] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:27:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.82, 6.62, 6.50 [14:29:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.29, 6.54, 6.48 [14:29:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.28, 3.12 [14:33:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.67, 3.79, 3.35 [14:37:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.61, 3.70, 3.44 [14:39:47] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.08, 6.74, 5.83 [14:41:39] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48948 MB (10% inode=98%); [14:41:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.74, 7.03, 6.04 [14:42:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [14:43:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.95, 3.63 [14:43:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [14:43:41] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.45, 6.21, 5.87 [14:44:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.98, 6.80, 6.46 [14:44:53] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.91, 6.53, 6.30 [14:45:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.75, 3.58 [14:46:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.66, 6.11, 6.25 [14:46:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:46:49] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.95, 6.36, 6.27 [14:49:15] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:50:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.81, 7.07, 6.55 [14:51:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.57, 3.13, 3.35 [14:54:20] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.93, 6.70, 6.53 [14:55:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.98, 3.38, 3.42 [15:05:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.74, 3.59 [15:07:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.59, 3.38, 3.48 [15:09:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.67, 2.83, 3.27 [15:14:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.49, 6.83, 6.24 [15:16:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.84, 6.54, 6.21 [15:17:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.47, 3.39 [15:19:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.48, 3.12, 3.27 [15:22:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.72, 6.77, 6.35 [15:22:46] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.76, 7.25, 6.52 [15:24:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [15:24:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.82, 7.31, 6.63 [15:27:22] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:27:24] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:27:27] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:28:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.26, 7.40, 6.76 [15:29:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [15:29:24] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.130 second response time [15:29:29] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 7.478 second response time [15:29:31] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.700 second response time [15:30:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:30:32] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.32, 6.36, 6.45 [15:31:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:34:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.58, 7.54, 7.04 [15:44:46] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.65, 1.76, 1.05 [15:46:45] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.54, 1.33, 0.98 [15:47:13] [02puppet] 07paladox closed pull request 03#2178: runner: support multi DC - 13https://git.io/JyibY [15:47:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jy1f1 [15:47:16] [02miraheze/puppet] 07RhinosF1 03d28c1a5 - runner: support multi DC (#2178) [15:49:04] ty paladox [15:49:12] yw [15:50:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.37, 7.32, 7.11 [15:50:54] paladox: testing mw9, should have auto pulled at :49 [15:51:15] ok [15:51:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [15:51:49] mw8 even [15:52:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.77, 7.13, 7.07 [15:52:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [15:53:05] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:55:02] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.221 second response time [15:55:13] RhinosF1: could you do me a small favor (when you have time)? [15:55:31] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.476 second response time [15:55:32] i don't think it's worth a whole phab task for this [15:56:08] PROBLEM - nl.xliving.tk - LetsEncrypt on sslhost is CRITICAL: connect to address nl.xliving.tk and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [15:56:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.10, 6.35, 5.88 [15:57:23] what? [15:57:40] PROBLEM - nl.xliving.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nl.xliving.tk could not be found [15:58:23] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.26, 7.10, 6.53 [15:58:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.25, 7.48, 7.20 [15:58:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.32, 6.31, 5.92 [15:58:34] lgtm paladox [15:58:42] ok [15:58:43] @Lakelimbo [16:00:15] could you set $egChameleonLayoutFile= __DIR__ . '/skins/chameleon/layouts/navhead.xml'; on koopacabanawiki? [16:00:19] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.63, 7.18, 6.62 [16:00:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.77, 6.89, 7.03 [16:00:43] it's one of the available layouts of chameleon and I personally despise this default layout, navhead is better [16:01:39] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49610 MB (11% inode=98%); [16:02:15] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.29, 7.58, 6.83 [16:03:13] looks [16:04:12] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.71, 7.61, 6.94 [16:04:32] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:05:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:06:08] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.09, 7.56, 6.99 [16:06:54] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy1qi [16:06:55] [02miraheze/mw-config] 07RhinosF1 03b94b78a - set $egChameleonLayoutFile on koopacabanawiki? [16:07:21] @Lakelimbo ^, should deploy within 30 minutes [16:07:27] ok thanks ❤️ [16:08:04] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.81, 6.83, 6.79 [16:08:05] miraheze/mw-config - RhinosF1 the build passed. [16:10:00] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.56, 6.18, 6.56 [16:10:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.59, 5.90, 6.57 [16:11:42] !log [@mw11] starting deploy of {'config': True} to ovlon [16:12:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:12:13] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 30s [16:12:30] deployed @Lakelimbo [16:12:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:42] nice, thanks [16:14:19] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [16:14:36] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:14:48] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:15:30] [02puppet] 07RhinosF1 synchronize pull request 03#2177: mediawiki: add new servers - 13https://git.io/JyiSs [16:16:08] PROBLEM - jobchron1 JobChron Service on jobchron1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [16:16:15] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:16:35] paladox: [16:16:36] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.751 second response time [16:16:44] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.347 second response time [16:17:46] RhinosF1: yes? [16:18:03] paladox: jobchron failed, fixing [16:18:09] ok [16:18:50] [02puppet] 07RhinosF1 opened pull request 03#2180: Update chron.pp - 13https://git.io/Jy1YS [16:19:00] paladox: ^ [16:19:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.13, 3.89, 3.39 [16:19:14] [02puppet] 07paladox closed pull request 03#2180: Update chron.pp - 13https://git.io/Jy1YS [16:19:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy1YH [16:19:17] [02miraheze/puppet] 07RhinosF1 0373e09a7 - Update chron.pp (#2180) [16:19:20] pls can you deploy & resart service as we don't get access [16:20:08] RECOVERY - jobchron1 JobChron Service on jobchron1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [16:20:12] done [16:21:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.78, 3.40 [16:22:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [16:22:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [16:22:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.17, 7.16, 6.81 [16:23:18] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.004 second response time [16:24:00] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:24:22] !log [@test3] starting deploy of {'config': True} to skip [16:24:23] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [16:24:36] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.66, 6.58, 6.63 [16:25:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.11, 3.33, 3.29 [16:25:14] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.789 second response time [16:25:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:01] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.867 second response time [16:26:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:37] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:27:03] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:27:37] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.08, 1.69, 1.31 [16:28:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:28:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:28:36] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.314 second response time [16:28:59] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.255 second response time [16:31:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.79, 3.91, 3.50 [16:31:27] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.50, 1.62, 1.37 [16:33:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.15, 7.05, 6.38 [16:33:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.72, 3.48 [16:36:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.23, 6.75, 6.63 [16:36:56] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.82, 6.23, 6.22 [16:37:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.61, 3.28, 3.37 [16:38:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.45, 6.30, 6.49 [16:42:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.47, 7.17, 6.82 [16:44:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.58, 6.16, 6.49 [16:51:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.00, 6.14, 5.60 [16:52:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.83, 6.19, 6.26 [16:53:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.99, 5.89, 5.56 [16:54:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.79, 6.11, 6.21 [16:56:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.46, 3.80, 3.44 [16:58:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.11, 3.16, 3.26 [17:02:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.32, 7.26, 6.71 [17:03:00] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.12, 6.89, 6.44 [17:04:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.27, 6.70, 6.56 [17:04:56] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.21, 6.57, 6.38 [17:13:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.31, 6.90, 6.66 [17:15:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.84, 3.41 [17:16:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.49, 6.78, 6.68 [17:17:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.05, 7.10, 6.25 [17:17:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.68, 3.40 [17:18:37] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.89, 7.31, 6.83 [17:19:06] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.46, 6.53, 6.15 [17:20:33] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.12, 6.77, 6.69 [17:23:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.33, 3.33, 3.32 [17:24:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.92, 6.68, 6.36 [17:25:00] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:25:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [17:26:39] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.94, 6.36, 6.28 [17:29:12] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.02, 7.27, 6.91 [17:29:30] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.18, 6.38, 6.06 [17:29:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:30:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:31:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.55, 3.44 [17:31:26] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.53, 6.11, 6.01 [17:33:00] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.20, 6.90, 6.85 [17:33:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.24, 3.34 [17:40:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.73, 6.54, 6.78 [17:44:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.94, 7.23, 7.00 [17:45:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.79, 6.89, 6.45 [17:45:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.04, 3.89, 3.49 [17:47:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.16, 7.45, 6.72 [17:47:19] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.35, 6.98, 6.81 [17:47:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.62, 3.42 [17:49:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.64, 3.44 [17:51:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.01, 7.22, 6.83 [17:51:12] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.37, 6.44, 6.64 [17:51:51] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.70, 3.68, 3.55 [17:53:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:53:47] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.374 second response time [17:58:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.95, 7.98, 7.42 [17:59:02] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.50, 6.92, 6.78 [17:59:20] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 3.95, 3.66 [18:00:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.79, 7.76, 7.41 [18:00:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.30, 6.62, 6.68 [18:01:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.25, 6.59, 6.74 [18:01:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.58, 3.37, 3.48 [18:01:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [18:03:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.63, 3.56 [18:03:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:07:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.84, 3.67 [18:09:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb [18:10:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:10:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.50, 6.19, 6.79 [18:11:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 4.01, 3.76 [18:11:53] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:12:05] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:16] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:03] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.59, 3.64 [18:13:36] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:15:05] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.946 second response time [18:15:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.90, 3.97, 3.76 [18:15:39] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.763 second response time [18:15:59] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.663 second response time [18:16:15] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 6.019 second response time [18:16:25] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.428 second response time [18:16:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.17, 6.65, 6.26 [18:18:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.99, 6.56, 6.29 [18:18:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:19:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.97, 3.83 [18:21:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.68, 4.02, 3.88 [18:21:30] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:48] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:22:56] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:01] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:23:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.99, 3.89 [18:24:20] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:25:01] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.878 second response time [18:25:02] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 6.685 second response time [18:25:40] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.320 second response time [18:25:58] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.915 second response time [18:26:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:28:22] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.340 second response time [18:37:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.21, 2.75, 3.26 [18:43:37] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.85, 6.64, 6.38 [18:44:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [18:45:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.38, 6.63, 6.41 [18:45:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [18:47:38] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:48:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:49:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.43, 6.46, 5.96 [18:51:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.35, 6.44, 6.02 [18:51:32] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:39] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:51:39] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48958 MB (10% inode=98%); [18:51:52] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:53:12] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.09, 7.20, 6.79 [18:53:17] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:53:40] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 8.995 second response time [18:53:58] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.836 second response time [18:54:29] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:55:18] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.402 second response time [18:55:30] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [18:55:43] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.306 second response time [18:58:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:58:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.06, 6.08, 6.45 [18:59:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:11:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.45, 6.78, 6.38 [19:13:35] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.29, 6.29, 6.26 [19:20:36] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:20:43] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:21:12] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:19] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [19:21:46] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:20] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:16] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.865 second response time [19:23:22] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.274 second response time [19:23:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:23:43] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.321 second response time [19:24:19] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 0.492 second response time [19:24:36] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:24:44] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.058 second response time [19:28:33] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.27, 6.90, 6.63 [19:29:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.45, 3.12 [19:31:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.37, 3.42, 3.15 [19:33:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:35:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.32, 3.95, 3.40 [19:35:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.14, 7.52, 6.67 [19:37:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.60, 3.34 [19:37:18] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.66, 7.01, 6.58 [19:39:12] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.05, 6.78, 6.55 [19:39:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:42:32] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.72, 7.79, 7.18 [19:44:33] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.58, 7.40, 7.11 [19:47:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.72, 3.48 [19:49:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.61, 3.47 [19:50:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.29, 6.78, 6.52 [19:51:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 3.30, 3.38 [19:51:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.35, 7.11, 6.57 [19:51:57] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:00] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.52, 6.73, 6.53 [19:53:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.48, 6.22, 6.31 [19:54:03] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.118 second response time [19:54:04] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.266 second response time [19:54:33] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.43, 7.38, 7.09 [19:56:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.36, 7.28, 6.84 [19:59:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.22, 3.87, 3.49 [20:00:32] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.62, 7.71, 7.39 [20:02:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.25, 6.28, 6.56 [20:03:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.62, 3.90, 3.63 [20:05:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 4.18, 3.77 [20:08:33] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.89, 6.08, 6.70 [20:09:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.66, 3.67 [20:20:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:20:12] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:20:20] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.53, 7.44, 6.69 [20:20:26] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:21:36] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:51] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:01] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.28, 7.34, 6.74 [20:22:21] [02puppet] 07Universal-Omega reviewed pull request 03#2177 commit - 13https://git.io/JyDlm [20:22:23] [02puppet] 07Universal-Omega reviewed pull request 03#2177 commit - 13https://git.io/JyDlY [20:22:24] [02puppet] 07Universal-Omega reviewed pull request 03#2177 commit - 13https://git.io/JyDlO [20:22:26] [02puppet] 07Universal-Omega reviewed pull request 03#2177 commit - 13https://git.io/JyDl3 [20:23:41] RhinosF1: ^ [20:23:44] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 9.071 second response time [20:23:55] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:24:06] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.632 second response time [20:24:24] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.950 second response time [20:24:38] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.165 second response time [20:25:13] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:25:54] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.013 second response time [20:27:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.98, 3.07, 3.36 [20:27:50] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:27:53] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:27:53] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:28:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:28:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:28:12] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.09, 7.11, 6.80 [20:28:46] [02puppet] 07Universal-Omega reviewed pull request 03#2179 commit - 13https://git.io/JyDl4 [20:28:47] [02puppet] 07Universal-Omega reviewed pull request 03#2179 commit - 13https://git.io/JyDlB [20:29:17] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.374 second response time [20:30:03] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.06, 7.07, 6.47 [20:30:08] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.11, 6.61, 6.65 [20:31:55] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.93, 6.11, 6.51 [20:32:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.98, 6.89, 6.47 [20:32:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [20:32:15] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /home/universalomega/MigrateToAbstractSchema.php again, after making another fix to script — verified successfully ran this time [20:32:26] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 9.286 second response time [20:32:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:33:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [20:34:09] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.490 second response time [20:34:10] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.538 second response time [20:34:18] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.826 second response time [20:36:03] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.40, 6.50, 6.45 [20:37:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:38:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:38:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 3.75, 3.46 [20:39:39] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49547 MB (11% inode=98%); [20:42:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.94, 6.99, 6.78 [20:43:33] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.13, 6.89, 6.58 [20:44:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 4.00, 3.69 [20:45:24] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [20:47:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:47:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.16, 7.00, 6.73 [20:48:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.87, 4.83, 4.06 [20:48:38] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.97, 7.09, 6.83 [20:49:20] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.62, 6.63, 6.61 [20:50:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::929a/cpweb [20:50:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.21, 6.62, 6.72 [20:51:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:52:14] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:59] [02puppet] 07RhinosF1 reviewed pull request 03#2177 commit - 13https://git.io/JyDlA [20:53:24] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:29] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.95, 3.90 [20:54:16] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.54, 7.19, 6.92 [20:56:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 4.02, 3.93 [20:56:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyD8e [20:56:21] [02miraheze/mw-config] 07Universal-Omega 031fe3c57 - Remove wgServicesRepo [20:57:20] miraheze/mw-config - Universal-Omega the build passed. [20:57:32] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.527 second response time [20:57:39] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.850 second response time [20:58:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.28, 3.28, 3.67 [20:58:33] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.250 second response time [20:58:36] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 7.706 second response time [20:59:41] [02puppet] 07Universal-Omega reviewed pull request 03#2177 commit - 13https://git.io/JyD8L [21:00:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:00:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.22, 6.60, 6.80 [21:00:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:03:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 3.54, 3.61 [21:04:33] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.14, 7.47, 7.15 [21:05:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.61, 3.16, 3.46 [21:06:33] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.35, 7.05, 7.02 [21:07:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.49, 3.55 [21:09:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.32, 3.06, 3.39 [21:11:44] !log [@mw11] starting deploy of {'config': True} to ovlon [21:12:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:23] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 38s [21:12:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.68, 7.04, 6.72 [21:12:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:12:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.83, 6.70, 6.53 [21:14:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:16:26] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.61, 6.31, 6.41 [21:16:32] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.09, 7.27, 7.05 [21:17:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.47, 3.45 [21:18:32] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.90, 6.98, 6.96 [21:19:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.59, 3.06, 3.29 [21:20:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:22:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:22:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.02, 6.76, 6.78 [21:24:25] !log [@test3] starting deploy of {'config': True} to skip [21:24:26] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:24:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:25:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:26:33] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.48, 6.29, 6.71 [21:26:45] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 2.00, 1.53, 1.02 [21:27:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.34, 6.85, 6.64 [21:28:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.53, 6.97, 6.81 [21:28:45] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.45, 1.54, 1.09 [21:29:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.64, 6.50, 6.54 [21:30:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.30, 6.58, 6.33 [21:30:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.27, 6.74, 6.75 [21:31:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 3.86, 3.55 [21:32:03] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.48, 6.34, 6.26 [21:33:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.50, 3.45 [21:39:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.02, 3.27, 3.38 [21:45:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.46, 3.42 [21:47:24] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.94, 3.81, 3.54 [21:49:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.60, 3.49 [21:51:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.26, 3.17, 3.34 [21:51:16] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [21:51:46] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:52:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [21:52:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.65, 7.13, 6.61 [21:52:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.77, 6.79, 6.50 [21:53:15] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 1.923 second response time [21:53:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:53:46] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.065 second response time [21:53:52] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.84, 6.80, 6.57 [21:54:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:54:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.78, 6.55, 6.46 [21:54:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.78, 6.64, 6.47 [21:55:49] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.34, 6.53, 6.49 [21:57:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.68, 3.50 [21:59:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.00, 3.31, 3.37 [22:11:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 3.66, 3.32 [22:13:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.78, 7.12, 6.55 [22:13:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.66, 3.37 [22:13:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [22:14:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.10, 6.76, 6.49 [22:15:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.15, 3.38, 3.29 [22:15:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:17:56] [02puppet] 07RhinosF1 synchronize pull request 03#2177: mediawiki: add new servers - 13https://git.io/JyiSs [22:20:59] [02puppet] 07RhinosF1 synchronize pull request 03#2179: site: add memcache + jobchron in new DC - 13https://git.io/JyXIS [22:21:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.76, 6.66, 6.24 [22:23:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.61, 6.63, 6.28 [22:23:45] CosmicAlpha: regex should be ok [22:24:57] [02puppet] 07Universal-Omega reviewed pull request 03#2179 commit - 13https://git.io/JyDB2 [22:24:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.82, 6.61, 6.42 [22:28:21] [02puppet] 07RhinosF1 synchronize pull request 03#2179: site: add memcache + jobchron in new DC - 13https://git.io/JyXIS [22:28:52] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.18, 7.76, 6.89 [22:31:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.47, 7.36, 6.64 [22:31:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.81, 3.57, 3.22 [22:32:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.36, 6.69, 5.86 [22:32:10] RhinosF1: both look good now [22:32:44] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.78, 7.02, 6.81 [22:33:00] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.02, 7.24, 6.69 [22:35:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.45, 3.21, 3.16 [22:36:03] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.74, 6.04, 5.81 [22:37:04] [02puppet] 07RhinosF1 opened pull request 03#2181: MediaWiki: auto generate deploy known_hosts - 13https://git.io/JyDBN [22:37:32] paladox: also, ^, that'll make it easieer [22:38:00] copied from salt [22:38:02] !log update xvfb on mw8/mwtask1 [22:38:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:38:33] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.08, 6.45, 6.69 [22:38:40] !log update xserver-xorg-core on mw8/mwtask1 [22:38:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:39:08] RECOVERY - mw8 APT on mw8 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [22:39:27] RECOVERY - mwtask1 APT on mwtask1 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [22:41:00] [02puppet] 07paladox synchronize pull request 03#2181: MediaWiki: auto generate deploy known_hosts - 13https://git.io/JyDBN [22:41:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.85, 6.73, 6.28 [22:41:26] RhinosF1: need to add githubs i think [22:41:30] for the ssl repo [22:41:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [22:42:10] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [22:43:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.06, 6.33, 6.18 [22:43:29] paladox: doesn't seem to use /var/www/.ssh [22:43:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:43:42] there's no known hosts file i can see on task [22:44:20] oh right [22:44:30] [02puppet] 07paladox closed pull request 03#2181: MediaWiki: auto generate deploy known_hosts - 13https://git.io/JyDBN [22:44:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JyDRn [22:44:33] [02miraheze/puppet] 07RhinosF1 0335d0e7a - MediaWiki: auto generate deploy known_hosts (#2181) [22:45:47] will test when puppet pulls [22:46:07] pulled [22:46:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:47:27] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.89, 7.26, 6.90 [22:47:52] !log [rhinos@mw11] starting deploy of {'config': True} to all [22:48:11] paladox: works [22:48:17] !log [rhinos@mw11] finished deploy of {'config': True} to all - SUCCESS in 24s [22:48:17] nice [22:48:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:48:27] https://phabricator.miraheze.org/T8370#172441 [22:48:28] [url] ⚓ T8370 Generate known_hosts for deploy tool automatically | phabricator.miraheze.org [22:48:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:50:51] paladox: can you poke test3 apt too [22:50:51] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:58] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:00] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:39] !log update xvfb on test3 [22:51:46] !log update xserver-xorg-core on test3 [22:51:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.68, 7.71, 6.76 [22:52:24] RECOVERY - test3 APT on test3 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [22:52:28] [02ssl] 07RhinosF1 opened pull request 03#458: remove nl.xliving.tk - 13https://git.io/JyDR6 [22:52:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:52:51] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.816 second response time [22:52:53] [02ssl] 07RhinosF1 synchronize pull request 03#458: remove nl.xliving.tk - 13https://git.io/JyDR6 [22:53:01] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.273 second response time [22:53:01] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.948 second response time [22:53:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:43] paladox: https://github.com/miraheze/ssl/pull/458 is generating a big BT danger page [22:53:43] [url] remove nl.xliving.tk by RhinosF1 · Pull Request #458 · miraheze/ssl · GitHub | github.com [22:53:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.27, 7.41, 6.76 [22:54:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.42, 6.67, 6.79 [22:55:15] [02ssl] 07paladox closed pull request 03#458: remove nl.xliving.tk - 13https://git.io/JyDR6 [22:55:16] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JyDRQ [22:55:18] [02miraheze/ssl] 07RhinosF1 03ef42eb3 - remove nl.xliving.tk (#458) [22:55:24] you'll need to remove it from cw_wikis [22:55:31] actually i will [22:55:35] have to do it via the UI [22:55:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.53, 3.64, 3.20 [22:55:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.04, 6.60, 6.55 [22:56:53] done [22:57:05] ty [22:57:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.45, 7.36, 7.05 [22:57:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.49, 3.19 [22:59:05] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.18, 7.04, 6.98 [22:59:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.69, 3.29 [23:00:02] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:03] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.71, 3.34 [23:02:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.31, 6.28, 6.70 [23:05:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 4.04, 3.55 [23:07:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.69, 3.64, 3.46 [23:09:39] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48944 MB (10% inode=98%); [23:10:17] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.19, 7.20, 6.56 [23:11:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.82, 3.56 [23:13:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.63, 3.52 [23:13:43] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.92, 6.93, 6.58 [23:14:10] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.96, 6.33, 6.38 [23:15:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.08, 3.16, 3.37 [23:15:37] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.31, 6.45, 6.45 [23:18:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.68, 7.33, 6.80 [23:20:03] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.46, 7.57, 6.95 [23:20:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [23:21:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.80, 6.83, 6.27 [23:22:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.60, 7.01, 6.84 [23:22:17] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.63, 7.34, 6.85 [23:22:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [23:23:19] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.78, 6.97, 6.60 [23:23:58] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.67, 6.24, 6.58 [23:24:05] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:24:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.21, 7.52, 6.99 [23:24:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:25:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.69, 6.63, 6.36 [23:25:13] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.18, 6.63, 6.52 [23:25:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.43, 3.39 [23:25:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JyD0o [23:25:58] [02miraheze/mw-config] 07Universal-Omega 038317ca9 - Update CreateWiki subdomain blacklist [23:25:59] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [23:26:01] [02mw-config] 07Universal-Omega opened pull request 03#4312: Update CreateWiki subdomain blacklist - 13https://git.io/JyD0K [23:26:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.25, 7.66, 7.10 [23:26:57] miraheze/mw-config - Universal-Omega the build passed. [23:27:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.00, 3.38, 3.38 [23:27:34] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:41] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:56] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:28:00] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [23:28:08] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.25, 7.26, 7.02 [23:28:20] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:22] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:28:29] [02mw-config] 07RhinosF1 commented on pull request 03#4312: Update CreateWiki subdomain blacklist - 13https://git.io/JyD01 [23:28:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [23:29:20] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 8.23, 4.59, 2.10 [23:29:31] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.82, 21.53, 18.68 [23:29:33] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.917 second response time [23:29:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.73, 6.96, 6.82 [23:29:44] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.198 second response time [23:29:46] PROBLEM - test3 Puppet on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:29:47] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:30:02] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:30:04] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:30:05] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.27, 6.17, 6.65 [23:30:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JyD0M [23:30:12] [02miraheze/puppet] 07paladox 03c894eef - cloud: Add support for ferm [23:30:14] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:30:15] [02puppet] 07paladox opened pull request 03#2182: cloud: Add support for ferm - 13https://git.io/JyD0D [23:30:45] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:16] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.12, 7.51, 6.49 [23:31:26] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:32:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JyD09 [23:32:26] [02miraheze/puppet] 07paladox 03ac2540e - Create forward-ferm [23:32:27] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 5.124 second response time [23:32:27] [02puppet] 07paladox synchronize pull request 03#2182: cloud: Add support for ferm - 13https://git.io/JyD0D [23:32:37] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.276 second response time [23:33:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.56, 3.74, 3.52 [23:33:15] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.35, 7.73, 6.68 [23:33:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JyD0H [23:33:26] [02miraheze/puppet] 07paladox 038615bd5 - Update forward-ferm [23:33:28] [02puppet] 07paladox synchronize pull request 03#2182: cloud: Add support for ferm - 13https://git.io/JyD0D [23:33:30] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.66, 22.61, 19.77 [23:35:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.00, 3.36, 3.41 [23:35:01] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.371 second response time [23:35:10] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.04, 6.75, 6.45 [23:35:22] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.548 second response time [23:35:27] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.314 second response time [23:35:53] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 2151s [23:36:00] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.838 second response time [23:36:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:04] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 41 minutes ago with 0 failures [23:36:08] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.360 second response time [23:36:10] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.716 second response time [23:36:16] RECOVERY - test3 APT on test3 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [23:36:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:36:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.55, 3.46 [23:37:22] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.34, 6.22, 6.58 [23:37:41] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:38:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.28, 3.38 [23:39:28] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.40, 18.93, 19.22 [23:39:38] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2376s [23:39:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:39:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:42:50] PROBLEM - test3 Current Load on test3 is WARNING: WARNING - load average: 0.08, 2.75, 3.67 [23:44:45] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 0.01, 1.87, 3.24 [23:45:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.35, 6.28, 5.58 [23:47:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.72, 5.66, 5.44 [23:48:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JyDEg [23:48:32] [02miraheze/mw-config] 07Universal-Omega 038d38acb - Cleanup existing blacklist [23:48:33] [02mw-config] 07Universal-Omega synchronize pull request 03#4312: Update CreateWiki subdomain blacklist - 13https://git.io/JyD0K [23:49:26] [02mw-config] 07Universal-Omega edited pull request 03#4312: Update CreateWiki subdomain blacklist - 13https://git.io/JyD0K [23:49:35] miraheze/mw-config - Universal-Omega the build passed. [23:49:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:50:57] RhinosF1: ^ FYI, updated that PR to redo other entries in blacklist also, as it was kinda a mess. [23:53:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:56:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JyDE5 [23:56:32] [02miraheze/mw-config] 07Universal-Omega 03c859aa5 - Update LocalSettings.php [23:56:33] [02mw-config] 07Universal-Omega synchronize pull request 03#4312: Update CreateWiki subdomain blacklist - 13https://git.io/JyD0K [23:57:33] miraheze/mw-config - Universal-Omega the build passed.