[00:03:43] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 8362 MB (21% inode=96%); [00:05:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.25, 6.89, 5.34 [00:06:46] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.18, 20.50, 17.49 [00:07:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.91, 7.06, 5.84 [00:07:33] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.63, 6.32, 5.32 [00:08:42] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.70, 18.13, 16.98 [00:09:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.22, 6.17, 5.65 [00:12:35] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.29, 22.39, 19.01 [00:13:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.11, 7.00, 6.12 [00:14:30] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.58, 21.68, 19.11 [00:15:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.97, 6.31, 5.97 [00:18:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.21, 22.57, 19.87 [00:21:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.23, 8.32, 6.86 [00:21:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.37, 6.83, 5.93 [00:22:12] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.16, 21.20, 19.92 [00:23:00] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.54, 7.94, 6.92 [00:23:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.19, 6.42, 5.90 [00:24:07] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.44, 19.64, 19.52 [00:28:52] [02ssl] 07Universal-Omega commented on pull request 03#419: kk.uncyc.tk: wiki does not exist - 13https://git.io/JWg2g [00:29:00] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.70, 6.75, 6.75 [00:34:55] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.59, 7.25, 6.86 [00:35:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.99, 7.34, 6.56 [00:36:54] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.38, 6.94, 6.77 [00:37:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.53, 6.69, 6.41 [00:38:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.21, 6.33, 6.56 [01:03:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.56, 6.98, 6.23 [01:05:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.19, 7.64, 6.47 [01:05:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.05, 6.77, 5.95 [01:05:38] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.36, 6.34, 6.10 [01:06:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.83, 6.87, 6.33 [01:07:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.11, 2.74 [01:08:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.66, 6.04, 6.08 [01:09:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.38, 3.02, 2.74 [01:09:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.35, 6.33, 5.97 [01:34:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.96, 7.22, 6.20 [01:35:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.31, 6.87, 6.18 [01:36:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.82, 6.43, 6.02 [01:37:33] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.91, 6.39, 6.08 [02:06:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.88, 6.92, 6.35 [02:08:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.92, 6.66, 6.33 [03:23:40] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [03:23:42] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [03:23:48] [02mw-config] 07Universal-Omega deleted branch 03The-Voidwalker-patch-1 - 13https://git.io/vbvb3 [03:23:50] [02miraheze/mw-config] 07Universal-Omega deleted branch 03The-Voidwalker-patch-1 [04:47:20] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Reception123-patch-1 [04:47:21] [02mw-config] 07Universal-Omega deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [06:24:58] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-11] [06:52:58] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:46] @bukkit@localhost: T&S can do anything that can be done from the web [07:54:17] RhinosF1: Very true [08:00:59] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:01:06] PROBLEM - cp3 Stunnel Http for mw11 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:01:13] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:23] PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 100% [08:01:26] PROBLEM - cp3 Stunnel Http for test3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:01:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [08:01:32] * RhinosF1 here [08:01:43] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:01:47] PROBLEM - cp3 SSH on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:50] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:01:59] PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:02:01] PROBLEM - cp3 NTP time on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:02:02] PROBLEM - cp3 APT on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:02:09] PROBLEM - cp3 Stunnel Http for mon2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:02:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:02:16] cp3 is down [08:02:20] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:02:25] PROBLEM - cp3 Stunnel Http for mw9 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:02:53] PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:02:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [08:02:58] PROBLEM - cp3 PowerDNS Recursor on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:03:11] PROBLEM - Host cp3 is DOWN: PING CRITICAL - Packet loss = 100% [08:03:13] [02dns] 07RhinosF1 opened pull request 03#208: Depool cp3 - 13https://git.io/JWVRs [08:03:23] Reception123: Southparkfan [08:04:40] .task 7647 [08:04:42] https://phabricator.miraheze.org/T7647 - cp3 down - 2021-07-16 09:00 UK time, authored by RhinosF1, assigned to None, Priority: Unbreak Now!, Status: Open [08:04:50] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:05:12] RECOVERY - Host cp3 is UP: PING OK - Packet loss = 0%, RTA = 244.99 ms [08:05:16] RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 0.946 second response time [08:05:16] RECOVERY - cp3 PowerDNS Recursor on cp3 is OK: DNS OK: 0.502 seconds response time. miraheze.org returns 128.199.139.216,2400:6180:0:d0::403:f001 [08:05:19] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.07, 0.13, 0.27 [08:05:21] RECOVERY - cp3 Stunnel Http for mw11 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 0.919 second response time [08:05:24] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2270 bytes in 0.936 second response time [08:05:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:05:37] RECOVERY - cp3 Stunnel Http for test3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15405 bytes in 0.953 second response time [08:05:43] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 245.57 ms [08:05:51] RECOVERY - cp3 SSH on cp3 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [08:05:54] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [08:05:56] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures [08:06:01] RECOVERY - cp3 APT on cp3 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [08:06:01] RECOVERY - cp3 NTP time on cp3 is OK: NTP OK: Offset -0.002666354179 secs [08:06:06] RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 1.038 second response time [08:06:06] RECOVERY - cp3 Stunnel Http for mon2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 32976 bytes in 0.969 second response time [08:06:09] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 3% [08:06:11] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 4864 MB (20% inode=94%); [08:06:12] Oh it's coming back [08:06:16] RECOVERY - cp3 Stunnel Http for mw9 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15417 bytes in 1.025 second response time [08:11:55] Universal_Omega: let's hope it stays up [08:12:38] RhinosF1: yeah, if it goes down again, I guess UBN it again and leave it like that. [08:12:53] Universal_Omega: yep [08:13:09] I left patch to depool sg if needed [08:13:31] Sounds good. [09:12:04] around now [09:58:46] [02dns] 07RhinosF1 commented on pull request 03#208: Depool cp3 - 13https://git.io/JWwW5 [09:58:48] [02dns] 07RhinosF1 closed pull request 03#208: Depool cp3 - 13https://git.io/JWVRs [10:12:00] Reception123: FYI bullseye freezes tommorow [10:30:34] ack [10:34:37] Reception123: I'm gonna dry run category collation on all wikis at some point [10:34:44] Just to see how long it might take [10:35:04] I want to make mwscript record time taken first [10:35:16] I will send my update to that soon [10:35:31] ok, sounds good [10:35:33] Probably tonight [10:36:05] Reception123: once everywhere is on bullseye, we'll need to refresh categories because ICU changes [10:36:23] ah [10:36:33] Well it's not a need [10:36:53] But otherwise some might be out of order until purges happen [10:41:21] Just needs one run after all upgraded [10:41:27] I'll do test3 first of course [11:36:42] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.06, 7.06, 5.89 [11:38:28] Go away icinga-miraheze [11:38:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.53, 6.04, 5.66 [11:38:59] Good [11:39:04] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.89, 6.29, 5.38 [11:41:04] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.42, 7.52, 5.95 [11:43:01] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.61, 6.90, 5.93 [11:44:36] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.04, 7.52, 6.44 [11:44:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.77, 6.80, 5.99 [11:46:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.88, 6.98, 6.38 [11:47:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.10, 7.06, 5.96 [11:48:37] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.49, 6.63, 6.31 [11:48:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.56, 7.04, 6.31 [11:49:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.43, 7.13, 6.14 [11:50:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.32, 6.39, 6.15 [11:51:09] PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found [11:51:10] PROBLEM - wiki.thehall.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thehall.xyz could not be found [11:51:10] PROBLEM - rupowerrangersfanon.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rupowerrangersfanon.tk could not be found [11:51:30] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.40, 7.34, 6.32 [11:52:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.20, 7.16, 6.57 [11:53:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.11, 7.10, 6.37 [11:53:34] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.35, 6.74, 5.99 [11:56:35] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.30, 6.57, 6.46 [11:57:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.78, 6.60, 6.36 [11:57:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.61, 7.89, 6.63 [11:57:55] RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: SSL OK - erikapedia.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [11:57:58] RECOVERY - wiki.thehall.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.thehall.xyz reverse DNS resolves to cp13.miraheze.org - CNAME OK [11:57:59] RECOVERY - rupowerrangersfanon.tk - reverse DNS on sslhost is OK: SSL OK - rupowerrangersfanon.tk reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [11:59:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.04, 7.88, 6.80 [12:06:26] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.36, 7.51, 6.90 [12:08:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.58, 7.52, 6.98 [12:09:34] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.02, 6.77, 6.71 [12:12:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.97, 8.20, 7.39 [12:14:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.94, 7.44, 7.22 [12:15:30] PROBLEM - wiki.aqua22.hu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.22672989800256:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1544:SSL alert number 80 [12:15:42] PROBLEM - wiki.aqua22.hu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.aqua22.hu could not be found [12:16:33] Reception123: not pointing at us ^ [12:18:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.20, 7.32, 7.18 [12:23:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.20, 7.08, 6.74 [12:24:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.03, 7.53, 7.38 [12:25:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.62, 6.59, 6.61 [12:28:24] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.23, 5.65, 6.64 [12:30:30] will look/remove [12:35:07] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:35:13] PROBLEM - cp3 Stunnel Http for mw11 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:35:14] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [12:35:20] Reception123: we're off again [12:35:32] PROBLEM - cp3 Stunnel Http for test3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:35:42] [02dns] 07RhinosF1 reopened pull request 03#208: Depool cp3 - 13https://git.io/JWVRs [12:35:46] PROBLEM - cp3 SSH on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:50] PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:35:57] PROBLEM - cp3 NTP time on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:36:00] PROBLEM - cp3 Stunnel Http for mon2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:36:01] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:36:07] Reception123: please depool it [12:36:09] PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:36:13] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:36:14] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:36:22] PROBLEM - cp3 APT on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:36:30] PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 302.48 ms [12:36:44] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw11 [12:36:50] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [12:37:16] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.01, 0.16, 0.29 [12:37:17] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2271 bytes in 1.245 second response time [12:37:18] RECOVERY - cp3 Stunnel Http for mw11 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 1.298 second response time [12:37:31] RECOVERY - cp3 Stunnel Http for test3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15405 bytes in 1.406 second response time [12:37:43] RECOVERY - cp3 SSH on cp3 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [12:37:50] RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 1.311 second response time [12:37:56] RECOVERY - cp3 NTP time on cp3 is OK: NTP OK: Offset -0.003626316786 secs [12:37:59] RECOVERY - cp3 Stunnel Http for mon2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 33007 bytes in 1.422 second response time [12:38:00] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 2% [12:38:09] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 4251 MB (17% inode=94%); [12:38:11] RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 1.359 second response time [12:38:13] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [12:38:17] Or not [12:38:21] RECOVERY - cp3 APT on cp3 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [12:38:23] It's flappy I guess [12:38:33] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 336.65 ms [12:38:44] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [12:38:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:39:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:40:16] @R4356th: cp3 went again [12:44:44] ergh [12:44:55] I think I'll depool and reboot [12:45:16] [02dns] 07Reception123 closed pull request 03#208: Depool cp3 - 13https://git.io/JWVRs [12:45:18] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JWrEP [12:45:19] [02miraheze/dns] 07RhinosF1 03c81c313 - Depool cp3 (#208) [12:45:32] Reception123: okay [12:45:38] I have no clue of cause though [12:45:48] Might be worth leaving out until void shows [12:46:35] A reboot can't hurt and probably not great to put stress on the other cp*s either [12:46:39] We can keep an eye on it [12:46:59] We can cope with a cp down [12:47:06] It's mw that get stressed [12:47:38] I think we can try a reboot + repool and if something happens again we can keep it depooled and wait for Void to see if he can figure something out [12:47:45] usually reboots somehow magically fix things [12:48:04] Ok [12:57:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.02, 7.44, 6.56 [12:57:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.63, 7.12, 6.47 [12:59:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.49, 7.38, 6.67 [12:59:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.78, 6.75, 6.43 [13:01:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.68, 6.73, 6.51 [13:05:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.67, 7.69, 6.89 [13:06:29] Reception123: how did reboot do? [13:06:39] !log rebooted cp3 [13:06:45] RhinosF1: we'll have to see [13:06:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:07:14] [02miraheze/dns] 07Reception123 pushed 031 commit to 03revert-208-patch-23 [+0/-0/±1] 13https://git.io/JWroW [13:07:14] There we go [13:07:15] [02miraheze/dns] 07Reception123 036603c0f - Revert "Depool cp3 (#208)" [13:07:17] [02dns] 07Reception123 created branch 03revert-208-patch-23 - 13https://git.io/vbQXl [13:07:19] [02dns] 07Reception123 opened pull request 03#209: Repool cp3 - 13https://git.io/JWroR [13:07:23] Please don't crash [13:07:24] [02dns] 07Reception123 closed pull request 03#209: Repool cp3 - 13https://git.io/JWroR [13:07:26] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JWrou [13:07:27] [02miraheze/dns] 07Reception123 034da3f1f - Revert "Depool cp3 (#208)" (#209) [13:07:29] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 243.79 ms [13:08:27] Well a RECOVERY is good [13:08:44] let's hope it stays like that [13:09:11] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.99, 6.75, 6.74 [13:23:09] Reception123: wikimedia lost a rack [13:23:10] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.34, 6.92, 6.48 [13:25:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.68, 6.49, 6.38 [13:25:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.44, 7.02, 6.27 [13:26:00] Reception123: codfw A2 down. Impact limited [13:27:24] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.40, 6.84, 6.31 [13:29:04] ack [13:29:22] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.54, 6.69, 6.31 [13:35:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.03, 8.29, 7.04 [13:36:09] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.25, 21.32, 17.56 [13:37:09] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.25, 7.37, 6.87 [13:37:27] PROBLEM - wiki.ciptamedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ciptamedia.org could not be found [13:38:05] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.83, 20.04, 17.54 [13:41:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.09, 6.78, 6.52 [13:42:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.27, 6.66, 6.73 [13:44:08] RECOVERY - wiki.ciptamedia.org - reverse DNS on sslhost is OK: SSL OK - wiki.ciptamedia.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [13:45:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.28, 6.56, 6.52 [13:45:12] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.86, 7.36, 6.62 [13:46:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 7.19, 6.97 [13:47:10] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.18, 6.52, 6.40 [13:50:48] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.75, 7.97, 7.29 [13:52:45] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.02, 7.47, 7.18 [13:54:42] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.43, 6.34, 6.80 [13:58:58] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.20, 7.29, 6.59 [14:00:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.75, 7.06, 6.60 [14:02:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.07, 7.24, 6.71 [14:04:55] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.97, 7.59, 6.90 [14:04:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.95, 6.87, 6.52 [14:06:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.83, 7.36, 7.04 [14:10:54] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.62, 6.54, 6.67 [14:13:00] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.66, 6.40, 6.56 [14:16:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.64, 6.89, 6.68 [14:18:58] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.41, 6.67, 6.62 [14:23:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.13, 6.14, 6.68 [14:34:40] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.69, 7.78, 7.07 [14:36:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.52, 7.67, 7.12 [14:37:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.42, 7.39, 6.77 [14:40:29] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.48, 6.55, 6.75 [14:43:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.17, 6.32, 6.51 [14:46:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.04, 6.86, 6.87 [14:50:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.63, 7.66, 7.18 [14:51:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.25, 7.93, 6.99 [14:52:28] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.14, 7.52, 6.73 [14:53:24] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.50, 6.87, 6.60 [14:53:46] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.83, 7.50, 6.95 [14:54:09] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.16, 6.91, 7.02 [14:54:26] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.57, 7.35, 6.76 [14:55:23] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.13, 7.12, 6.71 [14:57:22] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.68, 7.30, 6.83 [14:57:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.64, 6.68, 6.77 [14:58:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.23, 6.79, 6.67 [14:59:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.94, 8.12, 7.17 [15:01:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.84, 7.51, 7.06 [15:03:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.42, 7.92, 7.39 [15:05:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.28, 7.41, 7.28 [15:07:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.89, 8.05, 7.53 [15:09:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.00, 7.76, 7.49 [15:13:05] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.49, 6.34, 6.80 [15:15:38] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.34, 6.07, 6.79 [15:16:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.20, 6.88, 6.64 [15:20:19] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.36, 6.27, 6.45 [15:23:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.11, 7.68, 7.20 [15:25:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.03, 7.52, 7.20 [15:26:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.75, 7.13, 6.77 [15:28:17] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.87, 6.43, 6.56 [15:33:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.89, 7.70, 7.22 [15:35:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.95, 7.84, 7.35 [15:37:39] PROBLEM - files.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for files.pornwiki.org could not be found [15:37:41] PROBLEM - wiki.park-city.club - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.park-city.club could not be found [15:37:41] PROBLEM - hr.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hr.petrawiki.org could not be found [15:37:42] PROBLEM - wiki.cyberfurs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cyberfurs.org could not be found [15:37:42] PROBLEM - wiki.counterculturelabs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.counterculturelabs.org could not be found [15:37:43] PROBLEM - debilepedie.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for debilepedie.gq could not be found [15:37:43] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.05, 7.42, 6.64 [15:37:46] PROBLEM - en.famepedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.famepedia.org could not be found [15:37:59] PROBLEM - wiki.newnormal.fans - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.newnormal.fans could not be found [15:41:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.54, 6.02, 6.69 [15:41:41] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.10, 7.37, 6.83 [15:44:30] RECOVERY - files.pornwiki.org - reverse DNS on sslhost is OK: SSL OK - files.pornwiki.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [15:44:33] RECOVERY - wiki.park-city.club - reverse DNS on sslhost is OK: SSL OK - wiki.park-city.club reverse DNS resolves to cp13.miraheze.org - CNAME OK [15:44:33] RECOVERY - hr.petrawiki.org - reverse DNS on sslhost is OK: SSL OK - hr.petrawiki.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [15:44:36] RECOVERY - wiki.counterculturelabs.org - reverse DNS on sslhost is OK: SSL OK - wiki.counterculturelabs.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [15:44:36] RECOVERY - wiki.cyberfurs.org - reverse DNS on sslhost is OK: SSL OK - wiki.cyberfurs.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [15:44:37] RECOVERY - debilepedie.gq - reverse DNS on sslhost is OK: SSL OK - debilepedie.gq reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [15:44:45] RECOVERY - en.famepedia.org - reverse DNS on sslhost is OK: SSL OK - en.famepedia.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [15:44:51] RECOVERY - wiki.newnormal.fans - reverse DNS on sslhost is OK: SSL OK - wiki.newnormal.fans reverse DNS resolves to cp13.miraheze.org - CNAME OK [15:45:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.86, 7.28, 7.11 [15:45:36] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.71, 6.64, 6.66 [15:51:30] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.04, 7.37, 6.80 [15:57:27] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.01, 6.74, 6.70 [15:59:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.85, 5.83, 6.59 [16:05:25] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.19, 7.66, 6.99 [16:07:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.68, 7.36, 6.97 [16:09:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.48, 6.60, 6.73 [16:21:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.85, 6.69, 6.46 [16:21:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.14, 6.19, 6.00 [16:23:33] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.49, 5.99, 5.96 [16:25:13] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.63, 6.21, 6.35 [16:33:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.32, 7.43, 6.42 [16:34:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.41, 6.79, 6.34 [16:35:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.58, 7.36, 6.54 [16:36:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.85, 7.79, 6.92 [16:36:39] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.42, 6.46, 6.27 [16:37:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.31, 6.54, 6.34 [16:38:08] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.39, 7.30, 6.82 [16:40:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.82, 6.32, 6.52 [17:30:38] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.31, 6.67, 6.36 [17:32:34] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.82, 6.05, 6.17 [17:34:36] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.86, 7.54, 6.59 [17:36:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.79, 6.97, 6.50 [17:40:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.07, 6.64, 6.48 [17:42:22] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.27, 7.71, 6.94 [17:44:18] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.35, 7.37, 6.89 [17:45:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.38, 7.34, 6.41 [17:48:12] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.64, 7.89, 7.18 [17:49:01] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.86, 7.19, 6.57 [17:52:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.92, 7.48, 7.23 [17:52:26] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.14, 7.84, 7.05 [17:54:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.32, 7.75, 7.35 [17:54:25] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.18, 7.41, 6.98 [17:54:56] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.36, 6.76, 6.63 [17:55:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.67, 7.47, 7.31 [18:03:47] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.33, 8.76, 7.80 [18:04:25] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 15.49, 9.85, 7.97 [18:04:59] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.20, 7.42, 6.91 [18:06:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.75, 7.16, 6.87 [18:09:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.22, 7.73, 7.72 [18:10:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.03, 7.27, 6.87 [18:10:58] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.18, 6.57, 6.70 [18:12:46] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.51, 6.69, 6.69 [18:14:27] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.87, 7.95, 7.91 [18:15:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.09, 7.69, 7.64 [18:17:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.63, 6.84, 6.74 [18:17:15] [02PDFEmbed] 07R4356th opened pull request 03#5: perf: Lazy-load PDFs - 13https://git.io/JWKyJ [18:17:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.00, 7.03, 7.41 [18:19:00] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.22, 6.61, 6.67 [18:19:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.16, 8.20, 7.76 [18:20:27] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.97, 7.86, 7.80 [18:21:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.94, 7.46, 7.54 [18:22:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.41, 7.42, 7.63 [18:29:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.03, 5.61, 6.57 [18:30:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.40, 5.70, 6.74 [18:34:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.36, 9.59, 7.99 [18:35:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.23, 7.70, 7.12 [18:37:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.47, 7.63, 7.18 [18:38:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.18, 7.88, 7.67 [18:47:30] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.58, 6.87, 6.35 [18:49:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.94, 6.49, 6.28 [18:51:34] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.56, 7.93, 7.44 [18:52:08] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.22, 7.50, 7.35 [18:53:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.82, 7.85, 7.48 [18:54:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.86, 6.54, 7.02 [19:00:07] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.16, 7.46, 7.17 [19:01:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.87, 7.75, 7.41 [19:02:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.49, 7.01, 7.04 [19:04:07] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.41, 7.63, 7.26 [19:05:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.24, 7.70, 7.47 [19:06:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.45, 7.22, 7.16 [19:10:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.17, 6.12, 6.69 [19:11:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.96, 5.65, 6.60 [19:21:35]