[00:00:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.47, 1.69, 1.73 [00:01:13] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.32, 1.92, 1.91 [00:02:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.16, 11.71, 11.44 [00:02:35] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.35, 1.86, 1.78 [00:03:08] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.39, 1.69, 1.83 [00:04:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.72, 1.85, 1.79 [00:08:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.87, 10.76, 11.22 [00:08:35] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.55, 1.69 [00:10:51] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.09, 1.36, 1.62 [00:12:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.97, 11.98, 11.56 [00:12:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.30, 9.09, 8.58 [00:14:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.54, 3.71, 4.00 [00:14:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.78, 8.36, 8.38 [00:16:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.08, 4.08, 4.09 [00:18:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.42, 11.29, 11.47 [00:20:34] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.50, 1.99, 1.54 [00:22:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.66, 3.93 [00:22:32] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.45, 1.71, 1.48 [00:23:53] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 11.49, 11.45 [00:24:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.99, 4.02 [00:24:30] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.83, 1.39, 1.39 [00:25:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.87, 11.41, 11.42 [00:26:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.37, 3.79, 3.94 [00:27:31] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.12, 1.83, 1.72 [00:27:48] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.57, 11.78, 11.55 [00:28:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.66, 4.11, 4.04 [00:29:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.96, 11.61, 11.50 [00:30:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.93, 3.98 [00:31:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 2.00, 1.82 [00:31:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.63, 12.12, 11.71 [00:32:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.87, 4.56, 4.19 [00:33:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.26, 2.15, 1.90 [00:33:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.90, 11.73, 11.62 [00:35:27] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.31, 1.75, 1.78 [00:35:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.30, 12.57, 11.93 [00:36:15] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.90, 1.56, 1.38 [00:37:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.12, 1.93, 1.84 [00:38:13] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.53, 1.61, 1.42 [00:39:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.93, 1.58, 1.73 [00:39:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 11.54, 11.66 [00:41:34] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [00:41:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.26, 11.92, 11.76 [00:42:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.50, 10.26, 9.35 [00:44:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 10.06, 10.05, 9.38 [00:45:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.04, 11.72, 11.74 [00:46:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.58, 3.99 [00:48:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.21, 4.65, 4.33 [00:49:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.45, 1.57, 1.68 [00:49:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.58, 12.00, 11.79 [00:51:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.94, 10.43, 9.45 [00:53:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.92, 1.91, 1.80 [00:53:33] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.42, 1.88, 1.54 [00:57:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.87, 1.44, 1.64 [00:57:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.60, 10.17, 9.68 [01:01:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.96, 1.75 [01:01:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.15, 11.40, 11.97 [01:05:07] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.99, 1.81 [01:05:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.77, 1.75 [01:05:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.11, 12.18, 12.09 [01:07:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 8.39, 2.60, 1.25 [01:07:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.54, 11.56, 11.85 [01:09:07] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.98, 1.58, 1.70 [01:09:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.29, 1.83, 1.13 [01:11:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.46, 1.40, 1.06 [01:11:19] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:11:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.33, 11.51, 11.74 [01:18:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.48, 1.79, 1.57 [01:20:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.27, 1.64, 1.54 [01:21:59] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.87, 2.04, 1.81 [01:22:43] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.13, 3.30, 2.11 [01:23:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.29, 1.81, 1.76 [01:25:57] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.62, 1.41, 1.62 [01:26:43] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.43, 3.59, 2.54 [01:26:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.95, 1.98 [01:27:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.34, 3.24, 4.00 [01:27:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.35, 2.52, 1.89 [01:28:43] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.14, 3.16, 2.52 [01:31:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.43, 1.89, 1.77 [01:32:31] PROBLEM - bast121 NTP time on bast121 is CRITICAL: NTP CRITICAL: Offset 0.5532204211 secs [01:33:07] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.54, 1.40, 1.60 [01:39:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.81, 3.81 [01:41:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.07, 3.80, 3.82 [01:41:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.10, 10.80, 11.92 [01:41:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.15, 2.05, 1.85 [01:42:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.32, 1.46, 1.69 [01:43:45] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.83, 1.92, 1.82 [01:45:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.79, 4.09, 3.95 [01:45:44] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.53, 1.69 [01:58:41] PROBLEM - bast101 PowerDNS Recursor on bast101 is CRITICAL: connect to address 2a10:6740::6:101 port 5666: No route to hostconnect to host 2a10:6740::6:101 port 5666: No route to host [01:58:41] PROBLEM - bast101 Current Load on bast101 is CRITICAL: connect to address 2a10:6740::6:101 port 5666: No route to hostconnect to host 2a10:6740::6:101 port 5666: No route to host [01:58:41] PROBLEM - bast101 conntrack_table_size on bast101 is CRITICAL: connect to address 2a10:6740::6:101 port 5666: No route to hostconnect to host 2a10:6740::6:101 port 5666: No route to host [01:59:08] !log depool mw101, set cpu sockets to 2 and repool [01:59:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:40] RECOVERY - bast101 Current Load on bast101 is OK: OK - load average: 0.66, 0.26, 0.10 [02:00:40] RECOVERY - bast101 conntrack_table_size on bast101 is OK: OK: nf_conntrack is 0 % full [02:00:41] RECOVERY - bast101 PowerDNS Recursor on bast101 is OK: DNS OK: 0.108 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [02:01:55] !log revert [02:02:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:02:22] PROBLEM - mw101 SSH on mw101 is CRITICAL: connect to address 2a10:6740::6:107 and port 22: No route to host [02:02:31] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [02:02:34] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: connect to address 2a10:6740::6:107 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:03:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.73, 11.69, 11.12 [02:03:49] PROBLEM - mw101 Check Gluster Clients on mw101 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [02:03:51] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:04:18] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw101 [02:04:18] RECOVERY - mw101 SSH on mw101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [02:04:21] RECOVERY - bast121 NTP time on bast121 is OK: NTP OK: Offset -0.007566183805 secs [02:04:26] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [02:04:30] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [02:04:51] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw101 [02:05:10] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw101 [02:06:53] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:07:36] !log depool mw101, set cpu sockets to 2 & vpcus to 6 and repool [02:07:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:08:32] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.365 second response time [02:08:50] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.337 second response time [02:09:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.22, 11.68, 11.39 [02:09:50] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [02:10:16] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [02:10:25] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [02:10:51] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [02:11:10] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [02:11:49] RECOVERY - mw101 Check Gluster Clients on mw101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [02:13:26] !log depool mw102, set cpu sockets to 2 & vpcus to 6 and repool [02:13:31] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.56, 1.46 [02:13:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.39, 2.01, 1.64 [02:16:47] !log depool mw111, set cpu sockets to 2 & vpcus to 6 and repool [02:16:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:17:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.59, 1.53 [02:17:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: connect to address 2a10:6740::6:206 port 5666: No route to hostconnect to host 2a10:6740::6:206 port 5666: No route to host [02:19:16] !log depool mw112, set cpu sockets to 2 & vpcus to 6 and repool [02:19:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.68, 2.07, 0.72 [02:20:40] !log depool mw121, set cpu sockets to 2 & vpcus to 6 and repool [02:20:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:14] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.91, 1.60, 1.29 [02:21:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.96, 7.41, 3.33 [02:23:06] !log depool mw122, set cpu sockets to 2 & vpcus to 6 and repool [02:23:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:23:12] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.16, 1.46, 1.28 [02:23:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.64, 8.04, 4.08 [02:23:24] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.95, 2.07, 1.75 [02:23:29] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:35] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [02:23:43] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [02:23:47] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: connect to address 2a10:6740::6:309 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:24:48] PROBLEM - mw121 Check Gluster Clients on mw121 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [02:25:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.32, 1.75, 1.67 [02:25:29] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.378 second response time [02:25:33] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.329 second response time [02:25:43] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.017 second response time [02:25:46] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.014 second response time [02:26:12] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw121 [02:26:24] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw121 [02:26:51] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw121 [02:27:10] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw121 [02:27:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.08, 1.99, 1.78 [02:27:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.22, 9.35, 4.76 [02:28:12] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [02:28:24] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [02:28:48] RECOVERY - mw121 Check Gluster Clients on mw121 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [02:28:51] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [02:29:10] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [02:29:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.45, 1.71, 1.69 [02:29:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.82, 9.50, 5.38 [02:31:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.46, 1.65, 1.67 [02:36:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.30, 10.59, 7.35 [02:37:16] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.92, 2.09, 1.83 [02:38:30] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.52, 10.01, 7.52 [02:38:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.40, 2.01, 1.62 [02:39:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.51, 1.94, 1.81 [02:40:34] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [02:40:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.37, 1.75, 1.58 [02:41:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.02, 1.59, 1.69 [02:44:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.21, 1.91, 1.67 [02:46:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.89, 1.69 [02:48:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.33, 2.05, 1.77 [02:55:52] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.94, 1.76, 1.47 [02:57:52] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.67, 1.35, 1.35 [03:02:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.83, 1.67, 1.58 [03:04:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.37, 1.51, 1.53 [03:05:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.67, 11.34, 9.79 [03:07:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.80, 10.80, 9.79 [03:09:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.46, 9.99, 9.61 [03:19:50] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.81, 2.42, 1.86 [03:20:32] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.31, 2.69, 1.89 [03:22:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.66, 1.91, 2.00 [03:23:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.43, 10.71, 10.02 [03:24:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.87, 1.72, 1.68 [03:26:21] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:26] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:26] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.09, 1.60, 1.64 [03:26:59] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:54] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:29] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.965 second response time [03:29:04] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.779 second response time [03:29:57] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.988 second response time [03:32:30] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.525 second response time [03:32:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.28, 1.43, 1.70 [03:33:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.15, 11.60, 10.75 [03:33:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.72, 10.04, 9.09 [03:35:52] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.42, 9.72, 9.09 [03:35:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.84, 9.78, 8.27 [03:37:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.53, 11.96, 11.12 [03:37:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.83, 9.00, 8.17 [03:38:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.83, 3.60, 3.99 [03:39:40] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.29, 1.52, 1.95 [03:40:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.71, 3.86, 4.05 [03:41:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.85, 10.26, 9.47 [03:41:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.87, 9.75, 8.52 [03:43:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.84, 9.93, 9.46 [03:43:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.37, 9.40, 8.56 [03:45:36] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.04, 1.95, 1.95 [03:47:31] PROBLEM - mw112 NTP time on mw112 is WARNING: NTP WARNING: Offset -0.1071713865 secs [03:47:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.54, 1.71, 1.86 [03:49:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.89, 9.45, 10.18 [03:53:32] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.31, 1.40, 1.66 [03:53:34] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:38] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:49] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:56] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:58] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:58] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:01] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [03:54:15] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:17] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:22] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:25] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:28] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:28] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:29] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:31] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:46] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:47] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:54:53] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:53] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:58] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:55:10] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:55:10] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:55:37] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.410 second response time [03:55:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.36, 11.48, 10.82 [03:55:44] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.827 second response time [03:55:45] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [03:55:55] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.106 second response time [03:55:57] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.482 second response time [03:55:59] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.602 second response time [03:56:04] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.674 second response time [03:56:15] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [03:56:17] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.038 second response time [03:56:22] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.026 second response time [03:56:24] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 1.119 second response time [03:56:24] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.120 second response time [03:56:27] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [03:56:29] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.448 second response time [03:56:29] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [03:56:45] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [03:56:47] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.281 second response time [03:56:51] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw102 [03:56:52] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.484 second response time [03:56:54] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.441 second response time [03:56:54] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 4.914 second response time [03:57:07] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.070 second response time [03:57:10] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.329 second response time [03:57:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.22, 1.95, 1.82 [03:57:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.61, 10.11, 10.41 [03:58:51] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [03:59:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.65, 1.84, 1.79 [04:04:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.36, 10.24, 9.08 [04:06:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.95, 9.43, 8.91 [04:07:23] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.71, 1.34, 1.61 [04:09:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.20, 9.60, 10.18 [04:18:30] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.46, 1.71, 1.44 [04:20:28] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.00, 1.51, 1.40 [04:22:06] RECOVERY - mw112 NTP time on mw112 is OK: NTP OK: Offset -0.09539541602 secs [04:22:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.86, 1.59 [04:31:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.97, 1.80, 1.65 [04:35:06] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.56, 1.54, 1.57 [04:47:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.03, 1.36, 0.94 [04:49:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.93, 1.65, 1.10 [04:49:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.79, 10.77, 9.94 [04:51:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.24, 2.17, 1.37 [04:51:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.69, 9.91, 9.73 [04:53:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.77, 1.66, 1.28 [04:55:53] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.94, 1.73, 1.46 [04:57:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.09, 1.54, 1.42 [04:58:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.29, 11.34, 10.41 [05:00:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.86, 11.32, 10.51 [05:03:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.01, 11.32, 10.66 [05:05:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.91, 11.71, 10.89 [05:08:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.62, 1.85, 1.99 [05:13:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.39, 9.06, 9.90 [05:16:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.17, 1.34, 1.08 [05:18:26] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.41, 1.31, 1.10 [05:22:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.29, 1.45, 1.68 [05:53:46] PROBLEM - mw112 NTP time on mw112 is WARNING: NTP WARNING: Offset -0.1020084918 secs [05:56:15] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.69, 1.79, 1.32 [05:56:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.89, 10.61, 9.25 [05:58:11] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.31, 1.96, 1.44 [06:00:08] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.68, 1.49, 1.33 [06:03:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.59, 3.93 [06:03:56] RECOVERY - mw112 NTP time on mw112 is OK: NTP OK: Offset -0.0963049829 secs [06:04:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.33, 11.12, 9.98 [06:06:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.01, 11.06, 10.09 [06:06:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.40, 2.20, 1.68 [06:07:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.77, 4.16, 4.06 [06:10:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.13, 9.87, 9.82 [06:19:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.66, 3.99 [06:21:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 10.56, 10.00 [06:22:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.91, 10.81, 10.15 [06:24:16] [02dns] 07Universal-Omega opened pull request 03#307: Add arcanewaterswiki.org zone - 13https://github.com/miraheze/dns/pull/307 [06:24:16] [url] Page not found · GitHub · GitHub | github.com [06:24:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.27, 11.20, 10.36 [06:26:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.59, 11.23, 10.47 [06:30:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.66, 4.00 [06:31:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.66, 3.96, 3.74 [06:32:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.24, 3.79, 4.02 [06:34:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.98, 11.30, 10.70 [06:36:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.60, 11.84, 11.00 [06:44:34] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [06:46:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.09, 8.97, 9.92 [06:53:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.20, 10.30, 10.18 [06:55:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.67, 10.18, 10.16 [06:57:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.91, 4.00 [06:59:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.89, 4.25, 4.14 [07:07:03] PROBLEM - roblox-wiki.tk - LetsEncrypt on sslhost is CRITICAL: connect to address roblox-wiki.tk and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:07:06] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.36, 2.96, 1.92 [07:11:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.05, 1.83, 1.69 [07:13:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.56, 1.38, 1.54 [07:22:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 10.23, 9.32 [07:24:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.01, 10.28, 9.47 [07:26:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.85, 9.78, 9.38 [07:36:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.54, 10.03, 9.53 [07:36:48] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.40, 1.99, 1.57 [07:38:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.94, 1.58, 1.47 [07:40:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.48, 9.57, 9.50 [07:53:30] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.43, 2.29, 1.71 [07:55:26] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.34, 1.93, 1.64 [07:59:20] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.33, 1.56, 1.56 [08:03:10] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.53, 2.10, 1.77 [08:05:56] RECOVERY - roblox-wiki.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'roblox-wiki.tk' will expire on Thu 27 Oct 2022 15:10:45 GMT +0000. [08:13:49] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [08:15:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.10, 1.67, 1.84 [08:20:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.98, 10.67, 9.80 [08:24:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.54, 11.48, 10.35 [08:26:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 11.79, 10.60 [08:29:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.90, 1.80, 1.77 [08:30:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.15, 11.74, 10.85 [08:43:53] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.63, 8.63, 9.79 [08:45:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.50, 1.65, 1.98 [08:47:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.58, 1.83, 2.02 [08:48:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.40, 1.55, 1.97 [08:49:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.37, 1.57, 1.90 [08:53:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.07, 1.27, 1.70 [08:57:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.40, 1.84, 1.82 [08:58:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.37, 1.36, 1.67 [09:02:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 1.61, 1.68 [09:04:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.36, 1.47, 1.62 [09:07:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.08, 1.69, 1.91 [09:11:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.23, 1.53, 1.77 [09:29:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.70, 1.39, 1.85 [09:33:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.70, 1.06, 1.61 [09:33:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.26, 10.19, 9.64 [09:34:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 1.78, 1.53 [09:35:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 10.75, 9.90 [09:37:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.39, 10.57, 9.93 [09:39:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.53, 2.47, 1.97 [09:39:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.50, 10.09, 9.83 [09:42:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.93, 1.99, 1.77 [09:43:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.21, 1.79, 1.82 [09:46:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.20, 2.03, 1.84 [09:51:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.22, 10.18, 9.91 [09:53:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.04, 11.50, 10.44 [09:55:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.84, 1.88, 1.80 [09:55:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 11.54, 10.59 [09:59:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.17, 9.95, 10.14 [10:01:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.38, 1.95, 1.93 [10:03:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.45, 10.17, 10.18 [10:07:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 5.21, 2.97, 2.26 [10:09:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.46, 9.04, 9.77 [10:11:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.94, 1.89, 1.98 [10:14:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.92, 1.99 [10:15:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.29, 2.03, 2.01 [10:16:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 2.07, 2.04 [10:17:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.35, 1.76, 1.91 [10:18:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.92, 1.98 [10:19:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.12, 2.12, 2.03 [10:21:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.40, 1.83, 1.93 [10:27:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.03, 1.37, 1.68 [10:30:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.47, 1.70 [10:47:42] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.73, 1.98, 1.61 [10:49:39] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.08, 1.57, 1.50 [10:53:29] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.60, 1.75, 1.61 [10:55:26] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.80, 2.20, 1.77 [11:01:16] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.82, 1.93, 1.87 [11:05:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.25, 1.50, 1.70 [11:18:56] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.34, 1.92, 1.59 [11:25:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 10.34, 9.43 [11:27:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.83, 1.82, 1.57 [11:27:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.20, 9.75, 9.32 [11:28:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.95, 1.81 [11:29:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.81, 1.91, 1.62 [11:34:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.35, 2.10, 1.92 [11:37:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.68, 3.96 [11:39:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.11, 4.07, 4.06 [11:41:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.83, 3.97 [11:43:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.34, 4.72, 4.25 [11:45:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.30, 1.69, 1.93 [11:49:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.04, 2.11, 1.99 [11:53:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.33, 9.93, 9.41 [11:55:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.41, 1.91, 2.00 [11:57:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.93, 1.98, 2.01 [11:57:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.99, 9.43, 9.33 [11:59:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.61, 1.96, 2.00 [11:59:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.79, 4.00 [12:01:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.22, 2.09, 2.04 [12:02:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.91, 1.99 [12:03:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.00, 1.76, 1.94 [12:03:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.88, 3.96 [12:05:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.82, 3.94 [12:11:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.86, 3.96, 3.89 [12:12:38] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [12:13:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.04, 1.29, 1.62 [12:22:38] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [12:22:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.41, 1.50, 1.67 [12:23:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.21, 3.91, 3.98 [12:25:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.50, 5.08, 4.39 [12:25:59] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.81, 1.62, 1.57 [12:27:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.94, 1.42, 1.51 [12:31:45] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.28, 1.71, 1.64 [12:37:35] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.21, 1.54, 1.61 [12:37:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 11.16, 10.05 [12:39:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.26, 10.49, 9.94 [12:47:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.34, 10.15, 10.10 [12:55:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.54, 10.98, 10.44 [12:55:59] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.32, 2.40, 1.83 [12:59:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.38, 10.03, 10.17 [12:59:52] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.01, 1.80, 1.72 [13:00:47] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [13:01:49] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.10, 1.95, 1.79 [13:03:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.55, 1.91, 1.56 [13:03:46] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.77, 1.93, 1.81 [13:05:42] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.96, 1.56, 1.68 [13:05:47] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [13:09:32] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.56, 1.69, 1.71 [13:11:29] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.23, 1.52, 1.64 [13:15:19] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.81, 1.79, 1.72 [13:19:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.83, 2.00, 1.81 [13:21:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.73, 1.86, 1.78 [13:25:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.02, 1.79, 1.76 [13:29:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.12, 1.67, 1.74 [13:29:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.32, 11.18, 10.11 [13:31:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.30, 10.03, 9.81 [13:33:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.42, 1.43, 1.61 [13:51:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 10.16, 9.74 [13:53:18] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.23, 9.88, 9.69 [13:54:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.90, 1.98 [14:06:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.39, 1.47, 1.68 [14:07:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.43, 3.91 [14:09:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.75, 4.06, 4.07 [14:12:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.50, 10.07, 9.11 [14:14:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.52, 10.01, 9.21 [14:15:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.74, 3.95 [14:21:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.87, 4.25, 4.04 [14:25:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.22, 3.58, 3.86 [14:25:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.42, 10.34, 9.72 [14:25:53] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.20, 2.25, 1.63 [14:27:50] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.02, 1.80, 1.54 [14:29:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 3.98, 3.95 [14:29:46] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.10, 1.70, 1.54 [14:33:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.16, 9.87, 9.81 [14:37:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.04, 10.49, 10.08 [14:39:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.09, 3.66, 3.85 [14:39:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.45, 10.00, 9.95 [14:41:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.93, 3.85, 3.88 [14:42:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.26, 9.75, 8.96 [14:44:06] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.82, 9.00, 8.78 [14:45:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.78, 3.89 [14:47:33] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.48, 1.91, 1.58 [14:50:08] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [14:56:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.93, 10.39, 10.10 [14:57:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.94, 1.81 [14:57:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.07, 4.13, 3.85 [14:59:07] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.36, 2.09, 1.88 [14:59:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.53, 1.86, 1.51 [15:00:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.37, 11.64, 10.64 [15:01:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.79, 1.76, 1.51 [15:02:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.27, 11.60, 10.75 [15:04:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.36, 12.36, 11.12 [15:05:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.33, 1.64, 1.52 [15:07:10] PROBLEM - txt-dex.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'txt-dex.com' expires in 15 day(s) (Tue 16 Aug 2022 14:45:57 GMT +0000). [15:08:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.06, 11.79, 11.18 [15:10:21] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.13, 10.47, 9.53 [15:12:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.85, 12.21, 11.48 [15:13:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/59c873ca7028...3d14c1e9f461 [15:13:41] [url] Comparing 59c873ca7028...3d14c1e9f461 · miraheze/ssl · GitHub | github.com [15:13:42] [02miraheze/ssl] 07MirahezeSSLBot 033d14c1e - Bot: Update SSL cert for txt-dex.com [15:14:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.27, 11.61, 11.33 [15:14:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.84, 11.42, 10.14 [15:18:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.36, 10.00, 9.85 [15:21:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.40, 11.54, 11.21 [15:21:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.51, 10.27, 10.03 [15:23:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.28, 11.65, 11.31 [15:27:56] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 7.50, 7.05, 3.72 [15:30:13] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:14] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:18] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.85, 2.00 [15:31:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.54, 11.78, 11.41 [15:32:17] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.374 second response time [15:32:18] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.948 second response time [15:32:20] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.066 second response time [15:33:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.97, 11.69, 11.42 [15:35:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.61, 9.44, 9.94 [15:35:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 12.12, 11.61 [15:36:21] RECOVERY - txt-dex.com - LetsEncrypt on sslhost is OK: OK - Certificate 'txt-dex.com' will expire on Sat 29 Oct 2022 14:13:35 GMT +0000. [15:36:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.94, 2.00 [15:37:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.84, 11.26, 11.35 [15:37:57] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.236 second response time [15:38:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.80, 1.94 [15:38:45] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:45] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:49] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.80, 11.72, 11.50 [15:39:57] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.605 second response time [15:40:40] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [15:40:44] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.041 second response time [15:40:48] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.022 second response time [15:41:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.09, 10.94, 10.38 [15:41:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.15, 11.16, 11.32 [15:43:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.34, 10.53, 10.32 [15:45:08] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [15:45:26] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.91, 1.24, 1.93 [15:49:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.57, 0.97, 1.67 [15:49:21] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.09, 10.78, 10.45 [15:51:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.62, 9.99, 10.18 [15:51:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.03, 10.76, 10.87 [15:52:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.40, 1.50, 1.67 [15:53:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.80, 1.42, 1.70 [15:55:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.52, 1.47, 1.68 [15:59:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.88, 11.03, 11.16 [16:03:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.31, 1.79, 1.70 [16:05:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.27, 11.15, 11.10 [16:07:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.97, 1.56, 1.65 [16:07:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.27, 10.88, 11.01 [16:12:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.96, 11.10, 10.35 [16:12:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.96, 10.18, 9.38 [16:13:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.00, 10.36, 10.16 [16:14:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.18, 9.08, 9.08 [16:17:08] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [16:17:51] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.42, 9.49, 9.87 [16:20:24] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:33] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:37] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.234 second response time [16:24:28] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.700 second response time [16:25:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.86, 11.77, 11.00 [16:27:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.90, 1.59, 1.40 [16:27:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.89, 11.29, 10.91 [16:29:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.82, 1.35, 1.34 [16:29:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.41, 11.79, 11.14 [16:30:54] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.78, 1.50 [16:31:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.31, 11.44, 11.09 [16:33:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.35, 11.49, 11.14 [16:36:03] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.26, 11.56, 11.27 [16:37:59] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.001 second response time [16:42:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.98, 1.82 [16:44:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.30, 2.07, 1.87 [16:47:52] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.18, 1.86, 1.35 [16:51:52] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.69, 1.38, 1.29 [16:57:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.13, 2.11, 1.61 [16:59:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 2.00, 1.93, 1.60 [17:01:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.66, 2.08, 1.69 [17:03:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.17, 1.74, 1.61 [17:03:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.71, 11.20, 10.94 [17:05:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.18, 1.50, 1.54 [17:05:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.71, 11.46, 11.07 [17:07:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.72, 11.98, 11.32 [17:09:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.90, 11.26, 11.14 [17:18:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.01, 9.85, 9.41 [17:24:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.85, 1.98 [17:25:42] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.04, 1.82, 1.54 [17:25:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.89, 9.33, 9.46 [17:27:39] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.79, 1.85, 1.58 [17:29:35] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.60, 1.39, 1.44 [17:31:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.00, 10.22, 9.83 [17:32:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.35, 9.80, 9.02 [17:34:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.78, 9.18, 8.90 [17:38:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.42, 1.65 [17:41:26] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.48, 11.37, 10.55 [17:43:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.73, 10.60, 10.39 [17:47:06] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.79, 2.13, 1.63 [17:49:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.15, 1.78, 1.56 [17:53:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.63, 10.90, 10.59 [17:55:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.50, 11.09, 10.70 [17:57:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.92, 1.45, 1.54 [17:57:52] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.31, 1.79, 1.39 [17:59:52] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.24, 1.61, 1.37 [18:01:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.01, 10.93, 10.66 [18:05:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.89, 9.77, 10.12 [18:05:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.62, 11.37, 10.88 [18:08:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.76, 1.56 [18:11:21] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.32, 10.63, 10.33 [18:11:23] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:33] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:40] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:44] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:16] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:19] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [18:13:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.83, 9.75, 10.04 [18:13:32] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.111 second response time [18:13:39] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.791 second response time [18:13:40] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.317 second response time [18:14:21] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.204 second response time [18:15:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.39, 3.96 [18:16:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.33, 8.81, 8.57 [18:18:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.35, 8.71, 8.57 [18:19:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 11.78, 11.32 [18:23:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.89, 11.97, 11.51 [18:25:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.43, 12.17, 11.62 [18:27:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.83, 3.89 [18:27:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 11.99, 11.62 [18:29:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.69, 3.82 [18:29:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.09, 12.13, 11.72 [18:31:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.97, 3.90 [18:31:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.66, 11.93, 11.70 [18:32:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.75, 3.34, 3.91 [18:35:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.96, 11.82, 11.66 [18:35:52] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:37] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:55] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.603 second response time [18:38:43] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.080 second response time [18:39:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.91, 1.61, 1.36 [18:39:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.78, 11.53, 11.61 [18:40:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.45, 3.78, 3.83 [18:40:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.04, 2.15, 1.46 [18:41:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.92, 1.35, 1.29 [18:42:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.17, 3.48, 3.71 [18:42:34] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.78, 1.64, 1.35 [18:43:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.63, 12.08, 11.76 [18:45:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.45, 11.06, 11.43 [18:48:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.32, 3.77, 3.73 [18:49:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.94, 11.68, 11.58 [18:51:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.97, 11.38, 11.45 [18:53:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 11.54, 11.50 [18:56:16] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:30] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:30] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.40, 11.81, 11.65 [18:59:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.83, 11.92, 11.69 [19:01:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.45, 11.68, 11.64 [19:02:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.55, 3.87 [19:02:31] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.256 second response time [19:02:52] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.339 second response time [19:02:53] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.105 second response time [19:05:21] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.24, 9.83, 8.98 [19:05:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.71, 11.73, 11.63 [19:07:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.03, 10.09, 9.17 [19:08:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 3.74, 3.82 [19:11:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.09, 11.34, 11.63 [19:14:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.00, 3.43, 3.72 [19:15:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.61, 10.18, 9.62 [19:18:12] !log cloud10: cp /etc/pve/corosync.conf.new /etc/pve/corosync.conf && restart corosync [19:18:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:38] !log cloud11 & 12 restart corosync [19:20:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.80, 4.41, 3.97 [19:20:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:21:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 10.99, 11.08 [19:22:47] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.19, 1.55, 1.23 [19:23:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.88, 10.84, 11.01 [19:24:45] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.08, 1.29, 1.16 [19:27:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 11.48, 11.21 [19:29:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.37, 10.94, 11.04 [19:35:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.89, 11.59, 11.22 [19:39:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.56, 11.45, 11.28 [19:41:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.27, 9.66, 8.91 [19:43:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.02, 9.37, 8.89 [19:51:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.69, 9.14, 10.11 [19:52:11] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:16] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:28] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:46] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:08] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:19] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.023 second response time [19:55:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.66, 11.22, 10.78 [19:56:13] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.786 second response time [19:56:23] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.533 second response time [19:59:34] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.35, 1.93, 1.41 [20:00:00] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.321 second response time [20:01:36] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.827 second response time [20:03:28] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.91, 1.51, 1.37 [20:04:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.64, 10.03, 9.36 [20:05:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.11, 11.41, 10.89 [20:06:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.85, 9.58, 9.28 [20:07:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.22, 11.21, 10.90 [20:10:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.81, 10.54, 9.73 [20:12:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.27, 11.19, 10.07 [20:14:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.77, 10.99, 10.13 [20:17:09] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [20:17:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 11.03, 10.78 [20:19:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.98, 11.08, 10.81 [20:21:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.12, 11.70, 11.07 [20:22:06] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.64, 10.20, 10.11 [20:23:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.99, 11.16, 10.98 [20:25:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.41, 11.55, 11.14 [20:27:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.82, 1.57, 1.40 [20:27:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.77, 10.25, 10.13 [20:30:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.31, 10.64, 10.91 [20:30:29] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.74, 1.22, 1.29 [20:31:28] PROBLEM - cloud14 SMART on cloud14 is WARNING: WARNING: [cciss,0] - No health status line found, Self-test log contains errors --- [cciss,1] - No health status line found, Self-test log contains errors --- [cciss,2] - No health status line found, Self-test log contains errors --- [cciss,3] - No health status line found, Self-test log contains errors --- [cciss,4] - No health status line found, Self-test log contains errors --- [cciss,5 [20:31:28] th status line found, Self-test log contains errors --- [cciss,6] - No health status line found, Self-test log contains errors| [20:31:36] PROBLEM - cloud14 IPMI Sensors on cloud14 is UNKNOWN: Error: ipmimonitoring/ipmi-sensors command not found! [20:34:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 11.52, 11.22 [20:34:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.69, 9.43, 9.87 [20:37:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.40, 3.17, 3.88 [20:39:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.47, 11.53, 11.41 [20:42:09] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [20:43:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.06, 1.85, 1.40 [20:43:33] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [20:45:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.67, 1.77, 1.42 [20:45:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.89, 3.44, 3.65 [20:46:59] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.40, 10.65, 10.00 [20:47:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.04, 1.54, 1.38 [20:47:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.62, 9.93, 8.86 [20:49:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.61, 9.03, 8.66 [20:52:52] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.69, 9.78, 9.89 [20:55:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.75, 3.83 [20:55:44] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:52] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:54] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:57] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.238 second response time [20:58:59] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 7.071 second response time [20:59:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 10.97, 10.79 [20:59:46] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.649 second response time [20:59:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.32, 1.55, 1.95 [21:01:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.92, 10.71, 10.71 [21:03:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.39, 8.86, 10.04 [21:03:54] PROBLEM - prometheus101 Current Load on prometheus101 is CRITICAL: CRITICAL - load average: 2.94, 2.08, 1.08 [21:05:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.63, 4.13, 3.91 [21:05:51] RECOVERY - prometheus101 Current Load on prometheus101 is OK: OK - load average: 0.80, 1.54, 1.00 [21:09:38] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.22, 1.32, 1.64 [21:10:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.33, 10.16, 9.78 [21:12:40] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:14:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.73, 11.21, 10.27 [21:15:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.71, 3.90 [21:16:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.28, 11.35, 10.45 [21:17:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.80, 10.20, 9.32 [21:19:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.83, 3.82, 3.86 [21:19:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.90, 10.10, 9.94 [21:21:06] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.13, 9.50, 9.30 [21:21:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.89, 3.52, 3.75 [21:21:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.28, 1.86, 1.71 [21:23:38] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.18, 1.58, 1.62 [21:23:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.42, 9.84, 9.88 [21:24:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.84, 9.58, 10.04 [21:25:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.32, 3.43, 3.63 [21:25:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.82, 10.09, 9.22 [21:27:30] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.13, 9.52, 9.10 [21:28:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.44, 10.45, 10.34 [21:29:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.84, 3.76 [21:30:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.40, 9.81, 10.13 [21:33:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.13, 10.99, 10.46 [21:35:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.87, 3.83, 3.73 [21:37:22] !log change retention on graylog to 14 days [21:37:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.83, 3.79 [21:39:38] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [21:41:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.61, 4.33, 3.97 [21:41:48] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [21:43:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.38, 9.39, 10.04 [21:43:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.72, 1.53 [21:47:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.67, 3.81 [21:49:04] !log change retention on graylog to 10 days [21:49:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.91, 4.24, 3.99 [21:54:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.25, 10.29, 9.93 [21:55:54] !log create es131/141 on cloud13 & 14 with 2 cores, 4g of ram and 210G of disk [21:56:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:56:28] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.12, 9.40, 9.65 [21:57:58] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.94, 1.42, 0.94 [21:59:23] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:29] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:51] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.95, 1.16, 0.90 [22:00:43] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:20] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.312 second response time [22:01:32] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.139 second response time [22:01:49] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.439 second response time [22:01:57] PROBLEM - mw101 NTP time on mw101 is CRITICAL: NTP CRITICAL: Offset 0.9778315425 secs [22:02:41] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.371 second response time [22:03:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.17, 3.44, 3.91 [22:06:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.35, 10.51, 10.13 [22:08:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.34, 10.10, 10.03 [22:09:38] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [22:13:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.45, 3.61 [22:17:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.67, 3.58, 3.64 [22:21:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.53, 4.07, 3.82 [22:21:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 10.19, 9.96 [22:23:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.73, 3.58, 3.68 [22:23:23] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.72, 10.13, 9.97 [22:27:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.00, 4.48, 4.00 [22:33:12] RECOVERY - mw101 NTP time on mw101 is OK: NTP OK: Offset 0.000146061182 secs [22:34:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.00, 10.37, 9.58 [22:35:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.09, 10.37, 10.03 [22:37:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.75, 1.93, 2.00 [22:39:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.72, 9.05, 9.62 [22:39:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.33, 3.60, 3.86 [22:40:26] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.33, 9.97, 9.67 [22:43:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.01, 4.01, 3.93 [22:44:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.81, 10.82, 10.15 [22:47:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.95, 3.93 [22:48:14] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.42, 10.08, 10.00 [22:51:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.82, 4.16, 3.98 [22:53:38] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.29, 1.44, 1.65 [23:04:03] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 11.23, 6.49, 3.57 [23:05:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.96, 10.54, 10.01 [23:05:26] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/1eea61b4d21f...cad2bd595b3c [23:05:27] [url] Comparing 1eea61b4d21f...cad2bd595b3c · miraheze/dns · GitHub | github.com [23:05:28] [02miraheze/dns] 07paladox 03cad2bd5 - Add es131 and es141 to dns [23:05:49] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:22] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:24] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:25] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:53] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:07:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 11.10, 10.27 [23:07:53] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.417 second response time [23:08:17] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.342 second response time [23:08:20] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.816 second response time [23:08:28] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.537 second response time [23:08:51] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.337 second response time [23:09:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.53, 10.79, 10.26 [23:12:03] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.14, 5.76, 4.76 [23:14:03] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.84, 4.07, 4.26 [23:21:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.84, 1.67, 1.40 [23:23:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/d98f3217dbd5 [23:23:01] [02miraheze/puppet] 07paladox 03d98f321 - Install es131 / es141 [23:23:03] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [23:23:03] [url] Page not found · GitHub · GitHub | github.com [23:23:04] [02puppet] 07paladox opened pull request 03#2715: Install es131 / es141 - 13https://github.com/miraheze/puppet/pull/2715 [23:23:05] [url] Page not found · GitHub · GitHub | github.com [23:23:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.63, 1.66, 1.43 [23:23:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/d98f3217dbd5...9b863518bcb8 [23:23:44] [url] Comparing d98f3217dbd5...9b863518bcb8 · miraheze/puppet · GitHub | github.com [23:23:45] [02miraheze/puppet] 07paladox 039b86351 - Create es141.yaml [23:23:47] [02puppet] 07paladox synchronize pull request 03#2715: Install es131 / es141 - 13https://github.com/miraheze/puppet/pull/2715 [23:23:47] [url] Page not found · GitHub · GitHub | github.com [23:24:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9b863518bcb8...650938241977 [23:24:08] [url] Comparing 9b863518bcb8...650938241977 · miraheze/puppet · GitHub | github.com [23:24:09] [02miraheze/puppet] 07paladox 036509382 - Update es121.yaml [23:24:11] [02puppet] 07paladox synchronize pull request 03#2715: Install es131 / es141 - 13https://github.com/miraheze/puppet/pull/2715 [23:24:12] [url] Page not found · GitHub · GitHub | github.com [23:24:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/650938241977...3f032be57d71 [23:24:25] [url] Comparing 650938241977...3f032be57d71 · miraheze/puppet · GitHub | github.com [23:24:26] [02miraheze/puppet] 07paladox 033f032be - Update site.pp [23:24:27] [02puppet] 07paladox synchronize pull request 03#2715: Install es131 / es141 - 13https://github.com/miraheze/puppet/pull/2715 [23:24:27] [url] Page not found · GitHub · GitHub | github.com [23:25:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.09, 10.94, 10.52 [23:27:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.38, 10.00, 10.23 [23:27:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.28, 1.75, 1.52 [23:30:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.92, 9.61, 10.02 [23:32:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.53, 3.93 [23:36:22] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 4.06, 4.06 [23:38:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.76, 3.91, 4.00 [23:42:03] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.45, 1.79, 1.58 [23:42:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.87, 4.20, 4.07 [23:44:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.48, 1.49 [23:46:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.94, 10.18, 9.90 [23:48:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.28, 10.72, 10.11 [23:49:12] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.08, 1.74, 1.39 [23:50:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.43, 10.88, 10.24 [23:51:07] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.75, 1.70, 1.42 [23:52:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.79, 12.09, 10.75 [23:52:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.34, 1.94, 1.68 [23:53:02] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.14, 1.47, 1.36 [23:54:44] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.66, 1.60 [23:55:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.00, 11.67, 10.90