[00:00:00] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.45, 7.38, 7.68 [00:00:01] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.78, 7.35, 9.97 [00:00:15] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 4.49, 7.05, 9.51 [00:01:17] RECOVERY - istpcomputing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'istpcomputing.com' will expire on Sat 07 Oct 2023 17:48:31 GMT +0000. [00:01:17] RECOVERY - wiki.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.graalmilitary.com' will expire on Thu 05 Oct 2023 12:11:40 GMT +0000. [00:01:29] RECOVERY - wiki.anglish.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.anglish.info' will expire on Thu 05 Oct 2023 21:58:09 GMT +0000. [00:01:44] RECOVERY - dragonquestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dragonquestwiki.com' will expire on Fri 06 Oct 2023 18:17:02 GMT +0000. [00:02:08] RECOVERY - gtfo-wiki.cn - LetsEncrypt on sslhost is OK: OK - Certificate 'gtfo-wiki.cn' will expire on Mon 23 Oct 2023 09:23:07 GMT +0000. [00:10:40] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.83, 3.77, 3.96 [00:12:55] RECOVERY - bushcraftwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'bushcraftwiki.com' will expire on Tue 24 Oct 2023 15:04:26 GMT +0000. [00:12:57] RECOVERY - pastport.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pastport.org' will expire on Fri 06 Oct 2023 20:11:53 GMT +0000. [00:13:05] RECOVERY - www.rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Fri 06 Oct 2023 03:13:28 GMT +0000. [00:13:25] PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 0.60, 1.03, 3.95 [00:14:01] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.60, 5.94, 6.72 [00:14:38] RECOVERY - smashbroswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'smashbroswiki.com' will expire on Fri 06 Oct 2023 18:12:02 GMT +0000. [00:16:52] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.70, 6.48, 6.73 [00:17:25] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 0.92, 0.87, 3.20 [00:23:00] PROBLEM - cp32 Current Load on cp32 is WARNING: LOAD WARNING - total load average: 0.75, 0.85, 3.59 [00:24:32] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.20, 0.75, 3.87 [00:25:00] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 0.97, 0.92, 3.28 [00:26:40] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 4.02, 3.64, 3.66 [00:28:01] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 0.35, 1.19, 3.92 [00:28:32] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.17, 0.45, 3.03 [00:28:40] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.19, 3.56, 3.63 [00:32:01] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.30, 0.81, 3.14 [00:36:40] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 3.00, 2.97, 3.33 [00:48:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.85, 6.63, 7.75 [00:56:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.81, 5.60, 6.75 [01:00:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.29, 8.11, 7.59 [01:03:36] PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 3.73, 3.02, 2.03 [01:04:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.01, 6.60, 7.71 [01:05:31] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 1.12, 2.29, 1.87 [01:06:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.14, 7.42, 7.54 [01:16:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.69, 7.54 [01:18:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 7.76, 7.59 [01:20:19] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.03, 6.12, 6.70 [01:28:38] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.42, 5.99, 6.74 [01:29:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:30:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.99, 6.23, 6.29 [01:34:19] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.77, 6.48, 6.38 [01:36:15] PROBLEM - cp23 NTP time on cp23 is CRITICAL: NTP CRITICAL: Offset 0.5000528693 secs [01:36:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.39, 6.74, 6.81 [01:38:38] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.78, 6.38, 6.67 [01:46:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.62, 6.91, 6.77 [01:48:38] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.50, 6.31, 6.56 [01:56:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.15, 6.50, 6.11 [01:58:16] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [01:58:19] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.49, 5.96, 5.95 [02:20:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.22, 7.56, 6.66 [02:22:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.18, 6.80, 6.50 [02:31:17] !log [void@puppet141] banned a set of IP addresses in firewall due to suspicious activity [02:31:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:10] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [03:15:50] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:02:48] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.53, 2.84, 2.22 [04:03:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.73, 3.62, 3.00 [04:04:46] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 1.74, 2.34, 2.10 [04:05:20] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.08, 3.75, 3.13 [04:07:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.16, 3.52, 3.12 [04:19:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.19, 3.36, 3.26 [04:35:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.16, 7.03, 6.03 [04:35:55] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 41% [04:36:58] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.75, 6.06, 5.79 [04:37:50] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 21% [04:47:00] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14609 MB (10% inode=99%); [06:16:23] PROBLEM - mcdev.grantlmul.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mcdev.grantlmul.xyz could not be found [06:21:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.30, 3.73, 3.08 [06:23:20] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.44, 3.53, 3.08 [06:28:01] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 2.57, 3.44, 1.88 [06:30:01] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 1.10, 2.59, 1.76 [06:31:16] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.07, 3.35, 3.20 [06:44:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 4.00, 3.61, 3.34 [06:54:05] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 3.93, 4.04, 3.70 [06:56:04] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.51, 3.89, 3.69 [06:58:03] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.13, 3.91, 3.72 [07:00:02] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.56, 3.79, 3.70 [07:09:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.01, 3.71, 3.64 [07:11:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.86, 3.47, 3.56 [07:15:55] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.13, 2.82, 3.29 [07:47:10] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [08:15:49] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [09:04:58] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.02126944065 secs [09:25:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: LOAD CRITICAL - total load average: 27.69, 17.65, 7.26 [09:25:37] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [09:25:40] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [09:25:46] PROBLEM - smashbros.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:51] PROBLEM - cp22 HTTPS on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:53] PROBLEM - www.mh142.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 95.17, 59.31, 23.85 [09:25:58] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:00] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [09:26:00] PROBLEM - history.estill.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:01] PROBLEM - allthetropes.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:05] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 74.36, 77.53, 34.00 [09:26:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 217.174.247.33/cpweb, 109.228.51.216/cpweb, 2a00:da00:1800:326::1/cpweb, 2a00:da00:1800:328::1/cpweb, 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb, 2607:f1c0:1800:26f::1/cpweb [09:26:06] PROBLEM - cp23 HTTPS on cp23 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:09] PROBLEM - cp32 Stunnel for puppet141 on cp32 is CRITICAL: connect to address localhost and port 8204: Connection refused [09:26:11] PROBLEM - lgbta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:13] PROBLEM - closinglogosgroup.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 217.174.247.33/cpweb, 109.228.51.216/cpweb, 2a00:da00:1800:326::1/cpweb, 2a00:da00:1800:328::1/cpweb, 108.175.15.182/cpweb, 74.208.203.152/cpweb [09:26:14] PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:14] PROBLEM - www.johanloopmans.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:18] PROBLEM - www.sidem.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:22] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 8191 bytes in 4.142 second response time [09:26:27] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:27] PROBLEM - www.allthetropes.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:27] PROBLEM - miraheze.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:28] PROBLEM - wiki.rsf.world - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:30] PROBLEM - iceria.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:30] PROBLEM - cp33 HTTPS on cp33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:30] PROBLEM - franchise.franchising.org.ua - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:31] PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:32] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 11.57, 15.51, 9.11 [09:26:32] PROBLEM - en.omniversalis.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:32] PROBLEM - wiki.overwood.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:34] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 42.54, 26.18, 11.29 [09:26:37] PROBLEM - elsterclopedie.marithaime.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:37] PROBLEM - journeytheword.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:38] PROBLEM - fanon.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:39] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:41] PROBLEM - wiki.denby.tech - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:48] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [09:26:54] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:05] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 13.44, 7.75, 3.52 [09:27:06] PROBLEM - cp32 Stunnel for mw143 on cp32 is CRITICAL: connect to address localhost and port 8112: Connection refused [09:27:06] PROBLEM - cp32 Stunnel for phab121 on cp32 is CRITICAL: connect to address localhost and port 8202: Connection refused [09:27:08] PROBLEM - cp32 Stunnel for mw142 on cp32 is CRITICAL: connect to address localhost and port 8109: Connection refused [09:27:09] PROBLEM - swiftproxy111 Current Load on swiftproxy111 is CRITICAL: CRITICAL - load average: 28.66, 13.78, 5.75 [09:27:26] PROBLEM - cp32 Stunnel for mon141 on cp32 is CRITICAL: connect to address localhost and port 8201: Connection refused [09:27:38] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 31% [09:27:47] RECOVERY - cp22 HTTPS on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3572 bytes in 0.136 second response time [09:27:50] PROBLEM - cp32 Stunnel for mail121 on cp32 is CRITICAL: connect to address localhost and port 8200: Connection refused [09:27:51] PROBLEM - cp32 Stunnel for mw131 on cp32 is CRITICAL: connect to address localhost and port 8106: Connection refused [09:27:57] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 9% [09:28:02] RECOVERY - cp23 HTTPS on cp23 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3571 bytes in 0.163 second response time [09:28:04] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [09:28:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:28:07] PROBLEM - cp32 Stunnel for mwtask141 on cp32 is CRITICAL: connect to address localhost and port 8150: Connection refused [09:28:09] PROBLEM - cp32 Stunnel for mw133 on cp32 is CRITICAL: connect to address localhost and port 8110: Connection refused [09:28:12] PROBLEM - cp32 Stunnel for mw132 on cp32 is CRITICAL: connect to address localhost and port 8107: Connection refused [09:28:14] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:28:18] PROBLEM - cp32 Stunnel for reports121 on cp32 is CRITICAL: connect to address localhost and port 8205: Connection refused [09:28:27] RECOVERY - cp33 HTTPS on cp33 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3733 bytes in 0.534 second response time [09:28:29] PROBLEM - cp32 Stunnel for mw141 on cp32 is CRITICAL: connect to address localhost and port 8108: Connection refused [09:28:34] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Sun 05 Nov 2023 16:43:26 GMT +0000. [09:28:36] PROBLEM - cp32 Stunnel for matomo121 on cp32 is CRITICAL: connect to address localhost and port 8203: Connection refused [09:28:44] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 27% [09:28:53] PROBLEM - cp32 Stunnel for test131 on cp32 is CRITICAL: connect to address localhost and port 8180: Connection refused [09:28:56] PROBLEM - cp32 Stunnel for mw134 on cp32 is CRITICAL: connect to address localhost and port 8111: Connection refused [09:29:05] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 2.64, 5.56, 3.23 [09:30:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.19, 6.82, 5.20 [09:32:32] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 4.04, 7.59, 7.53 [09:32:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.74, 7.37, 5.74 [09:34:19] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.21, 7.56, 5.98 [09:34:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.26, 6.93, 5.78 [09:36:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 3.25, 6.17, 5.68 [09:36:32] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 13.35, 9.34, 8.09 [09:36:37] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 3.64, 5.69, 5.46 [09:36:40] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 4.41, 3.59, 2.83 [09:36:44] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [09:36:59] PROBLEM - wiki.shaazzz.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:14] PROBLEM - wiki.puucraft.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:14] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:16] PROBLEM - studio.niuboss123.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:29] PROBLEM - translate.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:30] PROBLEM - wiki.litek.top - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:30] PROBLEM - vmklegacy.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:37] PROBLEM - wiki.yahyabd.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:37] PROBLEM - crustypedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:41] PROBLEM - wiki.potabi.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:41] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 96% [09:37:44] PROBLEM - kb.nena.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:47] PROBLEM - wiki.susqu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:47] PROBLEM - ping6 on swiftproxy111 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 104.08 ms [09:37:58] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [09:37:59] PROBLEM - cp22 HTTPS on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:00] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:00] PROBLEM - wiki.yuanpi.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:00] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:04] PROBLEM - corru.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 109.228.51.216/cpweb, 2a00:da00:1800:328::1/cpweb, 74.208.203.152/cpweb, 2607:f1c0:1800:26f::1/cpweb [09:38:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 109.228.51.216/cpweb, 74.208.203.152/cpweb, 2607:f1c0:1800:26f::1/cpweb [09:38:26] PROBLEM - cp32 Puppet on cp32 is UNKNOWN: NRPE: Unable to read output [09:38:40] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.90, 3.82, 3.02 [09:38:44] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 14% [09:39:38] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 5% [09:39:48] RECOVERY - ping6 on swiftproxy111 is OK: PING OK - Packet loss = 0%, RTA = 0.52 ms [09:39:55] RECOVERY - cp22 HTTPS on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3733 bytes in 0.134 second response time [09:40:12] PROBLEM - cp33 HTTPS on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [09:40:39] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 2.00, 3.27, 2.92 [09:42:32] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.82, 6.59, 7.47 [09:45:34] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 55% [09:46:06] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.04, 6.81, 4.10 [09:46:14] PROBLEM - cp22 HTTPS on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:15] PROBLEM - cp23 HTTPS on cp23 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:32] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.94, 4.72, 6.51 [09:46:40] PROBLEM - schizoidnightmares.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:42] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:46] PROBLEM - echoes-wiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:46] PROBLEM - heavyironmodding.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:48] PROBLEM - wiki.orvyn.ca - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:57] PROBLEM - wiki.otir.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:00] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 12.33, 7.33, 3.97 [09:47:06] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 15.38, 9.43, 5.51 [09:47:28] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 10.26, 8.23, 5.05 [09:47:29] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 10.75, 8.52, 5.20 [09:47:33] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 97% [09:47:38] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 95% [09:47:40] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.34, 10.03, 6.37 [09:48:06] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 7.12, 7.53, 4.75 [09:48:11] RECOVERY - cp23 HTTPS on cp23 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3734 bytes in 0.140 second response time [09:48:37] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Sun 05 Nov 2023 16:43:26 GMT +0000. [09:49:00] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 2.50, 5.23, 3.60 [09:49:06] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 4.72, 7.25, 5.18 [09:49:28] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 2.76, 6.12, 4.66 [09:49:29] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 2.98, 6.40, 4.83 [09:49:40] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 2.97, 7.39, 5.85 [09:53:57] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 7% [09:54:03] RECOVERY - cp33 HTTPS on cp33 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3572 bytes in 0.531 second response time [09:54:58] RECOVERY - www.project-patterns.com - LetsEncrypt on sslhost is OK: OK - Certificate 'project-patterns.com' will expire on Sun 05 Nov 2023 16:57:56 GMT +0000. [09:54:59] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Sun 05 Nov 2023 16:40:52 GMT +0000. [09:55:01] RECOVERY - wiki.rsf.world - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rsf.world' will expire on Wed 08 Nov 2023 17:39:39 GMT +0000. [09:55:01] RECOVERY - www.allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Thu 05 Oct 2023 01:35:28 GMT +0000. [09:55:06] RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Thu 05 Oct 2023 21:11:57 GMT +0000. [09:55:07] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Thu 05 Oct 2023 20:14:22 GMT +0000. [09:55:07] RECOVERY - franchise.franchising.org.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'franchise.franchising.org.ua' will expire on Thu 05 Oct 2023 20:49:16 GMT +0000. [09:55:07] RECOVERY - smashbros.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [09:55:07] RECOVERY - en.omniversalis.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.omniversalis.org' will expire on Sat 07 Oct 2023 05:05:00 GMT +0000. [09:55:08] RECOVERY - wiki.overwood.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.overwood.xyz' will expire on Tue 21 Nov 2023 15:11:54 GMT +0000. [09:55:18] RECOVERY - fanon.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fanon.polandballwiki.com' will expire on Fri 06 Oct 2023 18:04:46 GMT +0000. [09:55:18] RECOVERY - journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Fri 06 Oct 2023 13:02:47 GMT +0000. [09:55:19] RECOVERY - elsterclopedie.marithaime.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'elsterclopedie.marithaime.nl' will expire on Sat 07 Oct 2023 23:02:11 GMT +0000. [09:55:23] RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Thu 05 Oct 2023 12:30:38 GMT +0000. [09:55:29] RECOVERY - wiki.denby.tech - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.denby.tech' will expire on Mon 23 Oct 2023 18:30:19 GMT +0000. [09:55:32] RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Thu 05 Oct 2023 12:53:34 GMT +0000. [09:55:33] RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Thu 05 Oct 2023 20:04:14 GMT +0000. [09:55:34] RECOVERY - allthetropes.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [09:55:36] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.54, 7.26, 6.25 [09:55:54] RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Thu 05 Oct 2023 01:42:35 GMT +0000. [09:56:00] RECOVERY - closinglogosgroup.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [09:56:02] RECOVERY - wiki.conworlds.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.conworlds.org' will expire on Thu 05 Oct 2023 13:35:56 GMT +0000. [09:56:02] RECOVERY - www.johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Fri 06 Oct 2023 05:58:06 GMT +0000. [09:56:08] RECOVERY - www.sidem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.sidem.wiki' will expire on Fri 06 Oct 2023 18:43:43 GMT +0000. [09:56:09] RECOVERY - lgbta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [09:56:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 7.27, 6.36 [09:57:31] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.64, 7.17, 6.36 [10:01:19] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 12.77, 8.24, 6.84 [10:02:01] PROBLEM - cp32 Current Load on cp32 is WARNING: LOAD WARNING - total load average: 0.03, 0.48, 3.97 [10:02:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.77, 8.32, 7.06 [10:03:14] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.75, 7.04, 6.56 [10:03:25] PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 0.22, 0.42, 3.78 [10:03:59] RECOVERY - cp32 Stunnel for puppet141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8204 [10:04:07] RECOVERY - cp32 Stunnel for mwtask141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8150 [10:04:09] RECOVERY - cp32 Stunnel for mw133 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8110 [10:04:12] RECOVERY - cp32 Stunnel for mw132 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8107 [10:04:18] RECOVERY - cp32 Stunnel for reports121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8205 [10:04:29] RECOVERY - cp32 Stunnel for mw141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8108 [10:04:33] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [10:04:34] RECOVERY - cp32 Stunnel for test131 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8180 [10:04:36] RECOVERY - cp32 Stunnel for matomo121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8203 [10:04:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 7.77, 7.01 [10:04:56] RECOVERY - cp32 Stunnel for mw134 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8111 [10:04:57] RECOVERY - cp32 Stunnel for mw143 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8112 [10:05:06] RECOVERY - cp32 Stunnel for phab121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8202 [10:05:08] RECOVERY - cp32 Stunnel for mw142 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8109 [10:05:08] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.77, 7.93, 6.93 [10:05:25] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 0.48, 0.44, 3.37 [10:05:26] RECOVERY - cp32 Stunnel for mon141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8201 [10:05:33] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 3% [10:05:38] RECOVERY - wiki.shaazzz.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.shaazzz.ir' will expire on Thu 05 Oct 2023 22:33:31 GMT +0000. [10:05:49] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 15 backends are healthy [10:05:50] RECOVERY - cp32 Stunnel for mail121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8200 [10:05:52] RECOVERY - cp32 Stunnel for mw131 on cp32 is OK: TCP OK - 0.001 second response time on localhost port 8106 [10:06:01] RECOVERY - wiki.puucraft.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.puucraft.net' will expire on Sat 07 Oct 2023 16:58:51 GMT +0000. [10:06:01] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 0.35, 0.44, 3.15 [10:06:05] RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Thu 05 Oct 2023 02:29:59 GMT +0000. [10:06:06] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3571 bytes in 0.535 second response time [10:06:09] RECOVERY - studio.niuboss123.com - LetsEncrypt on sslhost is OK: OK - Certificate 'studio.niuboss123.com' will expire on Mon 11 Dec 2023 09:43:36 GMT +0000. [10:06:30] RECOVERY - translate.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'translate.petrawiki.org' will expire on Sat 07 Oct 2023 23:12:53 GMT +0000. [10:06:32] RECOVERY - vmklegacy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vmklegacy.wiki' will expire on Fri 13 Oct 2023 13:43:15 GMT +0000. [10:06:34] RECOVERY - wiki.litek.top - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.litek.top' will expire on Fri 27 Oct 2023 08:58:10 GMT +0000. [10:06:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.99, 8.02, 7.18 [10:06:48] RECOVERY - wiki.yahyabd.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yahyabd.xyz' will expire on Sat 07 Oct 2023 04:04:00 GMT +0000. [10:06:49] RECOVERY - crustypedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'crustypedia.org' will expire on Fri 03 Nov 2023 09:01:30 GMT +0000. [10:06:58] RECOVERY - wiki.potabi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.potabi.com' will expire on Fri 06 Oct 2023 12:33:59 GMT +0000. [10:07:04] RECOVERY - kb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kb.nena.org' will expire on Wed 08 Nov 2023 07:00:57 GMT +0000. [10:07:08] RECOVERY - wiki.susqu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.susqu.org' will expire on Wed 08 Nov 2023 17:10:13 GMT +0000. [10:07:35] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Tue 21 Nov 2023 20:39:44 GMT +0000. [10:07:35] RECOVERY - wiki.yuanpi.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yuanpi.eu.org' will expire on Wed 06 Dec 2023 11:08:08 GMT +0000. [10:07:36] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Fri 06 Oct 2023 19:53:19 GMT +0000. [10:07:40] RECOVERY - corru.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'corru.wiki' will expire on Sun 05 Nov 2023 06:48:26 GMT +0000. [10:13:32] RECOVERY - cp22 HTTPS on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3734 bytes in 0.131 second response time [10:13:38] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 7% [10:14:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:14:14] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:14:32] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.58, 0.40, 3.62 [10:15:55] RECOVERY - schizoidnightmares.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'schizoidnightmares.wiki' will expire on Sat 07 Oct 2023 20:24:07 GMT +0000. [10:16:05] RECOVERY - heavyironmodding.org - LetsEncrypt on sslhost is OK: OK - Certificate 'heavyironmodding.org' will expire on Fri 06 Oct 2023 04:21:18 GMT +0000. [10:16:07] RECOVERY - echoes-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'echoes-wiki.com' will expire on Wed 18 Oct 2023 20:36:08 GMT +0000. [10:16:10] RECOVERY - wiki.orvyn.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.orvyn.ca' will expire on Fri 24 Nov 2023 17:01:57 GMT +0000. [10:16:31] RECOVERY - wiki.otir.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.otir.nl' will expire on Sat 07 Oct 2023 15:32:54 GMT +0000. [10:16:32] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.53, 0.46, 3.25 [10:18:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 7.93, 7.78 [10:20:21] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 95% [10:20:23] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 4.26, 7.21, 7.60 [10:20:33] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 23.76, 10.88, 6.54 [10:20:40] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 7.60, 5.02, 3.46 [10:20:41] PROBLEM - kirisame-kissaten.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:47] PROBLEM - bluearchive.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:49] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 12.41, 8.88, 5.65 [10:20:49] PROBLEM - cnt.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:49] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [10:20:57] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.008 second response time [10:21:06] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 15.27, 10.30, 6.74 [10:21:28] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 17.05, 10.42, 6.44 [10:21:29] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 17.59, 10.41, 6.38 [10:21:38] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 86% [10:21:40] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 18.47, 11.55, 7.47 [10:21:51] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 5 backends are down. mw132 mw141 mw133 mw134 mw143 [10:21:51] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 6 backends are down. mw131 mw132 mw141 mw133 mw134 mw143 [10:22:03] PROBLEM - cp32 Current Load on cp32 is WARNING: LOAD WARNING - total load average: 1.94, 3.58, 2.95 [10:22:03] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [10:22:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [10:22:06] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 17.11, 10.62, 6.31 [10:22:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [10:22:45] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 4% [10:22:49] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 7.25, 8.97, 6.15 [10:22:57] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.328 second response time [10:23:06] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.18, 9.53, 6.95 [10:23:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: LOAD CRITICAL - total load average: 6.51, 8.52, 5.18 [10:23:28] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 6.69, 8.75, 6.32 [10:23:29] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 5.69, 8.12, 6.01 [10:23:38] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 1% [10:23:40] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.03, 9.55, 7.21 [10:23:49] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 15 backends are healthy [10:23:51] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 15 backends are healthy [10:24:03] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 0.26, 2.39, 2.58 [10:24:06] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.76, 8.61, 6.11 [10:24:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.09, 7.25, 7.35 [10:26:40] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 2.43, 3.93, 3.61 [10:28:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 12.22, 8.67, 7.85 [10:29:25] PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 0.98, 3.17, 3.77 [10:30:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.92, 7.94, 7.69 [10:30:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.52, 7.98, 7.77 [10:30:40] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 2.37, 3.22, 3.40 [10:31:25] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 0.75, 2.34, 3.39 [10:32:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.81, 8.20, 7.86 [10:34:32] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.41, 1.75, 3.83 [10:35:56] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 4% [10:36:04] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3572 bytes in 0.532 second response time [10:36:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:36:14] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:38:32] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.45, 1.23, 3.16 [10:38:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.84, 7.91, 7.97 [10:40:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.89, 7.93, 7.71 [10:40:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.46, 8.33, 8.12 [10:46:39] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 4.01, 3.55, 3.38 [10:48:40] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 2.57, 3.24, 3.29 [10:49:29] RECOVERY - kirisame-kissaten.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'kirisame-kissaten.cf' will expire on Sat 14 Oct 2023 15:40:58 GMT +0000. [10:49:41] RECOVERY - bluearchive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bluearchive.wiki' will expire on Fri 06 Oct 2023 13:46:10 GMT +0000. [10:49:43] RECOVERY - cnt.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [10:52:19] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.88, 7.25, 7.94 [10:56:01] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 0.79, 0.73, 3.92 [11:00:01] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.99, 1.17, 3.38 [11:00:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.43, 7.38, 7.98 [11:06:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.00, 8.03, 7.83 [11:06:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.57, 7.66, 7.85 [11:08:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 7.39, 7.72 [11:10:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.60, 7.81, 7.78 [11:18:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.39, 7.29, 7.34 [11:20:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.43, 6.89, 7.19 [11:26:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.42, 6.23, 6.80 [11:26:38] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.14, 6.28, 6.80 [11:32:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.03, 6.78, 6.80 [11:34:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.59, 6.31, 6.62 [11:37:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.19, 7.32, 6.86 [11:39:17] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.57, 6.77, 6.71 [11:45:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.89, 6.80, 6.67 [11:47:00] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.43, 6.37, 6.53 [11:57:19] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.89, 6.57, 6.39 [11:59:13] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.06, 6.39, 6.34 [12:08:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.89, 3.03, 2.47 [12:12:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.33, 3.32, 2.73 [12:16:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.37, 3.58, 3.00 [12:16:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.06, 7.64, 6.74 [12:18:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.03, 3.33, 2.97 [12:18:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.99, 7.38, 6.73 [12:20:53] !log restart swift-proxy on swiftproxy111 [12:21:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:22:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.02, 6.37, 6.48 [12:26:10] !log increase swap on cp* to 3g [12:26:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:36:15] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.75, 3.49, 3.23 [12:37:09] PROBLEM - swiftproxy111 Current Load on swiftproxy111 is WARNING: WARNING - load average: 0.35, 1.34, 7.16 [12:38:14] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.04, 3.62, 3.30 [12:39:09] RECOVERY - swiftproxy111 Current Load on swiftproxy111 is OK: OK - load average: 0.35, 1.02, 6.36 [12:40:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.98, 3.71, 3.38 [12:48:10] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.28, 3.39, 3.35 [12:54:06] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.91, 3.97, 3.58 [12:54:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.03, 7.71, 6.73 [12:56:11]