[00:01:34] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:03:34] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:03:40] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 18434MiB (19% inode=98%); [00:04:58] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21357MiB (24% inode=98%); [00:06:06] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21326MiB (24% inode=98%); [00:45:35] PROBLEM - cp27 HTTPS on cp27 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:45:50] PROBLEM - db181 SSH on db181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:54] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:46:06] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 6 backends are down. mw151 mw161 mw162 mw172 mw181 mw182 [00:46:09] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 7 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw182 [00:46:17] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:46:21] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 161.16, 88.10, 35.94 [00:46:23] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [00:46:28] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [00:47:07] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw152 [00:47:31] RECOVERY - cp27 HTTPS on cp27 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3613 bytes in 0.884 second response time [00:47:45] RECOVERY - db181 SSH on db181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [00:47:53] RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3635 bytes in 0.894 second response time [00:48:06] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [00:48:09] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [00:48:11] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.064 second response time [00:48:19] RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3636 bytes in 0.798 second response time [00:48:28] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [00:49:07] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [01:04:14] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.17, 2.64, 11.48 [01:06:13] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.34, 1.88, 10.12 [02:17:15] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:47:09] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:00:27] RECOVERY - db162 Backups SQL on db162 is OK: FILE_AGE OK: /var/log/sql-backup.log is 14 seconds old and 1218 bytes [03:00:51] PROBLEM - mon181 Backups Grafana on mon181 is CRITICAL: FILE_AGE CRITICAL: /var/log/grafana-backup.log is 1209633 seconds old and 92 bytes [03:45:49] PROBLEM - ff8.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ff8.wiki' expires in 15 day(s) (Wed 24 Apr 2024 03:37:07 AM GMT +0000). [03:46:04] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/df5a0f97cba5...ad1a55ad7349 [03:46:06] [02miraheze/ssl] 07WikiTideSSLBot 03ad1a55a - Bot: Update SSL cert for ff8.wiki [04:00:13] PROBLEM - www.satepedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.satepedia.com' expires in 15 day(s) (Wed 24 Apr 2024 03:37:31 AM GMT +0000). [04:00:24] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ad1a55ad7349...ba5bc8a7c9a8 [04:00:27] [02miraheze/ssl] 07WikiTideSSLBot 03ba5bc8a - Bot: Update SSL cert for www.satepedia.com [04:14:38] RECOVERY - ff8.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ff8.wiki' will expire on Sun 07 Jul 2024 02:45:57 AM GMT +0000. [04:29:37] RECOVERY - www.satepedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.satepedia.com' will expire on Sun 07 Jul 2024 03:00:18 AM GMT +0000. [06:18:06] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=crosspathsofdestinywiki (END - exit=0) [06:18:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:18:18] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:48:11] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [10:24:01] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 166.07 ms [10:26:03] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 182.66 ms [10:40:22] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.14 ms [10:42:24] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 180.25 ms [10:56:18] PROBLEM - bn.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [11:12:47] PROBLEM - wikiyri.org - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [11:18:05] PROBLEM - data.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [11:18:42] PROBLEM - dft.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [11:19:32] PROBLEM - data.wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for data.wikiyri.org could not be found [11:20:47] PROBLEM - bn.wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bn.wikiyri.org could not be found [11:21:57] PROBLEM - en.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [11:24:39] PROBLEM - en.wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.wikiyri.org could not be found [11:41:58] PROBLEM - dft.wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dft.wikiyri.org could not be found [11:42:59] PROBLEM - wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiyri.org could not be found [12:00:59] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.99 ms [12:03:01] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.36 ms [13:02:06] PROBLEM - db162 MariaDB on db162 is UNKNOWN: [13:04:05] PROBLEM - db162 MariaDB on db162 is CRITICAL: Access denied for user 'icinga'@'2602:294:0:b12::110' (using password: YES) [13:05:06] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:09:14] PROBLEM - swiftproxy161 Puppet on swiftproxy161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:31:06] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:31:15] RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [15:52:10] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:53:10] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/namespaceDupes.php --wiki=1d6chanwiki (END - exit=0) [15:53:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:54:12] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/namespaceDupes.php --wiki=1d6chanwiki --fix (END - exit=256) [15:54:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:15] [1/4] ``` [15:56:15] [2/4] id=76595 ns=0 dbk=Story:_Holy_Opposites_Chapter_1 -> Story:Holy_Opposites_Chapter_1 (no conflict) [15:56:16] [3/4] Wikimedia\Rdbms\DBQueryError from line 1236 of /srv/mediawiki/1.41/includes/libs/rdbms/database/Database.php: Error 1062: Duplicate entry '3006-Holy_Opposites_Chapter_1' for key 'page_name_title'``` [15:56:16] [4/4] fun [15:57:15] Joyous [15:57:41] it's monday dude, it's too soon for these kinds of exceptions smh :/ [15:58:41] It’s always either too soon or too late [15:59:54] !log created Swift container miraheze-loadouts (T11901) [16:00:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:19:40] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 10226MiB (10% inode=98%); [16:22:06] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [16:26:55] I think you need to rebuild search index [16:27:12] I assume it's a normal not cirrus wiki [16:27:32] isn't everyone running cirrus now? [16:27:35] I'll check [16:28:12] hmm, doesn't seem to be running cirrus according to Special:Version [16:29:40] @bluemoon0332 try rebuild the search index then [16:32:05] it's on it [16:32:55] @rhinosf1 can you make rebuildtextindex one of those scripts that log when they start as well as when they end? [16:33:11] @bluemoon0332 ye [16:33:14] pls [16:37:35] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildtextindex.php --wiki=1d6chanwiki (END - exit=0) [16:37:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:42:12] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/namespaceDupes.php --wiki=1d6chanwiki --fix (END - exit=0) [16:42:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:52:01] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:12:25] Perfect @bluemoon0332 [17:21:56] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [17:34:47] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.00 ms [17:36:49] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.67 ms [17:57:21] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 164.44 ms [17:59:22] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.80 ms [18:25:53] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.17 ms [18:34:03] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 176.28 ms [18:43:18] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.76 ms [18:49:27] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.47 ms [19:09:59] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.05 ms [19:12:00] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.46 ms [19:16:29] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [19:16:34] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [19:16:41] PROBLEM - landofliberos.com - LetsEncrypt on sslhost is CRITICAL: connect to address landofliberos.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:16:41] PROBLEM - 9b9t.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address 9b9t.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:16:46] PROBLEM - cp37 Nginx Backend for test151 on cp37 is CRITICAL: connect to address localhost and port 8181: Connection refused [19:16:46] PROBLEM - wikappedia.cc - LetsEncrypt on sslhost is CRITICAL: connect to address wikappedia.cc and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:16:48] PROBLEM - cp37 Nginx Backend for mw162 on cp37 is CRITICAL: connect to address localhost and port 8116: Connection refused [19:16:55] PROBLEM - cp37 Nginx Backend for mw151 on cp37 is CRITICAL: connect to address localhost and port 8113: Connection refused [19:16:56] PROBLEM - acgn.sfdev.eu.org - LetsEncrypt on sslhost is CRITICAL: connect to address acgn.sfdev.eu.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:16:59] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp37.wikitide.net port 443 after 0 ms: Couldn't connect to server [19:17:00] PROBLEM - buildabearwiki.info - LetsEncrypt on sslhost is CRITICAL: connect to address buildabearwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:04] PROBLEM - rockrevolution.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address rockrevolution.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:04] PROBLEM - cp37 Nginx Backend for reports171 on cp37 is CRITICAL: connect to address localhost and port 8205: Connection refused [19:17:04] PROBLEM - wiki.aclevo.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.aclevo.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:05] PROBLEM - cp37 Nginx Backend for mw182 on cp37 is CRITICAL: connect to address localhost and port 8120: Connection refused [19:17:07] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [19:17:07] PROBLEM - cp37 Nginx Backend for mwtask181 on cp37 is CRITICAL: connect to address localhost and port 8160: Connection refused [19:17:07] PROBLEM - publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address publictestwiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:15] PROBLEM - cp37 Nginx Backend for puppet181 on cp37 is CRITICAL: connect to address localhost and port 8204: Connection refused [19:17:21] PROBLEM - cp37 Nginx Backend for mw181 on cp37 is CRITICAL: connect to address localhost and port 8119: Connection refused [19:17:21] PROBLEM - wiki.simorgh.me - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [19:17:21] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [19:17:25] PROBLEM - history.sdtef.org - LetsEncrypt on sslhost is CRITICAL: connect to address history.sdtef.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:26] PROBLEM - wiki.elgeis.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:27] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:27] PROBLEM - removededm.com - LetsEncrypt on sslhost is CRITICAL: connect to address removededm.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:27] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.nowchess.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:27] PROBLEM - guia.cineastas.pt - LetsEncrypt on sslhost is CRITICAL: connect to address guia.cineastas.pt and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:28] PROBLEM - cp37 Nginx Backend for swiftproxy171 on cp37 is CRITICAL: connect to address localhost and port 8207: Connection refused [19:17:29] PROBLEM - www.themagical.world - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:29] PROBLEM - sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:29] PROBLEM - cp37 Nginx Backend for mw172 on cp37 is CRITICAL: connect to address localhost and port 8118: Connection refused [19:17:29] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [19:17:39] PROBLEM - cp37 Nginx Backend for phorge171 on cp37 is CRITICAL: connect to address localhost and port 8202: Connection refused [19:17:39] PROBLEM - cp51 Current Load on cp51 is CRITICAL: LOAD CRITICAL - total load average: 10.64, 6.88, 3.03 [19:17:43] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp36.wikitide.net port 443 after 0 ms: Couldn't connect to server [19:17:44] PROBLEM - education.gced.in - LetsEncrypt on sslhost is CRITICAL: connect to address education.gced.in and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:49] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [19:17:51] PROBLEM - wiki.overwood.xyz - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.overwood.xyz and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:51] PROBLEM - en.noblework.org - LetsEncrypt on sslhost is CRITICAL: connect to address en.noblework.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:51] PROBLEM - metroid.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address metroid.miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:51] PROBLEM - www.mh142.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.mh142.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:52] PROBLEM - cp37 Nginx Backend for mw152 on cp37 is CRITICAL: connect to address localhost and port 8114: Connection refused [19:17:52] PROBLEM - patrykstan.veles-crimea.ru - LetsEncrypt on sslhost is CRITICAL: connect to address patrykstan.veles-crimea.ru and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:53] PROBLEM - cp37 Nginx Backend for matomo151 on cp37 is CRITICAL: connect to address localhost and port 8203: Connection refused [19:17:53] PROBLEM - cp37 Nginx Backend for mon181 on cp37 is CRITICAL: connect to address localhost and port 8201: Connection refused [19:17:54] PROBLEM - wiki.tulpa.info - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.tulpa.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:54] PROBLEM - wiki.dwarvesrp.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.dwarvesrp.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:55] PROBLEM - kombinat.eu.org - LetsEncrypt on sslhost is CRITICAL: connect to address kombinat.eu.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:55] PROBLEM - minescape.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address minescape.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:17:55] PROBLEM - ncuwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address ncuwiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:01] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.mahdiruiz.line.pm and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:01] PROBLEM - www.ebcat.me - LetsEncrypt on sslhost is CRITICAL: connect to address www.ebcat.me and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:04] PROBLEM - fanonpedia.com - LetsEncrypt on sslhost is CRITICAL: connect to address fanonpedia.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:04] PROBLEM - wiki.buryland.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.buryland.net and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:04] PROBLEM - wiki.ooer.ooo - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.ooer.ooo and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:06] PROBLEM - wiki.eversd.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.eversd.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:06] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [19:18:09] PROBLEM - allthetropes.org - LetsEncrypt on sslhost is CRITICAL: connect to address allthetropes.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:09] PROBLEM - journeytheword.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address journeytheword.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:10] PROBLEM - www.johanloopmans.nl - LetsEncrypt on sslhost is CRITICAL: connect to address www.johanloopmans.nl and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:12] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: connect to address enc.for.uz and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:15] PROBLEM - wiki.cosmicdeathfungus.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.cosmicdeathfungus.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:16] PROBLEM - schizoidnightmares.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address schizoidnightmares.miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:17] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [19:18:17] PROBLEM - test2.specialchat.org - LetsEncrypt on sslhost is CRITICAL: connect to address test2.specialchat.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:17] PROBLEM - richterian.com - LetsEncrypt on sslhost is CRITICAL: connect to address richterian.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:17] PROBLEM - ndgkb.nena.org - LetsEncrypt on sslhost is CRITICAL: connect to address ndgkb.nena.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:20] PROBLEM - www.tintinpedia.fr - LetsEncrypt on sslhost is CRITICAL: connect to address www.tintinpedia.fr and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:20] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.publictestwiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:20] PROBLEM - you.r-fit.cc - LetsEncrypt on sslhost is CRITICAL: connect to address you.r-fit.cc and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:20] PROBLEM - cp37 Nginx Backend for swiftproxy161 on cp37 is CRITICAL: connect to address localhost and port 8206: Connection refused [19:18:21] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp41.wikitide.net port 443 after 127 ms: Couldn't connect to server [19:18:26] PROBLEM - wiki.iurevar.world - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.iurevar.world and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:29] PROBLEM - cp37 Nginx Backend for mwtask171 on cp37 is CRITICAL: connect to address localhost and port 8161: Connection refused [19:18:31] PROBLEM - cp37 Nginx Backend for mw161 on cp37 is CRITICAL: connect to address localhost and port 8115: Connection refused [19:18:33] PROBLEM - cp37 Nginx Backend for mw171 on cp37 is CRITICAL: connect to address localhost and port 8117: Connection refused [19:18:38] PROBLEM - hololive.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address hololive.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:41] PROBLEM - wiki.newnormal.fans - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.newnormal.fans and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:42] PROBLEM - fanon.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address fanon.polandballwiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:42] PROBLEM - cp36 Nginx Backend for mwtask181 on cp36 is CRITICAL: connect to address localhost and port 8160: Connection refused [19:18:45] PROBLEM - dragdown.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address dragdown.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:45] PROBLEM - grayravens.com - LetsEncrypt on sslhost is CRITICAL: connect to address grayravens.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:50] PROBLEM - civwiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address civwiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:51] PROBLEM - tech.exputra.com - LetsEncrypt on sslhost is CRITICAL: connect to address tech.exputra.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:52] PROBLEM - wiki.zamnhacking.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.zamnhacking.net and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:57] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: connect to address m.miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:58] PROBLEM - bebaskanpengetahuan.id - LetsEncrypt on sslhost is CRITICAL: connect to address bebaskanpengetahuan.id and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:18:58] PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.autocountsoft.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:19:00] PROBLEM - cp36 Nginx Backend for mw151 on cp36 is CRITICAL: connect to address localhost and port 8113: Connection refused [19:19:02] PROBLEM - legacygt.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address legacygt.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:19:06] PROBLEM - cp36 Nginx Backend for matomo151 on cp36 is CRITICAL: connect to address localhost and port 8203: Connection refused [19:19:07] PROBLEM - cp36 Nginx Backend for mon181 on cp36 is CRITICAL: connect to address localhost and port 8201: Connection refused [19:19:07] PROBLEM - cp36 Nginx Backend for mw181 on cp36 is CRITICAL: connect to address localhost and port 8119: Connection refused [19:19:07] PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.pyramidgames.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:19:09] PROBLEM - psychevos.org - LetsEncrypt on sslhost is CRITICAL: connect to address psychevos.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:19:19] PROBLEM - kb.nena.org - LetsEncrypt on sslhost is CRITICAL: connect to address kb.nena.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:19:29] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.make717.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:19:30] PROBLEM - tno.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address tno.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:19:34] PROBLEM - cp41 Nginx Backend for mw151 on cp41 is CRITICAL: connect to address localhost and port 8113: Connection refused [19:19:36] PROBLEM - cp41 Nginx Backend for mw162 on cp41 is CRITICAL: connect to address localhost and port 8116: Connection refused [19:19:39] PROBLEM - cp51 Current Load on cp51 is WARNING: LOAD WARNING - total load average: 7.34, 6.92, 3.52 [19:19:41] PROBLEM - cp41 Nginx Backend for mw161 on cp41 is CRITICAL: connect to address localhost and port 8115: Connection refused [19:19:47] PROBLEM - podpedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address podpedia.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:19:50] PROBLEM - pandorastale.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address pandorastale.miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:01] PROBLEM - cp41 Nginx Backend for mon181 on cp41 is CRITICAL: connect to address localhost and port 8201: Connection refused [19:20:02] PROBLEM - uk.religiononfire.mar.in.ua - LetsEncrypt on sslhost is CRITICAL: connect to address uk.religiononfire.mar.in.ua and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:06] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [19:20:07] PROBLEM - cp41 Nginx Backend for phorge171 on cp41 is CRITICAL: connect to address localhost and port 8202: Connection refused [19:20:09] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [19:20:10] PROBLEM - cp41 Nginx Backend for puppet181 on cp41 is CRITICAL: connect to address localhost and port 8204: Connection refused [19:20:13] PROBLEM - pokemon.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address pokemon.miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:16] PROBLEM - runzeppelin.ru - LetsEncrypt on sslhost is CRITICAL: connect to address runzeppelin.ru and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:20] PROBLEM - nintendowiki.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address nintendowiki.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:21] PROBLEM - pokemonwiki.info - LetsEncrypt on sslhost is CRITICAL: connect to address pokemonwiki.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:22] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [19:20:24] PROBLEM - cp41 Nginx Backend for mw182 on cp41 is CRITICAL: connect to address localhost and port 8120: Connection refused [19:20:34] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 2% [19:20:35] PROBLEM - cp41 Nginx Backend for reports171 on cp41 is CRITICAL: connect to address localhost and port 8205: Connection refused [19:20:36] PROBLEM - cp41 Nginx Backend for mwtask171 on cp41 is CRITICAL: connect to address localhost and port 8161: Connection refused [19:20:37] PROBLEM - www.burnout.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.burnout.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:37] PROBLEM - apeirology.com - LetsEncrypt on sslhost is CRITICAL: connect to address apeirology.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:37] PROBLEM - osfirsttimer.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address osfirsttimer.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:39] PROBLEM - kaylaandrachel.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address kaylaandrachel.miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:39] PROBLEM - nintendo.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address nintendo.miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:40] PROBLEM - cp41 Nginx Backend for mwtask181 on cp41 is CRITICAL: connect to address localhost and port 8160: Connection refused [19:20:40] PROBLEM - cp41 Nginx Backend for matomo151 on cp41 is CRITICAL: connect to address localhost and port 8203: Connection refused [19:20:40] PROBLEM - cp41 Nginx Backend for mw171 on cp41 is CRITICAL: connect to address localhost and port 8117: Connection refused [19:20:42] RECOVERY - cp36 Nginx Backend for mwtask181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8160 [19:20:43] PROBLEM - uncyclopediakr.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address uncyclopediakr.miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:44] PROBLEM - iol.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address iol.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:52] PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address monarchists.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:52] PROBLEM - www.iceria.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.iceria.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:20:53] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Wed 26 Jun 2024 05:32:24 PM GMT +0000. [19:21:00] RECOVERY - cp36 Nginx Backend for mw151 on cp36 is OK: TCP OK - 0.001 second response time on localhost port 8113 [19:21:00] PROBLEM - hr.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address hr.petrawiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:21:01] PROBLEM - cp41 Nginx Backend for swiftproxy161 on cp41 is CRITICAL: connect to address localhost and port 8206: Connection refused [19:21:05] PROBLEM - cp41 Nginx Backend for mw172 on cp41 is CRITICAL: connect to address localhost and port 8118: Connection refused [19:21:06] PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is CRITICAL: connect to address thestarsareright.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:21:06] RECOVERY - cp36 Nginx Backend for matomo151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8203 [19:21:07] RECOVERY - cp36 Nginx Backend for mon181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8201 [19:21:07] RECOVERY - cp36 Nginx Backend for mw181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8119 [19:21:08] PROBLEM - cp41 Nginx Backend for test151 on cp41 is CRITICAL: connect to address localhost and port 8181: Connection refused [19:21:13] PROBLEM - cp41 Nginx Backend for mw181 on cp41 is CRITICAL: connect to address localhost and port 8119: Connection refused [19:21:15] PROBLEM - cp41 Nginx Backend for swiftproxy171 on cp41 is CRITICAL: connect to address localhost and port 8207: Connection refused [19:21:16] PROBLEM - cp41 Nginx Backend for mw152 on cp41 is CRITICAL: connect to address localhost and port 8114: Connection refused [19:21:39] RECOVERY - cp51 Current Load on cp51 is OK: LOAD OK - total load average: 6.12, 6.69, 3.86 [19:21:39] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3614 bytes in 0.065 second response time [19:23:23] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [19:25:03] PROBLEM - chobots.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address chobots.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:25:41] RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3634 bytes in 6.187 second response time [19:26:17] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 4% [19:26:25] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [19:28:55] RECOVERY - cp37 Nginx Backend for mw151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8113 [19:29:04] RECOVERY - cp37 Nginx Backend for reports171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8205 [19:29:05] RECOVERY - cp37 Nginx Backend for mw182 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8120 [19:29:07] RECOVERY - cp37 Nginx Backend for mwtask181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8160 [19:29:15] RECOVERY - cp37 Nginx Backend for puppet181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8204 [19:29:21] RECOVERY - cp37 Nginx Backend for mw181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8119 [19:29:28] RECOVERY - cp37 Nginx Backend for swiftproxy171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8207 [19:29:29] RECOVERY - cp37 Nginx Backend for mw172 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8118 [19:29:39] RECOVERY - cp37 Nginx Backend for phorge171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8202 [19:29:49] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 2% [19:29:52] RECOVERY - cp37 Nginx Backend for mw152 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8114 [19:29:53] RECOVERY - cp37 Nginx Backend for matomo151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8203 [19:29:53] RECOVERY - cp37 Nginx Backend for mon181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8201 [19:30:20] RECOVERY - cp37 Nginx Backend for swiftproxy161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8206 [19:30:29] RECOVERY - cp37 Nginx Backend for mwtask171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8161 [19:30:31] RECOVERY - cp37 Nginx Backend for mw161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8115 [19:30:33] RECOVERY - cp37 Nginx Backend for mw171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8117 [19:30:46] RECOVERY - cp37 Nginx Backend for test151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8181 [19:30:48] RECOVERY - cp37 Nginx Backend for mw162 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8116 [19:30:59] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3613 bytes in 0.067 second response time [19:31:07] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [19:31:17] @cosmicalpha or @agentisai, can any of you see if nginx is running on cp41? [19:33:08] RECOVERY - cp41 Nginx Backend for test151 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8181 [19:33:13] RECOVERY - cp41 Nginx Backend for mw181 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8119 [19:33:15] RECOVERY - cp41 Nginx Backend for swiftproxy171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8207 [19:33:16] RECOVERY - cp41 Nginx Backend for mw152 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8114 [19:33:24] cp41 was hit so hard by the DDoS that nginx got killed [19:33:34] RECOVERY - cp41 Nginx Backend for mw151 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8113 [19:33:35] RECOVERY - cp41 Nginx Backend for mw162 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8116 [19:33:42] RECOVERY - cp41 Nginx Backend for mw161 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8115 [19:34:01] RECOVERY - cp41 Nginx Backend for mon181 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8201 [19:34:07] RECOVERY - cp41 Nginx Backend for phorge171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8202 [19:34:09] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [19:34:10] RECOVERY - cp41 Nginx Backend for puppet181 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8204 [19:34:14] RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3636 bytes in 0.841 second response time [19:34:24] RECOVERY - cp41 Nginx Backend for mw182 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8120 [19:34:29] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 1% [19:34:34] RECOVERY - cp41 Nginx Backend for reports171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8205 [19:34:36] RECOVERY - cp41 Nginx Backend for mwtask171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8161 [19:34:39] RECOVERY - cp41 Nginx Backend for mw171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8117 [19:34:40] RECOVERY - cp41 Nginx Backend for matomo151 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8203 [19:34:40] RECOVERY - cp41 Nginx Backend for mwtask181 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8160 [19:35:01] RECOVERY - cp41 Nginx Backend for swiftproxy161 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8206 [19:35:05] RECOVERY - cp41 Nginx Backend for mw172 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8118 [19:38:17] PROBLEM - www.lostidols.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lostidols.wiki' expires in 15 day(s) (Wed 24 Apr 2024 07:20:15 PM GMT +0000). [19:39:49] wowza [19:42:04] PROBLEM - lostidols.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lostidols.wiki' expires in 15 day(s) (Wed 24 Apr 2024 07:20:15 PM GMT +0000). [19:42:19] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ba5bc8a7c9a8...a9bdb822bd89 [19:42:20] [02miraheze/ssl] 07WikiTideSSLBot 03a9bdb82 - Bot: Update SSL cert for lostidols.wiki [19:45:52] RECOVERY - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.teessidehackspace.org.uk' will expire on Wed 15 May 2024 05:07:21 PM GMT +0000. [19:45:54] RECOVERY - wiki.simorgh.me - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.simorgh.me' will expire on Wed 15 May 2024 12:16:18 PM GMT +0000. [19:45:57] RECOVERY - wiki.elgeis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.elgeis.com' will expire on Wed 15 May 2024 01:24:45 AM GMT +0000. [19:46:00] RECOVERY - www.themagical.world - LetsEncrypt on sslhost is OK: OK - Certificate 'themagical.world' will expire on Fri 05 Jul 2024 08:53:14 PM GMT +0000. [19:46:00] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Thu 09 May 2024 05:43:19 PM GMT +0000. [19:46:05] RECOVERY - history.sdtef.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.sdtef.org' will expire on Tue 14 May 2024 05:27:07 PM GMT +0000. [19:46:06] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Tue 14 May 2024 12:41:04 PM GMT +0000. [19:46:06] RECOVERY - guia.cineastas.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.cineastas.pt' will expire on Tue 14 May 2024 12:07:18 PM GMT +0000. [19:46:06] RECOVERY - 9b9t.wiki - LetsEncrypt on sslhost is OK: OK - Certificate '9b9t.wiki' will expire on Mon 27 May 2024 06:41:03 PM GMT +0000. [19:46:06] RECOVERY - landofliberos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'landofliberos.com' will expire on Thu 16 May 2024 11:52:19 AM GMT +0000. [19:46:18] RECOVERY - wikappedia.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'wikappedia.cc' will expire on Thu 30 May 2024 04:50:37 PM GMT +0000. [19:46:49] RECOVERY - acgn.sfdev.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'acgn.sfdev.eu.org' will expire on Thu 16 May 2024 09:41:43 PM GMT +0000. [19:46:50] RECOVERY - education.gced.in - LetsEncrypt on sslhost is OK: OK - Certificate 'education.gced.in' will expire on Wed 08 May 2024 07:23:30 AM GMT +0000. [19:46:50] RECOVERY - buildabearwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'buildabearwiki.info' will expire on Thu 16 May 2024 02:23:17 AM GMT +0000. [19:46:53] RECOVERY - metroid.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [19:46:53] RECOVERY - wiki.aclevo.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aclevo.com' will expire on Tue 30 Apr 2024 06:30:41 PM GMT +0000. [19:46:53] RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Tue 14 May 2024 10:25:39 AM GMT +0000. [19:46:55] RECOVERY - patrykstan.veles-crimea.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'patrykstan.veles-crimea.ru' will expire on Mon 13 May 2024 05:28:21 AM GMT +0000. [19:46:59] RECOVERY - publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Mon 13 May 2024 11:43:32 PM GMT +0000. [19:47:02] RECOVERY - wiki.overwood.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.overwood.xyz' will expire on Sun 30 Jun 2024 12:56:35 PM GMT +0000. [19:47:02] RECOVERY - en.noblework.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.noblework.org' will expire on Wed 03 Jul 2024 11:08:27 PM GMT +0000. [19:47:02] RECOVERY - rockrevolution.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rockrevolution.wiki' will expire on Fri 03 May 2024 01:54:39 AM GMT +0000. [19:47:04] RECOVERY - minescape.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'minescape.wiki' will expire on Thu 16 May 2024 08:08:35 PM GMT +0000. [19:47:05] RECOVERY - ncuwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'ncuwiki.com' will expire on Thu 16 May 2024 08:09:12 PM GMT +0000. [19:47:05] RECOVERY - kombinat.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kombinat.eu.org' will expire on Thu 27 Jun 2024 11:55:32 AM GMT +0000. [19:47:09] RECOVERY - removededm.com - LetsEncrypt on sslhost is OK: OK - Certificate 'removededm.com' will expire on Sun 30 Jun 2024 03:28:06 PM GMT +0000. [19:47:15] RECOVERY - wiki.dwarvesrp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dwarvesrp.com' will expire on Thu 16 May 2024 07:24:11 PM GMT +0000. [19:47:15] RECOVERY - wiki.tulpa.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tulpa.info' will expire on Tue 14 May 2024 12:32:31 AM GMT +0000. [19:47:17] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [19:47:23] RECOVERY - wiki.newnormal.fans - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.newnormal.fans' will expire on Wed 15 May 2024 11:50:37 AM GMT +0000. [19:47:23] RECOVERY - fanon.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fanon.polandballwiki.com' will expire on Wed 15 May 2024 03:29:06 PM GMT +0000. [19:47:25] RECOVERY - fanonpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fanonpedia.com' will expire on Fri 07 Jun 2024 03:24:36 PM GMT +0000. [19:47:27] RECOVERY - dragdown.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'dragdown.wiki' will expire on Thu 16 May 2024 01:14:47 AM GMT +0000. [19:47:27] RECOVERY - grayravens.com - LetsEncrypt on sslhost is OK: OK - Certificate 'grayravens.com' will expire on Wed 15 May 2024 06:16:15 PM GMT +0000. [19:47:31] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Wed 15 May 2024 02:05:20 AM GMT +0000. [19:47:32] RECOVERY - journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Wed 15 May 2024 10:45:45 AM GMT +0000. [19:47:32] RECOVERY - www.johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Mon 20 May 2024 11:20:22 AM GMT +0000. [19:47:32] RECOVERY - allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Mon 13 May 2024 11:30:59 PM GMT +0000. [19:47:32] RECOVERY - wiki.eversd.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.eversd.com' will expire on Wed 15 May 2024 10:04:04 AM GMT +0000. [19:47:32] RECOVERY - wiki.ooer.ooo - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ooer.ooo' will expire on Thu 06 Jun 2024 02:45:10 PM GMT +0000. [19:47:33] RECOVERY - wiki.buryland.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.buryland.net' will expire on Thu 20 Jun 2024 01:58:22 AM GMT +0000. [19:47:37] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Tue 14 May 2024 09:31:34 AM GMT +0000. [19:47:41] RECOVERY - tech.exputra.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tech.exputra.com' will expire on Thu 16 May 2024 06:21:31 PM GMT +0000. [19:47:42] RECOVERY - schizoidnightmares.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [19:47:48] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Tue 14 May 2024 01:11:29 PM GMT +0000. [19:47:48] RECOVERY - richterian.com - LetsEncrypt on sslhost is OK: OK - Certificate 'richterian.com' will expire on Sat 29 Jun 2024 10:54:46 PM GMT +0000. [19:47:48] RECOVERY - test2.specialchat.org - LetsEncrypt on sslhost is OK: OK - Certificate 'test2.specialchat.org' will expire on Wed 03 Jul 2024 12:39:43 PM GMT +0000. [19:47:48] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Mon 13 May 2024 11:56:21 PM GMT +0000. [19:47:49] RECOVERY - ndgkb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ndgkb.nena.org' will expire on Wed 19 Jun 2024 11:54:46 AM GMT +0000. [19:47:54] RECOVERY - www.tintinpedia.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'tintinpedia.fr' will expire on Tue 21 May 2024 11:22:47 AM GMT +0000. [19:47:54] RECOVERY - you.r-fit.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'you.r-fit.cc' will expire on Wed 15 May 2024 04:22:02 AM GMT +0000. [19:47:57] RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Mon 13 May 2024 11:43:32 PM GMT +0000. [19:48:00] RECOVERY - www.ebcat.me - LetsEncrypt on sslhost is OK: OK - Certificate 'www.ebcat.me' will expire on Wed 08 May 2024 12:06:52 PM GMT +0000. [19:48:04] RECOVERY - legacygt.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'legacygt.wiki' will expire on Wed 15 May 2024 11:55:58 PM GMT +0000. [19:48:04] RECOVERY - wiki.iurevar.world - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.iurevar.world' will expire on Wed 15 May 2024 05:16:03 PM GMT +0000. [19:48:06] RECOVERY - www.pyramidgames.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pyramidgames.wiki' will expire on Fri 05 Jul 2024 08:41:07 PM GMT +0000. [19:48:10] RECOVERY - wiki.cosmicdeathfungus.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cosmicdeathfungus.com' will expire on Thu 30 May 2024 05:07:25 PM GMT +0000. [19:48:11] RECOVERY - psychevos.org - LetsEncrypt on sslhost is OK: OK - Certificate 'psychevos.org' will expire on Thu 16 May 2024 09:27:10 PM GMT +0000. [19:48:30] RECOVERY - hololive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hololive.wiki' will expire on Tue 14 May 2024 08:59:19 PM GMT +0000. [19:48:34] RECOVERY - uk.religiononfire.mar.in.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'uk.religiononfire.mar.in.ua' will expire on Wed 05 Jun 2024 03:42:41 PM GMT +0000. [19:48:42] RECOVERY - wiki.zamnhacking.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zamnhacking.net' will expire on Mon 03 Jun 2024 05:22:12 PM GMT +0000. [19:48:58] RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Tue 14 May 2024 12:35:28 AM GMT +0000. [19:49:02] RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Tue 14 May 2024 10:02:34 AM GMT +0000. [19:49:11] RECOVERY - nintendowiki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nintendowiki.wiki' will expire on Thu 16 May 2024 06:22:10 PM GMT +0000. [19:49:11] RECOVERY - pokemonwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'pokemonwiki.info' will expire on Wed 15 May 2024 04:40:49 PM GMT +0000. [19:49:19] RECOVERY - kb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kb.nena.org' will expire on Wed 19 Jun 2024 12:03:41 PM GMT +0000. [19:49:28] RECOVERY - tno.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tno.wiki' will expire on Sun 30 Jun 2024 03:28:22 PM GMT +0000. [19:49:34]