[00:03:14] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [00:04:26] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [00:04:39] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [00:04:56] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [00:05:44] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [00:05:55] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [00:06:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 0.24, 2.03, 3.97 [00:06:26] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 12% [00:06:39] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 14% [00:07:18] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 51% [00:08:10] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:08:50] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:09:14] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [00:09:16] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 6% [00:09:18] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:09:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 21.14, 12.53, 9.84 [00:09:44] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [00:09:55] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [00:10:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.98, 3.83, 4.17 [00:10:05] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [00:10:54] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.923 second response time [00:10:56] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [00:11:17] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.310 second response time [00:11:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.41, 11.98, 10.00 [00:15:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.51, 9.71, 9.53 [00:20:03] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 9.91, 12.20, 7.59 [00:22:03] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 3.00, 8.85, 6.92 [00:22:30] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:32] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:42] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:44] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:55] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.68, 3.00, 4.00 [00:24:05] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:27] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:27] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.289 second response time [00:24:40] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:13] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 4 backends are down. mw121 mw122 mw131 mw142 [00:25:31] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.318 second response time [00:25:40] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.588 second response time [00:25:41] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.346 second response time [00:25:44] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw122 mw131 [00:25:51] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [00:26:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.64, 3.76, 4.13 [00:26:56] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 3 backends are down. mw121 mw131 mw141 [00:27:27] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [00:28:00] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.308 second response time [00:28:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.35, 2.82, 3.74 [00:28:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 32.03, 22.23, 12.62 [00:28:33] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.321 second response time [00:28:38] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 0.475 second response time [00:29:37] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 10.27, 18.46, 11.92 [00:29:44] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [00:30:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.21, 2.51, 3.98 [00:31:13] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [00:31:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.60, 11.23, 9.54 [00:33:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.65, 9.92, 9.25 [00:33:37] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 4.14, 10.89, 10.33 [00:34:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.94, 4.06, 3.91 [00:34:03] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 4.61, 10.70, 10.56 [00:34:22] RECOVERY - mon141 IRCEcho on mon141 is OK: PROCS OK: 1 process with args '/usr/local/bin/ircecho' [00:34:31] RECOVERY - mon141 php-fpm on mon141 is OK: PROCS OK: 3 processes with command name 'php-fpm7.4' [00:34:47] PROBLEM - mon141 Check correctness of the icinga configuration on mon141 is CRITICAL: Icinga configuration contains errors [00:34:56] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [00:35:37] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.35, 8.82, 9.64 [00:36:03] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 4.10, 8.57, 9.79 [00:36:58] RECOVERY - mon141 IRC Log Bot on mon141 is OK: PROCS OK: 1 process with args 'adminlogbot.py' [00:37:32] RECOVERY - mon141 icinga.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.167 second response time [00:40:49] RECOVERY - mon141 Check correctness of the icinga configuration on mon141 is OK: Icinga configuration is correct [00:42:13] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 51.35, 16.35, 6.00 [00:42:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.95, 2.83, 3.34 [00:43:07] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:52:18] PROBLEM - gluster101 PowerDNS Recursor on gluster101 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a. Check system logs on 2a10:6740::4c3a:433a:4b3a:413a [00:52:19] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a. Check system logs on 2a10:6740::4c3a:433a:4b3a:413a [00:52:20] PROBLEM - cp21 Stunnel HTTP for puppet111 on cp21 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2001:41d0:801:2000:4c3a:433a:4b3a:413a. Check system logs on 2001:41d0:801:2000:4c3a:433a:4b3a:413a [00:52:20] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2001:41d0:801:2000:4c3a:433a:4b3a:413a. Check system logs on 2001:41d0:801:2000:4c3a:433a:4b3a:413a [00:52:21] PROBLEM - cp30 Stunnel HTTP for puppet111 on cp30 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2607:5300:201:3100:4c3a:433a:4b3a:413a: Connection reset by peer [00:52:21] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2001:41d0:801:2000:4c3a:433a:4b3a:413a. Check system logs on 2001:41d0:801:2000:4c3a:433a:4b3a:413a [00:52:22] PROBLEM - es141 conntrack_table_size on es141 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a. Check system logs on 2a10:6740::4c3a:433a:4b3a:413a [00:52:23] PROBLEM - graylog121 PowerDNS Recursor on graylog121 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a. Check system logs on 2a10:6740::4c3a:433a:4b3a:413a [00:52:23] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a. Check system logs on 2a10:6740::4c3a:433a:4b3a:413a [00:52:24] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2001:41d0:801:2000:4c3a:433a:4b3a:413a. Check system logs on 2001:41d0:801:2000:4c3a:433a:4b3a:413a [00:52:24] PROBLEM - puppet111 ferm_active on puppet111 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a. Check system logs on 2a10:6740::4c3a:433a:4b3a:413a [00:52:24] PROBLEM - mw122 nutcracker process on mw122 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a. Check system logs on 2a10:6740::4c3a:433a:4b3a:413a [00:52:24] PROBLEM - phab121 phd on phab121 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a. Check system logs on 2a10:6740::4c3a:433a:4b3a:413a [00:52:24] PROBLEM - www.fraterniterebellion.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.fraterniterebellion.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:52:24] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2001:41d0:801:2000:4c3a:433a:4b3a:413a. Check system logs on 2001:41d0:801:2000:4c3a:433a:4b3a:413a [01:02:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cfae970d83eb...211d6d81aeea [01:02:14] [url] Comparing cfae970d83eb...211d6d81aeea · miraheze/puppet · GitHub | github.com [01:02:15] [02miraheze/puppet] 07paladox 03211d6d8 - nrpe: allow mon141 [01:03:05] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a. Check system logs on 2a10:6740::4c3a:433a:4b3a:413a [01:03:06] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2001:41d0:801:2000:4c3a:433a:4b3a:413a: Connection reset by peer [01:03:06] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2001:41d0:801:2000:4c3a:433a:4b3a:413a: Connection reset by peer [01:03:06] PROBLEM - cp21 Stunnel HTTP for puppet111 on cp21 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2001:41d0:801:2000:4c3a:433a:4b3a:413a. Check system logs on 2001:41d0:801:2000:4c3a:433a:4b3a:413a [01:03:07] PROBLEM - cp30 Stunnel HTTP for puppet111 on cp30 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2607:5300:201:3100:4c3a:433a:4b3a:413a: Connection reset by peer [01:03:07] PROBLEM - es141 conntrack_table_size on es141 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a. Check system logs on 2a10:6740::4c3a:433a:4b3a:413a [01:03:07] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2001:41d0:801:2000:4c3a:433a:4b3a:413a. Check system logs on 2001:41d0:801:2000:4c3a:433a:4b3a:413a [01:03:08] PROBLEM - graylog121 PowerDNS Recursor on graylog121 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a: Connection reset by peer [01:03:08] PROBLEM - gluster101 PowerDNS Recursor on gluster101 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a: Connection reset by peer [01:03:09] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a: Connection reset by peer [01:03:09] PROBLEM - puppet111 ferm_active on puppet111 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a: Connection reset by peer [01:03:09] PROBLEM - www.wikitipp.de - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikitipp.de and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:03:09] PROBLEM - phab121 phd on phab121 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2a10:6740::4c3a:433a:4b3a:413a: Connection reset by peer [01:04:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d94827d1097c [01:04:06] [02miraheze/puppet] 07paladox 03d94827d - Depool mon111 and pool in mon141 [01:04:08] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [01:04:09] [url] Page not found · GitHub · GitHub | github.com [01:04:09] [02puppet] 07paladox opened pull request 03#2746: Depool mon111 and pool in mon141 - 13https://github.com/miraheze/puppet/pull/2746 [01:04:10] ... [01:04:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d94827d1097c...364d74f55460 [01:04:45] [url] Comparing d94827d1097c...364d74f55460 · miraheze/puppet · GitHub | github.com [01:04:46] [02miraheze/puppet] 07paladox 03364d74f - Update default.vcl [01:04:48] [02puppet] 07paladox synchronize pull request 03#2746: Depool mon111 and pool in mon141 - 13https://github.com/miraheze/puppet/pull/2746 [01:04:48] [url] Page not found · GitHub · GitHub | github.com [01:04:58] [02puppet] 07paladox closed pull request 03#2746: Depool mon111 and pool in mon141 - 13https://github.com/miraheze/puppet/pull/2746 [01:04:58] [url] Page not found · GitHub · GitHub | github.com [01:05:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/211d6d81aeea...89eb62125bd5 [01:05:00] [url] Comparing 211d6d81aeea...89eb62125bd5 · miraheze/puppet · GitHub | github.com [01:05:01] [02miraheze/puppet] 07paladox 0389eb621 - Depool mon111 and pool in mon141 (#2746) [01:05:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [01:05:04] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [01:05:05] [url] Page not found · GitHub · GitHub | github.com [01:05:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b7f5658da28c...b36df435c5c6 [01:05:36] [url] Comparing b7f5658da28c...b36df435c5c6 · miraheze/ssl · GitHub | github.com [01:05:37] [02miraheze/ssl] 07MirahezeSSLBot 03b36df43 - Bot: Update SSL cert for www.zetapedia.com [01:07:37] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [01:07:37] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 3433 MB (47% inode=82%); [01:07:37] RECOVERY - prometheus131 APT on prometheus131 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [01:07:37] RECOVERY - mem131 NTP time on mem131 is OK: NTP OK: Offset 0.005078494549 secs [01:07:37] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [01:07:37] RECOVERY - reports121 conntrack_table_size on reports121 is OK: OK: nf_conntrack is 0 % full [01:07:38] RECOVERY - mem131 conntrack_table_size on mem131 is OK: OK: nf_conntrack is 0 % full [01:07:40] RECOVERY - mon141 IRCEcho on mon141 is OK: PROCS OK: 1 process with args '/usr/local/bin/ircecho' [01:07:46] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [01:07:48] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.89, 1.39, 1.28 [01:07:49] RECOVERY - mail121 conntrack_table_size on mail121 is OK: OK: nf_conntrack is 0 % full [01:07:50] RECOVERY - mem131 APT on mem131 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [01:07:51] RECOVERY - mon141 Check correctness of the icinga configuration on mon141 is OK: Icinga configuration is correct [01:07:52] RECOVERY - reports121 php-fpm on reports121 is OK: PROCS OK: 7 processes with command name 'php-fpm7.4' [01:07:54] RECOVERY - prometheus131 NTP time on prometheus131 is OK: NTP OK: Offset 0.0008599758148 secs [01:07:54] RECOVERY - mon141 php-fpm on mon141 is OK: PROCS OK: 4 processes with command name 'php-fpm7.4' [01:07:55] RECOVERY - reports121 ferm_active on reports121 is OK: OK ferm input default policy is set [01:07:55] RECOVERY - phab121 PowerDNS Recursor on phab121 is OK: DNS OK: 0.742 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [01:07:56] RECOVERY - mem101 NTP time on mem101 is OK: NTP OK: Offset 0.00132599473 secs [01:07:56] RECOVERY - mail121 ferm_active on mail121 is OK: OK ferm input default policy is set [01:07:57] RECOVERY - mail121 PowerDNS Recursor on mail121 is OK: DNS OK: 0.335 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [01:07:59] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28488 bytes in 5.038 second response time [01:07:59] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.05, 0.10, 0.11 [01:08:00] RECOVERY - phab121 Disk Space on phab121 is OK: DISK OK - free space: / 5787 MB (21% inode=84%); [01:08:00] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 15 processes with command name 'php-fpm7.4' [01:08:00] RECOVERY - mem101 Puppet on mem101 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [01:08:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:08:51] RECOVERY - mw122 nutcracker process on mw122 is OK: PROCS OK: 1 process with UID = 116 (nutcracker), command name 'nutcracker' [01:08:51] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:08:52] RECOVERY - mem101 PowerDNS Recursor on mem101 is OK: DNS OK: 0.223 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [01:08:52] RECOVERY - mw142 JobRunner Service on mw142 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [01:08:53] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.09, 0.08, 0.03 [01:08:53] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:08:53] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.340 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [01:08:55] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [01:08:55] RECOVERY - mail121 NTP time on mail121 is OK: NTP OK: Offset 0.001077860594 secs [01:08:55] RECOVERY - phab121 phd on phab121 is OK: PROCS OK: 1 process with args 'phd' [01:08:55] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.058 second response time [01:08:55] RECOVERY - ns2 ferm_active on ns2 is OK: OK ferm input default policy is set [01:08:55] RECOVERY - puppet111 ferm_active on puppet111 is OK: OK ferm input default policy is set [01:08:56] RECOVERY - ns2 Disk Space on ns2 is OK: DISK OK - free space: / 7302 MB (77% inode=87%); [01:17:36] !log shutdown mon111 [01:17:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:17:42] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 1% [01:17:42] RECOVERY - cp30 PowerDNS Recursor on cp30 is OK: DNS OK: 0.156 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [01:17:42] RECOVERY - cp31 Stunnel HTTP for puppet111 on cp31 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.239 second response time [01:17:42] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [01:17:42] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 10508 MB (27% inode=96%); [01:17:42] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.125 second response time [01:17:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.27, 1.80, 1.61 [01:17:45] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.669 second response time [01:17:50] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.118 second response time [01:17:51] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28488 bytes in 2.124 second response time [01:17:52] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [01:17:52] RECOVERY - mon111 Disk Space on mon111 is OK: DISK OK - free space: / 171880 MB (97% inode=98%); [01:17:54] RECOVERY - mon111 PowerDNS Recursor on mon111 is OK: DNS OK: 0.136 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [01:17:55] RECOVERY - mon111 IRC Log Bot on mon111 is OK: PROCS OK: 1 process with args 'adminlogbot.py' [01:19:06] PROBLEM - mon111 Puppet on mon111 is CRITICAL: connect to address 2a10:6740::6:205 port 5666: No route to hostconnect to host 2a10:6740::6:205 port 5666: No route to host [01:21:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.27, 2.09, 1.84 [01:21:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.11, 2.82, 3.17 [01:21:36] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 2% [01:21:37] RECOVERY - cp31 Stunnel HTTP for puppet111 on cp31 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.233 second response time [01:21:37] RECOVERY - cp30 PowerDNS Recursor on cp30 is OK: DNS OK: 0.270 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [01:21:39] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 10492 MB (27% inode=96%); [01:21:41] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.748 second response time [01:21:42] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.559 second response time [01:21:43] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.346 second response time [01:21:49] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.450 second response time [01:21:50] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 1.780 second response time [01:24:49] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/19716e0a7980...0477f8d737af [01:24:50] [url] Comparing 19716e0a7980...0477f8d737af · miraheze/mw-config · GitHub | github.com [01:24:50] [02miraheze/mw-config] 07paladox 030477f8d - Switch rc server to mon141 [01:24:54] PROBLEM - mon111 grafana.miraheze.org HTTPS on mon111 is CRITICAL: connect to address 2a10:6740::6:205 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [01:25:24] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.18, 1.74, 1.77 [01:26:00] PROBLEM - ping6 on mon111 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:205) [01:26:20] PROBLEM - mon111 SSH on mon111 is CRITICAL: connect to address 2a10:6740::6:205 and port 22: No route to host [01:26:40] PROBLEM - mon111 HTTPS on mon111 is CRITICAL: connect to address 2a10:6740::6:205 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [01:26:46] PROBLEM - Host mon111 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:205) [01:26:46] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: connect to address 2a10:6740::6:205 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [01:27:00] miraheze/mw-config - paladox the build passed. [01:27:18] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.03, 1.84, 1.79 [01:27:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.89, 3.09, 3.13 [01:27:31] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.41, 1.83, 1.83 [01:27:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/89eb62125bd5...9332f0d19945 [01:27:45] [url] Comparing 89eb62125bd5...9332f0d19945 · miraheze/puppet · GitHub | github.com [01:27:46] [02miraheze/puppet] 07paladox 039332f0d - Update logsalmsg [01:28:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9332f0d19945...b61f30a1d634 [01:28:17] [url] Comparing 9332f0d19945...b61f30a1d634 · miraheze/puppet · GitHub | github.com [01:28:18] [02miraheze/puppet] 07paladox 03b61f30a - nrpe: remove mon111 [01:31:07] PROBLEM - mon111 grafana.miraheze.org HTTPS on mon111 is CRITICAL: connect to address 2a10:6740::6:205 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [01:31:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.26, 1.50, 1.66 [01:32:42] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.36, 1.62 [01:34:14] PROBLEM - mon111 grafana.miraheze.org HTTPS on mon111 is CRITICAL: connect to address 2a10:6740::6:205 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [01:35:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d772a4558893 [01:35:47] [02miraheze/puppet] 07paladox 03d772a45 - mon141: lower MaxConcurrentChecks to 40 [01:35:49] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [01:35:49] [url] Page not found · GitHub · GitHub | github.com [01:35:50] [02puppet] 07paladox opened pull request 03#2747: mon141: lower MaxConcurrentChecks to 40 - 13https://github.com/miraheze/puppet/pull/2747 [01:35:51] [url] Page not found · GitHub · GitHub | github.com [01:35:56] [02puppet] 07paladox closed pull request 03#2747: mon141: lower MaxConcurrentChecks to 40 - 13https://github.com/miraheze/puppet/pull/2747 [01:35:56] [url] Page not found · GitHub · GitHub | github.com [01:35:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b61f30a1d634...381b986a51d1 [01:35:58] [url] Comparing b61f30a1d634...381b986a51d1 · miraheze/puppet · GitHub | github.com [01:35:59] [02miraheze/puppet] 07paladox 03381b986 - mon141: lower MaxConcurrentChecks to 40 (#2747) [01:36:00] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [01:36:00] [url] Page not found · GitHub · GitHub | github.com [01:36:02] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [01:36:29] PROBLEM - mon111 grafana.miraheze.org HTTPS on mon111 is CRITICAL: connect to address 2a10:6740::6:205 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [01:36:35] RECOVERY - www.zetapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zetapedia.com' will expire on Tue 01 Nov 2022 00:05:29 GMT +0000. [01:39:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.93, 1.16, 1.44 [01:39:46] RECOVERY - www.zetapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zetapedia.com' will expire on Tue 01 Nov 2022 00:05:29 GMT +0000. [01:39:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.15, 1.18, 1.41 [01:40:36] PROBLEM - mon111 SSH on mon111 is CRITICAL: connect to address 2a10:6740::6:205 and port 22: No route to host [01:40:45] PROBLEM - ping6 on mon111 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:205) [01:40:58] PROBLEM - mon111 HTTPS on mon111 is CRITICAL: connect to address 2a10:6740::6:205 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [01:41:08] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: connect to address 2a10:6740::6:205 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [01:41:08] PROBLEM - Host mon111 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:205) [01:41:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.71, 8.98, 11.98 [01:43:29] !log delete vm mon111 [01:43:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:46:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.04, 2.99, 3.84 [01:49:19] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [01:49:27] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:27] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:32] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [01:49:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 1.10, 4.44, 8.89 [01:49:33] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:33] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:40] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [01:49:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 1.09, 4.57, 9.78 [01:49:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.24, 1.70, 3.18 [01:50:00] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:03] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [01:50:03] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [01:50:09] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:10] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [01:50:13] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:14] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:14] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:15] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:16] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:17] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:17] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:19] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:20] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:22] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:33] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:35] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:13] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [01:51:19] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [01:51:20] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [01:51:22] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [01:51:23] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.462 second response time [01:51:29] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [01:51:29] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [01:51:29] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.020 second response time [01:51:31] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.312 second response time [01:51:37] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is WARNING: WARNING - NGINX Error Rate is 57% [01:51:39] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.023 second response time [01:51:56] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.320 second response time [01:52:00] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [01:52:00] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.026 second response time [01:52:02] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 52% [01:52:03] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [01:52:05] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.327 second response time [01:52:09] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [01:52:10] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.027 second response time [01:52:12] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.329 second response time [01:52:13] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.366 second response time [01:52:13] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.351 second response time [01:52:14] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.339 second response time [01:52:15] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.012 second response time [01:52:16] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.441 second response time [01:52:17] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.314 second response time [01:52:19] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.333 second response time [01:52:20] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.013 second response time [01:52:29] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.320 second response time [01:52:34] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.296 second response time [01:53:32] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 29% [01:54:01] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [01:55:11] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [01:55:11] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [01:55:26] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [01:56:00] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 4% [01:56:00] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [01:59:20] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:22] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:01:15] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.022 second response time [02:01:22] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.988 second response time [02:04:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-1/±0] 13https://github.com/miraheze/puppet/commit/650a0ea1e52f [02:04:06] [02miraheze/puppet] 07paladox 03650a0ea - Remove mon111 [02:04:08] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [02:04:08] [url] Page not found · GitHub · GitHub | github.com [02:04:09] [02puppet] 07paladox opened pull request 03#2748: Remove mon111 - 13https://github.com/miraheze/puppet/pull/2748 [02:04:10] ... [02:04:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/650a0ea1e52f...8239a8cae22a [02:04:36] [url] Comparing 650a0ea1e52f...8239a8cae22a · miraheze/puppet · GitHub | github.com [02:04:37] [02miraheze/puppet] 07paladox 038239a8c - Update site.pp [02:04:38] [02puppet] 07paladox synchronize pull request 03#2748: Remove mon111 - 13https://github.com/miraheze/puppet/pull/2748 [02:04:39] ... [02:04:43] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [02:04:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/puppet/compare/381b986a51d1...801c06893b36 [02:04:45] [url] Comparing 381b986a51d1...801c06893b36 · miraheze/puppet · GitHub | github.com [02:04:46] [02miraheze/puppet] 07paladox 03801c068 - Remove mon111 (#2748) [02:04:48] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [02:04:48] [url] Page not found · GitHub · GitHub | github.com [02:04:49] [02puppet] 07paladox closed pull request 03#2748: Remove mon111 - 13https://github.com/miraheze/puppet/pull/2748 [02:04:49] [url] Page not found · GitHub · GitHub | github.com [02:05:04] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/1e6db6b38b1d...d5d476527ac2 [02:05:05] [url] Comparing 1e6db6b38b1d...d5d476527ac2 · miraheze/dns · GitHub | github.com [02:05:06] [02miraheze/dns] 07paladox 03d5d4765 - Remove mon111 from dns [02:07:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.20, 11.00, 8.96 [02:07:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.82, 13.26, 10.68 [02:08:07] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:08:22] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:09:11] [02mediawiki] 07dependabot[bot] opened pull request 03#8433: Bump extensions/QuizGame from `a935e71` to `d4eb5f0` - 13https://github.com/miraheze/mediawiki/pull/8433 [02:09:12] [url] Page not found · GitHub · GitHub | github.com [02:09:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/QuizGame-d4eb5f0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/984a2b7bdd09 [02:09:14] [02miraheze/mediawiki] 07dependabot[bot] 03984a2b7 - Bump extensions/QuizGame from `a935e71` to `d4eb5f0` [02:09:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/QuizGame-d4eb5f0 - 13https://github.com/miraheze/mediawiki [02:09:17] [url] Page not found · GitHub · GitHub | github.com [02:09:17] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ShortDescription-4e24f71 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d1c82c0b7238 [02:09:19] [02miraheze/mediawiki] 07dependabot[bot] 03d1c82c0 - Bump extensions/ShortDescription from `bf89cf9` to `4e24f71` [02:09:20] [02mediawiki] 07dependabot[bot] opened pull request 03#8434: Bump extensions/ShortDescription from `bf89cf9` to `4e24f71` - 13https://github.com/miraheze/mediawiki/pull/8434 [02:09:21] [url] Page not found · GitHub · GitHub | github.com [02:09:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ShortDescription-4e24f71 - 13https://github.com/miraheze/mediawiki [02:09:22] ... [02:09:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Cosmos-ff3cc41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5f34cda881a5 [02:09:25] [02miraheze/mediawiki] 07dependabot[bot] 035f34cda - Bump skins/Cosmos from `d1163b5` to `ff3cc41` [02:09:26] [02mediawiki] 07dependabot[bot] opened pull request 03#8435: Bump skins/Cosmos from `d1163b5` to `ff3cc41` - 13https://github.com/miraheze/mediawiki/pull/8435 [02:09:27] ... [02:09:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Cosmos-ff3cc41 - 13https://github.com/miraheze/mediawiki [02:09:28] ... [02:09:29] [02mediawiki] 07dependabot[bot] opened pull request 03#8436: Bump extensions/MirahezeMagic from `8f6c75d` to `8fc0344` - 13https://github.com/miraheze/mediawiki/pull/8436 [02:09:30] [url] Page not found · GitHub · GitHub | github.com [02:09:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-8fc0344 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9c8d356fbd29 [02:09:32] [02miraheze/mediawiki] 07dependabot[bot] 039c8d356 - Bump extensions/MirahezeMagic from `8f6c75d` to `8fc0344` [02:09:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-8fc0344 - 13https://github.com/miraheze/mediawiki [02:09:34] [url] Page not found · GitHub · GitHub | github.com [02:09:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/DarkMode-e8ecbb1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/11d021b3ee07 [02:09:37] [02miraheze/mediawiki] 07dependabot[bot] 0311d021b - Bump extensions/DarkMode from `f9b0599` to `e8ecbb1` [02:09:38] [02mediawiki] 07dependabot[bot] opened pull request 03#8437: Bump extensions/DarkMode from `f9b0599` to `e8ecbb1` - 13https://github.com/miraheze/mediawiki/pull/8437 [02:09:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/DarkMode-e8ecbb1 - 13https://github.com/miraheze/mediawiki [02:09:41] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Report-d3fdcf3 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1ecf6724da91 [02:09:43] [02miraheze/mediawiki] 07dependabot[bot] 031ecf672 - Bump extensions/Report from `494fc48` to `d3fdcf3` [02:09:44] [02mediawiki] 07dependabot[bot] opened pull request 03#8438: Bump extensions/Report from `494fc48` to `d3fdcf3` - 13https://github.com/miraheze/mediawiki/pull/8438 [02:09:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Report-d3fdcf3 - 13https://github.com/miraheze/mediawiki [02:09:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/TabberNeue-2446821 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/76130fc972c1 [02:09:49] [02miraheze/mediawiki] 07dependabot[bot] 0376130fc - Bump extensions/TabberNeue from `6a40de7` to `2446821` [02:09:50] [02mediawiki] 07dependabot[bot] opened pull request 03#8439: Bump extensions/TabberNeue from `6a40de7` to `2446821` - 13https://github.com/miraheze/mediawiki/pull/8439 [02:09:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/TabberNeue-2446821 - 13https://github.com/miraheze/mediawiki [02:09:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/IncidentReporting-f4ca5f4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/54343d8fb76d [02:09:55] [02miraheze/mediawiki] 07dependabot[bot] 0354343d8 - Bump extensions/IncidentReporting from `5fee4ab` to `f4ca5f4` [02:09:56] [02mediawiki] 07dependabot[bot] opened pull request 03#8440: Bump extensions/IncidentReporting from `5fee4ab` to `f4ca5f4` - 13https://github.com/miraheze/mediawiki/pull/8440 [02:09:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/IncidentReporting-f4ca5f4 - 13https://github.com/miraheze/mediawiki [02:09:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-c76cddb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/140303d84193 [02:10:01] [02miraheze/mediawiki] 07dependabot[bot] 03140303d - Bump extensions/SemanticMediaWiki from `4fe6a36` to `c76cddb` [02:10:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#8434: Bump extensions/ShortDescription from `bf89cf9` to `4e24f71` - 13https://github.com/miraheze/mediawiki/pull/8434 [02:10:04] [02mediawiki] 07dependabot[bot] opened pull request 03#8441: Bump extensions/SemanticMediaWiki from `4fe6a36` to `c76cddb` - 13https://github.com/miraheze/mediawiki/pull/8441 [02:10:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-c76cddb - 13https://github.com/miraheze/mediawiki [02:10:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-a11aca4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/234964213476 [02:10:08] [02miraheze/mediawiki] 07dependabot[bot] 032349642 - Bump extensions/UniversalLanguageSelector from `06091ef` to `a11aca4` [02:10:10] [02mediawiki] 07dependabot[bot] opened pull request 03#8442: Bump extensions/UniversalLanguageSelector from `06091ef` to `a11aca4` - 13https://github.com/miraheze/mediawiki/pull/8442 [02:10:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-a11aca4 - 13https://github.com/miraheze/mediawiki [02:10:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#8435: Bump skins/Cosmos from `d1163b5` to `ff3cc41` - 13https://github.com/miraheze/mediawiki/pull/8435 [02:10:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#8433: Bump extensions/QuizGame from `a935e71` to `d4eb5f0` - 13https://github.com/miraheze/mediawiki/pull/8433 [02:10:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#8435: Bump skins/Cosmos from `d1163b5` to `ff3cc41` - 13https://github.com/miraheze/mediawiki/pull/8435 [02:10:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#8435: Bump skins/Cosmos from `d1163b5` to `ff3cc41` - 13https://github.com/miraheze/mediawiki/pull/8435 [02:10:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CreateWiki-fd29ec3 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3967e7ddf8ca [02:10:21] [02miraheze/mediawiki] 07dependabot[bot] 033967e7d - Bump extensions/CreateWiki from `0a62452` to `fd29ec3` [02:10:22] [02mediawiki] 07dependabot[bot] opened pull request 03#8443: Bump extensions/CreateWiki from `0a62452` to `fd29ec3` - 13https://github.com/miraheze/mediawiki/pull/8443 [02:10:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CreateWiki-fd29ec3 - 13https://github.com/miraheze/mediawiki [02:10:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#8436: Bump extensions/MirahezeMagic from `8f6c75d` to `8fc0344` - 13https://github.com/miraheze/mediawiki/pull/8436 [02:10:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Translate-4e84ff7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8f17becf7bf9 [02:10:28] [02miraheze/mediawiki] 07dependabot[bot] 038f17bec - Bump extensions/Translate from `d7618b5` to `4e84ff7` [02:10:29] [02mediawiki] 07dependabot[bot] opened pull request 03#8444: Bump extensions/Translate from `d7618b5` to `4e84ff7` - 13https://github.com/miraheze/mediawiki/pull/8444 [02:10:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Translate-4e84ff7 - 13https://github.com/miraheze/mediawiki [02:10:33] [02mediawiki] 07dependabot[bot] commented on pull request 03#8317: Bump extensions/Translate from `d7618b5` to `ffbe9ee` - 13https://github.com/miraheze/mediawiki/pull/8317#issuecomment-1203401700 [02:10:33] [url] Bump extensions/Translate from `d7618b5` to `ffbe9ee` by dependabot[bot] · Pull Request #8317 · miraheze/mediawiki · GitHub | github.com [02:10:34] [02mediawiki] 07dependabot[bot] closed pull request 03#8317: Bump extensions/Translate from `d7618b5` to `ffbe9ee` - 13https://github.com/miraheze/mediawiki/pull/8317 [02:10:34] [url] Page not found · GitHub · GitHub | github.com [02:10:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#8437: Bump extensions/DarkMode from `f9b0599` to `e8ecbb1` - 13https://github.com/miraheze/mediawiki/pull/8437 [02:10:36] [url] Page not found · GitHub · GitHub | github.com [02:10:37] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-ffbe9ee [02:10:39] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-ffbe9ee - 13https://github.com/miraheze/mediawiki [02:10:39] [url] Page not found · GitHub · GitHub | github.com [02:10:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#8437: Bump extensions/DarkMode from `f9b0599` to `e8ecbb1` - 13https://github.com/miraheze/mediawiki/pull/8437 [02:10:40] ... [02:10:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PageSchemas-a166354 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/60c2e1b5c8f3 [02:10:43] [02miraheze/mediawiki] 07dependabot[bot] 0360c2e1b - Bump extensions/PageSchemas from `ba1ec2b` to `a166354` [02:10:45] [02mediawiki] 07dependabot[bot] opened pull request 03#8445: Bump extensions/PageSchemas from `ba1ec2b` to `a166354` - 13https://github.com/miraheze/mediawiki/pull/8445 [02:10:45] ... [02:10:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#8437: Bump extensions/DarkMode from `f9b0599` to `e8ecbb1` - 13https://github.com/miraheze/mediawiki/pull/8437 [02:10:46] ... [02:10:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#8438: Bump extensions/Report from `494fc48` to `d3fdcf3` - 13https://github.com/miraheze/mediawiki/pull/8438 [02:10:48] [url] Page not found · GitHub · GitHub | github.com [02:10:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PageSchemas-a166354 - 13https://github.com/miraheze/mediawiki [02:10:49] [url] Page not found · GitHub · GitHub | github.com [02:10:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Cargo-409cf37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d612f1e290a3 [02:10:52] [02miraheze/mediawiki] 07dependabot[bot] 03d612f1e - Bump extensions/Cargo from `ee86e5f` to `409cf37` [02:10:54] [02mediawiki] 07dependabot[bot] opened pull request 03#8446: Bump extensions/Cargo from `ee86e5f` to `409cf37` - 13https://github.com/miraheze/mediawiki/pull/8446 [02:10:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Cargo-409cf37 - 13https://github.com/miraheze/mediawiki [02:10:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#8439: Bump extensions/TabberNeue from `6a40de7` to `2446821` - 13https://github.com/miraheze/mediawiki/pull/8439 [02:10:58] [02mediawiki] 07dependabot[bot] opened pull request 03#8447: Bump extensions/ApprovedRevs from `2c74834` to `4d2b002` - 13https://github.com/miraheze/mediawiki/pull/8447 [02:11:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ApprovedRevs-4d2b002 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/84e9fd710e26 [02:11:01] [02miraheze/mediawiki] 07dependabot[bot] 0384e9fd7 - Bump extensions/ApprovedRevs from `2c74834` to `4d2b002` [02:11:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#8439: Bump extensions/TabberNeue from `6a40de7` to `2446821` - 13https://github.com/miraheze/mediawiki/pull/8439 [02:11:04] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/d5d476527ac2...c0e592c0f2a9 [02:11:05] [url] Comparing d5d476527ac2...c0e592c0f2a9 · miraheze/dns · GitHub | github.com [02:11:06] [02miraheze/dns] 07paladox 03c0e592c - Add matomo131 to dns [02:11:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ApprovedRevs-4d2b002 - 13https://github.com/miraheze/mediawiki [02:11:07] [url] Page not found · GitHub · GitHub | github.com [02:11:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PollNY-b1c373f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/eb2a3da2a0fb [02:11:10] [02miraheze/mediawiki] 07dependabot[bot] 03eb2a3da - Bump extensions/PollNY from `6e009de` to `b1c373f` [02:11:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PollNY-b1c373f - 13https://github.com/miraheze/mediawiki [02:11:12] [url] Page not found · GitHub · GitHub | github.com [02:11:13] [02mediawiki] 07dependabot[bot] opened pull request 03#8448: Bump extensions/PollNY from `6e009de` to `b1c373f` - 13https://github.com/miraheze/mediawiki/pull/8448 [02:11:13] [url] Page not found · GitHub · GitHub | github.com [02:11:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#8440: Bump extensions/IncidentReporting from `5fee4ab` to `f4ca5f4` - 13https://github.com/miraheze/mediawiki/pull/8440 [02:11:15] ... [02:11:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SocialProfile-47728b1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2c09974b09c4 [02:11:18] [02miraheze/mediawiki] 07dependabot[bot] 032c09974 - Bump extensions/SocialProfile from `3c6ee3c` to `47728b1` [02:11:19] [02mediawiki] 07dependabot[bot] opened pull request 03#8449: Bump extensions/SocialProfile from `3c6ee3c` to `47728b1` - 13https://github.com/miraheze/mediawiki/pull/8449 [02:11:19] ... [02:11:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SocialProfile-47728b1 - 13https://github.com/miraheze/mediawiki [02:11:21] ... [02:11:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#8443: Bump extensions/CreateWiki from `0a62452` to `fd29ec3` - 13https://github.com/miraheze/mediawiki/pull/8443 [02:11:22] [url] Page not found · GitHub · GitHub | github.com [02:11:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#8442: Bump extensions/UniversalLanguageSelector from `06091ef` to `a11aca4` - 13https://github.com/miraheze/mediawiki/pull/8442 [02:11:24] [url] Page not found · GitHub · GitHub | github.com [02:11:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#8443: Bump extensions/CreateWiki from `0a62452` to `fd29ec3` - 13https://github.com/miraheze/mediawiki/pull/8443 [02:11:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#8445: Bump extensions/PageSchemas from `ba1ec2b` to `a166354` - 13https://github.com/miraheze/mediawiki/pull/8445 [02:11:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ImportDump-61eeea3 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ff31a61ecca4 [02:11:30] [02miraheze/mediawiki] 07dependabot[bot] 03ff31a61 - Bump extensions/ImportDump from `ab3032b` to `61eeea3` [02:11:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#8446: Bump extensions/Cargo from `ee86e5f` to `409cf37` - 13https://github.com/miraheze/mediawiki/pull/8446 [02:11:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ImportDump-61eeea3 - 13https://github.com/miraheze/mediawiki [02:11:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#8447: Bump extensions/ApprovedRevs from `2c74834` to `4d2b002` - 13https://github.com/miraheze/mediawiki/pull/8447 [02:11:36] [02mediawiki] 07dependabot[bot] opened pull request 03#8450: Bump extensions/ImportDump from `ab3032b` to `61eeea3` - 13https://github.com/miraheze/mediawiki/pull/8450 [02:11:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MatomoAnalytics-5791381 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e6efc29e8808 [02:11:39] [02miraheze/mediawiki] 07dependabot[bot] 03e6efc29 - Bump extensions/MatomoAnalytics from `1260bb1` to `5791381` [02:11:40] [02mediawiki] 07dependabot[bot] opened pull request 03#8451: Bump extensions/MatomoAnalytics from `1260bb1` to `5791381` - 13https://github.com/miraheze/mediawiki/pull/8451 [02:11:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MatomoAnalytics-5791381 - 13https://github.com/miraheze/mediawiki [02:11:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#8441: Bump extensions/SemanticMediaWiki from `4fe6a36` to `c76cddb` - 13https://github.com/miraheze/mediawiki/pull/8441 [02:11:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PageForms-e92137e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7ed039d07640 [02:11:46] [02miraheze/mediawiki] 07dependabot[bot] 037ed039d - Bump extensions/PageForms from `da3886e` to `e92137e` [02:11:48] [02mediawiki] 07dependabot[bot] opened pull request 03#8452: Bump extensions/PageForms from `da3886e` to `e92137e` - 13https://github.com/miraheze/mediawiki/pull/8452 [02:11:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PageForms-e92137e - 13https://github.com/miraheze/mediawiki [02:11:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/VoteNY-35b1d55 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ff7e8c1dec6a [02:11:52] [02miraheze/mediawiki] 07dependabot[bot] 03ff7e8c1 - Bump extensions/VoteNY from `34274da` to `35b1d55` [02:11:54] [02mediawiki] 07dependabot[bot] opened pull request 03#8453: Bump extensions/VoteNY from `34274da` to `35b1d55` - 13https://github.com/miraheze/mediawiki/pull/8453 [02:11:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#8444: Bump extensions/Translate from `d7618b5` to `4e84ff7` - 13https://github.com/miraheze/mediawiki/pull/8444 [02:11:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/VoteNY-35b1d55 - 13https://github.com/miraheze/mediawiki [02:11:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#8444: Bump extensions/Translate from `d7618b5` to `4e84ff7` - 13https://github.com/miraheze/mediawiki/pull/8444 [02:12:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#8444: Bump extensions/Translate from `d7618b5` to `4e84ff7` - 13https://github.com/miraheze/mediawiki/pull/8444 [02:12:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Citizen-6384d02 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ff5bf3459c56 [02:12:03] [02miraheze/mediawiki] 07dependabot[bot] 03ff5bf34 - Bump skins/Citizen from `e9139f0` to `6384d02` [02:12:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#8444: Bump extensions/Translate from `d7618b5` to `4e84ff7` - 13https://github.com/miraheze/mediawiki/pull/8444 [02:12:06] [02mediawiki] 07dependabot[bot] opened pull request 03#8454: Bump skins/Citizen from `e9139f0` to `6384d02` - 13https://github.com/miraheze/mediawiki/pull/8454 [02:12:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Citizen-6384d02 - 13https://github.com/miraheze/mediawiki [02:12:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ManageWiki-870ab9a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/371a9df61f3a [02:12:10] [02miraheze/mediawiki] 07dependabot[bot] 03371a9df - Bump extensions/ManageWiki from `aad1060` to `870ab9a` [02:12:12] [02mediawiki] 07dependabot[bot] opened pull request 03#8455: Bump extensions/ManageWiki from `aad1060` to `870ab9a` - 13https://github.com/miraheze/mediawiki/pull/8455 [02:12:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ManageWiki-870ab9a - 13https://github.com/miraheze/mediawiki [02:12:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#8448: Bump extensions/PollNY from `6e009de` to `b1c373f` - 13https://github.com/miraheze/mediawiki/pull/8448 [02:12:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ExternalData-e21a01d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/743027736a82 [02:12:18] [02miraheze/mediawiki] 07dependabot[bot] 037430277 - Bump extensions/ExternalData from `a5a4dc9` to `e21a01d` [02:12:19] [02mediawiki] 07dependabot[bot] opened pull request 03#8456: Bump extensions/ExternalData from `a5a4dc9` to `e21a01d` - 13https://github.com/miraheze/mediawiki/pull/8456 [02:12:20] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28480 bytes in 6.697 second response time [02:12:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ExternalData-e21a01d - 13https://github.com/miraheze/mediawiki [02:12:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#8449: Bump extensions/SocialProfile from `3c6ee3c` to `47728b1` - 13https://github.com/miraheze/mediawiki/pull/8449 [02:12:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Comments-d73d5be [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/84c65a7483cd [02:12:25] [02miraheze/mediawiki] 07dependabot[bot] 0384c65a7 - Bump extensions/Comments from `6d86851` to `d73d5be` [02:12:26] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.413 second response time [02:12:27] [02mediawiki] 07dependabot[bot] opened pull request 03#8457: Bump extensions/Comments from `6d86851` to `d73d5be` - 13https://github.com/miraheze/mediawiki/pull/8457 [02:12:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Comments-d73d5be - 13https://github.com/miraheze/mediawiki [02:12:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MintyDocs-f36d9b5 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3a665f6f4137 [02:12:31] [02miraheze/mediawiki] 07dependabot[bot] 033a665f6 - Bump extensions/MintyDocs from `591a2f2` to `f36d9b5` [02:12:33] [02mediawiki] 07dependabot[bot] opened pull request 03#8458: Bump extensions/MintyDocs from `591a2f2` to `f36d9b5` - 13https://github.com/miraheze/mediawiki/pull/8458 [02:12:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MintyDocs-f36d9b5 - 13https://github.com/miraheze/mediawiki [02:12:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#8450: Bump extensions/ImportDump from `ab3032b` to `61eeea3` - 13https://github.com/miraheze/mediawiki/pull/8450 [02:12:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#8451: Bump extensions/MatomoAnalytics from `1260bb1` to `5791381` - 13https://github.com/miraheze/mediawiki/pull/8451 [02:12:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#8453: Bump extensions/VoteNY from `34274da` to `35b1d55` - 13https://github.com/miraheze/mediawiki/pull/8453 [02:12:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#8455: Bump extensions/ManageWiki from `aad1060` to `870ab9a` - 13https://github.com/miraheze/mediawiki/pull/8455 [02:12:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#8455: Bump extensions/ManageWiki from `aad1060` to `870ab9a` - 13https://github.com/miraheze/mediawiki/pull/8455 [02:12:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#8452: Bump extensions/PageForms from `da3886e` to `e92137e` - 13https://github.com/miraheze/mediawiki/pull/8452 [02:12:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#8456: Bump extensions/ExternalData from `a5a4dc9` to `e21a01d` - 13https://github.com/miraheze/mediawiki/pull/8456 [02:12:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#8454: Bump skins/Citizen from `e9139f0` to `6384d02` - 13https://github.com/miraheze/mediawiki/pull/8454 [02:12:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#8454: Bump skins/Citizen from `e9139f0` to `6384d02` - 13https://github.com/miraheze/mediawiki/pull/8454 [02:12:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#8457: Bump extensions/Comments from `6d86851` to `d73d5be` - 13https://github.com/miraheze/mediawiki/pull/8457 [02:12:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#8458: Bump extensions/MintyDocs from `591a2f2` to `f36d9b5` - 13https://github.com/miraheze/mediawiki/pull/8458 [02:12:52] [02mediawiki] 07Universal-Omega labeled pull request 03#8433: Bump extensions/QuizGame from `a935e71` to `d4eb5f0` - 13https://github.com/miraheze/mediawiki/pull/8433 [02:12:54] [02mediawiki] 07Universal-Omega labeled pull request 03#8434: Bump extensions/ShortDescription from `bf89cf9` to `4e24f71` - 13https://github.com/miraheze/mediawiki/pull/8434 [02:12:55] [02mediawiki] 07Universal-Omega labeled pull request 03#8436: Bump extensions/MirahezeMagic from `8f6c75d` to `8fc0344` - 13https://github.com/miraheze/mediawiki/pull/8436 [02:12:57] [02mediawiki] 07Universal-Omega labeled pull request 03#8438: Bump extensions/Report from `494fc48` to `d3fdcf3` - 13https://github.com/miraheze/mediawiki/pull/8438 [02:12:58] [02mediawiki] 07Universal-Omega labeled pull request 03#8439: Bump extensions/TabberNeue from `6a40de7` to `2446821` - 13https://github.com/miraheze/mediawiki/pull/8439 [02:13:00] [02mediawiki] 07Universal-Omega labeled pull request 03#8440: Bump extensions/IncidentReporting from `5fee4ab` to `f4ca5f4` - 13https://github.com/miraheze/mediawiki/pull/8440 [02:13:01] [02mediawiki] 07Universal-Omega labeled pull request 03#8441: Bump extensions/SemanticMediaWiki from `4fe6a36` to `c76cddb` - 13https://github.com/miraheze/mediawiki/pull/8441 [02:13:03] [02mediawiki] 07Universal-Omega labeled pull request 03#8442: Bump extensions/UniversalLanguageSelector from `06091ef` to `a11aca4` - 13https://github.com/miraheze/mediawiki/pull/8442 [02:13:04] [02mediawiki] 07Universal-Omega labeled pull request 03#8443: Bump extensions/CreateWiki from `0a62452` to `fd29ec3` - 13https://github.com/miraheze/mediawiki/pull/8443 [02:13:06] [02mediawiki] 07Universal-Omega labeled pull request 03#8445: Bump extensions/PageSchemas from `ba1ec2b` to `a166354` - 13https://github.com/miraheze/mediawiki/pull/8445 [02:13:07] [02mediawiki] 07Universal-Omega labeled pull request 03#8446: Bump extensions/Cargo from `ee86e5f` to `409cf37` - 13https://github.com/miraheze/mediawiki/pull/8446 [02:13:09] [02mediawiki] 07Universal-Omega labeled pull request 03#8447: Bump extensions/ApprovedRevs from `2c74834` to `4d2b002` - 13https://github.com/miraheze/mediawiki/pull/8447 [02:13:10] [02mediawiki] 07Universal-Omega labeled pull request 03#8448: Bump extensions/PollNY from `6e009de` to `b1c373f` - 13https://github.com/miraheze/mediawiki/pull/8448 [02:13:12] [02mediawiki] 07Universal-Omega labeled pull request 03#8449: Bump extensions/SocialProfile from `3c6ee3c` to `47728b1` - 13https://github.com/miraheze/mediawiki/pull/8449 [02:13:13] [02mediawiki] 07Universal-Omega labeled pull request 03#8450: Bump extensions/ImportDump from `ab3032b` to `61eeea3` - 13https://github.com/miraheze/mediawiki/pull/8450 [02:13:15] [02mediawiki] 07Universal-Omega labeled pull request 03#8451: Bump extensions/MatomoAnalytics from `1260bb1` to `5791381` - 13https://github.com/miraheze/mediawiki/pull/8451 [02:13:16] [02mediawiki] 07Universal-Omega labeled pull request 03#8452: Bump extensions/PageForms from `da3886e` to `e92137e` - 13https://github.com/miraheze/mediawiki/pull/8452 [02:13:18] [02mediawiki] 07Universal-Omega labeled pull request 03#8453: Bump extensions/VoteNY from `34274da` to `35b1d55` - 13https://github.com/miraheze/mediawiki/pull/8453 [02:13:19] [02mediawiki] 07Universal-Omega labeled pull request 03#8455: Bump extensions/ManageWiki from `aad1060` to `870ab9a` - 13https://github.com/miraheze/mediawiki/pull/8455 [02:13:21] [02mediawiki] 07Universal-Omega labeled pull request 03#8456: Bump extensions/ExternalData from `a5a4dc9` to `e21a01d` - 13https://github.com/miraheze/mediawiki/pull/8456 [02:13:22] [02mediawiki] 07Universal-Omega labeled pull request 03#8457: Bump extensions/Comments from `6d86851` to `d73d5be` - 13https://github.com/miraheze/mediawiki/pull/8457 [02:13:24] [02mediawiki] 07Universal-Omega labeled pull request 03#8458: Bump extensions/MintyDocs from `591a2f2` to `f36d9b5` - 13https://github.com/miraheze/mediawiki/pull/8458 [02:13:24] [url] Page not found · GitHub · GitHub | github.com [02:13:25] [02mediawiki] 07github-actions[bot] closed pull request 03#8434: Bump extensions/ShortDescription from `bf89cf9` to `4e24f71` - 13https://github.com/miraheze/mediawiki/pull/8434 [02:13:27] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/9e64d48cb36e...4fd86444e2d0 [02:13:27] [url] Comparing 9e64d48cb36e...4fd86444e2d0 · miraheze/mediawiki · GitHub | github.com [02:13:28] [02miraheze/mediawiki] 07dependabot[bot] 034fd8644 - Bump extensions/ShortDescription from `bf89cf9` to `4e24f71` (#8434) [02:13:30] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ShortDescription-4e24f71 [02:13:31] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ShortDescription-4e24f71 - 13https://github.com/miraheze/mediawiki [02:13:32] [url] Page not found · GitHub · GitHub | github.com [02:13:33] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8434: Bump extensions/ShortDescription from `bf89cf9` to `4e24f71` - 13https://github.com/miraheze/mediawiki/pull/8434 [02:13:33] [url] Page not found · GitHub · GitHub | github.com [02:13:34] [02mediawiki] 07github-actions[bot] closed pull request 03#8440: Bump extensions/IncidentReporting from `5fee4ab` to `f4ca5f4` - 13https://github.com/miraheze/mediawiki/pull/8440 [02:13:35] ... [02:13:36] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/IncidentReporting-f4ca5f4 [02:13:37] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/4fd86444e2d0...05f7a8585893 [02:13:38] [url] Comparing 4fd86444e2d0...05f7a8585893 · miraheze/mediawiki · GitHub | github.com [02:13:39] [02miraheze/mediawiki] 07dependabot[bot] 0305f7a85 - Bump extensions/IncidentReporting from `5fee4ab` to `f4ca5f4` (#8440) [02:13:40] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/IncidentReporting-f4ca5f4 - 13https://github.com/miraheze/mediawiki [02:13:40] ... [02:13:42] [02mediawiki] 07github-actions[bot] closed pull request 03#8443: Bump extensions/CreateWiki from `0a62452` to `fd29ec3` - 13https://github.com/miraheze/mediawiki/pull/8443 [02:13:42] [url] Page not found · GitHub · GitHub | github.com [02:13:43] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/05f7a8585893...9687ec71e9cd [02:13:44] [url] Comparing 05f7a8585893...9687ec71e9cd · miraheze/mediawiki · GitHub | github.com [02:13:45] [02miraheze/mediawiki] 07dependabot[bot] 039687ec7 - Bump extensions/CreateWiki from `0a62452` to `fd29ec3` (#8443) [02:13:46] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8440: Bump extensions/IncidentReporting from `5fee4ab` to `f4ca5f4` - 13https://github.com/miraheze/mediawiki/pull/8440 [02:13:46] [url] Page not found · GitHub · GitHub | github.com [02:13:48] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/CreateWiki-fd29ec3 [02:13:49] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/CreateWiki-fd29ec3 - 13https://github.com/miraheze/mediawiki [02:13:49] [url] Page not found · GitHub · GitHub | github.com [02:13:51] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8443: Bump extensions/CreateWiki from `0a62452` to `fd29ec3` - 13https://github.com/miraheze/mediawiki/pull/8443 [02:13:51] [url] Page not found · GitHub · GitHub | github.com [02:13:52] [02mediawiki] 07github-actions[bot] closed pull request 03#8445: Bump extensions/PageSchemas from `ba1ec2b` to `a166354` - 13https://github.com/miraheze/mediawiki/pull/8445 [02:13:53] [url] Page not found · GitHub · GitHub | github.com [02:13:54] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageSchemas-a166354 [02:13:55] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/9687ec71e9cd...b23b40322151 [02:13:56] [url] Comparing 9687ec71e9cd...b23b40322151 · miraheze/mediawiki · GitHub | github.com [02:13:57] [02miraheze/mediawiki] 07dependabot[bot] 03b23b403 - Bump extensions/PageSchemas from `ba1ec2b` to `a166354` (#8445) [02:13:58] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageSchemas-a166354 - 13https://github.com/miraheze/mediawiki [02:13:58] ... [02:14:00] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8445: Bump extensions/PageSchemas from `ba1ec2b` to `a166354` - 13https://github.com/miraheze/mediawiki/pull/8445 [02:14:00] ... [02:14:01] [02mediawiki] 07github-actions[bot] closed pull request 03#8446: Bump extensions/Cargo from `ee86e5f` to `409cf37` - 13https://github.com/miraheze/mediawiki/pull/8446 [02:14:02] [url] Page not found · GitHub · GitHub | github.com [02:14:03] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/b23b40322151...d6727ba050c6 [02:14:03] [url] Comparing b23b40322151...d6727ba050c6 · miraheze/mediawiki · GitHub | github.com [02:14:04] [02miraheze/mediawiki] 07dependabot[bot] 03d6727ba - Bump extensions/Cargo from `ee86e5f` to `409cf37` (#8446) [02:14:06] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Cargo-409cf37 - 13https://github.com/miraheze/mediawiki [02:14:06] [url] Page not found · GitHub · GitHub | github.com [02:14:07] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Cargo-409cf37 [02:14:09] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8446: Bump extensions/Cargo from `ee86e5f` to `409cf37` - 13https://github.com/miraheze/mediawiki/pull/8446 [02:14:09] [url] Page not found · GitHub · GitHub | github.com [02:14:10] [02mediawiki] 07github-actions[bot] closed pull request 03#8439: Bump extensions/TabberNeue from `6a40de7` to `2446821` - 13https://github.com/miraheze/mediawiki/pull/8439 [02:14:11] [url] Page not found · GitHub · GitHub | github.com [02:14:12] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/d6727ba050c6...0e7aefbf7a30 [02:14:12] [url] Comparing d6727ba050c6...0e7aefbf7a30 · miraheze/mediawiki · GitHub | github.com [02:14:13] [02miraheze/mediawiki] 07dependabot[bot] 030e7aefb - Bump extensions/TabberNeue from `6a40de7` to `2446821` (#8439) [02:14:15] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/TabberNeue-2446821 [02:14:16] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/TabberNeue-2446821 - 13https://github.com/miraheze/mediawiki [02:14:16] [url] Page not found · GitHub · GitHub | github.com [02:14:18] [02mediawiki] 07github-actions[bot] closed pull request 03#8441: Bump extensions/SemanticMediaWiki from `4fe6a36` to `c76cddb` - 13https://github.com/miraheze/mediawiki/pull/8441 [02:14:18] ... [02:14:19] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8439: Bump extensions/TabberNeue from `6a40de7` to `2446821` - 13https://github.com/miraheze/mediawiki/pull/8439 [02:14:19] ... [02:14:21] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/0e7aefbf7a30...d87fbe0fa723 [02:14:21] [url] Comparing 0e7aefbf7a30...d87fbe0fa723 · miraheze/mediawiki · GitHub | github.com [02:14:22] [02miraheze/mediawiki] 07dependabot[bot] 03d87fbe0 - Bump extensions/SemanticMediaWiki from `4fe6a36` to `c76cddb` (#8441) [02:14:24] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-c76cddb [02:14:25] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8441: Bump extensions/SemanticMediaWiki from `4fe6a36` to `c76cddb` - 13https://github.com/miraheze/mediawiki/pull/8441 [02:14:25] [url] Page not found · GitHub · GitHub | github.com [02:14:27] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-c76cddb - 13https://github.com/miraheze/mediawiki [02:14:27] [url] Page not found · GitHub · GitHub | github.com [02:14:28] [02mediawiki] 07github-actions[bot] closed pull request 03#8455: Bump extensions/ManageWiki from `aad1060` to `870ab9a` - 13https://github.com/miraheze/mediawiki/pull/8455 [02:14:29] [url] Page not found · GitHub · GitHub | github.com [02:14:30] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/d87fbe0fa723...20ca8697a399 [02:14:30] [url] Comparing d87fbe0fa723...20ca8697a399 · miraheze/mediawiki · GitHub | github.com [02:14:31] [02miraheze/mediawiki] 07dependabot[bot] 0320ca869 - Bump extensions/ManageWiki from `aad1060` to `870ab9a` (#8455) [02:14:33] [02mediawiki] 07github-actions[bot] closed pull request 03#8438: Bump extensions/Report from `494fc48` to `d3fdcf3` - 13https://github.com/miraheze/mediawiki/pull/8438 [02:14:33] [url] Page not found · GitHub · GitHub | github.com [02:14:34] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ManageWiki-870ab9a [02:14:36] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/20ca8697a399...52996b3d6f03 [02:14:36] [url] Comparing 20ca8697a399...52996b3d6f03 · miraheze/mediawiki · GitHub | github.com [02:14:37] [02miraheze/mediawiki] 07dependabot[bot] 0352996b3 - Bump extensions/Report from `494fc48` to `d3fdcf3` (#8438) [02:14:39] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ManageWiki-870ab9a - 13https://github.com/miraheze/mediawiki [02:14:39] [url] Page not found · GitHub · GitHub | github.com [02:14:40] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8455: Bump extensions/ManageWiki from `aad1060` to `870ab9a` - 13https://github.com/miraheze/mediawiki/pull/8455 [02:14:40] ... [02:14:42] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Report-d3fdcf3 [02:14:43] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Report-d3fdcf3 - 13https://github.com/miraheze/mediawiki [02:14:43] ... [02:14:45] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8438: Bump extensions/Report from `494fc48` to `d3fdcf3` - 13https://github.com/miraheze/mediawiki/pull/8438 [02:14:45] [url] Page not found · GitHub · GitHub | github.com [02:14:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 3.76, 3.30 [02:14:46] [02mediawiki] 07github-actions[bot] closed pull request 03#8447: Bump extensions/ApprovedRevs from `2c74834` to `4d2b002` - 13https://github.com/miraheze/mediawiki/pull/8447 [02:14:47] [url] Page not found · GitHub · GitHub | github.com [02:14:48] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/52996b3d6f03...0d687cca8aff [02:14:48] [url] Comparing 52996b3d6f03...0d687cca8aff · miraheze/mediawiki · GitHub | github.com [02:14:49] [02miraheze/mediawiki] 07dependabot[bot] 030d687cc - Bump extensions/ApprovedRevs from `2c74834` to `4d2b002` (#8447) [02:14:51] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ApprovedRevs-4d2b002 [02:14:52] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ApprovedRevs-4d2b002 - 13https://github.com/miraheze/mediawiki [02:14:52] [url] Page not found · GitHub · GitHub | github.com [02:14:54] [02mediawiki] 07github-actions[bot] closed pull request 03#8456: Bump extensions/ExternalData from `a5a4dc9` to `e21a01d` - 13https://github.com/miraheze/mediawiki/pull/8456 [02:14:54] [url] Page not found · GitHub · GitHub | github.com [02:14:55] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8447: Bump extensions/ApprovedRevs from `2c74834` to `4d2b002` - 13https://github.com/miraheze/mediawiki/pull/8447 [02:14:55] ... [02:14:57] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/0d687cca8aff...6c9330aff976 [02:14:57] [url] Comparing 0d687cca8aff...6c9330aff976 · miraheze/mediawiki · GitHub | github.com [02:14:58] [02miraheze/mediawiki] 07dependabot[bot] 036c9330a - Bump extensions/ExternalData from `a5a4dc9` to `e21a01d` (#8456) [02:15:00] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ExternalData-e21a01d [02:15:01] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ExternalData-e21a01d - 13https://github.com/miraheze/mediawiki [02:15:01] [url] Page not found · GitHub · GitHub | github.com [02:15:03] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8456: Bump extensions/ExternalData from `a5a4dc9` to `e21a01d` - 13https://github.com/miraheze/mediawiki/pull/8456 [02:15:03] ... [02:15:04] [02mediawiki] 07github-actions[bot] closed pull request 03#8450: Bump extensions/ImportDump from `ab3032b` to `61eeea3` - 13https://github.com/miraheze/mediawiki/pull/8450 [02:15:05] [url] Page not found · GitHub · GitHub | github.com [02:15:06] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/6c9330aff976...0d2cf49dbb75 [02:15:06] [url] Comparing 6c9330aff976...0d2cf49dbb75 · miraheze/mediawiki · GitHub | github.com [02:15:07] [02miraheze/mediawiki] 07dependabot[bot] 030d2cf49 - Bump extensions/ImportDump from `ab3032b` to `61eeea3` (#8450) [02:15:09] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ImportDump-61eeea3 [02:15:10] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ImportDump-61eeea3 - 13https://github.com/miraheze/mediawiki [02:15:10] [url] Page not found · GitHub · GitHub | github.com [02:15:12] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8450: Bump extensions/ImportDump from `ab3032b` to `61eeea3` - 13https://github.com/miraheze/mediawiki/pull/8450 [02:15:12] [url] Page not found · GitHub · GitHub | github.com [02:15:13] [02mediawiki] 07github-actions[bot] closed pull request 03#8449: Bump extensions/SocialProfile from `3c6ee3c` to `47728b1` - 13https://github.com/miraheze/mediawiki/pull/8449 [02:15:14] [url] Page not found · GitHub · GitHub | github.com [02:15:15] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/0d2cf49dbb75...22c62e876c6c [02:15:15] [url] Comparing 0d2cf49dbb75...22c62e876c6c · miraheze/mediawiki · GitHub | github.com [02:15:16] [02miraheze/mediawiki] 07dependabot[bot] 0322c62e8 - Bump extensions/SocialProfile from `3c6ee3c` to `47728b1` (#8449) [02:15:18] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SocialProfile-47728b1 [02:15:19] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SocialProfile-47728b1 - 13https://github.com/miraheze/mediawiki [02:15:19] ... [02:15:21] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8449: Bump extensions/SocialProfile from `3c6ee3c` to `47728b1` - 13https://github.com/miraheze/mediawiki/pull/8449 [02:15:21] [url] Page not found · GitHub · GitHub | github.com [02:15:22] [02mediawiki] 07github-actions[bot] closed pull request 03#8433: Bump extensions/QuizGame from `a935e71` to `d4eb5f0` - 13https://github.com/miraheze/mediawiki/pull/8433 [02:15:23] ... [02:15:24] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/22c62e876c6c...77ccb5fda2d4 [02:15:24] [url] Comparing 22c62e876c6c...77ccb5fda2d4 · miraheze/mediawiki · GitHub | github.com [02:15:25] [02miraheze/mediawiki] 07dependabot[bot] 0377ccb5f - Bump extensions/QuizGame from `a935e71` to `d4eb5f0` (#8433) [02:15:27] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/QuizGame-d4eb5f0 [02:15:28] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/QuizGame-d4eb5f0 - 13https://github.com/miraheze/mediawiki [02:15:28] [url] Page not found · GitHub · GitHub | github.com [02:15:30] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8433: Bump extensions/QuizGame from `a935e71` to `d4eb5f0` - 13https://github.com/miraheze/mediawiki/pull/8433 [02:15:30] [url] Page not found · GitHub · GitHub | github.com [02:15:31] [02mediawiki] 07github-actions[bot] closed pull request 03#8458: Bump extensions/MintyDocs from `591a2f2` to `f36d9b5` - 13https://github.com/miraheze/mediawiki/pull/8458 [02:15:32] [url] Page not found · GitHub · GitHub | github.com [02:15:33] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/77ccb5fda2d4...efa219f651d2 [02:15:33] [url] Comparing 77ccb5fda2d4...efa219f651d2 · miraheze/mediawiki · GitHub | github.com [02:15:34] [02miraheze/mediawiki] 07dependabot[bot] 03efa219f - Bump extensions/MintyDocs from `591a2f2` to `f36d9b5` (#8458) [02:15:36] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MintyDocs-f36d9b5 [02:15:37] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MintyDocs-f36d9b5 - 13https://github.com/miraheze/mediawiki [02:15:37] [url] Page not found · GitHub · GitHub | github.com [02:15:39] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8458: Bump extensions/MintyDocs from `591a2f2` to `f36d9b5` - 13https://github.com/miraheze/mediawiki/pull/8458 [02:15:39] ... [02:15:40] [02mediawiki] 07github-actions[bot] closed pull request 03#8448: Bump extensions/PollNY from `6e009de` to `b1c373f` - 13https://github.com/miraheze/mediawiki/pull/8448 [02:15:41] [url] Page not found · GitHub · GitHub | github.com [02:15:42] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PollNY-b1c373f [02:15:43] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/efa219f651d2...7db381085e0e [02:15:44] [url] Comparing efa219f651d2...7db381085e0e · miraheze/mediawiki · GitHub | github.com [02:15:45] [02miraheze/mediawiki] 07dependabot[bot] 037db3810 - Bump extensions/PollNY from `6e009de` to `b1c373f` (#8448) [02:15:46] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PollNY-b1c373f - 13https://github.com/miraheze/mediawiki [02:15:46] ... [02:15:48] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8448: Bump extensions/PollNY from `6e009de` to `b1c373f` - 13https://github.com/miraheze/mediawiki/pull/8448 [02:15:48] [url] Page not found · GitHub · GitHub | github.com [02:15:49] [02mediawiki] 07github-actions[bot] closed pull request 03#8436: Bump extensions/MirahezeMagic from `8f6c75d` to `8fc0344` - 13https://github.com/miraheze/mediawiki/pull/8436 [02:15:50] [url] Page not found · GitHub · GitHub | github.com [02:15:51] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/7db381085e0e...4e4d6844cf50 [02:15:51] [url] Comparing 7db381085e0e...4e4d6844cf50 · miraheze/mediawiki · GitHub | github.com [02:15:52] [02miraheze/mediawiki] 07dependabot[bot] 034e4d684 - Bump extensions/MirahezeMagic from `8f6c75d` to `8fc0344` (#8436) [02:15:54] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-8fc0344 [02:15:55] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-8fc0344 - 13https://github.com/miraheze/mediawiki [02:15:55] [url] Page not found · GitHub · GitHub | github.com [02:15:57] [02mediawiki] 07github-actions[bot] closed pull request 03#8453: Bump extensions/VoteNY from `34274da` to `35b1d55` - 13https://github.com/miraheze/mediawiki/pull/8453 [02:15:57] [url] Page not found · GitHub · GitHub | github.com [02:15:58] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8436: Bump extensions/MirahezeMagic from `8f6c75d` to `8fc0344` - 13https://github.com/miraheze/mediawiki/pull/8436 [02:15:59] ... [02:16:00] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/4e4d6844cf50...c97ad83c56ac [02:16:00] [url] Comparing 4e4d6844cf50...c97ad83c56ac · miraheze/mediawiki · GitHub | github.com [02:16:01] [02miraheze/mediawiki] 07dependabot[bot] 03c97ad83 - Bump extensions/VoteNY from `34274da` to `35b1d55` (#8453) [02:16:03] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/VoteNY-35b1d55 - 13https://github.com/miraheze/mediawiki [02:16:03] [url] Page not found · GitHub · GitHub | github.com [02:16:04] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/VoteNY-35b1d55 [02:16:06] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8453: Bump extensions/VoteNY from `34274da` to `35b1d55` - 13https://github.com/miraheze/mediawiki/pull/8453 [02:16:06] ... [02:16:07] [02mediawiki] 07github-actions[bot] closed pull request 03#8451: Bump extensions/MatomoAnalytics from `1260bb1` to `5791381` - 13https://github.com/miraheze/mediawiki/pull/8451 [02:16:08] [url] Page not found · GitHub · GitHub | github.com [02:16:09] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MatomoAnalytics-5791381 [02:16:10] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c97ad83c56ac...f422424d88ec [02:16:11] [url] Comparing c97ad83c56ac...f422424d88ec · miraheze/mediawiki · GitHub | github.com [02:16:12] [02miraheze/mediawiki] 07dependabot[bot] 03f422424 - Bump extensions/MatomoAnalytics from `1260bb1` to `5791381` (#8451) [02:16:13] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MatomoAnalytics-5791381 - 13https://github.com/miraheze/mediawiki [02:16:14] [url] Page not found · GitHub · GitHub | github.com [02:16:15] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8451: Bump extensions/MatomoAnalytics from `1260bb1` to `5791381` - 13https://github.com/miraheze/mediawiki/pull/8451 [02:16:15] [url] Page not found · GitHub · GitHub | github.com [02:16:16] [02mediawiki] 07github-actions[bot] closed pull request 03#8457: Bump extensions/Comments from `6d86851` to `d73d5be` - 13https://github.com/miraheze/mediawiki/pull/8457 [02:16:17] [url] Page not found · GitHub · GitHub | github.com [02:16:18] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/f422424d88ec...f14314ebe7f8 [02:16:18] [url] Comparing f422424d88ec...f14314ebe7f8 · miraheze/mediawiki · GitHub | github.com [02:16:19] [02miraheze/mediawiki] 07dependabot[bot] 03f14314e - Bump extensions/Comments from `6d86851` to `d73d5be` (#8457) [02:16:21] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Comments-d73d5be - 13https://github.com/miraheze/mediawiki [02:16:21] ... [02:16:22] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8457: Bump extensions/Comments from `6d86851` to `d73d5be` - 13https://github.com/miraheze/mediawiki/pull/8457 [02:16:23] [url] Page not found · GitHub · GitHub | github.com [02:16:24] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Comments-d73d5be [02:16:25] [02mediawiki] 07github-actions[bot] closed pull request 03#8442: Bump extensions/UniversalLanguageSelector from `06091ef` to `a11aca4` - 13https://github.com/miraheze/mediawiki/pull/8442 [02:16:26] ... [02:16:27] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/f14314ebe7f8...75c14a8caca2 [02:16:27] [url] Comparing f14314ebe7f8...75c14a8caca2 · miraheze/mediawiki · GitHub | github.com [02:16:28] [02miraheze/mediawiki] 07dependabot[bot] 0375c14a8 - Bump extensions/UniversalLanguageSelector from `06091ef` to `a11aca4` (#8442) [02:16:30] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-a11aca4 [02:16:31] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-a11aca4 - 13https://github.com/miraheze/mediawiki [02:16:32] [url] Page not found · GitHub · GitHub | github.com [02:16:33] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8442: Bump extensions/UniversalLanguageSelector from `06091ef` to `a11aca4` - 13https://github.com/miraheze/mediawiki/pull/8442 [02:16:33] [url] Page not found · GitHub · GitHub | github.com [02:16:34] [02mediawiki] 07github-actions[bot] closed pull request 03#8452: Bump extensions/PageForms from `da3886e` to `e92137e` - 13https://github.com/miraheze/mediawiki/pull/8452 [02:16:35] [url] Page not found · GitHub · GitHub | github.com [02:16:36] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/75c14a8caca2...6f9ebae9369e [02:16:36] [url] Comparing 75c14a8caca2...6f9ebae9369e · miraheze/mediawiki · GitHub | github.com [02:16:37] [02miraheze/mediawiki] 07dependabot[bot] 036f9ebae - Bump extensions/PageForms from `da3886e` to `e92137e` (#8452) [02:16:39] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageForms-e92137e - 13https://github.com/miraheze/mediawiki [02:16:39] [url] Page not found · GitHub · GitHub | github.com [02:16:40] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageForms-e92137e [02:16:42] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#8452: Bump extensions/PageForms from `da3886e` to `e92137e` - 13https://github.com/miraheze/mediawiki/pull/8452 [02:16:42] ... [02:16:43] [02mediawiki] 07Universal-Omega closed pull request 03#8454: Bump skins/Citizen from `e9139f0` to `6384d02` - 13https://github.com/miraheze/mediawiki/pull/8454 [02:16:44] [url] Page not found · GitHub · GitHub | github.com [02:16:45] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/skins/Citizen-6384d02 [02:16:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.47, 3.25 [02:16:46] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/6f9ebae9369e...b68d8a9c85db [02:16:47] [url] Comparing 6f9ebae9369e...b68d8a9c85db · miraheze/mediawiki · GitHub | github.com [02:16:48] [02miraheze/mediawiki] 07dependabot[bot] 03b68d8a9 - Bump skins/Citizen from `e9139f0` to `6384d02` (#8454) [02:16:49] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/skins/Citizen-6384d02 - 13https://github.com/miraheze/mediawiki [02:16:50] ... [02:16:51] [02mediawiki] 07Universal-Omega closed pull request 03#8435: Bump skins/Cosmos from `d1163b5` to `ff3cc41` - 13https://github.com/miraheze/mediawiki/pull/8435 [02:16:52] [url] Page not found · GitHub · GitHub | github.com [02:16:52] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/b68d8a9c85db...19e5a42d1c14 [02:16:53] [url] Comparing b68d8a9c85db...19e5a42d1c14 · miraheze/mediawiki · GitHub | github.com [02:16:54] [02miraheze/mediawiki] 07dependabot[bot] 0319e5a42 - Bump skins/Cosmos from `d1163b5` to `ff3cc41` (#8435) [02:16:55] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/skins/Cosmos-ff3cc41 - 13https://github.com/miraheze/mediawiki [02:16:56] [url] Page not found · GitHub · GitHub | github.com [02:16:57] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/skins/Cosmos-ff3cc41 [02:16:58] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/extensions/DarkMode-e8ecbb1 [02:16:59] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.11, 2.18, 1.61 [02:17:00] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/19e5a42d1c14...4b394763c9e2 [02:17:00] [url] Comparing 19e5a42d1c14...4b394763c9e2 · miraheze/mediawiki · GitHub | github.com [02:17:01] [02miraheze/mediawiki] 07dependabot[bot] 034b39476 - Bump extensions/DarkMode from `f9b0599` to `e8ecbb1` (#8437) [02:17:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.20, 11.19, 10.86 [02:17:03] [02mediawiki] 07Universal-Omega closed pull request 03#8437: Bump extensions/DarkMode from `f9b0599` to `e8ecbb1` - 13https://github.com/miraheze/mediawiki/pull/8437 [02:17:04] [url] Page not found · GitHub · GitHub | github.com [02:17:04] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/extensions/DarkMode-e8ecbb1 - 13https://github.com/miraheze/mediawiki [02:17:05] [url] Page not found · GitHub · GitHub | github.com [02:18:13] !log start deploy of world and l10n on mwtask141 [02:18:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.82, 4.24, 3.55 [02:20:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.26, 3.77, 3.46 [02:20:50] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.68, 1.91, 1.62 [02:22:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.47, 4.48, 3.75 [02:22:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.07, 2.01, 1.69 [02:23:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.14, 8.91, 9.97 [02:24:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.37, 1.71, 1.62 [02:26:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 1.73, 1.63 [02:27:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.94, 13.58, 11.62 [02:27:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.76, 3.39, 2.82 [02:28:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.64, 1.68, 1.62 [02:29:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.98, 2.88, 2.70 [02:31:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.08, 10.73, 10.89 [02:31:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/a8deaac91ab2 [02:31:17] [02miraheze/puppet] 07paladox 03a8deaac - Install matomo131 [02:31:19] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [02:31:19] [url] Page not found · GitHub · GitHub | github.com [02:31:20] [02puppet] 07paladox opened pull request 03#2749: Install matomo131 - 13https://github.com/miraheze/puppet/pull/2749 [02:31:21] ... [02:31:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a8deaac91ab2...ffae797a6479 [02:31:48] [url] Comparing a8deaac91ab2...ffae797a6479 · miraheze/puppet · GitHub | github.com [02:31:49] [02miraheze/puppet] 07paladox 03ffae797 - Update site.pp [02:31:50] [02puppet] 07paladox synchronize pull request 03#2749: Install matomo131 - 13https://github.com/miraheze/puppet/pull/2749 [02:31:51] [url] Page not found · GitHub · GitHub | github.com [02:31:54] [02puppet] 07paladox closed pull request 03#2749: Install matomo131 - 13https://github.com/miraheze/puppet/pull/2749 [02:31:54] ... [02:31:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/801c06893b36...50298cbf4ed8 [02:31:56] [url] Comparing 801c06893b36...50298cbf4ed8 · miraheze/puppet · GitHub | github.com [02:31:57] [02miraheze/puppet] 07paladox 0350298cb - Install matomo131 (#2749) [02:31:58] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [02:32:00] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [02:32:00] [url] Page not found · GitHub · GitHub | github.com [02:36:02] paladox: MirahezdeLSBot is not connected. [02:36:15] hmm [02:36:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.92, 3.99 [02:36:56] what is MirahezdeLSBot connected to? [02:37:18] https://github.com/search?q=org%3Amiraheze+MirahezdeLSBot [02:37:18] [url] Search · org:miraheze MirahezdeLSBot · GitHub | github.com [02:37:35] CosmicAlpha: [02:38:21] paladox: https://github.com/miraheze/puppet/blob/master/modules/irc/manifests/irclogserverbot.pp and https://github.com/miraheze/puppet/blob/da930a29fe94ece7a97ec2ea94db38ea37883fee/modules/role/manifests/irc.pp#L17-L27 [02:38:22] [url] puppet/irclogserverbot.pp at master · miraheze/puppet · GitHub | github.com [02:38:22] [url] puppet/irc.pp at da930a29fe94ece7a97ec2ea94db38ea37883fee · miraheze/puppet · GitHub | github.com [02:38:29] I think [02:38:54] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.91, 1.82, 1.57 [02:39:39] !log finished deploy of world and l10n to all servers using deploy-mediawiki on mwtask141 [02:39:39] CosmicAlpha: ^ [02:39:49] thanks paladox! [02:39:55] yw [02:40:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:40:54] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.15, 1.56, 1.51 [02:41:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.01, 11.44, 10.84 [02:41:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ca7aced923c7 [02:41:57] [02miraheze/puppet] 07paladox 03ca7aced - Depool matomo101 and pool matomo131 [02:41:59] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [02:41:59] [url] Page not found · GitHub · GitHub | github.com [02:42:00] [02puppet] 07paladox opened pull request 03#2750: Depool matomo101 and pool matomo131 - 13https://github.com/miraheze/puppet/pull/2750 [02:42:01] [url] Page not found · GitHub · GitHub | github.com [02:42:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ca7aced923c7...907d8ae69a32 [02:42:21] [url] Comparing ca7aced923c7...907d8ae69a32 · miraheze/puppet · GitHub | github.com [02:42:21] [02miraheze/puppet] 07paladox 03907d8ae - Update default.vcl [02:42:23] [02puppet] 07paladox synchronize pull request 03#2750: Depool matomo101 and pool matomo131 - 13https://github.com/miraheze/puppet/pull/2750 [02:42:23] [url] Page not found · GitHub · GitHub | github.com [02:42:28] [02puppet] 07paladox closed pull request 03#2750: Depool matomo101 and pool matomo131 - 13https://github.com/miraheze/puppet/pull/2750 [02:42:29] [url] Page not found · GitHub · GitHub | github.com [02:42:30] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [02:42:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/50298cbf4ed8...f557a74e3813 [02:42:32] [url] Comparing 50298cbf4ed8...f557a74e3813 · miraheze/puppet · GitHub | github.com [02:42:33] [02miraheze/puppet] 07paladox 03f557a74 - Depool matomo101 and pool matomo131 (#2750) [02:42:34] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [02:42:35] [url] Page not found · GitHub · GitHub | github.com [02:43:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.45, 9.93, 10.35 [02:44:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 3.88, 3.88 [02:46:13] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.17, 1.90, 1.64 [02:47:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.27, 10.87, 10.61 [02:48:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.23, 1.57, 1.54 [02:49:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.93, 9.56, 10.15 [02:52:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.50, 3.60, 3.93 [02:52:47] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:56:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.71, 3.87 [02:56:25] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.405 second response time [02:57:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.95, 1.84, 1.66 [02:58:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.34, 3.71 [02:59:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.73, 11.46, 10.59 [02:59:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.47, 1.97, 1.72 [03:01:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.74, 10.44, 10.31 [03:02:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.63, 3.86, 3.80 [03:03:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.87, 9.90, 10.13 [03:04:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.70, 3.77 [03:04:43] !log shutdown and delete vm matomo101 [03:05:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.70, 10.82, 10.56 [03:07:37] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: connect to address 2a10:6740::6:106 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [03:07:52] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: connect to address 2a10:6740::6:106 port 5666: No route to hostconnect to host 2a10:6740::6:106 port 5666: No route to host [03:07:52] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: connect to address 2a10:6740::6:106 port 5666: No route to hostconnect to host 2a10:6740::6:106 port 5666: No route to host [03:07:52] PROBLEM - Host matomo101 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:106) [03:09:32] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php (START) [03:09:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.20, 1.74, 1.82 [03:11:17] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php (END - exit=65280) [03:11:18] !log [universalomega@mwtask141] starting deploy of {'files': '../mediawiki/w/extensions/SemanticMediaWiki/.smw.json'} to all [03:11:20] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.74, 8.87, 9.83 [03:11:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:24] !log [universalomega@mwtask141] finished deploy of {'files': '../mediawiki/w/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 7s [03:11:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:31] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php (START) [03:11:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.73, 1.43, 1.70 [03:11:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:53] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php (END - exit=0) [03:13:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:02] !log [universalomega@mwtask141] starting deploy of {'files': '../mediawiki/w/extensions/SemanticMediaWiki/.smw.json'} to all [03:14:22] !log [universalomega@mwtask141] finished deploy of {'files': '../mediawiki/w/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 19s [03:14:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:15:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.81, 4.24, 3.90 [03:17:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.66, 2.19, 1.98 [03:19:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.31, 1.87, 1.89 [03:21:02] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.54, 1.81, 1.38 [03:21:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.94, 3.91 [03:23:02] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.68, 1.70, 1.39 [03:25:11] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.79, 1.27, 1.63 [03:25:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.27, 4.21, 4.00 [03:29:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.46, 3.00 [03:29:45] [02puppet] 07paladox closed pull request 03#2745: Fix proxy for icinga - 13https://github.com/miraheze/puppet/pull/2745 [03:29:45] [url] Page not found · GitHub · GitHub | github.com [03:29:47] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [03:29:47] ... [03:29:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [03:30:11] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/c0e592c0f2a9...44f03f134aeb [03:30:11] [url] Comparing c0e592c0f2a9...44f03f134aeb · miraheze/dns · GitHub | github.com [03:30:12] [02miraheze/dns] 07paladox 0344f03f1 - Remove matomo101 from dns [03:30:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c1174f0672b8 [03:30:57] [02miraheze/puppet] 07paladox 03c1174f0 - Remove matomo101 [03:30:58] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [03:30:59] [url] Page not found · GitHub · GitHub | github.com [03:31:00] [02puppet] 07paladox opened pull request 03#2751: Remove matomo101 - 13https://github.com/miraheze/puppet/pull/2751 [03:31:00] ... [03:31:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/c1174f0672b8...472e01603329 [03:31:15] [url] Comparing c1174f0672b8...472e01603329 · miraheze/puppet · GitHub | github.com [03:31:16] [02miraheze/puppet] 07paladox 03472e016 - Delete matomo101.yaml [03:31:18] [02puppet] 07paladox synchronize pull request 03#2751: Remove matomo101 - 13https://github.com/miraheze/puppet/pull/2751 [03:31:18] [url] Page not found · GitHub · GitHub | github.com [03:31:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/puppet/compare/f557a74e3813...5dc3696cd548 [03:31:21] [url] Comparing f557a74e3813...5dc3696cd548 · miraheze/puppet · GitHub | github.com [03:31:22] [02miraheze/puppet] 07paladox 035dc3696 - Remove matomo101 (#2751) [03:31:24] [02puppet] 07paladox closed pull request 03#2751: Remove matomo101 - 13https://github.com/miraheze/puppet/pull/2751 [03:31:24] [url] Page not found · GitHub · GitHub | github.com [03:31:25] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [03:31:26] [url] Page not found · GitHub · GitHub | github.com [03:31:27] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [03:31:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.51, 3.07, 2.90 [03:31:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.84, 3.92 [03:33:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.02, 4.00, 3.95 [03:34:30] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/44f03f134aeb...3699c975a08e [03:34:30] [url] Comparing 44f03f134aeb...3699c975a08e · miraheze/dns · GitHub | github.com [03:34:31] [02miraheze/dns] 07paladox 033699c97 - Add ldap141 to dns [03:35:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.87, 3.91 [03:36:12] [02dns] 07Universal-Omega opened pull request 03#323: Add yokaiwatchwiki.com zone - 13https://github.com/miraheze/dns/pull/323 [03:36:12] [url] Page not found · GitHub · GitHub | github.com [03:36:22] paladox: do you mind merging ^ [03:36:40] [02dns] 07paladox closed pull request 03#323: Add yokaiwatchwiki.com zone - 13https://github.com/miraheze/dns/pull/323 [03:36:41] [url] Page not found · GitHub · GitHub | github.com [03:36:42] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/3699c975a08e...a923c5535512 [03:36:42] [url] Comparing 3699c975a08e...a923c5535512 · miraheze/dns · GitHub | github.com [03:36:43] [02miraheze/dns] 07Universal-Omega 03a923c55 - Add yokaiwatchwiki.com zone (#323) [03:36:46] Thanks! [03:37:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 4.12, 4.00 [03:39:56] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [03:48:39] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/1925d699c7d2 [03:48:40] [02miraheze/ssl] 07Universal-Omega 031925d69 - Add yokaiwatchwiki.com cert [03:48:42] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [03:48:42] [url] Page not found · GitHub · GitHub | github.com [03:48:43] [02ssl] 07Universal-Omega opened pull request 03#584: Add yokaiwatchwiki.com cert - 13https://github.com/miraheze/ssl/pull/584 [03:48:44] ... [03:49:04] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1925d699c7d2...d38bb67528ed [03:49:04] [url] Comparing 1925d699c7d2...d38bb67528ed · miraheze/ssl · GitHub | github.com [03:49:05] [02miraheze/ssl] 07Universal-Omega 03d38bb67 - Update certs.yaml [03:49:07] [02ssl] 07Universal-Omega synchronize pull request 03#584: Add yokaiwatchwiki.com cert - 13https://github.com/miraheze/ssl/pull/584 [03:49:07] [url] Page not found · GitHub · GitHub | github.com [03:49:26] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/b36df435c5c6...e6da1192cf76 [03:49:27] [url] Comparing b36df435c5c6...e6da1192cf76 · miraheze/ssl · GitHub | github.com [03:49:28] [02miraheze/ssl] 07Universal-Omega 03e6da119 - Add yokaiwatchwiki.com cert (#584) [03:49:29] [02ssl] 07Universal-Omega closed pull request 03#584: Add yokaiwatchwiki.com cert - 13https://github.com/miraheze/ssl/pull/584 [03:49:29] [url] Page not found · GitHub · GitHub | github.com [03:49:31] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [03:49:32] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [03:49:32] [url] Page not found · GitHub · GitHub | github.com [03:54:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/b41d5207811c [03:54:27] [02miraheze/puppet] 07paladox 03b41d520 - Install ldap141 [03:54:29] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [03:54:29] [url] Page not found · GitHub · GitHub | github.com [03:54:30] [02puppet] 07paladox opened pull request 03#2752: Install ldap141 - 13https://github.com/miraheze/puppet/pull/2752 [03:54:31] ... [03:54:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b41d5207811c...45c21c76fec3 [03:54:47] [url] Comparing b41d5207811c...45c21c76fec3 · miraheze/puppet · GitHub | github.com [03:54:48] [02miraheze/puppet] 07paladox 0345c21c7 - Update site.pp [03:54:49] [02puppet] 07paladox synchronize pull request 03#2752: Install ldap141 - 13https://github.com/miraheze/puppet/pull/2752 [03:54:50] [url] Page not found · GitHub · GitHub | github.com [03:54:53] [02puppet] 07paladox closed pull request 03#2752: Install ldap141 - 13https://github.com/miraheze/puppet/pull/2752 [03:54:54] ... [03:54:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/5dc3696cd548...c301553d1aff [03:54:55] [url] Comparing 5dc3696cd548...c301553d1aff · miraheze/puppet · GitHub | github.com [03:54:56] [02miraheze/puppet] 07paladox 03c301553 - Install ldap141 (#2752) [03:54:58] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [03:54:58] [url] Page not found · GitHub · GitHub | github.com [03:54:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [03:55:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.16, 3.29, 3.84 [03:56:47] PROBLEM - ldap141 Puppet on ldap141 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 1 minute ago with 3 failures. Failed resources (up to 3 shown): Openldap_global_conf[TLSCertificate],Openldap_global_conf[TLSCACertificateFile],Openldap_database[cn=monitor] [03:57:25] !log stop slapd on ldap111 to copy data to ldap141 [03:58:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:59:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.33, 3.70 [03:59:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.68, 3.18 [04:00:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.14, 2.88, 3.50 [04:01:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.05, 3.28, 3.08 [04:02:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.60, 3.62, 3.70 [04:03:19] PROBLEM - ldap111 LDAP on ldap111 is CRITICAL: Could not bind to the LDAP server [04:03:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/062088e6a02e [04:03:29] [02miraheze/puppet] 07paladox 03062088e - Switch all services from ldap111 to ldap141 [04:03:31] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [04:03:31] [url] Page not found · GitHub · GitHub | github.com [04:03:32] [02puppet] 07paladox opened pull request 03#2753: Switch all services from ldap111 to ldap141 - 13https://github.com/miraheze/puppet/pull/2753 [04:03:33] [url] Page not found · GitHub · GitHub | github.com [04:03:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/062088e6a02e...0afefc7d1bc1 [04:03:57] [url] Comparing 062088e6a02e...0afefc7d1bc1 · miraheze/puppet · GitHub | github.com [04:03:57] [02miraheze/puppet] 07paladox 030afefc7 - Update config.ini.php.erb [04:03:59] [02puppet] 07paladox synchronize pull request 03#2753: Switch all services from ldap111 to ldap141 - 13https://github.com/miraheze/puppet/pull/2753 [04:03:59] [url] Page not found · GitHub · GitHub | github.com [04:04:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0afefc7d1bc1...b10e1694d040 [04:04:13] [url] Comparing 0afefc7d1bc1...b10e1694d040 · miraheze/puppet · GitHub | github.com [04:04:14] [02miraheze/puppet] 07paladox 03b10e169 - Update icinga2.pp [04:04:15] [02puppet] 07paladox synchronize pull request 03#2753: Switch all services from ldap111 to ldap141 - 13https://github.com/miraheze/puppet/pull/2753 [04:04:16] [url] Page not found · GitHub · GitHub | github.com [04:04:25] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:04:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b10e1694d040...6b9a8fe730ec [04:04:28] [url] Comparing b10e1694d040...6b9a8fe730ec · miraheze/puppet · GitHub | github.com [04:04:29] [02miraheze/puppet] 07paladox 036b9a8fe - Update init.pp [04:04:31] [02puppet] 07paladox synchronize pull request 03#2753: Switch all services from ldap111 to ldap141 - 13https://github.com/miraheze/puppet/pull/2753 [04:04:31] ... [04:04:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6b9a8fe730ec...081d396395a1 [04:04:46] [url] Comparing 6b9a8fe730ec...081d396395a1 · miraheze/puppet · GitHub | github.com [04:04:47] [02miraheze/puppet] 07paladox 03081d396 - Update dovecot-ldap.conf [04:04:48] [02puppet] 07paladox synchronize pull request 03#2753: Switch all services from ldap111 to ldap141 - 13https://github.com/miraheze/puppet/pull/2753 [04:04:48] [url] Page not found · GitHub · GitHub | github.com [04:05:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/081d396395a1...7c9ee876a9e4 [04:05:02] [url] Comparing 081d396395a1...7c9ee876a9e4 · miraheze/puppet · GitHub | github.com [04:05:03] [02miraheze/puppet] 07paladox 037c9ee87 - Update smtpd_sender_login_maps [04:05:05] [02puppet] 07paladox synchronize pull request 03#2753: Switch all services from ldap111 to ldap141 - 13https://github.com/miraheze/puppet/pull/2753 [04:05:05] [url] Page not found · GitHub · GitHub | github.com [04:05:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7c9ee876a9e4...be46dfe0cf9b [04:05:18] [url] Comparing 7c9ee876a9e4...be46dfe0cf9b · miraheze/puppet · GitHub | github.com [04:05:19] [02miraheze/puppet] 07paladox 03be46dfe - Update virtual_alias_groups [04:05:20] [02puppet] 07paladox synchronize pull request 03#2753: Switch all services from ldap111 to ldap141 - 13https://github.com/miraheze/puppet/pull/2753 [04:05:20] [url] Page not found · GitHub · GitHub | github.com [04:05:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/be46dfe0cf9b...a6a9af9b4552 [04:05:33] [url] Comparing be46dfe0cf9b...a6a9af9b4552 · miraheze/puppet · GitHub | github.com [04:05:34] [02miraheze/puppet] 07paladox 03a6a9af9 - Update virtual_alias_domains [04:05:35] [02puppet] 07paladox synchronize pull request 03#2753: Switch all services from ldap111 to ldap141 - 13https://github.com/miraheze/puppet/pull/2753 [04:05:35] [url] Page not found · GitHub · GitHub | github.com [04:06:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a6a9af9b4552...b31b8be9b9c4 [04:06:12] [url] Comparing a6a9af9b4552...b31b8be9b9c4 · miraheze/puppet · GitHub | github.com [04:06:13] [02miraheze/puppet] 07paladox 03b31b8be - Update mon141.yaml [04:06:14] [02puppet] 07paladox synchronize pull request 03#2753: Switch all services from ldap111 to ldap141 - 13https://github.com/miraheze/puppet/pull/2753 [04:06:14] [url] Page not found · GitHub · GitHub | github.com [04:06:40] [02puppet] 07paladox closed pull request 03#2753: Switch all services from ldap111 to ldap141 - 13https://github.com/miraheze/puppet/pull/2753 [04:06:41] [url] Page not found · GitHub · GitHub | github.com [04:06:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±9] 13https://github.com/miraheze/puppet/compare/c301553d1aff...8d2d7072996e [04:06:42] [url] Comparing c301553d1aff...8d2d7072996e · miraheze/puppet · GitHub | github.com [04:06:43] [02miraheze/puppet] 07paladox 038d2d707 - Switch all services from ldap111 to ldap141 (#2753) [04:06:45] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [04:06:46] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [04:06:47] [url] Page not found · GitHub · GitHub | github.com [04:10:59] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:11:17] RECOVERY - ldap111 LDAP on ldap111 is OK: LDAP OK - 0.050 seconds response time [04:17:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8d2d7072996e...085ad6eb0e2c [04:17:56] [url] Comparing 8d2d7072996e...085ad6eb0e2c · miraheze/puppet · GitHub | github.com [04:17:57] [02miraheze/puppet] 07paladox 03085ad6e - Update virtual_alias_maps [04:18:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/085ad6eb0e2c...ad5fdfe4a7c6 [04:18:04] [url] Comparing 085ad6eb0e2c...ad5fdfe4a7c6 · miraheze/puppet · GitHub | github.com [04:18:05] [02miraheze/puppet] 07paladox 03ad5fdfe - Update virtual_mailbox_maps [04:18:44] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0477f8d737af...d53796c3f579 [04:18:45] [url] Comparing 0477f8d737af...d53796c3f579 · miraheze/mw-config · GitHub | github.com [04:18:46] [02miraheze/mw-config] 07paladox 03d53796c - Switch to ldap141 [04:19:16] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [04:19:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:26] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [04:19:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:48] miraheze/mw-config - paladox the build passed. [04:20:02] !log [paladox@test131] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@test141.miraheze.org [04:20:06] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to test131 [04:20:07] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to test131 - SUCCESS in 0s [04:20:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:20:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:20:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:20:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.28, 11.31, 11.99 [04:21:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8d1de74c039f [04:21:24] [02miraheze/puppet] 07paladox 038d1de74 - Remove ldap111 [04:21:25] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [04:21:26] [url] Page not found · GitHub · GitHub | github.com [04:21:27] [02puppet] 07paladox opened pull request 03#2754: Remove ldap111 - 13https://github.com/miraheze/puppet/pull/2754 [04:21:27] [url] Page not found · GitHub · GitHub | github.com [04:22:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/8d1de74c039f...28951d44bca8 [04:22:17] [url] Comparing 8d1de74c039f...28951d44bca8 · miraheze/puppet · GitHub | github.com [04:22:18] [02miraheze/puppet] 07paladox 0328951d4 - Delete ldap111.yaml [04:22:19] [02puppet] 07paladox synchronize pull request 03#2754: Remove ldap111 - 13https://github.com/miraheze/puppet/pull/2754 [04:22:20] [url] Page not found · GitHub · GitHub | github.com [04:22:32] !log shutdown ldap111 [04:22:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.88, 1.51 [04:22:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.70, 11.67, 12.01 [04:22:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:24:10] [02puppet] 07paladox closed pull request 03#2754: Remove ldap111 - 13https://github.com/miraheze/puppet/pull/2754 [04:24:10] [url] Page not found · GitHub · GitHub | github.com [04:24:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/puppet/compare/ad5fdfe4a7c6...32a629bdf111 [04:24:12] [url] Comparing ad5fdfe4a7c6...32a629bdf111 · miraheze/puppet · GitHub | github.com [04:24:13] [02miraheze/puppet] 07paladox 0332a629b - Remove ldap111 (#2754) [04:24:14] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [04:24:15] [url] Page not found · GitHub · GitHub | github.com [04:24:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [04:24:16] !log delete vm ldap111 [04:24:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:24:34] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.44, 1.67, 1.47 [04:25:53] PROBLEM - ldap111 NTP time on ldap111 is CRITICAL: connect to address 2a10:6740::6:204 port 5666: No route to hostconnect to host 2a10:6740::6:204 port 5666: No route to host [04:26:19] PROBLEM - ldap111 PowerDNS Recursor on ldap111 is CRITICAL: connect to address 2a10:6740::6:204 port 5666: No route to hostconnect to host 2a10:6740::6:204 port 5666: No route to host [04:26:19] PROBLEM - ldap111 ferm_active on ldap111 is CRITICAL: connect to address 2a10:6740::6:204 port 5666: No route to hostconnect to host 2a10:6740::6:204 port 5666: No route to host [04:26:19] PROBLEM - ldap111 Puppet on ldap111 is CRITICAL: connect to address 2a10:6740::6:204 port 5666: No route to hostconnect to host 2a10:6740::6:204 port 5666: No route to host [04:26:33] PROBLEM - ldap111 conntrack_table_size on ldap111 is CRITICAL: connect to address 2a10:6740::6:204 port 5666: No route to hostconnect to host 2a10:6740::6:204 port 5666: No route to host [04:26:39] PROBLEM - ping6 on ldap111 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:204) [04:26:51] PROBLEM - ldap111 LDAP on ldap111 is CRITICAL: Could not bind to the LDAP server PROBLEM - ldap111 SSH on ldap111 is CRITICAL: connect to address 2a10:6740::6:204 and port 22: No route to host [04:26:51] PROBLEM - Host ldap111 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:204) [04:34:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.35, 3.49, 3.93 [04:36:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 3.79, 3.99 [04:38:17] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/a923c5535512...31f043bdc63b [04:38:17] [url] Comparing a923c5535512...31f043bdc63b · miraheze/dns · GitHub | github.com [04:38:18] [02miraheze/dns] 07paladox 0331f043b - Remove ldap111 from dns [04:38:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.98, 3.68, 3.91 [04:40:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.11, 4.18, 4.07 [04:40:48] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [04:48:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.35, 2.98 [04:52:12] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.80, 3.07, 3.00 [05:07:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.68, 3.99 [05:08:18] PROBLEM - burnout.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'burnout.wiki' expires in 15 day(s) (Fri 19 Aug 2022 05:01:19 GMT +0000). [05:10:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e6da1192cf76...a86b3ac3f2e6 [05:10:35] [url] Comparing e6da1192cf76...a86b3ac3f2e6 · miraheze/ssl · GitHub | github.com [05:10:36] [02miraheze/ssl] 07MirahezeSSLBot 03a86b3ac - Bot: Update SSL cert for burnout.wiki [05:15:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.41, 2.51, 3.36 [05:15:39] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:17:16] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:17:38] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 2.034 second response time [05:18:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.24, 11.09, 9.52 [05:18:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.09, 2.85 [05:19:16] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.311 second response time [05:19:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.36, 3.27, 3.48 [05:20:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.19, 9.82, 9.25 [05:20:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.67, 3.21, 2.93 [05:22:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.25, 2.97, 2.88 [05:25:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 2.84, 3.29 [05:26:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.65, 3.20 [05:28:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 3.38, 3.18 [05:29:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.38, 3.46 [05:33:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 3.71, 3.55 [05:33:47] PROBLEM - entw.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'entw.ml' expires in 15 day(s) (Fri 19 Aug 2022 05:03:57 GMT +0000). [05:35:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.37, 3.44 [05:37:36] RECOVERY - burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'burnout.wiki' will expire on Tue 01 Nov 2022 04:10:29 GMT +0000. [05:38:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a86b3ac3f2e6...0a5e0dc3ff41 [05:38:40] [url] Comparing a86b3ac3f2e6...0a5e0dc3ff41 · miraheze/ssl · GitHub | github.com [05:38:41] [02miraheze/ssl] 07MirahezeSSLBot 030a5e0dc - Bot: Update SSL cert for entw.ml [05:41:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.84, 4.18, 3.70 [05:41:55] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:43:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.85, 3.47 [05:49:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.00, 3.40, 3.39 [05:51:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.65, 3.74 [05:57:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.69, 4.39, 3.97 [06:02:36] PROBLEM - entw.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:50] PROBLEM - wiki.wisrail.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:57] PROBLEM - it.famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:05:06] PROBLEM - wiki.harlonserver.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:15] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:16] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:09:17] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.061 second response time [06:09:24] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.125 second response time [06:10:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.39, 3.28 [06:10:58] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [06:14:10] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.09, 3.23, 3.23 [06:14:40] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:14:58] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw142 [06:16:53] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 2.043 second response time [06:16:55] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [06:17:26] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:19:20] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.029 second response time [06:21:48] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.01, 1.52, 1.11 [06:23:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.36, 3.59, 3.38 [06:23:44] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.03, 1.33, 1.08 [06:26:59] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.67, 4.14, 3.62 [06:27:22] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.72, 1.45, 1.11 [06:29:03] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:29:14] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw121 [06:29:17] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.66, 1.18, 1.05 [06:30:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.79, 3.60 [06:31:08] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [06:31:23] RECOVERY - entw.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'entw.ml' will expire on Tue 01 Nov 2022 04:38:34 GMT +0000. [06:31:51] RECOVERY - it.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'it.famepedia.org' will expire on Sat 08 Oct 2022 06:08:33 GMT +0000. [06:32:18] RECOVERY - wiki.wisrail.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wisrail.de' will expire on Tue 06 Sep 2022 21:01:07 GMT +0000. [06:33:10] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw141 mw142 [06:34:35] Reception123: copy data from mwtask111 [06:34:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 4.17, 3.80 [06:36:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.02, 10.25, 9.20 [06:36:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:38:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.12, 9.42, 9.03 [06:39:05] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [06:40:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.94, 3.85 [06:41:42] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [06:45:31] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [06:46:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.97, 3.85 [06:47:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.44, 9.33, 11.88 [06:48:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.74, 3.78 [06:49:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.90, 11.86, 12.48 [06:50:12] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:50:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.01, 1.37, 0.99 [06:52:12] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.021 second response time [06:52:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.68, 1.09, 0.94 [06:52:55] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw132 [06:57:27] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:57:48] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.96, 1.74, 1.23 [06:58:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/b30de5c12741 [06:58:32] [02miraheze/mw-config] 07Universal-Omega 03b30de5c - Raise `$wgMaxExecutionTimeForExpensiveQueries` to 30 seconds [06:58:33] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [06:58:34] [url] Page not found · GitHub · GitHub | github.com [06:58:35] [02mw-config] 07Universal-Omega opened pull request 03#4839: Raise `$wgMaxExecutionTimeForExpensiveQueries` to 30 seconds - 13https://github.com/miraheze/mw-config/pull/4839 [06:58:35] [url] Page not found · GitHub · GitHub | github.com [06:58:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 4.01, 3.81 [06:59:11] [02mw-config] 07Universal-Omega closed pull request 03#4839: Raise `$wgMaxExecutionTimeForExpensiveQueries` to 30 seconds - 13https://github.com/miraheze/mw-config/pull/4839 [06:59:11] [url] Page not found · GitHub · GitHub | github.com [06:59:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d53796c3f579...8da2c0c60e6f [06:59:13] [url] Comparing d53796c3f579...8da2c0c60e6f · miraheze/mw-config · GitHub | github.com [06:59:14] [02miraheze/mw-config] 07Universal-Omega 038da2c0c - Raise `$wgMaxExecutionTimeForExpensiveQueries` to 30 seconds (#4839) [06:59:15] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [06:59:17] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [06:59:17] [url] Page not found · GitHub · GitHub | github.com [06:59:41] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [06:59:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.75, 1.38, 1.16 [06:59:44] miraheze/mw-config - Universal-Omega the build passed. [07:00:20] miraheze/mw-config - Universal-Omega the build passed. [07:01:02] PROBLEM - puritwiki.p-e.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:01:10] CosmicAlpha: that won't work [07:01:18] We enforce DB max time at mysql level [07:01:48] RhinosF1: it worked on test... [07:02:12] PROBLEM - wiki.slimeapedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:02:16] Oh, it works even without that on test. Whoops. [07:02:22] CosmicAlpha: ye, mysql is cap 10 minutes [07:02:26] So actually that's fine [07:02:35] https://github.com/miraheze/puppet/blob/dc07c533bc2fc3019ed7d18642889dc41bc40597/modules/mariadb/templates/grants/mediawiki-grants.sql.erb#L7 [07:02:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.25, 3.66, 3.72 [07:02:36] [url] puppet/mediawiki-grants.sql.erb at dc07c533bc2fc3019ed7d18642889dc41bc40597 · miraheze/puppet · GitHub | github.com [07:02:44] So just don't go past 10 minutes [07:02:44] Wow, why do we have that so high.. [07:03:00] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [07:03:03] CosmicAlpha: because it's not when you take maint scripts and stuff into account [07:03:24] RECOVERY - wiki.harlonserver.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.harlonserver.net' will expire on Wed 24 Aug 2022 16:27:40 GMT +0000. [07:03:32] Also, yeah it worked without that on test because I forgot to deploy-mediawiki the revert to see if it did... lol. [07:03:40] Ah that makes sense. [07:03:57] CosmicAlpha: are both canaries working fine now? [07:04:06] RhinosF1: yep. [07:04:37] I will see if I can test the mwdeploy changes tomorrow also by the way. [07:04:47] Ace! [07:05:07] If it runs everything fine, I'm fairly confident [07:05:20] It's just that actual execution I can't easily test [07:05:30] All the prep we can test with the maps [07:05:43] I copied it over to mwtask141 into my home directory today but did not get to actually testing unfortantly. [07:06:02] PROBLEM - wiki.site97.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:43] RhinosF1: yep, will do tomorrow hopefully then. It will be nice to get done! [07:06:53] CosmicAlpha: yep [07:07:02] Then I going to debate splitting the config out [07:07:55] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw131 [07:08:42] RhinosF1: What about converting to a template controlled by hiera also, for servers, etc... taken from something else. [07:09:50] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [07:10:01] !log [@mwtask141] starting deploy of {'config': True} to all [07:10:06] Not now though still, I want to just test the current version before duing anything like that to it. [07:11:23] CosmicAlpha: I don't really care how you store the config. In my head the best way is to make it a pip package that can be just updated [07:11:45] And the config is just tried for in a number of places [07:11:45] Also, I deployed world and l10n today in 600 seconds, before on mwtask111 it took around 2,000 seconds for both [07:11:55] That's a lot better [07:12:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.72, 2.85, 3.31 [07:12:32] Why did the config deploy freeze? [07:12:49] It still does not say finished. [07:12:51] No idea [07:13:01] I have never seen it freeze [07:13:31] Let me check htop and see if it is still running then. [07:14:14] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw131 mw142 [07:14:14] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:15:17] Yep, it is, and so is puppet still running for some reason. [07:15:56] mwtask141 just kicked me off SSH also, and I can not log back in... [07:16:30] I have to go for an appointment [07:16:33] Oh, crap that was my internet this time, which just went down.... [07:16:43] Ok [07:17:23] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:17:36] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:18:17] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [07:19:12] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 2.044 second response time [07:19:35] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.015 second response time [07:20:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [07:21:32] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [07:23:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.14, 10.13, 11.87 [07:23:30] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [07:24:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.92, 3.59 [07:26:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.36, 3.62, 3.52 [07:26:30] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw121 mw122 [07:26:55] !log [@test131] starting deploy of {'config': True} to all [07:26:56] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [07:27:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:27:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:28:51] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw122 mw131 [07:29:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 3.87, 6.84, 9.90 [07:29:40] RECOVERY - puritwiki.p-e.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'puritwiki.p-e.kr' will expire on Thu 29 Sep 2022 02:01:07 GMT +0000. [07:30:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.73, 3.58 [07:30:33] PROBLEM - slupe.fp64.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:31:09] RECOVERY - wiki.slimeapedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimeapedia.org' will expire on Mon 10 Oct 2022 05:19:54 GMT +0000. [07:31:12] PROBLEM - elsterclopedie.marithaime.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:08] PROBLEM - savagepedia.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:32] PROBLEM - pwiki.drydraytonvillagehall.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:34:08] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:34:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:34:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.38, 3.68, 3.61 [07:35:12] RECOVERY - wiki.site97.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.site97.fr' will expire on Tue 18 Oct 2022 15:25:34 GMT +0000. [07:35:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.15, 11.52, 10.69 [07:35:27] PROBLEM - stablestate.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:50] PROBLEM - miraheze.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:36:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.40, 4.13, 3.77 [07:36:57] PROBLEM - rww2.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:37:11] PROBLEM - burnout.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:38:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.97, 3.76 [07:40:28] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:40:46] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [07:41:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.30, 9.90, 10.75 [07:42:23] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.60, 3.63 [07:43:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.24, 8.65, 10.20 [07:44:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.61, 3.63 [07:44:35] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.028 second response time [07:44:50] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [07:45:00] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:45:15] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [07:46:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [07:46:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.55, 3.00, 3.39 [07:46:58] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:47:31] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:52] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw131 mw132 [07:49:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.50, 11.37, 10.88 [07:50:02] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.070 second response time [07:50:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.32, 3.44 [07:51:47] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw131 mw142 [07:52:08] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.902 second response time [07:52:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.66, 3.21, 3.40 [07:53:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.40, 12.35, 11.36 [07:54:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:55:26] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:56:42] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:57:00] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:57:34] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.039 second response time [07:58:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.36, 4.42, 3.82 [07:58:57] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [07:59:34] PROBLEM - wiki.pacebi.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:36] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:41] PROBLEM - wiki.shaazzz.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:53] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:00:08] RECOVERY - elsterclopedie.marithaime.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'elsterclopedie.marithaime.nl' will expire on Sun 30 Oct 2022 20:08:40 GMT +0000. [08:00:10] PROBLEM - pokemonwiki.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:00:44] PROBLEM - civwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:00:50] PROBLEM - n64brew.dev - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:00:59] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.43, 9.59, 8.00 [08:01:11] RECOVERY - savagepedia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'savagepedia.wiki' will expire on Wed 24 Aug 2022 04:44:37 GMT +0000. [08:01:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.91, 3.66, 4.00 [08:01:22] PROBLEM - mariopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:46] PROBLEM - wikifencing.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:48] PROBLEM - thepolicyhub.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:51] PROBLEM - files.pornwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:58] RECOVERY - pwiki.drydraytonvillagehall.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.drydraytonvillagehall.org.uk' will expire on Wed 12 Oct 2022 13:33:40 GMT +0000. [08:01:58] PROBLEM - wiki.misslessfighters.club - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:20] PROBLEM - dragonquest2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:47] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Wed 24 Aug 2022 02:12:36 GMT +0000. [08:03:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.99, 10.87, 8.62 [08:03:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.39, 10.29, 11.67 [08:03:17] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:03:46] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:03:47] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.642 second response time [08:04:06] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.033 second response time [08:04:55] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:08] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 1.036 second response time [08:05:10] RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Tue 23 Aug 2022 08:41:48 GMT +0000. [08:05:15] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Wed 12 Oct 2022 13:21:03 GMT +0000. [08:05:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.79, 3.99, 4.07 [08:05:27] PROBLEM - wiki.otir.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:29] RECOVERY - rww2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rww2.org' will expire on Sat 17 Sep 2022 22:30:25 GMT +0000. [08:05:34] PROBLEM - lab.wikimimo.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:44] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.026 second response time [08:05:47] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:16] PROBLEM - metroidpedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:22] PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:28] RECOVERY - burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'burnout.wiki' will expire on Tue 01 Nov 2022 04:10:29 GMT +0000. [08:06:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.67, 3.70 [08:06:40] PROBLEM - miraheze.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:07:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.50, 9.68, 8.81 [08:07:10] PROBLEM - arru.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:07:14] PROBLEM - tot.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:07:26] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:20] PROBLEM - dc.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [08:09:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.45, 12.51, 11.84 [08:09:18] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [08:10:27] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.018 second response time [08:10:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.72, 3.71 [08:11:00] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:11:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.80, 11.55, 11.55 [08:12:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.90, 3.58, 3.65 [08:13:33] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:13:34] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:13:54] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.016 second response time [08:14:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.83, 3.73 [08:15:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 2.92, 7.89, 10.13 [08:15:33] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 1.038 second response time [08:15:50] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.059 second response time [08:16:21] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.25, 0.60, 0.79 [08:16:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.77, 3.17, 3.48 [08:18:16] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:18:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 3.01, 3.39 [08:20:44] PROBLEM - cp21 Stunnel HTTP for puppet111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:22:03] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:24:09] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 7.131 second response time [08:24:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.22, 10.09, 9.00 [08:25:16] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Wed 31 Aug 2022 21:21:26 GMT +0000. [08:25:27] RECOVERY - cp21 Stunnel HTTP for puppet111 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 5.073 second response time [08:25:58] PROBLEM - hellointernet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.86, 9.90, 9.06 [08:27:25] PROBLEM - ngsswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:28:00] PROBLEM - wikileague.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:28:07] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:28:18] RECOVERY - slupe.fp64.org - LetsEncrypt on sslhost is OK: OK - Certificate 'slupe.fp64.org' will expire on Tue 04 Oct 2022 18:47:51 GMT +0000. [08:28:19] PROBLEM - sarovia.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:28:24] RECOVERY - wiki.pacebi.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pacebi.org' will expire on Mon 10 Oct 2022 06:45:00 GMT +0000. [08:28:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:29:03] RECOVERY - pokemonwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'pokemonwiki.info' will expire on Wed 19 Oct 2022 06:57:42 GMT +0000. [08:29:26] PROBLEM - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:29:28] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Sat 17 Sep 2022 21:39:52 GMT +0000. [08:30:03] RECOVERY - n64brew.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'n64brew.dev' will expire on Tue 23 Aug 2022 09:10:11 GMT +0000. [08:30:18] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:21] RECOVERY - mariopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mariopedia.org' will expire on Fri 07 Oct 2022 11:49:47 GMT +0000. [08:30:28] PROBLEM - wiki.digitalefitheid.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.91, 3.64 [08:30:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.43, 11.04, 9.72 [08:30:40] RECOVERY - files.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'files.pornwiki.org' will expire on Sat 17 Sep 2022 19:10:31 GMT +0000. [08:30:57] RECOVERY - wiki.misslessfighters.club - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.misslessfighters.club' will expire on Tue 23 Aug 2022 13:06:24 GMT +0000. [08:31:15] RECOVERY - dragonquest2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:31:22] PROBLEM - wiki.slimeapedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:31:50] PROBLEM - arcanewaterswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:15] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:32:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.58, 10.78, 9.81 [08:32:34] PROBLEM - www.iceria.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:33:29] PROBLEM - gp.ct777.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:33:40] PROBLEM - wiki.cyberfurs.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:33:44] PROBLEM - wiki.globasa.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:34:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.61, 9.87, 9.56 [08:34:43] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Mon 10 Oct 2022 04:58:16 GMT +0000. [08:34:53] PROBLEM - wiki.parappa.dev - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:03] RECOVERY - lab.wikimimo.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lab.wikimimo.com' will expire on Sun 09 Oct 2022 03:42:01 GMT +0000. [08:35:04] PROBLEM - piwik.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:22] RECOVERY - wiki.otir.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.otir.nl' will expire on Fri 26 Aug 2022 04:20:05 GMT +0000. [08:35:44] PROBLEM - test2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:46] RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Tue 23 Aug 2022 02:51:23 GMT +0000. [08:35:56] RECOVERY - miraheze.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.gq' will expire on Wed 31 Aug 2022 20:33:05 GMT +0000. [08:36:01] RECOVERY - metroidpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'metroidpedia.com' will expire on Wed 26 Oct 2022 18:29:18 GMT +0000. [08:36:05] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.053 second response time [08:36:21] PROBLEM - replication-ops.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:25] RECOVERY - tot.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tot.wiki' will expire on Tue 23 Aug 2022 12:51:08 GMT +0000. [08:36:28] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 23 Aug 2022 00:31:22 GMT +0000. [08:36:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.28, 3.77, 3.62 [08:36:35] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:41] RECOVERY - arru.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'www.arru.xyz' will expire on Sat 17 Sep 2022 21:47:49 GMT +0000. [08:36:57] PROBLEM - rww2.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:57] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:37:07] PROBLEM - wiki.cjgh.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:38:14] RECOVERY - dc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:38:26] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:38:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [08:38:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.73, 3.63 [08:38:36] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:38:40] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:39:47] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:39:48] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:40:21] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:40:25] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [08:40:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.40, 3.88, 3.69 [08:40:43] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.293 second response time [08:40:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 22.80, 13.02, 10.50 [08:41:58] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11264 MB (29% inode=96%); [08:42:28] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.180 second response time [08:42:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.97, 3.68, 3.63 [08:42:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.22, 11.21, 10.17 [08:43:07] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:26] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:44:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.16, 8.80, 9.41 [08:45:13] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.075 second response time [08:46:54] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.034 second response time [08:48:00] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.810 second response time [08:48:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:48:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.48, 3.10, 3.40 [08:50:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.27, 11.28, 10.04 [08:52:11] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:24] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:52:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.27, 9.70, 9.61 [08:53:27] PROBLEM - pokemon2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:54:09] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.028 second response time [08:54:34] RECOVERY - hellointernet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:55:00] PROBLEM - wiki.danvs.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.40, 3.43 [08:56:12] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:30] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.029 second response time [08:56:53] RECOVERY - wikileague.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wikileague.cf' will expire on Mon 22 Aug 2022 22:21:46 GMT +0000. [08:57:20] RECOVERY - sarovia.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sarovia.graalmilitary.com' will expire on Tue 25 Oct 2022 07:05:18 GMT +0000. [08:57:23] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:57:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.35, 4.61, 3.83 [08:57:27] RECOVERY - wiki.shaazzz.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.shaazzz.ir' will expire on Tue 23 Aug 2022 09:06:44 GMT +0000. [08:58:02] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:02] RECOVERY - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.hoolehistoryheritagesociety.org.uk' will expire on Sat 01 Oct 2022 16:03:14 GMT +0000. [08:58:04] PROBLEM - www.rothwell-leeds.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:45] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.36, 3.58, 3.98 [08:59:25] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.093 second response time [09:00:19] RECOVERY - wiki.slimeapedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimeapedia.org' will expire on Mon 10 Oct 2022 05:19:54 GMT +0000. [09:00:55] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.918 second response time [09:00:59] PROBLEM - bluearchive.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:04] RECOVERY - arcanewaterswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'arcanewaterswiki.org' will expire on Sun 30 Oct 2022 20:54:06 GMT +0000. [09:01:05] PROBLEM - thestories.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:06] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [09:01:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.66, 3.65, 3.67 [09:01:38] PROBLEM - wiki.msnld.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:40] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Tue 23 Aug 2022 01:42:45 GMT +0000. [09:01:44] RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 17 Sep 2022 21:13:26 GMT +0000. [09:01:45] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:02:38] RECOVERY - gp.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'gp.ct777.cf' will expire on Sat 01 Oct 2022 16:25:42 GMT +0000. [09:03:02] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.008 second response time [09:03:03] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Wed 24 Aug 2022 05:10:07 GMT +0000. [09:03:08] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:03:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.64, 2.93, 3.40 [09:03:30] RECOVERY - wiki.globasa.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.globasa.net' will expire on Mon 10 Oct 2022 05:36:34 GMT +0000. [09:04:04] RECOVERY - wiki.parappa.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.parappa.dev' will expire on Sun 18 Sep 2022 13:44:45 GMT +0000. [09:04:26] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:04:28] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:04:31] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:04:34] RECOVERY - test2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:04:36] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:04:37] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:04:42] RECOVERY - piwik.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:04:52] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:05:05] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.341 second response time [09:05:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.17, 2.07, 3.19 [09:05:27] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:05:29] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Tue 23 Aug 2022 07:32:52 GMT +0000. [09:05:29] RECOVERY - rww2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rww2.org' will expire on Sat 17 Sep 2022 22:30:25 GMT +0000. [09:05:41] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:05:43] RECOVERY - replication-ops.com - LetsEncrypt on sslhost is OK: OK - Certificate 'replication-ops.com' will expire on Tue 18 Oct 2022 15:32:39 GMT +0000. [09:06:00] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.274 second response time [09:06:09] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [09:06:14] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [09:06:19] RECOVERY - wiki.cjgh.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cjgh.xyz' will expire on Mon 03 Oct 2022 20:57:39 GMT +0000. [09:06:23] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.230 second response time [09:06:24] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.316 second response time [09:06:27] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.038 second response time [09:06:29] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:31] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:34] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.556 second response time [09:06:38] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.322 second response time [09:06:50] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.535 second response time [09:07:41] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.033 second response time [09:07:43] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.087 second response time [09:08:09] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [09:08:10] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:08:26] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [09:09:01] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.925 second response time [09:10:10] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.036 second response time [09:10:14] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [09:11:03] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:11:28] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:12:10] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:12:57] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:12:58] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:17] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:27] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.05, 9.39, 8.35 [09:14:10] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [09:14:12] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:15:24] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.624 second response time [09:15:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.30, 9.33, 8.46 [09:16:14] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw132 mw142 [09:16:46] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.641 second response time [09:16:58] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.306 second response time [09:17:03] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.440 second response time [09:17:20] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.344 second response time [09:17:23] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:17:36] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:17:49] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:18:01] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:18:05] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:18:17] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.441 second response time [09:18:30] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.028 second response time [09:18:31] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:26] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:49] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:51] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:19:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.62, 10.43, 8.65 [09:20:01] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 2% [09:20:14] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [09:20:25] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.107 second response time [09:20:32] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.555 second response time [09:20:41] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:21:56] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 9.915 second response time [09:21:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.64, 9.31, 8.50 [09:22:04] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:26] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.045 second response time [09:22:59] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:00] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:23:15] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:21] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:24] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:28] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:29] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:36] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:52] PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:54] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.016 second response time [09:23:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.33, 8.47, 8.26 [09:24:10] PROBLEM - test1.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:35] RECOVERY - ngsswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'ngsswiki.com' will expire on Sun 02 Oct 2022 05:32:39 GMT +0000. [09:24:48] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.950 second response time [09:25:02] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.031 second response time [09:25:19] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.347 second response time [09:25:22] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.229 second response time [09:25:28] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.317 second response time [09:25:28] PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.48, 11.15, 9.52 [09:26:09] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw131 mw132 [09:26:14] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw121 [09:26:21] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.068 second response time [09:26:34] PROBLEM - cp21 Stunnel HTTP for puppet111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:53] RECOVERY - www.project-patterns.com - LetsEncrypt on sslhost is OK: OK - Certificate 'project-patterns.com' will expire on Fri 19 Aug 2022 17:20:05 GMT +0000. [09:26:56] RECOVERY - www.rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Sat 17 Sep 2022 18:56:39 GMT +0000. [09:27:15] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:27:26] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 6.121 second response time [09:27:31] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.012 second response time [09:27:35] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.335 second response time [09:27:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.05, 12.48, 10.18 [09:28:09] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [09:28:12] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [09:28:14] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [09:28:19] PROBLEM - vitriol.spoon.army - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:19] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:25] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [09:28:28] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:32] RECOVERY - cp21 Stunnel HTTP for puppet111 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.033 second response time [09:28:47] RECOVERY - wikifencing.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'wikifencing.ru' will expire on Tue 06 Sep 2022 20:55:14 GMT +0000. [09:29:16] PROBLEM - wiki.newnormal.fans - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:26] RECOVERY - thepolicyhub.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thepolicyhub.org.uk' will expire on Tue 23 Aug 2022 02:56:36 GMT +0000. [09:29:27] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:28] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:32] PROBLEM - vault.agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.59, 11.11, 9.95 [09:29:43] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fr.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:15:36 GMT +0000. [09:30:06] PROBLEM - wiki.wikimedia.cat - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:10] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:30:15] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:30:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [09:30:33] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:30:57] RECOVERY - wiki.msnld.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.msnld.com' will expire on Sun 18 Sep 2022 04:31:29 GMT +0000. [09:31:16] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:31:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.26, 10.06, 9.69 [09:31:45] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:31:47] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:31:50] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:31:53] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.023 second response time [09:32:04] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:15] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.036 second response time [09:32:20] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.115 second response time [09:32:25] PROBLEM - www.sidem.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:29] PROBLEM - www.portalsofphereon.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:37] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.811 second response time [09:33:16] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.019 second response time [09:33:30] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.672 second response time [09:33:31] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [09:33:34] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw141 mw142 [09:33:39] PROBLEM - steamdecklinux.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:33:42] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.321 second response time [09:33:46] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [09:33:49] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 1.469 second response time [09:33:54] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.027 second response time [09:33:59] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.406 second response time [09:34:15] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [09:34:29] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.318 second response time [09:35:50] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [09:36:05] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:36:09] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw122 [09:36:19] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.029 second response time [09:36:39] PROBLEM - miraheze.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:00] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:03] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.043 second response time [09:38:09] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [09:38:23] PROBLEM - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:40] PROBLEM - en.clockupwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:43] PROBLEM - staratlaswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.60, 11.45, 9.15 [09:39:02] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 3.038 second response time [09:39:09] PROBLEM - theapplesecretswiki.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:39:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 22.41, 16.33, 12.38 [09:39:29] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:39:30] PROBLEM - mcdev.grantlmul.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:40:34] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:41:36] PROBLEM - wiki.yumeka.icu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:43:05] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:43:08] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 3.078 second response time [09:43:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.91, 11.66, 11.51 [09:43:38] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.040 second response time [09:44:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:44:36] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:44:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.19, 11.29, 10.17 [09:46:36] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [09:46:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.76, 9.32, 9.59 [09:47:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 3.95, 7.28, 9.72 [09:47:40] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:49] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:48:00] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [09:49:19] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.25, 1.75, 1.14 [09:49:46] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.095 second response time [09:49:50] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.029 second response time [09:50:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [09:51:01] RECOVERY - pokemon2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:51:03] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:27] RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.warframestat.us' will expire on Wed 24 Aug 2022 01:37:17 GMT +0000. [09:52:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.58, 12.74, 10.89 [09:52:39] RECOVERY - test1.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:54:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.97, 10.34, 10.23 [09:55:24] PROBLEM - wiki.scvo.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:49] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:55:52] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:55] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 0.020 second response time [09:56:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.85, 8.91, 9.72 [09:56:49] RECOVERY - vitriol.spoon.army - LetsEncrypt on sslhost is OK: OK - Certificate 'vitriol.spoon.army' will expire on Tue 23 Aug 2022 01:40:06 GMT +0000. [09:57:28] RECOVERY - wiki.digitalefitheid.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.digitalefitheid.nl' will expire on Wed 21 Sep 2022 08:49:04 GMT +0000. [09:57:51] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [09:58:06] RECOVERY - wiki.newnormal.fans - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.newnormal.fans' will expire on Fri 14 Oct 2022 16:20:05 GMT +0000. [09:58:25] RECOVERY - vault.agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vault.agentisai.com' will expire on Thu 29 Sep 2022 08:34:24 GMT +0000. [09:58:36] RECOVERY - bluearchive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bluearchive.wiki' will expire on Mon 22 Aug 2022 15:22:46 GMT +0000. [09:58:47] RECOVERY - thestories.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'thestories.kr' will expire on Thu 01 Sep 2022 00:44:22 GMT +0000. [09:58:57] RECOVERY - wiki.wikimedia.cat - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Mon 10 Oct 2022 04:52:56 GMT +0000. [09:59:08] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.42, 1.18, 1.18 [10:00:19] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:00:21] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:56] RECOVERY - www.sidem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.sidem.wiki' will expire on Tue 23 Aug 2022 12:59:12 GMT +0000. [10:01:54] RECOVERY - www.portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Sat 17 Sep 2022 20:30:32 GMT +0000. [10:01:58] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:02:16] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [10:02:26] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.041 second response time [10:02:59] RECOVERY - steamdecklinux.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'steamdecklinux.wiki' will expire on Thu 01 Sep 2022 00:43:35 GMT +0000. [10:03:24] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:03:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.99, 3.81, 3.17 [10:04:12] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.029 second response time [10:04:28] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:06:26] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.059 second response time [10:06:31] RECOVERY - miraheze.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.cf' will expire on Sat 20 Aug 2022 20:31:27 GMT +0000. [10:07:40] PROBLEM - tl.awiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:13] PROBLEM - famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:18] PROBLEM - sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:19] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 0.034 second response time [10:08:24] PROBLEM - franchise.franchising.org.ua - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:25] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [10:08:31] RECOVERY - en.clockupwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.clockupwiki.org' will expire on Sat 10 Sep 2022 21:43:05 GMT +0000. [10:08:49] RECOVERY - theapplesecretswiki.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'theapplesecretswiki.ml' will expire on Mon 17 Oct 2022 19:49:07 GMT +0000. [10:09:03] RECOVERY - mcdev.grantlmul.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'mcdev.grantlmul.xyz' will expire on Fri 14 Oct 2022 16:18:23 GMT +0000. [10:09:18] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:29] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.72, 3.36 [10:11:12] RECOVERY - wiki.yumeka.icu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yumeka.icu' will expire on Sat 17 Sep 2022 18:24:18 GMT +0000. [10:11:14] PROBLEM - wiki.zergmafia.top - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:23] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:11:38] PROBLEM - dc.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:12:37] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.49, 10.06, 9.15 [10:13:29] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.22, 4.17, 3.61 [10:13:52] PROBLEM - nottingham.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.58, 3.66, 3.50 [10:15:58] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:16:45] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [10:16:48] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [10:17:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.68, 10.88, 9.83 [10:17:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.47, 3.27, 3.37 [10:17:49] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.049 second response time [10:17:55] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.024 second response time [10:19:35] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:19:56] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:20:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [10:21:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.28, 8.35, 9.14 [10:21:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.46, 3.43 [10:21:41] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.060 second response time [10:21:45] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 5.074 second response time [10:22:03] RECOVERY - wiki.danvs.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.danvs.net' will expire on Tue 23 Aug 2022 02:16:10 GMT +0000. [10:22:48] PROBLEM - wiki.slimelab.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:51] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.34, 2.55, 1.75 [10:22:56] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.12, 2.97, 3.25 [10:24:17] RECOVERY - wiki.scvo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scvo.org' will expire on Tue 23 Aug 2022 09:01:11 GMT +0000. [10:24:53] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.019 second response time [10:25:00] PROBLEM - wiki.jkrstudios.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:26:29] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [10:26:40] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:26:50] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.72, 1.96, 1.68 [10:27:01] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.030 second response time [10:28:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.20, 2.96, 3.88 [10:28:39] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:29:25] PROBLEM - wiki.misslessfighters.club - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 11.57, 7.00, 5.29 [10:30:45] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:31:00] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.207 second response time [10:31:09] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:47] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:32:16] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:18] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:32] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw122 mw141 [10:32:40] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:41] PROBLEM - pornwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:07] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.035 second response time [10:33:12] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:34:21] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.093 second response time [10:34:22] PROBLEM - m33ks.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:34:24] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.021 second response time [10:34:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:35:35] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:35:44] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.379 second response time [10:36:37] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.47, 0.96, 1.30 [10:36:45] PROBLEM - stablestate.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:37:04] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:37:13] PROBLEM - wiki.otir.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:37:19] RECOVERY - tl.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tl.awiki.org' will expire on Tue 23 Aug 2022 07:53:50 GMT +0000. [10:37:28] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [10:37:46] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.017 second response time [10:37:48] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:38:02] RECOVERY - famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.org' will expire on Mon 10 Oct 2022 05:45:28 GMT +0000. [10:38:15] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sat 17 Sep 2022 21:29:52 GMT +0000. [10:38:15] RECOVERY - franchise.franchising.org.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'franchise.franchising.org.ua' will expire on Tue 23 Aug 2022 02:08:17 GMT +0000. [10:38:27] RECOVERY - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beaconspace.unrestrictedlorefare.com' will expire on Tue 23 Aug 2022 00:32:44 GMT +0000. [10:38:29] RECOVERY - staratlaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'staratlaswiki.org' will expire on Mon 19 Sep 2022 03:34:09 GMT +0000. [10:38:36] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:38:40] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:38:53] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:15] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 22 Oct 2022 05:58:53 GMT +0000. [10:39:44] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:39:47] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.029 second response time [10:39:58] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 0.015 second response time [10:40:36] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 10929 MB (28% inode=96%); [10:40:43] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.048 second response time [10:40:56] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.022 second response time [10:40:59] PROBLEM - mystiverse.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:41:22] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.66, 9.72, 6.50 [10:41:34] RECOVERY - dc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [10:41:50] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [10:42:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.43, 3.45, 3.08 [10:42:31] PROBLEM - cp20 Stunnel HTTP for puppet111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:43:10] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [10:43:22] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.52, 9.21, 6.71 [10:43:39] RECOVERY - nottingham.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nottingham.wiki' will expire on Mon 26 Sep 2022 06:16:48 GMT +0000. [10:43:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.84, 11.13, 9.18 [10:44:22] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:01] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:45:25] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:43] PROBLEM - www.trollpasta.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.69, 8.93, 8.62 [10:46:19] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [10:46:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [10:47:16] !log sudo service ircrcbot restart on mon141 [10:47:24] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.026 second response time [10:47:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:47:57] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.037 second response time [10:48:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.18, 2.63 [10:48:28] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [10:49:04] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.037 second response time [10:49:23] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:50:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.54, 2.83 [10:50:10] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:50:17] RECOVERY - cp20 Stunnel HTTP for puppet111 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [10:50:18] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw122 mw132 mw142 [10:51:23] RECOVERY - wiki.slimelab.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimelab.net' will expire on Sat 01 Oct 2022 16:16:26 GMT +0000. [10:52:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.17, 3.33, 2.83 [10:52:27] RECOVERY - thestarsareright.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thestarsareright.org' will expire on Sat 17 Sep 2022 22:08:07 GMT +0000. [10:53:16] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=osconceptswiki --no-updates --username-prefix=wikia:osm /mnt/mediawiki-static/metawiki/ImportDump/osconceptswiki-20220803104809.xml (START) [10:53:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:53:37] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.123 second response time [10:54:53] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:22] RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Mon 22 Aug 2022 22:44:29 GMT +0000. [10:55:28] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:55:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.94, 3.73 [10:56:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.62, 3.08 [10:57:02] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:57:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.10, 2.99, 2.91 [10:58:07] PROBLEM - cp20 Stunnel HTTP for puppet111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:58:22] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:58:26] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.75, 3.64 [10:59:32] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:34] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [10:59:38] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [10:59:48] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:51] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:10] RECOVERY - cp20 Stunnel HTTP for puppet111 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.016 second response time [11:00:13] PROBLEM - pt.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:20] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [11:00:33] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:36] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [11:00:39] PROBLEM - gluster101 Disk Space on gluster101 is WARNING: DISK WARNING - free space: / 108806 MB (10% inode=82%); [11:01:04] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:01:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.64, 2.52, 3.20 [11:01:37] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 81% [11:01:39] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [11:01:45] PROBLEM - gluster101 SSH on gluster101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:01:48] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:01:49] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:01:53] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.215 second response time [11:01:54] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.092 second response time [11:02:01] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:03] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:07] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:09] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [11:02:09] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:10] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:11] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:12] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:12] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:14] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [11:02:16] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:17] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:17] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:17] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:18] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:19] PROBLEM - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:21] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:21] PROBLEM - gluster101 PowerDNS Recursor on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:02:21] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:22] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:30] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Wed 24 Aug 2022 01:59:18 GMT +0000. [11:02:34] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:35] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:43] PROBLEM - avas.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:43] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:47] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:55] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:58] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:03:07] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:03:23] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:03:27] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.024 second response time [11:03:30] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:03:38] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 45% [11:03:39] PROBLEM - gluster101 Disk Space on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:03:50] PROBLEM - gluster101 Puppet on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:03:51] PROBLEM - cloud10 Current Load on cloud10 is CRITICAL: CRITICAL - load average: 49.51, 34.62, 21.19 [11:03:55] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:04:16] PROBLEM - gluster101 Gluster Disk Space on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:04:38] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.016 second response time [11:04:59] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.93, 0.74, 0.82 [11:05:15] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.013 second response time [11:05:35] PROBLEM - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:37] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 96% [11:05:42] PROBLEM - wiki.pixlies.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:46] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.82, 5.22, 2.88 [11:05:49] PROBLEM - robloxapi.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:51] PROBLEM - miraheze.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:52] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.056 second response time [11:06:10] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.838 second response time [11:06:11] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 6.967 second response time [11:06:15] RECOVERY - gluster101 Gluster Disk Space on gluster101 is OK: DISK OK - free space: /mnt/mediawiki-static 1520435 MB (37% inode=84%); [11:06:16] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.084 second response time [11:06:16] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.578 second response time [11:06:16] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.511 second response time [11:06:17] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.005 second response time [11:06:18] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.212 second response time [11:06:18] RECOVERY - gluster101 Puppet on gluster101 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures [11:06:19] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.009 second response time [11:06:19] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.324 second response time [11:06:22] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.012 second response time [11:06:23] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.409 second response time [11:06:23] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.012 second response time [11:06:24] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.315 second response time [11:06:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:06:28] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.818 second response time [11:06:28] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.020 second response time [11:06:35] PROBLEM - gluster101 glusterd on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:06:36] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.013 second response time [11:06:38] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.029 second response time [11:06:42] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.324 second response time [11:06:42] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.181 second response time [11:06:50] RECOVERY - mw142 HTTPS on mw142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.011 second response time [11:06:54] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.70, 0.63, 0.77 [11:06:57] PROBLEM - rww2.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:07:03] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [11:07:20] RECOVERY - gluster101 PowerDNS Recursor on gluster101 is OK: DNS OK: 0.145 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [11:07:22] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [11:07:30] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [11:07:46] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.306 second response time [11:07:46] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.54, 4.35, 2.86 [11:07:51] RECOVERY - cloud10 Current Load on cloud10 is OK: OK - load average: 13.66, 31.18, 23.56 [11:07:54] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.306 second response time [11:07:55] RECOVERY - gluster101 SSH on gluster101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:08:02] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.336 second response time [11:08:04] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.289 second response time [11:08:04] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.314 second response time [11:08:13] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.307 second response time [11:08:27] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.035 second response time [11:08:33] PROBLEM - gluster101 Disk Space on gluster101 is WARNING: DISK WARNING - free space: / 108693 MB (10% inode=82%); [11:08:35] RECOVERY - gluster101 glusterd on gluster101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterd' [11:08:41] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.028 second response time [11:08:54] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.199 second response time [11:09:12] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [11:09:36] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 0% [11:09:37] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 2% [11:09:49] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:09:53] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.049 second response time [11:10:06] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [11:10:09] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [11:10:14] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [11:10:51] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:04] RECOVERY - mystiverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mystiverse.wiki' will expire on Sat 01 Oct 2022 04:26:41 GMT +0000. [11:11:35] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:11:52] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [11:12:16] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.204 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [11:13:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:13:28] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.233 second response time [11:13:32] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.060 second response time [11:13:36] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.043 second response time [11:16:26] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw132 mw142 [11:16:32] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 51% [11:16:43] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:18:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.38, 2.79 [11:18:55] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.038 second response time [11:19:29] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:20:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.72, 2.84, 2.67 [11:21:44] PROBLEM - chakra.moe - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:58] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 1% [11:22:06] PROBLEM - tep.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:41] RECOVERY - wiki.jkrstudios.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jkrstudios.org' will expire on Mon 26 Sep 2022 05:51:00 GMT +0000. [11:23:26] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:23:27] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:24:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:25:23] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [11:25:43] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:26:44] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:50] PROBLEM - jp.popira.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:37] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [11:27:49] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.55, 0.67, 0.56 [11:27:51] PROBLEM - cp20 Stunnel HTTP for puppet111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:14] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:31:29] RECOVERY - pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pornwiki.org' will expire on Sat 17 Sep 2022 18:32:03 GMT +0000. [11:31:35] PROBLEM - wiki.meower.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:31:38] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.024 second response time [11:31:44] RECOVERY - avas.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'avas.wiki' will expire on Mon 10 Oct 2022 04:52:24 GMT +0000. [11:31:51] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=osconceptswiki --no-updates --username-prefix=wikia:osm /mnt/mediawiki-static/metawiki/ImportDump/osconceptswiki-20220803104809.xml (END - exit=0) [11:31:52] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=osconceptswiki (START) [11:31:59] RECOVERY - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ricochetuniverse.com' will expire on Tue 23 Aug 2022 00:04:36 GMT +0000. [11:32:03] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:32:16] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:32:29] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw132 mw142 [11:32:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:32:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.24, 9.82, 7.74 [11:32:42] RECOVERY - cp20 Stunnel HTTP for puppet111 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [11:32:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:33:21] RECOVERY - m33ks.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'm33ks.ml' will expire on Thu 29 Sep 2022 04:18:17 GMT +0000. [11:33:48] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:33:49] PROBLEM - rarewiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:03] PROBLEM - meta.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:21] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:34:22] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 0.020 second response time [11:34:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.31, 11.76, 8.67 [11:34:38] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:34:40] PROBLEM - dragonquestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:11] RECOVERY - miraheze.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.gq' will expire on Wed 31 Aug 2022 20:33:05 GMT +0000. [11:35:15] RECOVERY - robloxapi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Mon 12 Sep 2022 18:36:58 GMT +0000. [11:35:30] RECOVERY - rww2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rww2.org' will expire on Sat 17 Sep 2022 22:30:25 GMT +0000. [11:35:39] PROBLEM - gametheorywiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:41] RECOVERY - wiki.pixlies.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pixlies.net' will expire on Tue 06 Sep 2022 20:56:25 GMT +0000. [11:35:44] PROBLEM - test2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:59] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 7.376 second response time [11:36:06] RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Tue 23 Aug 2022 08:41:48 GMT +0000. [11:36:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.15, 10.36, 8.51 [11:36:32] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:36:57] RECOVERY - wiki.otir.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.otir.nl' will expire on Fri 26 Aug 2022 04:20:05 GMT +0000. [11:37:15] PROBLEM - wiki.globasa.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:38:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.30, 12.78, 9.66 [11:39:41] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:40:39] PROBLEM - miraheze.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:41] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.078 second response time [11:41:05] RECOVERY - wiki.zergmafia.top - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zergmafia.top' will expire on Tue 06 Sep 2022 20:57:49 GMT +0000. [11:41:54] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.058 second response time [11:42:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.93, 9.56, 9.12 [11:45:21] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:45:34] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:47:20] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.019 second response time [11:48:03] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=osconceptswiki (END - exit=0) [11:48:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=osconceptswiki --active --update (END - exit=0) [11:48:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:50:05] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:50:24] RECOVERY - chakra.moe - LetsEncrypt on sslhost is OK: OK - Certificate 'chakra.moe' will expire on Wed 26 Oct 2022 18:41:37 GMT +0000. [11:50:54] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Tue 23 Aug 2022 08:40:10 GMT +0000. [11:51:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.29, 10.40, 9.54 [11:52:05] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Wed 31 Aug 2022 21:21:26 GMT +0000. [11:52:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.31, 9.88, 9.45 [11:53:41] PROBLEM - lgbta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:02] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.040 second response time [11:55:03] PROBLEM - wikiyri.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:32] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [11:55:47] RECOVERY - jp.popira.info - LetsEncrypt on sslhost is OK: OK - Certificate 'jp.popira.info' will expire on Mon 10 Oct 2022 06:13:03 GMT +0000. [11:56:15] PROBLEM - wiki.timewisely.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:23] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.022 second response time [11:56:33] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:56:34] PROBLEM - elsterclopedie.marithaime.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:42] RECOVERY - wiki.misslessfighters.club - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.misslessfighters.club' will expire on Tue 23 Aug 2022 13:06:24 GMT +0000. [11:57:06] PROBLEM - wiki.ameristraliagov.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:58:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:58:31] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.029 second response time [11:59:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.25, 3.85, 3.26 [11:59:33] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:31] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:58] RECOVERY - wiki.meower.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meower.org' will expire on Thu 27 Oct 2022 03:39:58 GMT +0000. [12:01:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 3.04, 3.03 [12:01:02] PROBLEM - solaswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:16] PROBLEM - fanon.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:32] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.054 second response time [12:01:46] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:02:34] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.059 second response time [12:02:35] PROBLEM - wiki.openhatch.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:56] RECOVERY - rarewiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewiki.org' will expire on Wed 28 Sep 2022 05:06:19 GMT +0000. [12:03:12] RECOVERY - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dc-multiverse.dcwikis.com' will expire on Wed 24 Aug 2022 05:08:53 GMT +0000. [12:03:33] RECOVERY - meta.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:12:19 GMT +0000. [12:03:47] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.049 second response time [12:04:06] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:44] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw121 mw142 [12:04:57] PROBLEM - wiki.jeremyboyd.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:15] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [12:05:24] RECOVERY - gametheorywiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'gametheorywiki.org' will expire on Sun 25 Sep 2022 20:40:02 GMT +0000. [12:05:40] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.99, 2.14, 1.56 [12:06:11] PROBLEM - erawikisource.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:06:21] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 3.068 second response time [12:06:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [12:06:42] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 3.58, 2.88 [12:07:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.64, 3.29 [12:08:13] PROBLEM - metroid.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:47] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39138 bytes in 1.043 second response time [12:08:48] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:09:24] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.01, 2.97, 2.74 [12:10:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:10:31] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.17, 1.59, 1.48 [12:10:43] PROBLEM - fc.songcontests.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:45] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [12:10:51] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw121 mw132 mw142 [12:11:03] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [12:13:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.66, 3.94, 3.49 [12:14:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [12:14:56] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:15:24] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw131 mw132 mw142 [12:15:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.27, 3.21, 3.27 [12:15:57] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:57] PROBLEM - electowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:11] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 19.14, 13.78, 10.80 [12:16:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:16:34] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:43] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:16:49] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:23] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:58] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 2.046 second response time [12:18:16] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:18:45] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.311 second response time [12:19:25] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:25] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.058 second response time [12:19:57] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.042 second response time [12:20:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.04, 11.37, 10.74 [12:20:49] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.026 second response time [12:21:23] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [12:21:38] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.085 second response time [12:22:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.25, 9.38, 10.09 [12:23:27] PROBLEM - pokemon2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:29] PROBLEM - hoodwiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:30] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:18] PROBLEM - sarovia.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:13] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:27:19] PROBLEM - wiki.minecraftathome.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 21.09, 15.04, 11.95 [12:28:17] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.039 second response time [12:28:19] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:29:57] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:20] PROBLEM - www.turtletown.ca - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:30] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [12:30:37] RECOVERY - fanon.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fanon.polandballwiki.com' will expire on Tue 25 Oct 2022 07:14:58 GMT +0000. [12:31:00] RECOVERY - drowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'drowiki.com' will expire on Tue 04 Oct 2022 18:56:56 GMT +0000. [12:31:05] PROBLEM - miraheze.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:11] PROBLEM - animefeet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:17] RECOVERY - wiki.openhatch.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.openhatch.org' will expire on Tue 23 Aug 2022 09:22:30 GMT +0000. [12:31:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [12:31:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.88, 12.33, 9.33 [12:32:12] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:32:44] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:14] RECOVERY - dragonquestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dragonquestwiki.com' will expire on Thu 15 Sep 2022 05:19:23 GMT +0000. [12:33:35] RECOVERY - test2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:33:37] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.83, 10.35, 8.96 [12:33:59] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:34:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:34:09] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:34:12] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.316 second response time [12:34:12] RECOVERY - wiki.jeremyboyd.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jeremyboyd.org' will expire on Tue 25 Oct 2022 07:00:25 GMT +0000. [12:34:17] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.021 second response time [12:34:45] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.061 second response time [12:35:51] RECOVERY - erawikisource.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:35:55] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.826 second response time [12:35:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.82, 9.04, 8.63 [12:36:03] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.585 second response time [12:36:54] RECOVERY - wiki.globasa.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.globasa.net' will expire on Mon 10 Oct 2022 05:36:34 GMT +0000. [12:37:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.46, 3.28 [12:37:27] RECOVERY - metroid.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:38:40] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.016 second response time [12:38:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.79, 3.40, 3.29 [12:39:40] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:39:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 18.06, 13.63, 10.56 [12:40:22] RECOVERY - miraheze.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.cf' will expire on Sat 20 Aug 2022 20:31:27 GMT +0000. [12:40:29] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [12:41:51] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [12:41:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.29, 11.53, 10.17 [12:42:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.86, 3.87, 3.49 [12:43:54] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.34, 9.45, 9.58 [12:44:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.98, 3.69, 3.48 [12:46:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.87, 3.56 [12:46:51] PROBLEM - proficientprofit.click - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:02] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 0.020 second response time [12:48:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.47, 3.46 [12:49:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 3.97, 8.59, 11.80 [12:50:40] PROBLEM - commonwealth.0um.io - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:45] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:51:19] RECOVERY - lgbta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:51:58] RECOVERY - pokemon2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:52:10] RECOVERY - hoodwiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'hoodwiki.xyz' will expire on Wed 19 Oct 2022 06:37:38 GMT +0000. [12:52:11] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Sat 17 Sep 2022 20:22:48 GMT +0000. [12:52:13] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.81, 3.89, 3.61 [12:52:38] RECOVERY - wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiyri.org' will expire on Thu 22 Sep 2022 04:19:34 GMT +0000. [12:53:05] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [12:53:31] PROBLEM - otogebase.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:20] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.027 second response time [12:54:24] RECOVERY - wiki.timewisely.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.timewisely.net' will expire on Mon 22 Aug 2022 22:01:18 GMT +0000. [12:54:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.00, 3.65, 3.56 [12:54:36] RECOVERY - elsterclopedie.marithaime.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'elsterclopedie.marithaime.nl' will expire on Sun 30 Oct 2022 20:08:40 GMT +0000. [12:54:46] PROBLEM - istpcomputing.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:00] PROBLEM - datacrondatabase.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:02] PROBLEM - cp21 Stunnel HTTP for puppet111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:05] PROBLEM - mario.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.43, 10.16, 11.26 [12:56:20] PROBLEM - largedu.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:21] RECOVERY - wiki.minecraftathome.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minecraftathome.com' will expire on Wed 24 Aug 2022 02:32:00 GMT +0000. [12:56:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.71, 3.93, 3.66 [12:57:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.72, 8.76, 10.61 [12:57:36] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:57:54] PROBLEM - www.cgradegames.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:04] RECOVERY - pt.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.polandballwiki.com' will expire on Thu 29 Sep 2022 08:59:53 GMT +0000. [12:58:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.99, 3.73 [12:58:35] RECOVERY - solaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'solaswiki.org' will expire on Sat 03 Sep 2022 15:10:56 GMT +0000. [12:59:07] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw122 mw141 mw142 [12:59:10] PROBLEM - www.iceria.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.37, 7.33, 9.87 [12:59:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [12:59:33] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.059 second response time [12:59:51] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.75, 11.58, 9.78 [13:00:09] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:00:17] PROBLEM - wiki.milkywayidle.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:29] PROBLEM - wiki.leapit.no - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:51] PROBLEM - roll-of-arms.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:01:00] PROBLEM - roblox-wiki.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:01:33] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:01:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.70, 8.93, 9.04 [13:02:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.16, 2.88, 3.35 [13:02:24] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [13:02:56] PROBLEM - miraheze.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:01] PROBLEM - wiki.potabi.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:04] PROBLEM - www.lab612.at - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:21] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [13:03:29] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [13:03:29] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:31] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:41] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:45] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:46] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:46] PROBLEM - cp31 Stunnel HTTP for phab121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [13:03:46] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [13:03:51] PROBLEM - cp30 Stunnel HTTP for mon141 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [13:03:51] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [13:03:51] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [13:03:52] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:54] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:02] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:06] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:08] PROBLEM - cp30 Stunnel HTTP for mail121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [13:04:15] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:16] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:17] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:21] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:22] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.050 second response time [13:04:22] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:04:22] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:22] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:57] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:02] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.024 second response time [13:05:05] PROBLEM - wiki.rtapp.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:25] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [13:05:25] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.319 second response time [13:05:25] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:25] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.012 second response time [13:05:26] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:05:30] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [13:05:37] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 86% [13:05:39] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.311 second response time [13:05:40] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.310 second response time [13:05:42] RECOVERY - mw142 HTTPS on mw142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [13:05:43] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:05:45] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 3.070 second response time [13:05:46] RECOVERY - cp31 Stunnel HTTP for phab121 on cp31 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.348 second response time [13:05:50] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:50] PROBLEM - az.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:50] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.333 second response time [13:05:51] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.335 second response time [13:05:51] RECOVERY - cp30 Stunnel HTTP for mon141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 0.332 second response time [13:05:51] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.215 second response time [13:05:51] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.318 second response time [13:06:00] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.308 second response time [13:06:03] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.329 second response time [13:06:08] RECOVERY - cp30 Stunnel HTTP for mail121 on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.234 second response time [13:06:09] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [13:06:11] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.012 second response time [13:06:13] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.025 second response time [13:06:14] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [13:06:18] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [13:06:19] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.382 second response time [13:06:19] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.256 second response time [13:06:20] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.412 second response time [13:06:26] RECOVERY - cp21 Stunnel HTTP for puppet111 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.014 second response time [13:07:00] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:07:02] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.030 second response time [13:07:11] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [13:07:31] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:07:48] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 1.054 second response time [13:08:03] PROBLEM - stablestate.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:08:27] PROBLEM - pastport.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:07] PROBLEM - tl.awiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:11] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 42% [13:09:44] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [13:09:50] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [13:10:02] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:18] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:19] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:25] RECOVERY - fc.songcontests.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'fc.songcontests.eu' will expire on Mon 22 Aug 2022 22:44:59 GMT +0000. [13:10:26] PROBLEM - wiki.triplescripts.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:37] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:40] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:40] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:45] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:49] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:55] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:55] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:57] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:58] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:58] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:06] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:07] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:11:07] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.029 second response time [13:11:11] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:11] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 20% [13:11:13] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:26] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [13:11:37] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is WARNING: WARNING - NGINX Error Rate is 50% [13:11:46] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.38, 7.43, 3.91 [13:11:52] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [13:11:59] RECOVERY - mw142 HTTPS on mw142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.013 second response time [13:12:06] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [13:12:09] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.156 second response time [13:12:11] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:15] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.321 second response time [13:12:15] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.310 second response time [13:12:26] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.055 second response time [13:12:26] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.039 second response time [13:12:32] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [13:12:36] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.011 second response time [13:12:37] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.185 second response time [13:12:43] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.225 second response time [13:12:43] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.029 second response time [13:12:45] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.319 second response time [13:12:51] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:12:52] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.309 second response time [13:12:53] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [13:12:53] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.014 second response time [13:12:53] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.437 second response time [13:12:56] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.307 second response time [13:13:02] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.043 second response time [13:13:03] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.316 second response time [13:13:11] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.039 second response time [13:13:13] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.327 second response time [13:13:27] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.045 second response time [13:13:35] PROBLEM - wiki.anglish.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:36] RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Tue 23 Aug 2022 00:50:24 GMT +0000. [13:13:37] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [13:13:46] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.30, 5.32, 3.56 [13:14:06] RECOVERY - www.trollpasta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'trollpasta.com' will expire on Mon 10 Oct 2022 06:29:05 GMT +0000. [13:14:12] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.044 second response time [13:14:52] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.048 second response time [13:15:35] RECOVERY - proficientprofit.click - LetsEncrypt on sslhost is OK: OK - Certificate 'www.proficientprofit.click' will expire on Sat 08 Oct 2022 06:14:24 GMT +0000. [13:15:37] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:37] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 3% [13:15:46] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 4.85, 4.98, 3.62 [13:16:09] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [13:16:14] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [13:16:21] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:31] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:18:13] PROBLEM - wiki.scpsharp.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:18:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [13:19:25] RECOVERY - commonwealth.0um.io - LetsEncrypt on sslhost is OK: OK - Certificate 'commonwealth.0um.io' will expire on Thu 20 Oct 2022 18:21:09 GMT +0000. [13:20:38] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 7.213 second response time [13:20:52] PROBLEM - tensegritywiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:21:10] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:21:32] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 4% [13:21:42] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:21:53] PROBLEM - wiki.wilderyogi.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:04] RECOVERY - otogebase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'otogebase.com' will expire on Mon 12 Sep 2022 09:53:37 GMT +0000. [13:22:17] RECOVERY - sarovia.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sarovia.graalmilitary.com' will expire on Tue 25 Oct 2022 07:05:18 GMT +0000. [13:22:25] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.204 second response time [13:22:48] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:23:07] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.035 second response time [13:23:10] PROBLEM - wiki.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:37] PROBLEM - feathercare.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:39] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 0.021 second response time [13:23:52] RECOVERY - istpcomputing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'istpcomputing.com' will expire on Tue 11 Oct 2022 04:37:06 GMT +0000. [13:24:34] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:24:58] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 30 minutes ago with 0 failures [13:25:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.14, 11.25, 8.25 [13:25:36] RECOVERY - largedu.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'largedu.eu.org' will expire on Sun 30 Oct 2022 14:46:40 GMT +0000. [13:27:05] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:27:10] RECOVERY - www.cgradegames.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.cgradegames.net' will expire on Sat 20 Aug 2022 20:03:51 GMT +0000. [13:28:20] RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 17 Sep 2022 21:13:26 GMT +0000. [13:28:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 3.51, 2.43 [13:28:38] RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Wed 24 Aug 2022 04:25:02 GMT +0000. [13:29:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.61, 9.60, 8.24 [13:29:27] RECOVERY - roll-of-arms.com - LetsEncrypt on sslhost is OK: OK - Certificate 'roll-of-arms.com' will expire on Thu 27 Oct 2022 15:02:13 GMT +0000. [13:29:32] RECOVERY - roblox-wiki.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'roblox-wiki.tk' will expire on Thu 27 Oct 2022 15:10:45 GMT +0000. [13:29:49] RECOVERY - wiki.milkywayidle.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.milkywayidle.com' will expire on Fri 21 Oct 2022 06:32:46 GMT +0000. [13:29:56] RECOVERY - wiki.leapit.no - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.leapit.no' will expire on Sun 30 Oct 2022 08:12:03 GMT +0000. [13:30:22] RECOVERY - animefeet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [13:30:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 3.75, 2.99 [13:30:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.49, 2.56 [13:30:39] PROBLEM - www.portalsofphereon.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:57] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.164 seconds response time. miraheze.org returns 198.244.148.90,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a,51.195.220.68 [13:32:04] RECOVERY - wiki.potabi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.potabi.com' will expire on Mon 10 Oct 2022 06:24:27 GMT +0000. [13:32:10] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:32:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.42, 3.10, 2.53 [13:32:52] RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Sat 17 Sep 2022 18:42:17 GMT +0000. [13:34:49] RECOVERY - wiki.rtapp.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rtapp.tk' will expire on Wed 26 Oct 2022 23:42:41 GMT +0000. [13:35:26] RECOVERY - az.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'az.wikiretet.ga' will expire on Tue 11 Oct 2022 04:40:37 GMT +0000. [13:37:03] PROBLEM - gametheorywiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:37:38] RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Tue 23 Aug 2022 08:41:48 GMT +0000. [13:38:51] RECOVERY - tl.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tl.awiki.org' will expire on Tue 23 Aug 2022 07:53:50 GMT +0000. [13:38:52] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:55] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.94, 3.57 [13:40:46] RECOVERY - sbs.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'sbs.wiki' will expire on Thu 01 Sep 2022 00:42:07 GMT +0000. [13:40:51] PROBLEM - mcdev.grantlmul.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:54] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:42:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.13, 4.53, 3.81 [13:42:21] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.043 second response time [13:42:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:42:34] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:43:30] RECOVERY - wiki.anglish.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.anglish.info' will expire on Wed 24 Aug 2022 03:18:35 GMT +0000. [13:43:30] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:26] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:31] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 7 % full [13:45:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.38, 13.79, 10.54 [13:45:52] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.85, 1.11, 0.75 [13:45:57] PROBLEM - www.istpcomputing.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:28] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.151 second response time [13:46:37] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.075 second response time [13:47:01] RECOVERY - wiki.scpsharp.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scpsharp.ga' will expire on Wed 31 Aug 2022 20:42:39 GMT +0000. [13:47:33] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:42] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3401 bytes in 3.048 second response time [13:47:51] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.11, 1.60, 0.95 [13:48:21] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:13] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:49:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.37, 11.22, 10.24 [13:49:38] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.024 second response time [13:49:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.79, 11.05, 8.15 [13:50:20] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [13:50:21] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.76, 1.25, 0.91 [13:51:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.55, 12.74, 10.92 [13:51:24] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:26] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:51:53] PROBLEM - houkai2.cyou - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.41, 9.70, 7.99 [13:52:00] PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:11] RECOVERY - wiki.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.graalmilitary.com' will expire on Tue 23 Aug 2022 03:02:35 GMT +0000. [13:52:17] RECOVERY - feathercare.net - LetsEncrypt on sslhost is OK: OK - Certificate 'feathercare.net' will expire on Sat 27 Aug 2022 14:00:44 GMT +0000. [13:52:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [13:53:01] RECOVERY - wiki.ameristraliagov.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ameristraliagov.com' will expire on Thu 20 Oct 2022 03:45:43 GMT +0000. [13:53:24] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.057 second response time [13:53:27] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:15] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.000 second response time [13:54:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:55:12] PROBLEM - es.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:13] PROBLEM - sims.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:18] PROBLEM - wiki.newnormal.fans - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:37] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:55:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.58, 11.50, 9.14 [13:56:02] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:44] PROBLEM - mazakwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:46] PROBLEM - pokemonwiki.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:57] PROBLEM - wiki.misslessfighters.club - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:57] PROBLEM - mariopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:33] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.025 second response time [13:57:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.69, 11.58, 9.45 [13:58:02] PROBLEM - www.mh142.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:08] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.219 second response time [13:58:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [13:58:39] RECOVERY - www.turtletown.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'www.turtletown.ca' will expire on Tue 20 Sep 2022 20:30:39 GMT +0000. [13:58:39] PROBLEM - mh142.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:07] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:59:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.26, 3.73, 3.18 [13:59:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.68, 10.15, 9.19 [14:00:17] PROBLEM - pwiki.drydraytonvillagehall.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:58] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:06] PROBLEM - baharna.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:19] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 4% [14:01:38] PROBLEM - wmworld.sktz.live - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:40] PROBLEM - wiki.meower.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:45] PROBLEM - buildabearwiki.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:55] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Wed 12 Oct 2022 13:21:03 GMT +0000. [14:02:19] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:01] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Mon 22 Aug 2022 23:58:24 GMT +0000. [14:03:12] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:14] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.51, 3.28, 3.13 [14:03:41] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.74, 10.52, 7.35 [14:03:58] PROBLEM - phgalaxy.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:05] PROBLEM - persist.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:16] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:05:15] PROBLEM - evil.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:40] PROBLEM - wiki.mcpirevival.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:41] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.03, 8.78, 7.10 [14:05:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.88, 12.97, 10.70 [14:06:04] RECOVERY - pastport.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pastport.org' will expire on Wed 31 Aug 2022 20:55:03 GMT +0000. [14:06:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:06:45] [discord] https://discord.gg/eX8RyxhD2x [14:06:45] [url] 18+ Discord Hub | discord.gg [14:06:45] [discord] @KhoAjA No invite links. [14:07:06] PROBLEM - reviwiki.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.15, 10.57, 11.61 [14:07:44] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [14:07:46] PROBLEM - replication-ops.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:57] PROBLEM - wiki.recaptime.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.37, 11.76, 10.51 [14:08:00] PROBLEM - lab.wikimimo.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:08:30] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.047 second response time [14:09:07] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:11] PROBLEM - wiki.exputra.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:43] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:09:55] PROBLEM - cp21 Stunnel HTTP for puppet111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:02] PROBLEM - theapplesecretswiki.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:18] RECOVERY - wiki.triplescripts.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.triplescripts.org' will expire on Wed 24 Aug 2022 04:49:04 GMT +0000. [14:10:31] RECOVERY - mcdev.grantlmul.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'mcdev.grantlmul.xyz' will expire on Fri 14 Oct 2022 16:18:23 GMT +0000. [14:11:05] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 1.048 second response time [14:11:41] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.771 second response time [14:12:05] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw122 mw131 [14:12:08] RECOVERY - cp21 Stunnel HTTP for puppet111 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.013 second response time [14:13:09] RECOVERY - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.teessidehackspace.org.uk' will expire on Wed 21 Sep 2022 09:01:23 GMT +0000. [14:13:09] PROBLEM - wiki.otir.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.67, 7.66, 9.91 [14:13:19] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:14:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [14:14:32] RECOVERY - www.istpcomputing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'istpcomputing.com' will expire on Tue 11 Oct 2022 04:37:06 GMT +0000. [14:14:33] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:15:15] PROBLEM - ambientguitar.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:19] PROBLEM - proficientprofit.click - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:37] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [14:16:38] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:17:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.27, 7.39, 9.40 [14:18:14] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.013 second response time [14:19:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.55, 14.15, 12.06 [14:19:42] RECOVERY - wiki.wilderyogi.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wilderyogi.eu' will expire on Mon 26 Sep 2022 06:35:10 GMT +0000. [14:20:12] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [14:20:27] RECOVERY - wiki.thunis.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thunis.eu' will expire on Tue 23 Aug 2022 01:47:24 GMT +0000. [14:20:34] RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.warframestat.us' will expire on Wed 24 Aug 2022 01:37:17 GMT +0000. [14:21:41] RECOVERY - mario.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:21:59] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Wed 31 Aug 2022 21:21:26 GMT +0000. [14:21:59] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fr.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:15:36 GMT +0000. [14:22:32] PROBLEM - agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:35] RECOVERY - datacrondatabase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'datacrondatabase.com' will expire on Mon 22 Aug 2022 15:10:26 GMT +0000. [14:22:56] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 1.038 second response time [14:24:00] RECOVERY - es.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'es.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:18:15 GMT +0000. [14:24:08] RECOVERY - wiki.newnormal.fans - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.newnormal.fans' will expire on Fri 14 Oct 2022 16:20:05 GMT +0000. [14:24:26] RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:24:33] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:24:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.50, 3.21 [14:25:02] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw132 mw141 [14:25:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.37, 11.85, 11.92 [14:25:39] RECOVERY - pokemonwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'pokemonwiki.info' will expire on Wed 19 Oct 2022 06:57:42 GMT +0000. [14:25:41] RECOVERY - mazakwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mazakwiki.org' will expire on Wed 26 Oct 2022 18:25:25 GMT +0000. [14:25:56] RECOVERY - wiki.misslessfighters.club - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.misslessfighters.club' will expire on Tue 23 Aug 2022 13:06:24 GMT +0000. [14:25:56] RECOVERY - mariopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mariopedia.org' will expire on Fri 07 Oct 2022 11:49:47 GMT +0000. [14:25:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.80, 10.22, 9.52 [14:26:39] RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Tue 23 Aug 2022 00:06:10 GMT +0000. [14:26:40] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.016 second response time [14:26:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.19, 3.35, 3.19 [14:27:02] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.95, 2.36, 1.67 [14:27:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.22, 9.99, 9.50 [14:27:59] RECOVERY - mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Tue 23 Aug 2022 00:06:10 GMT +0000. [14:28:20] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:29:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.69, 1.74, 1.53 [14:29:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.00, 13.41, 12.45 [14:29:21] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:29:25] RECOVERY - www.portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Sat 17 Sep 2022 20:30:32 GMT +0000. [14:29:46] RECOVERY - pwiki.drydraytonvillagehall.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.drydraytonvillagehall.org.uk' will expire on Wed 12 Oct 2022 13:33:40 GMT +0000. [14:29:47] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:29:58] PROBLEM - croc.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:29:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.84, 7.76, 8.76 [14:30:38] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 7.200 second response time [14:31:02] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.69, 1.40, 1.42 [14:31:03] RECOVERY - wiki.meower.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meower.org' will expire on Thu 27 Oct 2022 03:39:58 GMT +0000. [14:31:08] RECOVERY - wmworld.sktz.live - LetsEncrypt on sslhost is OK: OK - Certificate 'wmworld.sktz.live' will expire on Sat 01 Oct 2022 16:33:15 GMT +0000. [14:31:24] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:31:25] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.038 second response time [14:31:54] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 1.129 second response time [14:32:04] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:11] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:35] RECOVERY - phgalaxy.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'phgalaxy.eu.org' will expire on Sun 02 Oct 2022 05:33:18 GMT +0000. [14:32:50] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Wed 24 Aug 2022 01:38:48 GMT +0000. [14:32:58] PROBLEM - pmwiki.cubestudios.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:00] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:03] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:19] PROBLEM - rarewiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:52] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:34:06] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 5.775 second response time [14:34:12] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.440 second response time [14:34:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:34:46] RECOVERY - evil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'evil.wiki' will expire on Sun 23 Oct 2022 21:23:17 GMT +0000. [14:34:49] RECOVERY - wiki.mcpirevival.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcpirevival.tk' will expire on Wed 24 Aug 2022 03:07:55 GMT +0000. [14:34:53] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:34:53] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:35:31] PROBLEM - steamdecklinux.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:35:41] PROBLEM - aman.awiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:35:57] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:36:03] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.306 second response time [14:36:18] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:36:27] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:28] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:50] RECOVERY - gametheorywiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'gametheorywiki.org' will expire on Sun 25 Sep 2022 20:40:02 GMT +0000. [14:36:55] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [14:36:57] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:58] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:08] RECOVERY - replication-ops.com - LetsEncrypt on sslhost is OK: OK - Certificate 'replication-ops.com' will expire on Tue 18 Oct 2022 15:32:39 GMT +0000. [14:37:12] PROBLEM - wiki.recaptime.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:20] PROBLEM - wiki.thedev.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:32] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [14:37:34] RECOVERY - wiki.recaptime.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.tk' will expire on Wed 26 Oct 2022 23:10:27 GMT +0000. [14:37:39] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:44] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:49] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [14:38:09] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.954 second response time [14:38:09] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw121 mw122 mw132 mw141 mw142 [14:38:16] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw131 mw142 [14:38:19] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.028 second response time [14:38:24] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.345 second response time [14:38:26] RECOVERY - wiki.exputra.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.exputra.com' will expire on Fri 16 Sep 2022 18:09:57 GMT +0000. [14:38:32] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.695 second response time [14:38:45] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:53] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.324 second response time [14:39:01] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:05] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:06] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:14] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 1.303 second response time [14:39:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.03, 3.94 [14:39:26] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.044 second response time [14:39:27] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.846 second response time [14:39:34] RECOVERY - theapplesecretswiki.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'theapplesecretswiki.ml' will expire on Mon 17 Oct 2022 19:49:07 GMT +0000. [14:39:49] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.810 second response time [14:39:55] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:40:14] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [14:40:14] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:15] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:17] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 0.043 second response time [14:40:57] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:59] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.443 second response time [14:41:03] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.272 second response time [14:41:07] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 3.972 second response time [14:41:14] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.043 second response time [14:41:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.95, 9.85, 11.62 [14:41:17] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.053 second response time [14:41:22] PROBLEM - mediawiki.ecfin.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:29] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.016 second response time [14:41:45] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [14:41:46] PROBLEM - wiki.ravynos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:49] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.044 second response time [14:42:09] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [14:42:12] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.323 second response time [14:42:12] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.217 second response time [14:42:46] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.021 second response time [14:42:56] RECOVERY - wiki.otir.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.otir.nl' will expire on Fri 26 Aug 2022 04:20:05 GMT +0000. [14:43:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.71, 3.99 [14:43:25] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.037 second response time [14:44:00] RECOVERY - proficientprofit.click - LetsEncrypt on sslhost is OK: OK - Certificate 'www.proficientprofit.click' will expire on Sat 08 Oct 2022 06:14:24 GMT +0000. [14:44:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [14:45:03] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.017 second response time [14:45:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.35, 6.82, 9.96 [14:45:48] PROBLEM - revi.wiki - PositiveSSLDV on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:48] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.85, 12.48, 9.65 [14:47:23] PROBLEM - www.journeytheword.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:46] PROBLEM - gs.sidem.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:49:30] RECOVERY - houkai2.cyou - LetsEncrypt on sslhost is OK: OK - Certificate 'houkai2.cyou' will expire on Thu 27 Oct 2022 05:22:41 GMT +0000. [14:49:40] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:50:52] PROBLEM - wiki.dereric.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:11] RECOVERY - tensegritywiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tensegritywiki.com' will expire on Tue 23 Aug 2022 08:10:17 GMT +0000. [14:51:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 26.34, 16.58, 12.97 [14:51:29] RECOVERY - agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.agentisai.com' will expire on Wed 28 Sep 2022 11:24:06 GMT +0000. [14:51:36] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.030 second response time [14:51:46] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:55] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:00] PROBLEM - www.fossilfighterswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:06] PROBLEM - wiki.ciptamedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:11] PROBLEM - wiki.wikimedia.cat - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [14:52:51] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:02] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:53:27] PROBLEM - pokemon2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:24] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:54:50] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [14:55:01] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.024 second response time [14:55:07] PROBLEM - dkpedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:21] PROBLEM - wiki.digitalefitheid.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:26] PROBLEM - wiki.newnormal.fans - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:13] PROBLEM - unmade.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:19] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.272 second response time [14:56:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.97, 11.78, 11.26 [14:56:42] PROBLEM - civclassic.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:50] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:56:53] PROBLEM - scum-wiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:54] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.258 second response time [14:59:01] RECOVERY - croc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:59:51] PROBLEM - wikicryptos.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:07] RECOVERY - baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Sat 17 Sep 2022 21:09:35 GMT +0000. [15:01:29] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 0.018 second response time [15:01:38] RECOVERY - persist.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'persist.wiki' will expire on Mon 10 Oct 2022 04:01:57 GMT +0000. [15:02:34] PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:36] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:36] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:02:46] PROBLEM - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:52] PROBLEM - wiki-tintin.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:08] PROBLEM - portalsofphereon.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:08] PROBLEM - wiki.jeremyboyd.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:27] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Wed 24 Aug 2022 02:07:19 GMT +0000. [15:03:54] PROBLEM - cp20 Stunnel HTTP for puppet111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:03:58] PROBLEM - famedata.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:03] PROBLEM - liberhood.su - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:19] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Wed 24 Aug 2022 01:59:18 GMT +0000. [15:04:19] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.31, 8.58, 10.00 [15:04:33] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 0.031 second response time [15:04:34] PROBLEM - wiki.nevillepedia.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:35] PROBLEM - wiki.vinesh.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:50] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.085 second response time [15:04:52] RECOVERY - steamdecklinux.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'steamdecklinux.wiki' will expire on Thu 01 Sep 2022 00:43:35 GMT +0000. [15:05:20] RECOVERY - aman.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'aman.awiki.org' will expire on Mon 22 Aug 2022 23:45:19 GMT +0000. [15:05:29] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Tue 23 Aug 2022 07:32:52 GMT +0000. [15:05:35] PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:36] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:44] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:06] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:12] PROBLEM - iol.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:24] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39177 bytes in 1.071 second response time [15:06:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [15:06:29] RECOVERY - reviwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Sat 17 Sep 2022 19:46:36 GMT +0000. [15:07:12] RECOVERY - wiki.thedev.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thedev.gq' will expire on Fri 02 Sep 2022 12:36:20 GMT +0000. [15:07:14] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:07:15] RECOVERY - lab.wikimimo.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lab.wikimimo.com' will expire on Sun 09 Oct 2022 03:42:01 GMT +0000. [15:07:27] PROBLEM - wiki.pablojprieto.com.es - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:33] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.679 second response time [15:07:44] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 3.736 second response time [15:07:45] PROBLEM - erawikisource.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:01] RECOVERY - cp20 Stunnel HTTP for puppet111 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.015 second response time [15:08:13] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.703 second response time [15:09:15] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.184 second response time [15:09:49] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:54] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:11:07] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:11:19] RECOVERY - mediawiki.ecfin.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'mediawiki.ecfin.eu' will expire on Mon 22 Aug 2022 23:01:19 GMT +0000. [15:11:44] RECOVERY - wiki.ravynos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ravynos.com' will expire on Thu 01 Sep 2022 00:45:42 GMT +0000. [15:12:53] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 1.025 second response time [15:14:02] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:15:16] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.38, 1.98, 1.40 [15:15:56] RECOVERY - www.journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Wed 31 Aug 2022 18:13:33 GMT +0000. [15:17:45] RECOVERY - gs.sidem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'gs.sidem.wiki' will expire on Tue 23 Aug 2022 13:09:38 GMT +0000. [15:19:35] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.959 second response time [15:19:46] RECOVERY - wiki.dereric.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dereric.gq' will expire on Sat 22 Oct 2022 22:30:02 GMT +0000. [15:20:21] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:40] RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Mon 22 Aug 2022 22:45:26 GMT +0000. [15:21:04] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:21:09] RECOVERY - wiki.wikimedia.cat - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Mon 10 Oct 2022 04:52:56 GMT +0000. [15:21:27] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:36] PROBLEM - mario.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:43] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:01] RECOVERY - pokemon2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:22:02] RECOVERY - wiki.ciptamedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Tue 23 Aug 2022 08:44:56 GMT +0000. [15:22:13] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:43] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:23:05] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.036 second response time [15:23:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 22.47, 14.93, 11.68 [15:23:15] PROBLEM - wiki.minecraftathome.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:47] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:49] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:17] RECOVERY - wiki.newnormal.fans - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.newnormal.fans' will expire on Fri 14 Oct 2022 16:20:05 GMT +0000. [15:24:27] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 0.020 second response time [15:24:57] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.167 second response time [15:25:19] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:20] RECOVERY - unmade.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:25:33] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.427 second response time [15:25:33] RECOVERY - civclassic.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:25:54] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.086 second response time [15:26:06] RECOVERY - scum-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'scum-wiki.com' will expire on Sat 22 Oct 2022 05:35:09 GMT +0000. [15:26:20] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 4.694 second response time [15:26:37] PROBLEM - donkeykong.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:27:36] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [15:28:05] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.020 second response time [15:28:25] PROBLEM - wiki.ssangyongsports.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:28:48] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:59] PROBLEM - mh142.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.01, 1.86, 1.88 [15:29:17] RECOVERY - wikicryptos.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikicryptos.org' will expire on Wed 24 Aug 2022 05:45:16 GMT +0000. [15:29:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.14, 4.74, 3.57 [15:29:23] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:23] PROBLEM - popira.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:33] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 6.603 second response time [15:29:38] RECOVERY - buildabearwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'buildabearwiki.info' will expire on Fri 16 Sep 2022 17:59:19 GMT +0000. [15:29:39] PROBLEM - ndg.nenawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:56] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.352 second response time [15:31:03] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [15:31:15] RECOVERY - pmwiki.cubestudios.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'pmwiki.cubestudios.xyz' will expire on Mon 03 Oct 2022 20:32:38 GMT +0000. [15:31:18] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [15:31:44] RECOVERY - rarewiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewiki.org' will expire on Wed 28 Sep 2022 05:06:19 GMT +0000. [15:31:59] RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Wed 24 Aug 2022 03:06:17 GMT +0000. [15:32:41] RECOVERY - liberhood.su - LetsEncrypt on sslhost is OK: OK - Certificate 'liberhood.su' will expire on Tue 25 Oct 2022 15:28:02 GMT +0000. [15:33:02] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:33:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.88, 3.47 [15:33:32] RECOVERY - famedata.org - LetsEncrypt on sslhost is OK: OK - Certificate 'famedata.org' will expire on Thu 29 Sep 2022 07:42:43 GMT +0000. [15:33:43] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:33:52] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.052 second response time [15:34:00] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:34:19] RECOVERY - wiki.vinesh.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.vinesh.eu.org' will expire on Sat 17 Sep 2022 22:28:24 GMT +0000. [15:34:41] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:02] RECOVERY - wiki.recaptime.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.eu.org' will expire on Mon 03 Oct 2022 22:23:06 GMT +0000. [15:35:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 4.11, 9.80, 11.71 [15:35:02] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.40, 1.69 [15:35:04] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 9885 MB (25% inode=96%); [15:35:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.65, 4.18, 3.63 [15:35:34] PROBLEM - wiki.mcpirevival.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:39] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.051 second response time [15:35:47] PROBLEM - meta.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:36:37] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:36:57] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:12] PROBLEM - az.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:24] RECOVERY - erawikisource.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:37:31] PROBLEM - rww2.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:51] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:11] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 3.061 second response time [15:38:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [15:38:27] PROBLEM - pastport.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:52] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.092 second response time [15:39:41] PROBLEM - tl.awiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:45] PROBLEM - wiki.auraxis.co - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:35] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:55] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.141 second response time [15:40:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.15, 6.70, 9.70 [15:41:29] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:39] PROBLEM - wiki.sketch.town - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:42:39] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.28, 3.34, 3.53 [15:43:32] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:39] PROBLEM - miraheze.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:43] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.019 second response time [15:44:08] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:17] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:21] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:22] RECOVERY - ambientguitar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ambientguitar.org' will expire on Mon 22 Aug 2022 22:13:35 GMT +0000. [15:44:29] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:33] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:44:39] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:39] PROBLEM - hi.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:45] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 7.221 second response time [15:44:48] PROBLEM - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:52] PROBLEM - crocwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.86, 2.81, 3.31 [15:45:25] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.039 second response time [15:45:30] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [15:45:43] RECOVERY - revi.wiki - PositiveSSLDV on sslhost is OK: OK - Certificate 'revi.wiki' will expire on Thu 29 Dec 2022 23:59:59 GMT +0000. [15:45:51] PROBLEM - wiki.zergmafia.top - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:05] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 1.053 second response time [15:46:09] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw132 [15:46:16] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.339 second response time [15:46:16] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.321 second response time [15:46:26] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.026 second response time [15:46:37] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [15:46:39] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.509 second response time [15:46:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.46, 10.42, 10.35 [15:46:58] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [15:48:09] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [15:48:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:48:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.38, 9.46, 10.02 [15:49:12] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:21] RECOVERY - mario.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:51:02] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [15:52:09] RECOVERY - wiki.minecraftathome.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minecraftathome.com' will expire on Wed 24 Aug 2022 02:32:00 GMT +0000. [15:52:41] RECOVERY - dkpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dkpedia.com' will expire on Tue 04 Oct 2022 18:55:51 GMT +0000. [15:53:29] PROBLEM - hoodwiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:19] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:54:36] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.91, 3.05, 3.85 [15:56:36] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.073 second response time [15:56:44] PROBLEM - www.iceria.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:44] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:45] PROBLEM - civclassic.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:10] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.60, 7.10, 11.44 [15:57:35] RECOVERY - wiki.ssangyongsports.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ssangyongsports.eu.org' will expire on Wed 31 Aug 2022 22:12:11 GMT +0000. [15:57:39] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 1.021 second response time [15:57:54] RECOVERY - popira.info - LetsEncrypt on sslhost is OK: OK - Certificate 'popira.info' will expire on Sun 02 Oct 2022 11:12:21 GMT +0000. [15:58:17] RECOVERY - mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Tue 23 Aug 2022 00:06:10 GMT +0000. [15:58:23] RECOVERY - ndg.nenawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nenawiki.org' will expire on Tue 23 Aug 2022 01:35:44 GMT +0000. [15:58:40] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:59:13] PROBLEM - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.70, 10.82, 12.23 [15:59:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.56, 3.86 [15:59:32] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:00:06] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [16:00:20] PROBLEM - wiki.leapit.no - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:54] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.061 second response time [16:01:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.51, 3.81 [16:01:27] PROBLEM - avas.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:37] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.020 second response time [16:01:47] RECOVERY - wiki.jeremyboyd.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jeremyboyd.org' will expire on Tue 25 Oct 2022 07:00:25 GMT +0000. [16:01:48] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.65, 2.09, 1.75 [16:02:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [16:02:44] RECOVERY - portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Sat 17 Sep 2022 20:30:32 GMT +0000. [16:02:50] PROBLEM - webkinzguide.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:03:08] RECOVERY - wiki.grottocenter.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.grottocenter.org' will expire on Mon 22 Aug 2022 22:30:31 GMT +0000. [16:03:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.05, 1.74, 1.66 [16:03:52] RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Mon 22 Aug 2022 23:35:07 GMT +0000. [16:03:53] RECOVERY - wiki.nevillepedia.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nevillepedia.eu' will expire on Wed 24 Aug 2022 04:41:20 GMT +0000. [16:04:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.20, 10.92, 9.34 [16:04:42] RECOVERY - wiki.mcpirevival.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcpirevival.tk' will expire on Wed 24 Aug 2022 03:07:55 GMT +0000. [16:05:14] RECOVERY - meta.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:12:19 GMT +0000. [16:05:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.05, 3.91, 3.89 [16:05:26] RECOVERY - iol.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'iol.wiki' will expire on Thu 27 Oct 2022 14:58:25 GMT +0000. [16:05:29] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Tue 23 Aug 2022 07:32:52 GMT +0000. [16:05:40] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.96, 3.64, 2.42 [16:06:03] RECOVERY - rww2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rww2.org' will expire on Sat 17 Sep 2022 22:30:25 GMT +0000. [16:06:13] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:41] RECOVERY - az.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'az.wikiretet.ga' will expire on Tue 11 Oct 2022 04:40:37 GMT +0000. [16:06:49] RECOVERY - wiki.pablojprieto.com.es - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pablojprieto.com.es' will expire on Tue 23 Aug 2022 09:39:26 GMT +0000. [16:07:03] RECOVERY - pastport.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pastport.org' will expire on Wed 31 Aug 2022 20:55:03 GMT +0000. [16:07:07] PROBLEM - swests.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:09] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:07:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 3.37, 8.67, 11.37 [16:07:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.64, 3.79 [16:07:22] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Tue 23 Aug 2022 01:05:30 GMT +0000. [16:07:57] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:08:09] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw122 [16:08:15] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.942 second response time [16:08:23] PROBLEM - Host mwtask111 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:208) [16:08:26] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.419 second response time [16:09:13] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.147 second response time [16:09:17] RECOVERY - tl.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tl.awiki.org' will expire on Tue 23 Aug 2022 07:53:50 GMT +0000. [16:09:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 4.03, 3.92 [16:10:00] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 7.188 second response time [16:10:09] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [16:10:20] RECOVERY - wiki.sketch.town - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sketch.town' will expire on Thu 13 Oct 2022 04:34:16 GMT +0000. [16:11:03] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:11:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.56, 10.08, 11.21 [16:12:32] !log shutdown mwtask111 [16:12:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:23] RECOVERY - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.umbranet.mandelbrotgames.com' will expire on Sun 02 Oct 2022 11:07:49 GMT +0000. [16:13:30] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.150 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [16:13:32] RECOVERY - crocwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crocwiki.com' will expire on Thu 15 Sep 2022 05:33:41 GMT +0000. [16:14:37] RECOVERY - hi.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.gyaanipedia.com' will expire on Thu 08 Sep 2022 06:44:45 GMT +0000. [16:15:18] !log delete vm mwtask111 [16:15:23] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:49] RECOVERY - wiki.zergmafia.top - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zergmafia.top' will expire on Tue 06 Sep 2022 20:57:49 GMT +0000. [16:16:22] PROBLEM - ambientguitar.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.89, 11.14, 11.78 [16:17:02] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [16:17:07] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:17:08] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/31f043bdc63b...e69bba3266ce [16:17:08] [url] Comparing 31f043bdc63b...e69bba3266ce · miraheze/dns · GitHub | github.com [16:17:09] [02miraheze/dns] 07paladox 03e69bba3 - Remove mwtask111 from dns [16:17:33] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:17:46] PROBLEM - www.scum-wiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:18:48] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/7f3932f9378e [16:18:49] [02miraheze/ImportDump] 07Universal-Omega 037f3932f - Convert DB select queries to use SelectQueryBuilder [16:18:51] [02ImportDump] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ImportDump [16:18:51] [url] Page not found · GitHub · GitHub | github.com [16:18:52] [02ImportDump] 07Universal-Omega opened pull request 03#24: Convert DB select queries to use SelectQueryBuilder - 13https://github.com/miraheze/ImportDump/pull/24 [16:18:53] [url] Page not found · GitHub · GitHub | github.com [16:19:58] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/7f3932f9378e...6f4d6c078c19 [16:19:59] [url] Comparing 7f3932f9378e...6f4d6c078c19 · miraheze/ImportDump · GitHub | github.com [16:20:00] [02miraheze/ImportDump] 07Universal-Omega 036f4d6c0 - Update SpecialRequestImportDump.php [16:20:01] [02ImportDump] 07Universal-Omega synchronize pull request 03#24: Convert DB select queries to use SelectQueryBuilder - 13https://github.com/miraheze/ImportDump/pull/24 [16:20:02] [url] Page not found · GitHub · GitHub | github.com [16:20:16] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.26, 1.53, 1.98 [16:20:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.06, 7.43, 10.11 [16:20:29] RECOVERY - www.fossilfighterswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.fossilfighterswiki.com' will expire on Sat 24 Sep 2022 13:17:51 GMT +0000. [16:20:45] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.050 second response time [16:20:46] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:20:55] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:28] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.135 second response time [16:21:29] PROBLEM - bn.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:52] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.044 second response time [16:22:58] PROBLEM - pubwiki.lab.co.pl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:00] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [16:23:25] PROBLEM - intp.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:27] PROBLEM - wiki.mastodon.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:27] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw122 mw132 mw142 [16:23:28] PROBLEM - electrowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:28] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:32] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:50] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:24:14] RECOVERY - donkeykong.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [16:24:22] miraheze/ImportDump - Universal-Omega the build has errored. [16:24:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.74, 3.97 [16:24:52] [02ImportDump] 07Universal-Omega synchronize pull request 03#24: Convert DB select queries to use SelectQueryBuilder - 13https://github.com/miraheze/ImportDump/pull/24 [16:24:53] [url] Page not found · GitHub · GitHub | github.com [16:24:54] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/6f4d6c078c19...f7373b7d3026 [16:24:54] [url] Comparing 6f4d6c078c19...f7373b7d3026 · miraheze/ImportDump · GitHub | github.com [16:24:55] [02miraheze/ImportDump] 07Universal-Omega 03f7373b7 - Update ImportDumpRequestManager.php [16:24:59] PROBLEM - datacrondatabase.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:29] RECOVERY - civclassic.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [16:25:31] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [16:25:56] RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 17 Sep 2022 21:13:26 GMT +0000. [16:26:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.15, 1.86, 1.96 [16:26:06] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.106 second response time [16:26:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.18, 10.93, 10.87 [16:26:31] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:26:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.94, 4.01 [16:27:56] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.33, 1.67, 1.88 [16:28:30] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39177 bytes in 0.049 second response time [16:28:34] RECOVERY - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bikinibottomdiscord.org' will expire on Mon 22 Aug 2022 21:26:08 GMT +0000. [16:28:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.68, 3.91 [16:29:01] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [16:30:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.92, 7.95, 9.67 [16:30:22] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.062 second response time [16:30:33] RECOVERY - wiki-tintin.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki-tintin.cf' will expire on Mon 10 Oct 2022 03:55:03 GMT +0000. [16:30:35] RECOVERY - avas.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'avas.wiki' will expire on Mon 10 Oct 2022 04:52:24 GMT +0000. [16:30:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [16:30:43] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:58] RECOVERY - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worlduniversityandschool.org' will expire on Tue 23 Aug 2022 07:52:18 GMT +0000. [16:31:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.68, 1.13, 1.61 [16:32:03] PROBLEM - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:42] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.093 second response time [16:32:57] PROBLEM - commonswiki.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:33:00] PROBLEM - wiki.potabi.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:33:13] PROBLEM - wikipanel.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:56] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:58] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:04] miraheze/ImportDump - Universal-Omega the build passed. [16:35:07] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:15] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:41] PROBLEM - m33ks.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:47] PROBLEM - sammaditthi.cyou - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:31] RECOVERY - swests.org - LetsEncrypt on sslhost is OK: OK - Certificate 'swests.org' will expire on Thu 20 Oct 2022 03:33:41 GMT +0000. [16:36:34] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:18] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:38:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 4.01, 3.93 [16:39:21] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [16:39:44] RECOVERY - wiki.auraxis.co - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.auraxis.co' will expire on Fri 21 Oct 2022 06:34:16 GMT +0000. [16:40:07] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:40] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:05] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.329 second response time [16:41:14] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [16:41:14] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.374 second response time [16:41:57] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.046 second response time [16:42:32] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:36] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.041 second response time [16:42:37] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.055 second response time [16:43:22] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:43:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.48, 9.01, 8.63 [16:44:34] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.134 second response time [16:45:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.88, 1.96, 1.67 [16:45:58] RECOVERY - ambientguitar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ambientguitar.org' will expire on Mon 22 Aug 2022 22:13:35 GMT +0000. [16:46:48] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.051 second response time [16:46:57] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.44, 2.30, 1.84 [16:47:45] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:48:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.81, 3.96 [16:48:27] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 13.10, 9.92, 7.31 [16:49:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.64, 1.96, 1.76 [16:49:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.85, 11.92, 10.37 [16:49:45] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.182 second response time [16:50:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.88, 4.25, 4.10 [16:50:26] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 8.04, 9.35, 7.43 [16:50:42] RECOVERY - bn.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'bn.gyaanipedia.com' will expire on Thu 08 Sep 2022 06:11:48 GMT +0000. [16:50:45] RECOVERY - hoodwiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'hoodwiki.xyz' will expire on Wed 19 Oct 2022 06:37:38 GMT +0000. [16:50:47] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:51:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 2.02, 1.80 [16:51:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 18.10, 14.38, 11.45 [16:51:42] RECOVERY - wiki.digitalefitheid.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.digitalefitheid.nl' will expire on Wed 21 Sep 2022 08:49:04 GMT +0000. [16:52:09] PROBLEM - ethene.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:39] PROBLEM - wikimoma.art - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:44] PROBLEM - vrcdev.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:46] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:52:58] PROBLEM - wiki.pbptanarchy.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:00] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Wed 31 Aug 2022 21:21:26 GMT +0000. [16:53:02] PROBLEM - commons.ballmedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:07] PROBLEM - it.famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:07] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.342 second response time [16:53:18] PROBLEM - feathercare.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:29] PROBLEM - thenintendowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:41] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:54:29] PROBLEM - perpustaxaan.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:30] PROBLEM - wiki.creaturathegame.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:55:38] PROBLEM - polandball.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.17, 9.68, 10.23 [16:55:41] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.117 second response time [16:56:47] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:57:24] PROBLEM - studyabroad.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'studyabroad.wiki' expires in 15 day(s) (Fri 19 Aug 2022 16:50:34 GMT +0000). [16:57:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.93, 1.94, 1.82 [16:57:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.30, 7.94, 9.50 [16:58:13] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.78, 1.63, 1.26 [16:58:30] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [16:59:25] PROBLEM - cp20 Stunnel HTTP for puppet111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:10] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.32, 2.00, 1.42 [17:00:43] PROBLEM - wiki.ct777.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:47] PROBLEM - opendatascot.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:08] RECOVERY - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ricochetuniverse.com' will expire on Tue 23 Aug 2022 00:04:36 GMT +0000. [17:01:09] RECOVERY - webkinzguide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'webkinzguide.com' will expire on Mon 29 Aug 2022 18:06:45 GMT +0000. [17:01:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.75, 12.79, 11.15 [17:01:56] PROBLEM - miraheze.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:02:06] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.51, 1.77, 1.41 [17:02:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [17:02:34] RECOVERY - commonswiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:02:39] PROBLEM - phgalaxy.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:02:47] RECOVERY - wiki.potabi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.potabi.com' will expire on Mon 10 Oct 2022 06:24:27 GMT +0000. [17:02:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0a5e0dc3ff41...f393374e3f66 [17:02:51] [url] Comparing 0a5e0dc3ff41...f393374e3f66 · miraheze/ssl · GitHub | github.com [17:02:52] [02miraheze/ssl] 07MirahezeSSLBot 03f393374 - Bot: Update SSL cert for studyabroad.wiki [17:03:24] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.55, 1.69 [17:03:59] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:01] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.07, 1.63, 1.41 [17:04:22] PROBLEM - kunwok.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:23] RECOVERY - cp20 Stunnel HTTP for puppet111 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [17:04:24] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:04:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:04:40] RECOVERY - m33ks.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'm33ks.ml' will expire on Thu 29 Sep 2022 04:18:17 GMT +0000. [17:05:31] RECOVERY - sammaditthi.cyou - LetsEncrypt on sslhost is OK: OK - Certificate 'sammaditthi.cyou' will expire on Sat 08 Oct 2022 06:03:19 GMT +0000. [17:05:54] PROBLEM - metroid.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:02] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 3.158 second response time [17:06:11] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.080 second response time [17:06:26] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:37] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [17:07:50] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:08:20] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:22] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [17:08:23] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.017 second response time [17:09:25] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.526 second response time [17:09:27] PROBLEM - patternarchive.online - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:09:54] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [17:10:03] PROBLEM - wiki.seamly.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:13] PROBLEM - lab.wikimimo.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:36] PROBLEM - ngsswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:35] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw121 mw122 mw142 [17:12:01] PROBLEM - sbs.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:42] RECOVERY - miraheze.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.cf' will expire on Sat 20 Aug 2022 20:31:27 GMT +0000. [17:13:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.14, 9.91, 11.56 [17:14:19] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:14:27] PROBLEM - wiki.otir.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:42] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:15:50] PROBLEM - wiki.veloren.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:19] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:17:14] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.54, 11.57, 11.65 [17:17:41] PROBLEM - mystiverse.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:43] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/96305641654e [17:17:45] [02miraheze/dns] 07paladox 039630564 - Add puppet141 to dns [17:17:46] [02dns] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [17:17:47] [url] Page not found · GitHub · GitHub | github.com [17:17:47] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.209 second response time [17:17:48] [02dns] 07paladox opened pull request 03#324: Add puppet141 to dns - 13https://github.com/miraheze/dns/pull/324 [17:17:48] ... [17:17:56] [02dns] 07paladox closed pull request 03#324: Add puppet141 to dns - 13https://github.com/miraheze/dns/pull/324 [17:17:56] [url] Page not found · GitHub · GitHub | github.com [17:17:58] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/e69bba3266ce...2e73aefb8d37 [17:17:58] [url] Comparing e69bba3266ce...2e73aefb8d37 · miraheze/dns · GitHub | github.com [17:17:59] [02miraheze/dns] 07paladox 032e73aef - Add puppet141 to dns (#324) [17:18:01] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-3 [17:18:02] [02dns] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [17:18:02] [url] Page not found · GitHub · GitHub | github.com [17:18:19] PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:40] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.45, 3.20, 3.85 [17:20:21] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:20:29] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:20:32] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:33] PROBLEM - dcmultiverse.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:39] RECOVERY - ethene.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Sat 24 Sep 2022 13:58:45 GMT +0000. [17:20:51] RECOVERY - electrowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electrowiki.org' will expire on Sat 17 Sep 2022 19:15:15 GMT +0000. [17:21:18] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:21:48] RECOVERY - intp.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:21:48] RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Tue 23 Aug 2022 08:56:40 GMT +0000. [17:21:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.44, 9.81, 11.82 [17:21:52] RECOVERY - it.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'it.famepedia.org' will expire on Sat 08 Oct 2022 06:08:33 GMT +0000. [17:21:54] RECOVERY - wiki.pbptanarchy.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pbptanarchy.tk' will expire on Tue 06 Sep 2022 20:52:59 GMT +0000. [17:22:03] PROBLEM - gluster121 Disk Space on gluster121 is WARNING: DISK WARNING - free space: / 109278 MB (10% inode=82%); [17:22:16] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 4 % full [17:22:24] PROBLEM - nottingham.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:28] PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:34] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 2.056 second response time [17:22:37] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:22:47] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:22:50] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:23:00] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [17:23:07] RECOVERY - feathercare.net - LetsEncrypt on sslhost is OK: OK - Certificate 'feathercare.net' will expire on Sat 27 Aug 2022 14:00:44 GMT +0000. [17:23:13] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [17:23:20] RECOVERY - wiki.mastodon.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mastodon.kr' will expire on Mon 05 Sep 2022 00:02:52 GMT +0000. [17:23:26] RECOVERY - thenintendowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thenintendowiki.org' will expire on Sat 22 Oct 2022 05:06:23 GMT +0000. [17:24:25] RECOVERY - wiki.creaturathegame.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.creaturathegame.com' will expire on Mon 22 Aug 2022 20:43:25 GMT +0000. [17:24:32] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.023 second response time [17:24:43] RECOVERY - polandball.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:24:48] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.018 second response time [17:25:13] PROBLEM - squareenix.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:20] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:21] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:25:29] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.197 second response time [17:25:30] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.103 second response time [17:25:44] RECOVERY - wiki.mineland.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mineland.eu' will expire on Tue 23 Aug 2022 01:29:10 GMT +0000. [17:26:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.80, 3.89, 3.84 [17:26:49] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:54] PROBLEM - studyabroad.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:28] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:27:33] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.021 second response time [17:27:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.85, 11.60, 11.72 [17:28:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.10, 3.07, 3.54 [17:28:34] RECOVERY - wiki.leapit.no - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.leapit.no' will expire on Sun 30 Oct 2022 08:12:03 GMT +0000. [17:28:54] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:54] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.040 second response time [17:29:05] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:34] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:53] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:58] RECOVERY - wiki.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ct777.cf' will expire on Sun 02 Oct 2022 10:59:59 GMT +0000. [17:29:58] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.015 second response time [17:30:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.52, 2.20, 3.16 [17:30:24] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:26] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:31] RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Thu 01 Sep 2022 00:46:11 GMT +0000. [17:30:50] RECOVERY - miraheze.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.gq' will expire on Wed 31 Aug 2022 20:33:05 GMT +0000. [17:31:00] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.465 second response time [17:31:07] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.234 second response time [17:31:39] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.030 second response time [17:31:48] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.177 second response time [17:32:04] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [17:32:23] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.373 second response time [17:32:23] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.381 second response time [17:32:30] RECOVERY - wikipanel.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipanel.org' will expire on Sun 02 Oct 2022 04:51:38 GMT +0000. [17:33:07] RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Wed 24 Aug 2022 04:11:39 GMT +0000. [17:33:19] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:33:22] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 12.38, 8.86, 6.96 [17:33:38] PROBLEM - tot.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:34:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 3.36, 3.40 [17:34:32] RECOVERY - metroid.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:34:33] PROBLEM - www.arru.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:34:48] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=amazingideaswiki --no-updates --username-prefix=wikia:amazingideas /mnt/mediawiki-static/metawiki/ImportDump/amazingideaswiki-20220803122217.xml (START) [17:35:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=osconceptswiki --no-updates --username-prefix=wikia:osm /mnt/mediawiki-static/metawiki/ImportDump/osconceptswiki-20220803071517.xml (START) [17:35:18] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.047 second response time [17:35:21] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 7.84, 8.50, 7.07 [17:35:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.61, 1.86, 1.54 [17:36:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:36:40] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:36:51] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Wed 24 Aug 2022 02:59:35 GMT +0000. [17:37:31] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:37:31] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:38:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.62, 3.53 [17:38:10] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:38:38] RECOVERY - patternarchive.online - LetsEncrypt on sslhost is OK: OK - Certificate 'patternarchive.online' will expire on Sat 17 Sep 2022 18:49:33 GMT +0000. [17:38:41] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [17:38:43] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:38:45] RECOVERY - wiki.seamly.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seamly.net' will expire on Tue 23 Aug 2022 07:40:05 GMT +0000. [17:38:49] [02puppet] 07Universal-Omega opened pull request 03#2755: Convert memcached_servers to single nutcracker config in hiera - 13https://github.com/miraheze/puppet/pull/2755 [17:38:49] [url] Page not found · GitHub · GitHub | github.com [17:38:57] RECOVERY - lab.wikimimo.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lab.wikimimo.com' will expire on Sun 09 Oct 2022 03:42:01 GMT +0000. [17:39:01] PROBLEM - isv.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:39:02] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=amazingideaswiki --no-updates --username-prefix=wikia:amazingideas /mnt/mediawiki-static/metawiki/ImportDump/amazingideaswiki-20220803122217.xml (END - exit=0) [17:39:03] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=amazingideaswiki (START) [17:39:29] RECOVERY - ngsswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'ngsswiki.com' will expire on Sun 02 Oct 2022 05:32:39 GMT +0000. [17:39:33] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.274 second response time [17:39:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:39:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.69, 1.41, 1.45 [17:39:41] [02puppet] 07Universal-Omega opened pull request 03#2756: nutcracker: fix port in ferm rules - 13https://github.com/miraheze/puppet/pull/2756 [17:39:42] [url] Page not found · GitHub · GitHub | github.com [17:39:42] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.434 second response time [17:39:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:40:00] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:40:08] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:40:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.08, 4.21, 3.76 [17:40:40] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 1.051 second response time [17:40:52] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.16, 2.47 [17:42:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/32a629bdf111...155a925d7503 [17:42:04] [url] Comparing 32a629bdf111...155a925d7503 · miraheze/puppet · GitHub | github.com [17:42:05] [02miraheze/puppet] 07Universal-Omega 03155a925 - nutcracker: fix port in ferm rules (#2756) [17:42:07] [02puppet] 07paladox closed pull request 03#2756: nutcracker: fix port in ferm rules - 13https://github.com/miraheze/puppet/pull/2756 [17:42:07] ... [17:42:10] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39177 bytes in 0.052 second response time [17:42:26] PROBLEM - wiki.auraxis.co - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:42:36] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:42:46] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [17:42:53] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=amazingideaswiki (END - exit=0) [17:42:54] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=amazingideaswiki --active --update (END - exit=0) [17:42:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.57, 2.52, 2.31 [17:43:04] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:43:05] RECOVERY - wiki.otir.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.otir.nl' will expire on Fri 26 Aug 2022 04:20:05 GMT +0000. [17:43:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.64, 2.01, 1.71 [17:43:39] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:54] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:54] [02puppet] 07Universal-Omega synchronize pull request 03#2755: Convert memcached_servers to single nutcracker config in hiera - 13https://github.com/miraheze/puppet/pull/2755 [17:43:55] [url] Page not found · GitHub · GitHub | github.com [17:44:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:44:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.72, 3.70 [17:44:54] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.56, 1.80, 1.67 [17:45:38] PROBLEM - wiki.wilderyogi.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:38] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:45:50] PROBLEM - wiki.worldbuilding.network - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:46:48] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.043 second response time [17:46:53] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.043 second response time [17:46:54] RECOVERY - www.scum-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'scum-wiki.com' will expire on Sat 22 Oct 2022 05:35:09 GMT +0000. [17:47:03] PROBLEM - hoseki.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:06] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.071 second response time [17:47:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.58, 1.64, 1.62 [17:48:07] RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Tue 23 Aug 2022 08:39:11 GMT +0000. [17:48:45] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:49:00] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/7eeb4f17934d [17:49:33] RECOVERY - dcmultiverse.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:49:33] [02miraheze/puppet] 07paladox 037eeb4f1 - Install puppet141 [17:49:35] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [17:49:35] [url] Page not found · GitHub · GitHub | github.com [17:49:36] [02puppet] 07paladox opened pull request 03#2757: Install puppet141 - 13https://github.com/miraheze/puppet/pull/2757 [17:49:37] ... [17:49:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.71, 8.83, 11.50 [17:50:00] RECOVERY - wikimoma.art - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimoma.art' will expire on Sat 17 Sep 2022 19:04:53 GMT +0000. [17:50:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/31463d31172e [17:50:04] [02miraheze/puppet] 07paladox 0331463d3 - Update site.pp [17:50:06] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [17:50:06] ... [17:50:07] [02puppet] 07paladox opened pull request 03#2758: Update site.pp - 13https://github.com/miraheze/puppet/pull/2758 [17:50:08] [url] Page not found · GitHub · GitHub | github.com [17:50:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.33, 4.28, 3.90 [17:50:15] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [17:50:40] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.242 second response time [17:51:05] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.046 second response time [17:51:08] PROBLEM - sarovia.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:41] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Sun 18 Sep 2022 20:39:54 GMT +0000. [17:52:10] RECOVERY - nottingham.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nottingham.wiki' will expire on Mon 26 Sep 2022 06:16:48 GMT +0000. [17:53:38] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=osconceptswiki --no-updates --username-prefix=wikia:osm /mnt/mediawiki-static/metawiki/ImportDump/osconceptswiki-20220803071517.xml (END - exit=0) [17:53:39] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=osconceptswiki (START) [17:54:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:46] RECOVERY - squareenix.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:54:52] RECOVERY - datacrondatabase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'datacrondatabase.com' will expire on Mon 22 Aug 2022 15:10:26 GMT +0000. [17:54:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.59, 10.46, 11.20 [17:55:04] PROBLEM - istpcomputing.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:16] RECOVERY - commons.ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.ballmedia.org' will expire on Mon 10 Oct 2022 06:16:04 GMT +0000. [17:55:19] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.94, 1.43, 1.07 [17:55:29] RECOVERY - perpustaxaan.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'perpustaxaan.wiki' will expire on Tue 11 Oct 2022 04:34:44 GMT +0000. [17:55:36] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:56:14] RECOVERY - studyabroad.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'studyabroad.wiki' will expire on Tue 01 Nov 2022 16:02:45 GMT +0000. [17:56:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [17:56:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.95, 10.06, 10.99 [17:57:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.63, 3.93, 3.96 [17:57:21] PROBLEM - www.cgradegames.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:48] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.93, 1.28, 1.07 [17:57:50] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.326 second response time [17:57:51] PROBLEM - thestories.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:58:48] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 22.61, 13.98, 12.25 [17:59:29] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:30] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:54] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:16] RECOVERY - phgalaxy.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'phgalaxy.eu.org' will expire on Sun 02 Oct 2022 05:33:18 GMT +0000. [18:01:25] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.451 second response time [18:01:27] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.311 second response time [18:02:17] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Wed 24 Aug 2022 02:12:36 GMT +0000. [18:02:36] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[pgreload],Service[puppetdb] [18:04:12] RECOVERY - www.arru.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'www.arru.xyz' will expire on Sat 17 Sep 2022 21:47:49 GMT +0000. [18:04:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.13, 10.39, 11.41 [18:05:05] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=osconceptswiki (END - exit=0) [18:05:06] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=osconceptswiki --active --update (END - exit=0) [18:05:14] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:05:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:06:08] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 1.028 second response time [18:06:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.16, 2.34, 1.90 [18:07:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.78, 3.78 [18:07:13] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.019 second response time [18:07:38] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:28] RECOVERY - isv.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [18:09:41] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:09:51] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.056 second response time [18:10:01] RECOVERY - sbs.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'sbs.wiki' will expire on Thu 01 Sep 2022 00:42:07 GMT +0000. [18:10:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.20, 11.61, 11.41 [18:11:36] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [18:11:49] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:11:54] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:13] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:40] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:45] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [18:13:54] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.797 second response time [18:14:04] RECOVERY - wiki.veloren.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.veloren.net' will expire on Sat 15 Oct 2022 15:52:05 GMT +0000. [18:14:15] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:14:27] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.95, 1.72, 1.91 [18:15:05] PROBLEM - wiki.hsins.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.87, 3.96 [18:15:10] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:15:11] RECOVERY - wiki.wilderyogi.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wilderyogi.eu' will expire on Mon 26 Sep 2022 06:35:10 GMT +0000. [18:15:35] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.277 second response time [18:16:08] RECOVERY - hoseki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hoseki.wiki' will expire on Sun 30 Oct 2022 08:17:32 GMT +0000. [18:16:22] PROBLEM - wiki.yumeka.icu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.38, 11.33, 11.66 [18:17:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.79, 4.77, 4.26 [18:17:29] RECOVERY - mystiverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mystiverse.wiki' will expire on Sat 01 Oct 2022 04:26:41 GMT +0000. [18:17:38] PROBLEM - wiki.ameristraliagov.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:17:45] PROBLEM - dc.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:18:18] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.022 second response time [18:18:23] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:18:27] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:18:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.30, 1.33, 1.68 [18:18:52] RECOVERY - vrcdev.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vrcdev.wiki' will expire on Tue 20 Sep 2022 12:25:22 GMT +0000. [18:20:00] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:20:08] !log stopped puppetserver/puppetdb on puppet111 for copy of data to puppet141 [18:20:16] RECOVERY - sarovia.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sarovia.graalmilitary.com' will expire on Tue 25 Oct 2022 07:05:18 GMT +0000. [18:20:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.18, 13.18, 12.17 [18:20:29] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.023 second response time [18:20:37] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:21] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/4b13604597df [19:04:46] [02miraheze/mw-config] 07Universal-Omega 034b13604 - Split comments to seperate lines in LocalSettings.php [19:04:48] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [19:04:48] [url] Page not found · GitHub · GitHub | github.com [19:04:49] [02mw-config] 07Universal-Omega opened pull request 03#4840: Split comments to seperate lines in LocalSettings.php - 13https://github.com/miraheze/mw-config/pull/4840 [19:04:50] [url] Page not found · GitHub · GitHub | github.com [19:05:57] miraheze/mw-config - Universal-Omega the build passed. [19:10:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8da2c0c60e6f...d6aeb054eda3 [19:10:14] [url] Comparing 8da2c0c60e6f...d6aeb054eda3 · miraheze/mw-config · GitHub | github.com [19:10:15] [02miraheze/mw-config] 07Universal-Omega 03d6aeb05 - Use `::class` [19:11:30] miraheze/mw-config - Universal-Omega the build passed. [19:13:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d6aeb054eda3...c3530a1b9e2b [19:13:53] [url] Comparing d6aeb054eda3...c3530a1b9e2b · miraheze/mw-config · GitHub | github.com [19:13:54] [02miraheze/mw-config] 07Universal-Omega 03c3530a1 - Add comment about DBO_SSL being deprecated in 1.39 [19:15:02] miraheze/mw-config - Universal-Omega the build passed. [19:17:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c3530a1b9e2b...844650167bab [19:17:03] [url] Comparing c3530a1b9e2b...844650167bab · miraheze/mw-config · GitHub | github.com [19:17:04] [02miraheze/mw-config] 07Universal-Omega 038446501 - GlobalCache: split comments to new lines [19:18:09] miraheze/mw-config - Universal-Omega the build passed. [19:21:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/039164266d9d [19:21:48] [02miraheze/mw-config] 07Universal-Omega 030391642 - Database: split comments to new lines [19:21:50] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [19:21:50] [url] Page not found · GitHub · GitHub | github.com [19:21:51] [02mw-config] 07Universal-Omega opened pull request 03#4841: Database: split comments to new lines - 13https://github.com/miraheze/mw-config/pull/4841 [19:21:52] [url] Page not found · GitHub · GitHub | github.com [19:23:04] miraheze/mw-config - Universal-Omega the build passed. [19:24:23] [02mw-config] 07Universal-Omega closed pull request 03#4841: Database: split comments to new lines - 13https://github.com/miraheze/mw-config/pull/4841 [19:24:23] [url] Page not found · GitHub · GitHub | github.com [19:24:24] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [19:24:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/844650167bab...788eabde2a9e [19:24:26] [url] Comparing 844650167bab...788eabde2a9e · miraheze/mw-config · GitHub | github.com [19:24:27] [02miraheze/mw-config] 07Universal-Omega 03788eabd - Database: split comments to new lines (#4841) [19:24:29] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [19:24:29] [url] Page not found · GitHub · GitHub | github.com [19:25:30] miraheze/mw-config - Universal-Omega the build passed. [19:26:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/788eabde2a9e...0feb45b5077a [19:26:21] [url] Comparing 788eabde2a9e...0feb45b5077a · miraheze/mw-config · GitHub | github.com [19:26:22] [02miraheze/mw-config] 07Universal-Omega 030feb45b - Split comments to seperate lines in LocalSettings.php (#4840) [19:26:24] [02mw-config] 07Universal-Omega closed pull request 03#4840: Split comments to seperate lines in LocalSettings.php - 13https://github.com/miraheze/mw-config/pull/4840 [19:26:24] [url] Page not found · GitHub · GitHub | github.com [19:26:25] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:26:27] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [19:26:27] ... [19:27:26] miraheze/mw-config - Universal-Omega the build passed. [19:30:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0feb45b5077a...f5270cf105b5 [19:30:14] [url] Comparing 0feb45b5077a...f5270cf105b5 · miraheze/mw-config · GitHub | github.com [19:30:15] [02miraheze/mw-config] 07Universal-Omega 03f5270cf - Split comments to new lines [19:31:21] miraheze/mw-config - Universal-Omega the build passed. [19:32:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f5270cf105b5...7cdf6d03f1aa [19:32:09] [url] Comparing f5270cf105b5...7cdf6d03f1aa · miraheze/mw-config · GitHub | github.com [19:32:10] [02miraheze/mw-config] 07Universal-Omega 037cdf6d0 - Split comment to new line [19:33:11] miraheze/mw-config - Universal-Omega the build passed. [19:33:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7cdf6d03f1aa...8bb57ffe967b [19:33:17] [url] Comparing 7cdf6d03f1aa...8bb57ffe967b · miraheze/mw-config · GitHub | github.com [19:33:18] [02miraheze/mw-config] 07Universal-Omega 038bb57ff - Split comment to new line [19:34:32] miraheze/mw-config - Universal-Omega the build passed. [19:36:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8bb57ffe967b...21cd7ddf3d01 [19:36:16] [url] Comparing 8bb57ffe967b...21cd7ddf3d01 · miraheze/mw-config · GitHub | github.com [19:36:17] [02miraheze/mw-config] 07Universal-Omega 0321cd7dd - Split comments to new lines [19:37:24] miraheze/mw-config - Universal-Omega the build passed. [19:38:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/21cd7ddf3d01...5eadef96a72d [19:38:45] [url] Comparing 21cd7ddf3d01...5eadef96a72d · miraheze/mw-config · GitHub | github.com [19:38:46] [02miraheze/mw-config] 07Universal-Omega 035eadef9 - Split comments to new lines [19:39:58] miraheze/mw-config - Universal-Omega the build passed. [19:40:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a316bfad3f3e [19:40:58] [02miraheze/mw-config] 07Universal-Omega 03a316bfa - phpcs: enable NewLineComment rule [19:41:00] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [19:41:00] [url] Page not found · GitHub · GitHub | github.com [19:41:01] [02mw-config] 07Universal-Omega opened pull request 03#4842: phpcs: enable NewLineComment rule - 13https://github.com/miraheze/mw-config/pull/4842 [19:41:02] [url] Page not found · GitHub · GitHub | github.com [19:42:07] miraheze/mw-config - Universal-Omega the build passed. [19:42:36] [02mw-config] 07Universal-Omega closed pull request 03#4842: phpcs: enable NewLineComment rule - 13https://github.com/miraheze/mw-config/pull/4842 [19:42:36] [url] Page not found · GitHub · GitHub | github.com [19:42:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5eadef96a72d...151b8363301b [19:42:38] [url] Comparing 5eadef96a72d...151b8363301b · miraheze/mw-config · GitHub | github.com [19:42:39] [02miraheze/mw-config] 07Universal-Omega 03151b836 - phpcs: enable NewLineComment rule (#4842) [19:42:40] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:42:42] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [19:42:42] [url] Page not found · GitHub · GitHub | github.com [19:43:52] miraheze/mw-config - Universal-Omega the build passed. [20:17:07] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/f7ae49330ec9 [20:17:08] [02miraheze/CreateWiki] 07Universal-Omega 03f7ae493 - Actually do update on changes being commited [20:17:10] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/CreateWiki [20:17:10] [url] Page not found · GitHub · GitHub | github.com [20:17:11] [02CreateWiki] 07Universal-Omega opened pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [20:17:12] ... [20:18:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7eeb4f17934d...7dd07e0c1c47 [20:18:47] [url] Comparing 7eeb4f17934d...7dd07e0c1c47 · miraheze/puppet · GitHub | github.com [20:18:48] [02miraheze/puppet] 07paladox 037dd07e0 - Update site.pp [20:18:50] [02puppet] 07paladox synchronize pull request 03#2757: Install puppet141 - 13https://github.com/miraheze/puppet/pull/2757 [20:18:50] ... [20:18:52] [02puppet] 07paladox closed pull request 03#2758: Update site.pp - 13https://github.com/miraheze/puppet/pull/2758 [20:18:52] [url] Page not found · GitHub · GitHub | github.com [20:18:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [20:18:56] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [20:18:56] [url] Page not found · GitHub · GitHub | github.com [20:19:00] [02puppet] 07paladox closed pull request 03#2757: Install puppet141 - 13https://github.com/miraheze/puppet/pull/2757 [20:19:00] [url] Page not found · GitHub · GitHub | github.com [20:19:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/155a925d7503...99a91899766f [20:19:02] [url] Comparing 155a925d7503...99a91899766f · miraheze/puppet · GitHub | github.com [20:19:03] [02miraheze/puppet] 07paladox 0399a9189 - Install puppet141 (#2757) [20:19:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [20:19:06] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [20:19:06] [url] Page not found · GitHub · GitHub | github.com [20:20:28] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/f7ae49330ec9...29a27dc8a28b [20:20:28] [url] Comparing f7ae49330ec9...29a27dc8a28b · miraheze/CreateWiki · GitHub | github.com [20:20:29] [02miraheze/CreateWiki] 07Universal-Omega 0329a27dc - Update WikiManager.php [20:20:31] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [20:20:31] [url] Page not found · GitHub · GitHub | github.com [20:21:27] miraheze/CreateWiki - Universal-Omega the build has errored. [20:24:13] miraheze/CreateWiki - Universal-Omega the build has errored. [20:27:28] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/29a27dc8a28b...d9bfb23354d5 [20:27:28] [url] Comparing 29a27dc8a28b...d9bfb23354d5 · miraheze/CreateWiki · GitHub | github.com [20:27:29] [02miraheze/CreateWiki] 07Universal-Omega 03d9bfb23 - Update WikiRequest.php [20:27:31] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [20:27:31] ... [20:27:53] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d9bfb23354d5...13b6f6e124c1 [20:27:53] [url] Comparing d9bfb23354d5...13b6f6e124c1 · miraheze/CreateWiki · GitHub | github.com [20:27:54] [02miraheze/CreateWiki] 07Universal-Omega 0313b6f6e - Update RequestWikiRequestViewer.php [20:27:56] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [20:27:56] [url] Page not found · GitHub · GitHub | github.com [20:28:21] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/13b6f6e124c1...a9d7909602c3 [20:28:21] [url] Comparing 13b6f6e124c1...a9d7909602c3 · miraheze/CreateWiki · GitHub | github.com [20:28:22] [02miraheze/CreateWiki] 07Universal-Omega 03a9d7909 - Update deleteWikis.php [20:28:24] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [20:28:24] [url] Page not found · GitHub · GitHub | github.com [20:38:14] miraheze/CreateWiki - Universal-Omega the build passed. [21:15:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/2253dd14b8cc...ebbd0281dd0a [21:15:17] [url] Comparing 2253dd14b8cc...ebbd0281dd0a · miraheze/puppet · GitHub | github.com [21:15:18] [02miraheze/puppet] 07paladox 03ebbd028 - Delete mwtask111.yaml [21:16:50] !log [@mwtask141] starting deploy of {'config': True} to all [21:16:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:17:43] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 52s [21:18:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:49] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [21:18:49] [url] Page not found · GitHub · GitHub | github.com [21:18:50] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/aa071fa99076...e5bfb377977c [21:18:51] [url] Comparing aa071fa99076...e5bfb377977c · miraheze/CreateWiki · GitHub | github.com [21:18:52] [02miraheze/CreateWiki] 07Universal-Omega 03e5bfb37 - Update RequestWikiRequestViewer.php [21:18:56] !log [@test131] starting deploy of {'config': True} to all [21:18:57] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [21:19:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:19:17] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e5bfb377977c...3ad25a2f2c15 [21:19:17] [url] Comparing e5bfb377977c...3ad25a2f2c15 · miraheze/CreateWiki · GitHub | github.com [21:19:18] [02miraheze/CreateWiki] 07Universal-Omega 033ad25a2 - Update WikiRequest.php [21:19:20] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [21:19:20] [url] Page not found · GitHub · GitHub | github.com [21:19:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:20:02] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/3ad25a2f2c15...ad4429ab0c3f [21:20:03] [url] Comparing 3ad25a2f2c15...ad4429ab0c3f · miraheze/CreateWiki · GitHub | github.com [21:20:04] [02miraheze/CreateWiki] 07Universal-Omega 03ad4429a - Update deleteWikis.php [21:20:05] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [21:20:06] [url] Page not found · GitHub · GitHub | github.com [21:22:10] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.20, 12.19, 11.68 [21:22:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 3.61, 3.32 [21:24:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.96, 3.36, 3.27 [21:24:20] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:24] miraheze/CreateWiki - Universal-Omega the build has errored. [21:24:41] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 2 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[/etc/rsyslog.d],File[/etc/rsyslog.conf] [21:24:48] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:19] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:30] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 423 failures. Last run 2 minutes ago with 423 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/central.ballmedia.org.key],File[/etc/ssl/localcerts/wikileague.cf.crt],File[/etc/ssl/private/wikileague.cf.key],File[/etc/ssl/localcerts/www.connorjwat.net.crt] [21:25:37] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:42] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:25:45] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:25:54] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:26:22] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:27:05] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [21:27:05] [url] Page not found · GitHub · GitHub | github.com [21:27:06] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ad4429ab0c3f...914c07850afe [21:27:07] [url] Comparing ad4429ab0c3f...914c07850afe · miraheze/CreateWiki · GitHub | github.com [21:27:08] [02miraheze/CreateWiki] 07Universal-Omega 03914c078 - Update WikiManager.php [21:27:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.65, 1.90, 1.47 [21:27:50] RECOVERY - puppet141 Puppet on puppet141 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [21:29:16] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.25, 2.05, 1.57 [21:30:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.63, 3.42 [21:31:12] miraheze/CreateWiki - Universal-Omega the build has errored. [21:31:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.52, 1.79, 1.53 [21:33:16] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.00, 1.69, 1.54 [21:35:18] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:35:26] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:28] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [21:35:34] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw122 mw141 [21:35:40] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [21:35:40] [url] Page not found · GitHub · GitHub | github.com [21:35:41] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/914c07850afe...6433fb6ce6c0 [21:35:42] [url] Comparing 914c07850afe...6433fb6ce6c0 · miraheze/CreateWiki · GitHub | github.com [21:35:43] [02miraheze/CreateWiki] 07Universal-Omega 036433fb6 - Update WikiManager.php [21:36:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 0.90, 2.52, 3.07 [21:36:21] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [21:36:22] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [21:36:25] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [21:36:25] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:36:26] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [21:36:26] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:36:28] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:36:33] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:36:41] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:36:53] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:36:58] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:37:33] PROBLEM - cp30 Stunnel HTTP for mail121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [21:37:59] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:38:02] PROBLEM - cp30 Varnish Backends on cp30 is WARNING: No backends detected. If this is an error, see readme.txt [21:38:09] PROBLEM - cp31 Stunnel HTTP for mail121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:38:14] PROBLEM - cp31 Stunnel HTTP for matomo131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:38:15] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [21:38:16] PROBLEM - cp31 Stunnel HTTP for mwtask141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:38:22] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [21:38:29] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:38:30] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.311 second response time [21:38:37] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.528 second response time [21:38:39] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 3.503 second response time [21:38:41] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.047 second response time [21:38:48] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [21:38:51] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.022 second response time [21:38:57] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:38:57] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.266 second response time [21:39:03] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [21:39:03] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.332 second response time [21:39:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 0.30, 2.01, 3.70 [21:39:30] RECOVERY - cp30 Stunnel HTTP for mail121 on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.246 second response time [21:39:31] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.030 second response time [21:39:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 4.37, 8.61, 11.15 [21:39:39] miraheze/CreateWiki - Universal-Omega the build has errored. [21:39:53] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.628 second response time [21:39:54] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.405 second response time [21:39:56] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw122 mw131 mw142 [21:40:00] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.54, 2.43, 1.73 [21:40:02] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [21:40:05] RECOVERY - cp31 Stunnel HTTP for mail121 on cp31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.242 second response time [21:40:09] RECOVERY - cp31 Stunnel HTTP for matomo131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 0.726 second response time [21:40:09] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is WARNING: WARNING - NGINX Error Rate is 57% [21:40:11] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [21:40:12] PROBLEM - cp30 HTTPS on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:40:15] RECOVERY - cp31 Stunnel HTTP for mwtask141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.315 second response time [21:40:24] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.009 second response time [21:40:28] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [21:41:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.40, 1.44, 3.28 [21:41:27] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:41:30] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 2.58, 6.54, 10.09 [21:41:33] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [21:41:34] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:41:42] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:41:53] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:41:54] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 98% [21:42:20] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.307 second response time [21:42:23] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 6% [21:42:25] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.709 second response time [21:42:26] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [21:42:27] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [21:42:32] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.738 second response time [21:42:51] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.350 second response time [21:43:03] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.408 second response time [21:43:22] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.374 second response time [21:43:28] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.006 second response time [21:43:40] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [21:43:53] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.47, 1.59, 1.56 [21:43:59] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 12% [21:44:09] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [21:44:10] [url] Page not found · GitHub · GitHub | github.com [21:44:11] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/6433fb6ce6c0...8eefece13ae0 [21:44:11] [url] Comparing 6433fb6ce6c0...8eefece13ae0 · miraheze/CreateWiki · GitHub | github.com [21:44:12] [02miraheze/CreateWiki] 07Universal-Omega 038eefece - Update WikiManager.php [21:44:14] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:31] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:58] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [21:45:09] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [21:45:14] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:20] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:29] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 10.03, 5.85, 3.33 [21:45:36] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [21:45:44] PROBLEM - gluster101 PowerDNS Recursor on gluster101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:45:52] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [21:46:00] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:08] Ah no wonder why server down [21:46:17] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:29] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.896 second response time [21:46:43] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [21:46:45] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:47] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.237 second response time [21:46:58] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:00] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.233 second response time [21:47:06] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:07] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:11] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.308 second response time [21:47:15] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.331 second response time [21:47:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.29, 4.67, 11.16 [21:47:41] RECOVERY - gluster101 PowerDNS Recursor on gluster101 is OK: DNS OK: 0.141 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [21:47:47] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [21:48:00] miraheze/CreateWiki - Universal-Omega the build has errored. [21:48:14] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.418 second response time [21:48:39] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.029 second response time [21:48:40] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.541 second response time [21:48:46] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.318 second response time [21:48:55] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.208 second response time [21:49:00] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.318 second response time [21:49:01] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [21:49:04] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.017 second response time [21:49:06] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.030 second response time [21:49:07] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.317 second response time [21:49:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 3.76, 4.02, 10.11 [21:49:37] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [21:49:44] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.320 second response time [21:49:59] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.383 second response time [21:50:34] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [21:50:43] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:44] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:50:58] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [21:51:50] I love clg wiki [21:52:10] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.673 second response time [21:52:26] That's not for me [21:52:30] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.023 second response time [21:52:31] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.026 second response time [21:52:34] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [21:52:40] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.314 second response time [21:52:42] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.321 second response time [21:52:45] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/8eefece13ae0...0ddcefc5db7f [21:52:46] [url] Comparing 8eefece13ae0...0ddcefc5db7f · miraheze/CreateWiki · GitHub | github.com [21:52:47] No!!!!!!!!!!! [21:52:47] [02miraheze/CreateWiki] 07Universal-Omega 030ddcefc - Update CreateWikiJson.php [21:52:48] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [21:52:49] [url] Page not found · GitHub · GitHub | github.com [21:52:58] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:53:18] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:53:22] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:53:22] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.37, 4.93, 4.06 [21:53:36] https://devantart.com [21:53:43] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 3% [21:53:46] RECOVERY - cp30 HTTPS on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3387 bytes in 0.396 second response time [21:54:00] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [21:54:08] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:54:10] 👎!!!-!!!!!!!!!!!!!!!!!!! [21:54:58] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.323 second response time [21:54:59] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.335 second response time [21:54:59] It's NOOOOOO!!!!!!!!!!!!!! [21:55:18] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.311 second response time [21:55:20] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [21:55:34] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [21:55:37] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 87% [21:56:00] STOPPPPPPPIT!!!!!!!!!!!!!!!!!!!!! [21:56:18] miraheze/CreateWiki - Universal-Omega the build has errored. [21:56:23] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:56:32] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [21:57:13] GGGGGGGGGGGRRRRRRRRRRRRRRRR!!!!!!!!!!!!!!!!!!!!!!!!!!  STOP!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [21:57:37] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 4% [21:57:47] PROBLEM - cp30 Stunnel HTTP for reports121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [21:57:54] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:24] I'm not saying it again I will beat you! [21:58:51] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:51] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:52] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:54] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:56] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.21, 2.60 [21:59:10] .kb Guest32 spam [21:59:10] Please wait... [21:59:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.74, 9.90, 9.54 [21:59:42] RECOVERY - cp30 Stunnel HTTP for reports121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.280 second response time [21:59:52] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.291 second response time [22:00:46] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.324 second response time [22:00:47] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.017 second response time [22:01:03] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/0ddcefc5db7f...a38dc2b3041b [22:01:04] [url] Comparing 0ddcefc5db7f...a38dc2b3041b · miraheze/CreateWiki · GitHub | github.com [22:01:05] [02miraheze/CreateWiki] 07Universal-Omega 03a38dc2b - Update WikiManager.php [22:01:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 10.26, 5.70, 3.57 [22:01:06] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [22:01:07] [url] Page not found · GitHub · GitHub | github.com [22:01:08] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/a38dc2b3041b...ff53dc064b32 [22:01:08] [url] Comparing a38dc2b3041b...ff53dc064b32 · miraheze/CreateWiki · GitHub | github.com [22:01:09] [02miraheze/CreateWiki] 07Universal-Omega 03ff53dc0 - Update CreateWikiJson.php [22:01:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.09, 10.22, 9.74 [22:02:08] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [22:02:13] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [22:02:15] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:16] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:16] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:17] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [22:02:18] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [22:02:22] PROBLEM - cp30 Stunnel HTTP for phab121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:02:26] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [22:02:32] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 86% [22:02:33] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:36] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [22:03:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 3.86, 7.83, 8.92 [22:03:42] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:44] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:44] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:47] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:51] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:58] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:03] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:04] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 57% [22:04:06] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [22:04:06] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:11] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:15] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:17] RECOVERY - cp30 Stunnel HTTP for phab121 on cp30 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.292 second response time [22:04:17] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:30] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.214 second response time [22:04:33] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 3.996 second response time [22:04:36] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.328 second response time [22:04:56] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [22:05:01] miraheze/CreateWiki - Universal-Omega the build has errored. [22:05:08] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.017 second response time [22:05:13] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.325 second response time [22:05:34] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 4 backends are down. mw121 mw122 mw132 mw141 [22:05:39] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.329 second response time [22:06:08] PROBLEM - gluster101 Disk Space on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:06:12] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.033 second response time [22:06:14] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.336 second response time [22:06:16] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.313 second response time [22:06:18] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.329 second response time [22:06:21] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw122 mw141 [22:06:21] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.336 second response time [22:06:27] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [22:07:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.34, 2.77, 3.07 [22:07:47] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.324 second response time [22:07:53] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.334 second response time [22:08:02] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.176 second response time [22:08:04] PROBLEM - gluster101 Disk Space on gluster101 is WARNING: DISK WARNING - free space: / 104528 MB (10% inode=82%); [22:08:06] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.318 second response time [22:08:07] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [22:08:09] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.306 second response time [22:08:10] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.467 second response time [22:08:11] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 5.096 second response time [22:08:21] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 27% [22:08:22] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.412 second response time [22:08:25] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.018 second response time [22:08:26] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 29% [22:08:29] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.307 second response time [22:09:09] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:09:52] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.882 second response time [22:09:59] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 7% [22:10:00] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 9.017 second response time [22:10:20] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:10:20] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [22:10:29] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:51] PROBLEM - cp30 Stunnel HTTP for mail121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:11:16] PROBLEM - cp30 Stunnel HTTP for matomo131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:11:17] PROBLEM - cp30 Stunnel HTTP for reports121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:11:20] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [22:11:46] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:11:48] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:12:11] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 93% [22:12:16] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.174 second response time [22:12:20] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.315 second response time [22:12:22] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.017 second response time [22:12:22] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [22:12:24] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.344 second response time [22:12:26] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:12:44] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:12:48] RECOVERY - cp30 Stunnel HTTP for mail121 on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.246 second response time [22:13:05] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [22:13:12] RECOVERY - cp30 Stunnel HTTP for matomo131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 0.680 second response time [22:13:12] RECOVERY - cp30 Stunnel HTTP for reports121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.279 second response time [22:13:53] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.368 second response time [22:14:43] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:16] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.753 second response time [22:15:53] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ff53dc064b32...9101818db8fb [22:15:54] [url] Comparing ff53dc064b32...9101818db8fb · miraheze/CreateWiki · GitHub | github.com [22:15:55] [02miraheze/CreateWiki] 07Universal-Omega 039101818 - Update WikiManager.php [22:15:56] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [22:15:56] [url] Page not found · GitHub · GitHub | github.com [22:16:01] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 18% [22:16:03] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:07] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [22:16:08] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:12] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:17:02] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.75, 6.63, 4.71 [22:18:02] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 7% [22:18:05] PROBLEM - cp30 Stunnel HTTP for mon141 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:18:06] PROBLEM - cp30 Stunnel HTTP for test131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:18:10] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:18:11] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:18:12] PROBLEM - cp30 Stunnel HTTP for phab121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:18:13] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:23] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:28] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:36] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:40] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:50] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.343 second response time [22:18:58] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 9.132 second response time [22:19:20] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [22:19:31] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:19:34] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.015 second response time [22:19:34] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [22:19:51] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is WARNING: WARNING - NGINX Error Rate is 58% [22:20:03] RECOVERY - cp30 Stunnel HTTP for test131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.340 second response time [22:20:03] RECOVERY - cp30 Stunnel HTTP for mon141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.352 second response time [22:20:04] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.346 second response time [22:20:07] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.311 second response time [22:20:07] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [22:20:09] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.316 second response time [22:20:10] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.339 second response time [22:20:13] RECOVERY - cp30 Stunnel HTTP for phab121 on cp30 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.317 second response time [22:20:13] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.026 second response time [22:20:18] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.011 second response time [22:20:21] !log shutdown puppet111 [22:20:22] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.019 second response time [22:20:32] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.310 second response time [22:20:39] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.332 second response time [22:20:40] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [22:20:59] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.44, 5.03, 4.54 [22:21:18] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [22:21:28] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.171 second response time [22:21:32] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [22:21:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:21:42] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.315 second response time [22:21:46] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 10% [22:21:56] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:08] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 23.82, 11.20, 6.73 [22:22:36] RECOVERY - nexus.wiki - reverse DNS on sslhost is OK: SSL OK - nexus.wiki reverse DNS resolves to cp20.miraheze.org - CNAME FLAT [22:23:43] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [22:24:03] miraheze/CreateWiki - Universal-Omega the build passed. [22:24:04] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.028 second response time [22:24:16] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [22:24:22] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 2% [22:24:48] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [22:25:16] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.02, 1.54, 1.00 [22:25:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 22.45, 11.27, 7.61 [22:25:34] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [22:25:38] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [22:26:33] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [22:26:33] [url] Page not found · GitHub · GitHub | github.com [22:26:34] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/9101818db8fb...228b32f5654e [22:26:35] [url] Comparing 9101818db8fb...228b32f5654e · miraheze/CreateWiki · GitHub | github.com [22:26:36] [02miraheze/CreateWiki] 07Universal-Omega 03228b32f - Update WikiManager.php [22:26:43] PROBLEM - puppet111 SSH on puppet111 is CRITICAL: connect to address 2a10:6740::6:210 and port 22: No route to host [22:26:47] PROBLEM - puppet111 Disk Space on puppet111 is CRITICAL: connect to address 2a10:6740::6:210 port 5666: No route to hostconnect to host 2a10:6740::6:210 port 5666: No route to host [22:26:51] PROBLEM - puppet111 ferm_active on puppet111 is CRITICAL: connect to address 2a10:6740::6:210 port 5666: No route to hostconnect to host 2a10:6740::6:210 port 5666: No route to host [22:26:51] PROBLEM - ping6 on puppet111 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:210) [22:27:00] PROBLEM - cp30 Stunnel HTTP for puppet111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:27:00] PROBLEM - cp20 Stunnel HTTP for puppet111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:27:16] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.67, 1.18, 0.93 [22:27:22] PROBLEM - cp31 Stunnel HTTP for puppet111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:27:24] PROBLEM - puppet111 conntrack_table_size on puppet111 is CRITICAL: connect to address 2a10:6740::6:210 port 5666: No route to hostconnect to host 2a10:6740::6:210 port 5666: No route to host [22:27:30] PROBLEM - puppet111 MirahezeRenewSsl on puppet111 is CRITICAL: connect to address 2a10:6740::6:210 and port 5000: No route to host [22:27:33] PROBLEM - cp21 Stunnel HTTP for puppet111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:27:36] PROBLEM - Host puppet111 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:210) [22:27:53] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.328 second response time [22:28:12] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.315 second response time [22:28:27] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:29:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.28, 1.48, 1.07 [22:30:29] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 5.175 second response time [22:30:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.22, 12.00, 9.13 [22:30:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.70, 3.41, 2.49 [22:32:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.14, 11.81, 9.39 [22:32:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.76, 2.74 [22:34:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.25, 11.84, 9.71 [22:34:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.03, 4.63, 3.19 [22:35:32] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.30, 1.59, 1.27 [22:36:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.79, 11.94, 9.99 [22:37:27] miraheze/CreateWiki - Universal-Omega the build passed. [22:38:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.29, 11.39, 10.04 [22:39:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.42, 1.80, 1.45 [22:40:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.95, 12.09, 10.43 [22:42:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.14, 11.17, 10.29 [22:42:49] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/228b32f5654e...43e06be70582 [22:42:50] [url] Comparing 228b32f5654e...43e06be70582 · miraheze/CreateWiki · GitHub | github.com [22:42:50] [02miraheze/CreateWiki] 07Universal-Omega 0343e06be - Update Hooks.php [22:42:52] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [22:42:52] [url] Page not found · GitHub · GitHub | github.com [22:43:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.64, 1.49 [22:44:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.00, 11.55, 10.49 [22:48:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.10, 11.09, 10.55 [22:48:42] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [22:48:43] [url] Page not found · GitHub · GitHub | github.com [22:48:44] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/43e06be70582...23b529d6158b [22:48:44] [url] Comparing 43e06be70582...23b529d6158b · miraheze/CreateWiki · GitHub | github.com [22:48:45] [02miraheze/CreateWiki] 07Universal-Omega 0323b529d - Update Hooks.php [22:49:01] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [22:49:01] [url] Page not found · GitHub · GitHub | github.com [22:49:03] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/23b529d6158b...4bd5a41bd475 [22:49:03] [url] Comparing 23b529d6158b...4bd5a41bd475 · miraheze/CreateWiki · GitHub | github.com [22:49:04] [02miraheze/CreateWiki] 07Universal-Omega 034bd5a41 - Update Hooks.php [22:50:11] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.31, 1.76, 1.34 [22:50:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.35, 11.82, 10.89 [22:51:52] [02CreateWiki] 07Universal-Omega edited pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [22:51:52] [url] Page not found · GitHub · GitHub | github.com [22:52:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.97, 3.68, 2.99 [22:52:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.27, 1.60, 1.33 [22:52:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.93, 11.88, 11.03 [22:54:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.98, 3.20, 2.91 [22:54:53] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:54:55] PROBLEM - cp30 Stunnel HTTP for matomo131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:54:56] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [22:55:28] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.009 second response time [22:55:38] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:39] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:57] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:12] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:25] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [22:56:32] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:38] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:47] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:47] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [22:56:47] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [22:56:56] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:56] RECOVERY - cp30 Stunnel HTTP for matomo131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.691 second response time [22:57:14] miraheze/CreateWiki - Universal-Omega the build has errored. [22:57:26] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.029 second response time [22:57:33] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.200 second response time [22:57:38] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.306 second response time [22:57:43] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [22:57:58] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.007 second response time [22:58:08] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [22:58:16] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw121 mw122 [22:58:22] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 81% [22:58:24] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.319 second response time [22:58:27] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [22:58:29] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.279 second response time [22:58:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.01, 7.28, 9.37 [22:58:32] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.229 second response time [22:58:37] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [22:58:42] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [22:58:42] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.321 second response time [22:58:53] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.313 second response time [22:58:59] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.730 second response time [22:59:01] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.599 second response time [23:00:22] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 6% [23:00:22] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 1% [23:00:50] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [23:01:43] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [23:02:16] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [23:02:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.37, 2.48, 3.63 [23:02:36] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [23:04:19] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:47] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:56] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:01] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:22] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.120 second response time [23:06:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.86, 3.30, 3.66 [23:06:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.38, 9.57, 9.39 [23:06:46] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 2.079 second response time [23:06:51] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [23:06:56] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.312 second response time [23:08:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.26, 3.60 [23:10:04] miraheze/CreateWiki - Universal-Omega the build passed. [23:10:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.31, 4.19, 3.89 [23:10:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.39, 12.54, 10.58 [23:27:38] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [23:27:50] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.38, 3.21, 3.80 [23:28:28] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:29] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.21, 11.19, 11.47 [23:28:37] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:43] PROBLEM - cp30 Stunnel HTTP for reports121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [23:28:47] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:48] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:49] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:49] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:50] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:52] PROBLEM - cp30 Stunnel HTTP for mwtask141 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [23:28:53] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:53] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [23:28:54] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:55] PROBLEM - cp30 Stunnel HTTP for matomo131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [23:28:56] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [23:28:57] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [23:29:01] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:29:33] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.312 second response time [23:30:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.94, 3.62, 3.86 [23:30:29] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.048 second response time [23:30:31] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.779 second response time [23:30:33] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.306 second response time [23:30:43] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.008 second response time [23:30:43] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.237 second response time [23:30:43] RECOVERY - cp30 Stunnel HTTP for reports121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.261 second response time [23:30:45] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.263 second response time [23:30:45] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.008 second response time [23:30:47] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.008 second response time [23:30:49] RECOVERY - cp30 Stunnel HTTP for mwtask141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.335 second response time [23:30:49] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [23:30:50] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [23:30:51] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.224 second response time [23:30:54] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 5% [23:30:57] RECOVERY - cp30 Stunnel HTTP for matomo131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 2.150 second response time [23:31:43] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw121 mw131 [23:32:00] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.803 second response time [23:32:26] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw122 mw131 [23:33:01] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.378 second response time [23:33:07] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [23:33:43] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [23:34:25] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [23:36:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.23, 10.13, 10.67 [23:38:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.68, 9.17, 10.25 [23:40:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.48, 11.70, 11.04 [23:44:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 11.02, 10.99 [23:46:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.82, 11.96, 11.32 [23:48:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.05, 10.89, 11.02 [23:52:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.37, 11.36, 11.11 [23:52:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.87, 1.69, 1.36 [23:54:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.76, 10.33, 10.82 [23:54:40] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [23:55:03] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:19] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:25] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:22] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:30] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.24, 12.65, 11.62 [23:57:02] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:58:18] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [23:58:29] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.333 second response time [23:58:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.82, 11.97, 11.56 [23:58:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 3.26, 2.67