[00:01:41] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [00:03:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.03, 3.29, 3.41 [00:05:07] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.73, 3.16, 3.36 [00:14:20] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.61, 3.38, 2.53 [00:18:09] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.32, 3.37, 2.71 [00:22:55] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [00:23:53] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.95, 3.80, 3.11 [00:29:36] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.86, 3.27, 3.11 [00:37:31] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.36, 3.66, 3.28 [00:39:31] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.28, 3.13, 3.13 [00:41:07] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.00, 4.23, 3.60 [00:45:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.64, 3.76, 3.59 [00:53:07] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.73, 3.19, 3.38 [02:00:08] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:58:59] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [03:04:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.02, 3.77, 2.52 [03:06:02] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.38, 2.87, 2.34 [03:28:25] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:01:43] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:09:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.70, 3.03, 2.52 [04:09:17] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9b497e2e39b1...87227ba63b3f [04:09:19] [02miraheze/dns] 07paladox 0387227ba - Revert "Re-add cp32" [04:11:07] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.19, 3.09, 2.61 [04:14:11] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 100% [04:15:31] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:15:36] PROBLEM - cp32 APT on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:15:43] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:15:43] PROBLEM - cp32 Stunnel for phab121 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:15:43] PROBLEM - cp32 Stunnel for mw134 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:15:45] PROBLEM - cp32 Stunnel for reports121 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:15:46] PROBLEM - cp32 Stunnel for mail121 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:15:47] PROBLEM - cp32 Stunnel for mon141 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:15:48] PROBLEM - cp32 Stunnel for mw132 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:15:49] PROBLEM - cp32 Disk Space on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:15:49] PROBLEM - cp32 Stunnel for mw131 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:15:54] PROBLEM - cp32 ferm_active on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:15:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:00] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:02] PROBLEM - cp32 Stunnel for mw133 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:04] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:05] PROBLEM - cp32 Stunnel for mw143 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:11] PROBLEM - cp32 Stunnel for test131 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:12] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 105.68 ms [04:16:14] PROBLEM - cp32 Stunnel for matomo121 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:15] PROBLEM - cp32 Puppet on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:16] PROBLEM - cp32 Stunnel for mw141 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:19] PROBLEM - cp32 NTP time on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:21] PROBLEM - cp32 Stunnel for mwtask141 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:22] PROBLEM - cp32 conntrack_table_size on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:24] PROBLEM - cp32 Stunnel for mw142 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:16:27] PROBLEM - cp32 Stunnel for puppet141 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:29:15] RECOVERY - cp32 Stunnel for mw132 on cp32 is OK: TCP OK - 0.001 second response time on localhost port 8107 [04:29:23] PROBLEM - cp32 Varnish Backends on cp32 is WARNING: No backends detected. If this is an error, see readme.txt [04:29:24] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:29:32] RECOVERY - cp32 Stunnel for phab121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8202 [04:29:34] RECOVERY - cp32 Stunnel for mw134 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8111 [04:29:35] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 0.40, 0.42, 0.29 [04:29:37] RECOVERY - cp32 Stunnel for mon141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8201 [04:29:38] RECOVERY - cp32 Stunnel for mw131 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8106 [04:29:38] RECOVERY - cp32 Disk Space on cp32 is OK: DISK OK - free space: / 71886MiB (96% inode=98%); [04:29:39] RECOVERY - cp32 Stunnel for reports121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8205 [04:29:43] PROBLEM - cp32 ferm_active on cp32 is UNKNOWN: NRPE: Unable to read output [04:29:43] RECOVERY - cp32 Stunnel for mail121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8200 [04:29:48] RECOVERY - cp32 Stunnel for mw142 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8109 [04:29:49] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 0.043 seconds response time. miraheze.org returns 2607:f1c0:1800:26f::1,74.208.203.152 [04:29:51] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset -0.2064827979 secs [04:29:54] RECOVERY - cp32 Stunnel for mw133 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8110 [04:30:01] RECOVERY - cp32 Stunnel for mwtask141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8150 [04:30:04] RECOVERY - cp32 Stunnel for mw143 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8112 [04:30:06] RECOVERY - cp32 Stunnel for puppet141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8204 [04:30:07] RECOVERY - cp32 Stunnel for matomo121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8203 [04:30:08] RECOVERY - cp32 Stunnel for test131 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8180 [04:30:10] PROBLEM - cp32 Puppet on cp32 is UNKNOWN: NRPE: Unable to read output [04:30:12] PROBLEM - cp32 conntrack_table_size on cp32 is UNKNOWN: NRPE: Unable to read output [04:30:13] RECOVERY - cp32 Stunnel for mw141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8108 [04:31:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/6d30ac2085c9 [04:31:18] [02miraheze/puppet] 07paladox 036d30ac2 - base::puppet: force the puppet repo to use bullseye [04:31:20] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [04:31:23] [02puppet] 07paladox opened pull request 03#3362: base::puppet: force the puppet repo to use bullseye - 13https://github.com/miraheze/puppet/pull/3362 [04:31:24] [02puppet] 07paladox closed pull request 03#3362: base::puppet: force the puppet repo to use bullseye - 13https://github.com/miraheze/puppet/pull/3362 [04:31:25] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:31:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1db9cf98d7a3...f5647b5a5419 [04:31:29] [02miraheze/puppet] 07paladox 03f5647b5 - base::puppet: force the puppet repo to use bullseye (#3362) [04:31:30] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [04:31:32] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [04:33:32] RECOVERY - cp32 ferm_active on cp32 is OK: OK ferm input default policy is set [04:34:09] RECOVERY - cp32 conntrack_table_size on cp32 is OK: OK: nf_conntrack is 0 % full [04:35:10] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 15 backends are healthy [04:37:47] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3713 bytes in 0.538 second response time [04:37:57] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 3% [04:41:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/9c089954b758 [04:41:44] [02miraheze/puppet] 07paladox 039c08995 - check_puppetrun: Drop safe_yaml [04:41:47] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [04:41:50] [02puppet] 07paladox opened pull request 03#3363: check_puppetrun: Drop safe_yaml - 13https://github.com/miraheze/puppet/pull/3363 [04:44:12] [02puppet] 07paladox closed pull request 03#3363: check_puppetrun: Drop safe_yaml - 13https://github.com/miraheze/puppet/pull/3363 [04:44:14] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [04:44:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f5647b5a5419...07926b7418b6 [04:44:17] [02miraheze/puppet] 07paladox 0307926b7 - check_puppetrun: Drop safe_yaml (#3363) [04:44:20] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [04:45:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/6d6eb06b71f2 [04:45:02] [02miraheze/puppet] 07paladox 036d6eb06 - Stop installing ruby-safe-yaml [04:45:05] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [04:45:07] [02puppet] 07paladox opened pull request 03#3364: Stop installing ruby-safe-yaml - 13https://github.com/miraheze/puppet/pull/3364 [04:45:34] [02puppet] 07paladox closed pull request 03#3364: Stop installing ruby-safe-yaml - 13https://github.com/miraheze/puppet/pull/3364 [04:45:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/07926b7418b6...b212230f4e11 [04:45:38] [02miraheze/puppet] 07paladox 03b212230 - Stop installing ruby-safe-yaml (#3364) [04:45:40] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [04:45:43] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [04:49:15] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/87227ba63b3f...6e19b935af99 [04:49:16] [02miraheze/dns] 07paladox 036e19b93 - Revert "Revert "Re-add cp32"" [04:49:48] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:49:56] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/6e19b935af99...14de0ab4ef5e [04:49:57] [02miraheze/dns] 07paladox 0314de0ab - Remove cp33 temp [04:56:04] PROBLEM - cp33 SSH on cp33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:56:09] PROBLEM - cp33 HTTPS on cp33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:56:18] PROBLEM - ping6 on cp33 is CRITICAL: PING CRITICAL - Packet loss = 100% [04:57:03] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:09] PROBLEM - cp33 Stunnel for mw134 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:14] PROBLEM - cp33 PowerDNS Recursor on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:15] PROBLEM - cp33 Stunnel for test131 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:20] PROBLEM - cp33 Stunnel for mw133 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:26] PROBLEM - cp33 Stunnel for mw141 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:27] PROBLEM - cp33 Stunnel for matomo121 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:27] PROBLEM - cp33 Stunnel for mw142 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:28] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:29] PROBLEM - cp33 Stunnel for mon141 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:30] PROBLEM - cp33 APT on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:31] PROBLEM - cp33 Stunnel for puppet141 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:32] PROBLEM - cp33 Stunnel for phab121 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:57:35] PROBLEM - cp33 Current Load on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:58:00] RECOVERY - cp33 SSH on cp33 is OK: SSH OK - OpenSSH_9.2p1 Debian-2 (protocol 2.0) [04:58:01] PROBLEM - cp33 Stunnel for mw143 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:58:01] PROBLEM - cp33 NTP time on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:58:02] PROBLEM - cp33 Stunnel for mw132 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:58:06] PROBLEM - cp33 Stunnel for reports121 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:58:07] PROBLEM - cp33 Stunnel for mwtask141 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:58:07] PROBLEM - cp33 conntrack_table_size on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:58:09] PROBLEM - cp33 Stunnel for mail121 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:58:10] PROBLEM - cp33 Stunnel for mw131 on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:58:11] PROBLEM - cp33 ferm_active on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:58:12] PROBLEM - cp33 Puppet on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:58:15] PROBLEM - cp33 Disk Space on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:26f::1 port 5666: Connection refused [04:58:21] RECOVERY - ping6 on cp33 is OK: PING OK - Packet loss = 0%, RTA = 105.73 ms [04:58:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.51, 7.80, 5.31 [05:00:38] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 3.94, 6.49, 5.13 [05:06:03] RECOVERY - mon141 APT on mon141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:08:39] PROBLEM - cp33 Varnish Backends on cp33 is WARNING: No backends detected. If this is an error, see readme.txt [05:08:39] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 0.31, 0.45, 0.29 [05:08:41] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.03574070334 secs [05:08:42] RECOVERY - cp33 Stunnel for puppet141 on cp33 is OK: TCP OK - 0.001 second response time on localhost port 8204 [05:08:42] PROBLEM - cp33 ferm_active on cp33 is UNKNOWN: NRPE: Unable to read output [05:08:42] RECOVERY - cp33 Stunnel for mw141 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8108 [05:08:42] RECOVERY - cp33 Stunnel for matomo121 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8203 [05:08:42] RECOVERY - cp33 Disk Space on cp33 is OK: DISK OK - free space: / 71890MiB (96% inode=98%); [05:08:44] RECOVERY - cp33 Stunnel for test131 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8180 [05:08:46] RECOVERY - cp33 Stunnel for mw143 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8112 [05:08:48] RECOVERY - cp33 Stunnel for mw133 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8110 [05:08:49] PROBLEM - cp33 conntrack_table_size on cp33 is UNKNOWN: NRPE: Unable to read output [05:08:50] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:08:51] RECOVERY - cp33 Stunnel for mw142 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8109 [05:08:53] RECOVERY - cp33 Stunnel for mon141 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8201 [05:08:55] PROBLEM - cp33 Puppet on cp33 is UNKNOWN: NRPE: Unable to read output [05:08:58] RECOVERY - cp33 Stunnel for mw132 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8107 [05:08:59] RECOVERY - cp33 Stunnel for reports121 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8205 [05:09:02] RECOVERY - cp33 Stunnel for mail121 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8200 [05:09:11] RECOVERY - cp33 Stunnel for phab121 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8202 [05:09:14] RECOVERY - cp33 PowerDNS Recursor on cp33 is OK: DNS OK: 0.043 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:8100::1 [05:09:16] RECOVERY - cp33 Stunnel for mwtask141 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8150 [05:09:16] RECOVERY - cp33 Stunnel for mw131 on cp33 is OK: TCP OK - 0.000 second response time on localhost port 8106 [05:10:05] RECOVERY - cp33 HTTPS on cp33 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3713 bytes in 0.527 second response time [05:12:40] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/14de0ab4ef5e...51a1a8a25b3d [05:12:42] [02miraheze/dns] 07paladox 0351a1a8a - Revert "Remove cp33 temp" [05:15:13] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:17:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2607:f1c0:1800:8100::1/cpweb, 2607:f1c0:1800:26f::1/cpweb [05:17:20] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:17:27] PROBLEM - cp32 NTP time on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:17:30] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 15 backends are healthy [05:18:06] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 100% [05:18:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:f1c0:1800:8100::1/cpweb [05:18:15] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:19:27] PROBLEM - Host cp32 is DOWN: PING CRITICAL - Packet loss = 100% [05:21:05] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/51a1a8a25b3d...72a81099c27c [05:21:06] [02miraheze/dns] 07paladox 0372a8109 - Depool US [05:21:25] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw133 [05:24:15] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.29, 7.12, 5.81 [05:24:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.09, 7.31, 6.21 [05:26:13] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.20, 5.94, 5.53 [05:26:54] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 3.90, 6.06, 5.89 [05:35:14] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 15 backends are healthy [05:50:51] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [05:52:36] RECOVERY - Host cp32 is UP: PING OK - Packet loss = 0%, RTA = 104.75 ms [05:52:45] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_9.2p1 Debian-2 (protocol 2.0) [05:53:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:53:43] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:54:01] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.05330595374 secs [05:54:01] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:54:26] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 105.26 ms [05:54:36] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3725 bytes in 0.426 second response time [05:55:57] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [05:58:35] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/72a81099c27c...fd2ccefb3b7c [05:58:36] [02miraheze/dns] 07paladox 03fd2ccef - Revert "Depool US" [06:03:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.54, 6.51, 5.43 [06:06:25] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 11.31, 8.40, 6.00 [06:08:11] PROBLEM - theenglishlakes.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'theenglishlakes.uk' expires in 15 day(s) (Thu 24 Aug 2023 05:40:12 GMT +0000). [06:10:21] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.10, 7.14, 6.06 [06:10:49] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cad207cd6484...ee5dddb8d984 [06:10:52] [02miraheze/mw-config] 07paladox 03ee5dddb - Revert "Remove paradox from staffwiki" [06:10:54] [02mw-config] 07paladox closed pull request 03#5291: Fix wgSharedTables for srewiki - 13https://github.com/miraheze/mw-config/pull/5291 [06:10:56] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [06:10:58] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [06:11:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.27, 7.77, 6.60 [06:11:42] miraheze/mw-config - paladox the build passed. [06:12:19] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.12, 6.80, 6.07 [06:12:28] PROBLEM - wiki.gestaltonline.ca - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gestaltonline.ca' expires in 15 day(s) (Thu 24 Aug 2023 05:41:42 GMT +0000). [06:13:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ac379d3c3a4f...87c4450a4849 [06:13:18] [02miraheze/ssl] 07MirahezeSSLBot 0387c4450 - Bot: Update SSL cert for theenglishlakes.uk [06:13:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.76, 8.45, 6.96 [06:13:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/87c4450a4849...5b2a59a74840 [06:13:59] [02miraheze/ssl] 07MirahezeSSLBot 035b2a59a - Bot: Update SSL cert for wiki.gestaltonline.ca [06:15:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.91, 7.41, 6.76 [06:16:13] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.80, 7.70, 6.69 [06:17:34] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.38, 6.43, 6.48 [06:18:10] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.73, 6.64, 6.42 [06:20:39] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [06:28:55] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.59, 7.13, 6.67 [06:30:53] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.58, 6.13, 6.36 [06:37:02] RECOVERY - theenglishlakes.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'theenglishlakes.uk' will expire on Mon 06 Nov 2023 05:13:09 GMT +0000. [06:38:15] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1799515188 secs [06:41:44] RECOVERY - wiki.gestaltonline.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gestaltonline.ca' will expire on Mon 06 Nov 2023 05:13:51 GMT +0000. [07:06:05] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.05122864246 secs [07:40:05] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.118306011 secs [07:49:59] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [07:58:05] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.06259641051 secs [08:19:46] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [08:34:05] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [09:02:50] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.79, 3.53, 2.76 [09:06:50] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.31, 3.85, 3.06 [09:12:50] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.29, 3.76, 3.31 [09:19:20] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [09:20:50] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.36, 3.73, 3.42 [09:22:50] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.67, 3.61, 3.40 [09:26:50] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.81, 3.27, 3.31 [09:33:43] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [09:49:07] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [10:12:05] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.209949404 secs [10:16:42] PROBLEM - theagonian.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'theagonian.com' expires in 15 day(s) (Thu 24 Aug 2023 10:00:26 GMT +0000). [10:22:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5b2a59a74840...2a2ef6b22f0f [10:22:28] [02miraheze/ssl] 07MirahezeSSLBot 032a2ef6b - Bot: Update SSL cert for theagonian.com [10:28:05] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.01074492931 secs [10:48:42] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [10:59:14] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [11:14:45] RECOVERY - theagonian.com - LetsEncrypt on sslhost is OK: OK - Certificate 'theagonian.com' will expire on Mon 06 Nov 2023 09:22:19 GMT +0000. [11:18:29] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [11:29:05] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:16:50] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.12, 3.52, 2.69 [12:18:50] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.58, 3.45, 2.77 [12:20:50] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.04, 3.31, 2.81 [12:30:50] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.24, 3.48, 3.14 [12:32:50] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.03, 3.32, 3.12 [12:50:50] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.11, 3.52, 3.32 [12:52:50] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.36, 3.37, 3.28 [13:30:05] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1699786782 secs [13:35:30] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1000752747 secs [13:42:05] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.05822172761 secs [14:05:00] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [14:34:49] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:42:06] PROBLEM - wiki.songngu.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.songngu.xyz' expires in 15 day(s) (Thu 24 Aug 2023 15:28:11 GMT +0000). [15:47:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2a2ef6b22f0f...d69e784e0667 [15:47:18] [02miraheze/ssl] 07MirahezeSSLBot 03d69e784 - Bot: Update SSL cert for wiki.songngu.xyz [16:01:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/eef15cd9cfe8 [16:01:43] [02miraheze/puppet] 07paladox 03eef15cd - varnish: increase thread_pool_max to 1000 [16:01:46] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:01:48] [02puppet] 07paladox opened pull request 03#3365: varnish: increase thread_pool_max to 1000 - 13https://github.com/miraheze/puppet/pull/3365 [16:02:13] [02puppet] 07paladox synchronize pull request 03#3365: varnish: increase thread_pool_max to 1000 - 13https://github.com/miraheze/puppet/pull/3365 [16:02:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eef15cd9cfe8...5179b0010a2c [16:02:18] [02miraheze/puppet] 07paladox 035179b00 - Update varnish.systemd.erb [16:04:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5179b0010a2c...32d9e395876d [16:04:06] [02miraheze/puppet] 07paladox 0332d9e39 - Update init.pp [16:04:08] [02puppet] 07paladox synchronize pull request 03#3365: varnish: increase thread_pool_max to 1000 - 13https://github.com/miraheze/puppet/pull/3365 [16:04:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/32d9e395876d...2b0dd018f213 [16:04:20] [02miraheze/puppet] 07paladox 032b0dd01 - Update cp22.yaml [16:04:21] [02puppet] 07paladox synchronize pull request 03#3365: varnish: increase thread_pool_max to 1000 - 13https://github.com/miraheze/puppet/pull/3365 [16:04:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2b0dd018f213...3f0fd8bf6013 [16:04:35] [02miraheze/puppet] 07paladox 033f0fd8b - Update cp23.yaml [16:04:36] [02puppet] 07paladox synchronize pull request 03#3365: varnish: increase thread_pool_max to 1000 - 13https://github.com/miraheze/puppet/pull/3365 [16:04:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3f0fd8bf6013...6d5ae4d0724c [16:04:45] [02miraheze/puppet] 07paladox 036d5ae4d - Update cp33.yaml [16:04:46] [02puppet] 07paladox synchronize pull request 03#3365: varnish: increase thread_pool_max to 1000 - 13https://github.com/miraheze/puppet/pull/3365 [16:04:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6d5ae4d0724c...21aed98b0c10 [16:04:56] [02miraheze/puppet] 07paladox 0321aed98 - Update cp32.yaml [16:04:57] [02puppet] 07paladox synchronize pull request 03#3365: varnish: increase thread_pool_max to 1000 - 13https://github.com/miraheze/puppet/pull/3365 [16:06:57] [02puppet] 07paladox closed pull request 03#3365: varnish: increase thread_pool_max to 1000 - 13https://github.com/miraheze/puppet/pull/3365 [16:07:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [16:07:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/b212230f4e11...b0f4fdf2537f [16:07:02] [02miraheze/puppet] 07paladox 03b0f4fdf - varnish: increase thread_pool_max to 1000 (#3365) [16:07:03] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:07:14] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.72, 3.23, 2.23 [16:09:14] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.94, 3.51, 2.46 [16:13:14] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.17, 3.02, 2.52 [16:13:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0a5f7cc094b1 [16:14:00] [02miraheze/puppet] 07paladox 030a5f7cc - varnish: increase varnish::thread_pool_max to 500 [16:14:01] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:14:04] [02puppet] 07paladox opened pull request 03#3366: varnish: increase varnish::thread_pool_max to 500 - 13https://github.com/miraheze/puppet/pull/3366 [16:14:07] [02puppet] 07paladox edited pull request 03#3366: varnish: increase varnish::thread_pool_max to 5000 - 13https://github.com/miraheze/puppet/pull/3366 [16:14:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0a5f7cc094b1...2dfbacd6a106 [16:14:20] [02miraheze/puppet] 07paladox 032dfbacd - Update cp23.yaml [16:14:21] [02puppet] 07paladox synchronize pull request 03#3366: varnish: increase varnish::thread_pool_max to 5000 - 13https://github.com/miraheze/puppet/pull/3366 [16:14:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2dfbacd6a106...e13a49edf872 [16:14:27] [02miraheze/puppet] 07paladox 03e13a49e - Update cp32.yaml [16:14:30] [02puppet] 07paladox synchronize pull request 03#3366: varnish: increase varnish::thread_pool_max to 5000 - 13https://github.com/miraheze/puppet/pull/3366 [16:14:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e13a49edf872...20bdf3d7025a [16:14:33] [02miraheze/puppet] 07paladox 0320bdf3d - Update cp33.yaml [16:14:36] [02puppet] 07paladox synchronize pull request 03#3366: varnish: increase varnish::thread_pool_max to 5000 - 13https://github.com/miraheze/puppet/pull/3366 [16:15:17] [02puppet] 07paladox closed pull request 03#3366: varnish: increase varnish::thread_pool_max to 5000 - 13https://github.com/miraheze/puppet/pull/3366 [16:15:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/b0f4fdf2537f...b35f8cb98103 [16:15:19] [02miraheze/puppet] 07paladox 03b35f8cb - varnish: increase varnish::thread_pool_max to 5000 (#3366) [16:15:20] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [16:15:22] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:15:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.73, 7.83, 5.17 [16:16:40] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.13, 7.87, 5.17 [16:17:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.53, 7.38, 5.32 [16:19:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.43, 9.00, 6.19 [16:20:40] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.43, 7.56, 5.68 [16:23:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.14, 7.56, 6.35 [16:24:40] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.26, 6.60, 5.76 [16:25:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.90, 8.54, 6.85 [16:25:59] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.12, 3.41, 2.99 [16:27:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0565c4b266c4 [16:27:44] [02miraheze/puppet] 07paladox 030565c4b - varnish: increase thread_pool_max to 15000 [16:27:47] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:27:48] [02puppet] 07paladox opened pull request 03#3367: varnish: increase thread_pool_max to 15000 - 13https://github.com/miraheze/puppet/pull/3367 [16:27:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.04, 7.44, 6.67 [16:27:55] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.28, 3.41, 3.05 [16:28:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0565c4b266c4...236b450dadc9 [16:28:08] [02miraheze/puppet] 07paladox 03236b450 - Update cp22.yaml [16:28:10] [02puppet] 07paladox synchronize pull request 03#3367: varnish: increase thread_pool_max to 15000 - 13https://github.com/miraheze/puppet/pull/3367 [16:28:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/236b450dadc9...ac9bf9a05d37 [16:28:15] [02miraheze/puppet] 07paladox 03ac9bf9a - Update cp32.yaml [16:28:18] [02puppet] 07paladox synchronize pull request 03#3367: varnish: increase thread_pool_max to 15000 - 13https://github.com/miraheze/puppet/pull/3367 [16:28:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ac9bf9a05d37...851f255c734b [16:28:22] [02miraheze/puppet] 07paladox 03851f255 - Update cp33.yaml [16:28:23] [02puppet] 07paladox synchronize pull request 03#3367: varnish: increase thread_pool_max to 15000 - 13https://github.com/miraheze/puppet/pull/3367 [16:28:45] [02puppet] 07paladox closed pull request 03#3367: varnish: increase thread_pool_max to 15000 - 13https://github.com/miraheze/puppet/pull/3367 [16:28:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/b35f8cb98103...4385dad9bc41 [16:28:50] [02miraheze/puppet] 07paladox 034385dad - varnish: increase thread_pool_max to 15000 (#3367) [16:28:53] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [16:28:56] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:29:50] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 1.93, 2.99, 2.94 [16:30:40] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.40, 7.66, 6.44 [16:31:53] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.87, 6.64, 6.57 [16:35:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.46, 8.21, 7.18 [16:36:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/dec95992349a [16:36:20] [02miraheze/puppet] 07paladox 03dec9599 - varnish: increase thread_pool_max to 50000 [16:36:21] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:36:22] [02puppet] 07paladox opened pull request 03#3368: varnish: increase thread_pool_max to 50000 - 13https://github.com/miraheze/puppet/pull/3368 [16:36:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dec95992349a...d771a12b24ae [16:36:35] [02miraheze/puppet] 07paladox 03d771a12 - Update cp23.yaml [16:36:38] [02puppet] 07paladox synchronize pull request 03#3368: varnish: increase thread_pool_max to 50000 - 13https://github.com/miraheze/puppet/pull/3368 [16:36:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d771a12b24ae...06046056c6d9 [16:36:44] [02miraheze/puppet] 07paladox 030604605 - Update cp32.yaml [16:36:45] [02puppet] 07paladox synchronize pull request 03#3368: varnish: increase thread_pool_max to 50000 - 13https://github.com/miraheze/puppet/pull/3368 [16:36:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/06046056c6d9...f9d9907de862 [16:36:47] [02miraheze/puppet] 07paladox 03f9d9907 - Update cp33.yaml [16:36:49] [02puppet] 07paladox synchronize pull request 03#3368: varnish: increase thread_pool_max to 50000 - 13https://github.com/miraheze/puppet/pull/3368 [16:37:30] [02puppet] 07paladox closed pull request 03#3368: varnish: increase thread_pool_max to 50000 - 13https://github.com/miraheze/puppet/pull/3368 [16:37:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/4385dad9bc41...16bdf2598d7a [16:37:36] [02miraheze/puppet] 07paladox 0316bdf25 - varnish: increase thread_pool_max to 50000 (#3368) [16:37:39] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [16:37:42] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:37:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.23, 7.80, 7.17 [16:41:38] RECOVERY - wiki.songngu.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.songngu.xyz' will expire on Mon 06 Nov 2023 14:47:09 GMT +0000. [16:45:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.38, 9.28, 7.82 [16:45:57] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.47, 2.86, 1.83 [16:47:56] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.03, 2.19, 1.71 [16:51:01] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/fd2ccefb3b7c...f06396dae515 [16:51:03] [02miraheze/dns] 07paladox 03f06396d - Depool cp23 [16:52:40] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.05, 7.95, 7.92 [16:56:46] PROBLEM - cp23 HTTPS on cp23 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:00] PROBLEM - cp23 SSH on cp23 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:11] PROBLEM - cp23 conntrack_table_size on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:57:13] PROBLEM - cp23 Stunnel for mail121 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:57:21] PROBLEM - cp23 Stunnel for puppet141 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:57:21] PROBLEM - cp23 Stunnel for test131 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:57:23] PROBLEM - cp23 Disk Space on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:57:31] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:57:37] PROBLEM - cp23 Stunnel for mon141 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:57:45] PROBLEM - cp23 Stunnel for mw141 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:57:53] PROBLEM - cp23 Stunnel for reports121 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:57:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 3.64, 6.39, 7.84 [16:57:56] PROBLEM - cp23 Stunnel for mw133 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:57:58] PROBLEM - cp23 Stunnel for mw134 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:58:00] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:58:01] PROBLEM - cp23 Puppet on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:58:27] PROBLEM - cp23 Stunnel for mw132 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:58:28] PROBLEM - cp23 Stunnel for mwtask141 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:58:28] PROBLEM - cp23 Stunnel for mw143 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:58:30] PROBLEM - cp23 Stunnel for matomo121 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:58:30] PROBLEM - cp23 PowerDNS Recursor on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:58:30] PROBLEM - cp23 ferm_active on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:58:31] PROBLEM - cp23 Stunnel for mw131 on cp23 is CRITICAL: connect to address 2a00:da00:1800:328::1 port 5666: Connection refusedconnect to host 2a00:da00:1800:328::1 port 5666: Connection refused [16:58:33]