[00:03:32] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [00:06:31] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.55, 2.81, 1.90 [00:08:03] RECOVERY - cp33 Disk Space on cp33 is OK: DISK OK - free space: / 14247 MB (19% inode=98%); [00:08:31] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.06, 3.11, 2.11 [00:10:31] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.83, 3.52, 2.39 [00:16:31] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.18, 3.74, 2.83 [00:18:31] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.46, 3.46, 2.83 [00:20:31] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.62, 3.69, 2.98 [00:22:31] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.23, 3.47, 2.99 [00:24:31] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.22, 3.77, 3.16 [00:26:31] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.09, 3.47, 3.12 [00:28:31] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.70, 3.12, 3.03 [00:35:06] PROBLEM - infectowiki.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query infectowiki.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:38:31] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.69, 3.66, 3.32 [00:40:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.34, 3.43, 3.12 [00:42:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.96, 3.22, 3.08 [00:48:31] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.44, 3.20, 3.29 [00:50:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.74, 3.46, 3.99 [00:52:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.28, 4.16, 4.16 [01:02:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.44, 3.30, 3.88 [01:03:01] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [01:03:41] PROBLEM - infectowiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - infectowiki.com All nameservers failed to answer the query. [01:04:02] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [01:08:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.26, 3.46, 3.71 [01:10:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.74, 3.05, 3.53 [01:14:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.51, 2.89, 3.35 [01:23:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.33, 4.10, 3.75 [01:27:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.61, 3.85, 3.79 [01:32:53] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.11, 2.73, 3.31 [01:33:05] RECOVERY - infectowiki.com - reverse DNS on sslhost is OK: SSL OK - infectowiki.com reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [01:34:01] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [01:34:31] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.73, 3.03, 2.85 [01:36:31] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.30, 2.82, 2.80 [01:36:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.50, 3.91, 3.63 [01:38:42] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.21, 3.10, 3.36 [01:47:37] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.57, 3.46, 3.24 [01:49:34] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.16, 3.33, 3.22 [01:53:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.91, 3.60, 3.35 [01:54:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.55, 4.18, 3.44 [01:55:22] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.79, 3.27, 3.26 [01:58:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.38, 3.86, 3.47 [02:00:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.21, 3.23, 3.28 [02:03:07] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.74, 3.76, 3.42 [02:04:01] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [02:10:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.90, 4.07, 3.60 [02:12:52] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.38, 3.93, 3.75 [02:28:28] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.83, 3.04, 3.33 [02:30:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.82, 3.24, 3.87 [02:34:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.19, 3.53, 3.82 [02:35:01] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:36:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.63, 3.09, 3.62 [02:42:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.99, 5.30, 4.27 [02:56:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.81, 3.47, 3.84 [03:02:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.42, 5.03, 4.23 [03:03:32] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:04:01] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [03:12:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.03, 3.56, 3.29 [03:14:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.56, 3.53, 3.31 [03:16:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.96, 3.61, 4.00 [03:18:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.18, 3.35, 3.27 [03:20:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.55, 4.32, 4.19 [03:22:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.31, 3.41, 3.32 [03:24:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.80, 3.18, 3.25 [03:30:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.91, 2.84, 3.71 [03:34:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.53, 2.11, 3.20 [03:42:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.47, 5.73, 4.13 [03:42:45] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/21adf29eb902...06bccc8b9d37 [03:42:46] [02miraheze/dns] 07paladox 0306bccc8 - Revert "Repool US - Remove cp32 temporarily (#428)" [03:43:05] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/06bccc8b9d37...4c264d4436d0 [03:43:07] [02miraheze/dns] 07paladox 034c264d4 - Repool US [03:45:29] [02miraheze/dns] 07paladox pushed 032 commits to 03master [+0/-0/±3] 13https://github.com/miraheze/dns/compare/4c264d4436d0...388c3ecc49d2 [03:45:31] [02miraheze/dns] 07paladox 039850e56 - Revert "Repool US" [03:45:34] [02miraheze/dns] 07paladox 03388c3ec - Revert "Revert "Repool US - Remove cp32 temporarily (#428)"" [03:46:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 93.82, 29.92, 10.66 [03:46:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.54, 3.58, 3.58 [03:50:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.69, 4.63, 3.92 [03:52:12] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.97, 0.32, 0.11 [03:53:33] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:53:51] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [03:57:22] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:52] PROBLEM - cp32 Stunnel for mon141 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:58:56] PROBLEM - cp32 Stunnel for mw142 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:58:57] PROBLEM - cp32 Stunnel for mail121 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:28] PROBLEM - cp32 Puppet on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused PROBLEM - cp32 APT on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:29] PROBLEM - cp32 Stunnel for mwtask141 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:29] PROBLEM - cp32 Stunnel for mw141 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:30] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:30] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:31] PROBLEM - cp32 Stunnel for mw134 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:31] PROBLEM - cp32 Stunnel for test131 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:32] PROBLEM - cp32 Stunnel for mw143 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:32] PROBLEM - cp32 NTP time on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:33] PROBLEM - cp32 Stunnel for mw133 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:33] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:36] PROBLEM - cp32 Disk Space on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:36] PROBLEM - cp32 Stunnel for mw131 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:38] PROBLEM - cp32 ferm_active on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:40] PROBLEM - cp32 conntrack_table_size on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:40] PROBLEM - cp32 Stunnel for matomo121 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:41] PROBLEM - cp32 Stunnel for reports121 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:46] PROBLEM - cp32 Stunnel for puppet141 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:49] PROBLEM - cp32 Stunnel for mw132 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [03:59:58] PROBLEM - cp32 Stunnel for phab121 on cp32 is CRITICAL: connect to address 2607:f1c0:1800:8100::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:8100::1 port 5666: Connection refused [04:01:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.61, 3.48, 3.30 [04:03:18] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.84, 3.22, 3.22 [04:08:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.85, 3.63, 3.98 [04:10:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.33, 3.93, 4.05 [04:11:03] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.56, 3.59, 3.33 [04:13:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.16, 3.48, 3.33 [04:14:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.03, 3.31, 3.29 [04:22:01] RECOVERY - cp32 Stunnel for phab121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8202 [04:22:08] RECOVERY - cp32 Stunnel for matomo121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8203 [04:22:09] RECOVERY - cp32 Stunnel for puppet141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8204 [04:22:10] RECOVERY - cp32 Stunnel for reports121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8205 [04:22:11] RECOVERY - cp32 Stunnel for mw132 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8107 [04:22:28] RECOVERY - cp32 Stunnel for mw134 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8111 [04:22:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.63, 0.64, 0.45 [04:22:32] RECOVERY - cp32 Stunnel for mw143 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8112 [04:22:41] RECOVERY - cp32 Stunnel for test131 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8180 [04:22:43] RECOVERY - cp32 Stunnel for mw133 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8110 [04:22:47] RECOVERY - cp32 Stunnel for mw142 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8109 [04:22:51] RECOVERY - cp32 Stunnel for mail121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8200 [04:22:53] PROBLEM - cp32 ferm_active on cp32 is UNKNOWN: NRPE: Unable to read output [04:22:55] RECOVERY - cp32 Stunnel for mon141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8201 [04:22:55] PROBLEM - cp32 Puppet on cp32 is UNKNOWN: NRPE: Unable to read output [04:22:59] RECOVERY - cp32 Stunnel for mw141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8108 [04:23:00] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 15 backends are healthy [04:23:04] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.001678884029 secs [04:23:06] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:23:15] RECOVERY - cp32 Stunnel for mw131 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8106 [04:23:15] RECOVERY - cp32 Disk Space on cp32 is OK: DISK OK - free space: / 72147 MB (96% inode=98%); [04:23:16] PROBLEM - cp32 conntrack_table_size on cp32 is UNKNOWN: NRPE: Unable to read output [04:23:20] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 0.069 seconds response time. miraheze.org returns 2607:f1c0:1800:26f::1,74.208.203.152 [04:23:49] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:24:22] RECOVERY - cp32 Stunnel for mwtask141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8150 [04:26:46] RECOVERY - cp32 ferm_active on cp32 is OK: OK ferm input default policy is set [04:27:13] RECOVERY - cp32 conntrack_table_size on cp32 is OK: OK: nf_conntrack is 0 % full [04:28:49] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [04:28:59] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3711 bytes in 0.741 second response time [04:34:01] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [04:36:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.73, 2.97, 3.78 [04:38:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.95, 3.31, 3.78 [04:39:09] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 57% [04:40:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.11, 3.13, 3.65 [04:41:06] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/388c3ecc49d2...3f21db1dac57 [04:41:09] [02miraheze/dns] 07paladox 033f21db1 - Re-add cp32 [04:43:07] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 2% [04:44:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.68, 2.51, 3.29 [04:53:48] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:57:26] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:03:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/161082ff53c0 [05:03:53] [02miraheze/puppet] 07paladox 03161082f - cp32/cp33 lower storage size to 45g [05:03:55] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [05:03:56] [02puppet] 07paladox opened pull request 03#3358: cp32/cp33 lower storage size to 45g - 13https://github.com/miraheze/puppet/pull/3358 [05:04:01] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [05:04:03] [02puppet] 07paladox closed pull request 03#3358: cp32/cp33 lower storage size to 45g - 13https://github.com/miraheze/puppet/pull/3358 [05:04:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/be231f75f412...7f71521b4208 [05:04:07] [02miraheze/puppet] 07paladox 037f71521 - cp32/cp33 lower storage size to 45g (#3358) [05:04:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [05:04:11] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [05:10:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-3358-paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/36b1b91daf06 [05:10:38] [02miraheze/puppet] 07paladox 0336b1b91 - Revert "cp32/cp33 lower storage size to 45g (#3358)" [05:10:40] [02puppet] 07paladox created branch 03revert-3358-paladox-patch-10 - 13https://github.com/miraheze/puppet [05:10:45] [02puppet] 07paladox opened pull request 03#3359: Revert "cp32/cp33 lower storage size to 45g (#3358)" - 13https://github.com/miraheze/puppet/pull/3359 [05:10:49] [02puppet] 07paladox closed pull request 03#3359: Revert "cp32/cp33 lower storage size to 45g (#3358)" - 13https://github.com/miraheze/puppet/pull/3359 [05:10:52] [02miraheze/puppet] 07paladox deleted branch 03revert-3358-paladox-patch-10 [05:10:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7f71521b4208...7bd3dc6ed811 [05:10:57] [02miraheze/puppet] 07paladox 037bd3dc6 - Revert "cp32/cp33 lower storage size to 45g (#3358)" (#3359) [05:10:58] [02puppet] 07paladox deleted branch 03revert-3358-paladox-patch-10 - 13https://github.com/miraheze/puppet [05:11:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ef88fed6cbf7 [05:11:15] [02miraheze/puppet] 07paladox 03ef88fed - cp32/cp33 lower storage size to 45g [05:11:19] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [05:11:19] [02puppet] 07paladox opened pull request 03#3360: cp32/cp33 lower storage size to 45g - 13https://github.com/miraheze/puppet/pull/3360 [05:11:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ef88fed6cbf7...3051d91447f8 [05:11:31] [02miraheze/puppet] 07paladox 033051d91 - Update cp33.yaml [05:11:34] [02puppet] 07paladox synchronize pull request 03#3360: cp32/cp33 lower storage size to 45g - 13https://github.com/miraheze/puppet/pull/3360 [05:11:37] [02puppet] 07paladox closed pull request 03#3360: cp32/cp33 lower storage size to 45g - 13https://github.com/miraheze/puppet/pull/3360 [05:11:39] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [05:11:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/7bd3dc6ed811...e1fbc7c3f95a [05:11:42] [02miraheze/puppet] 07paladox 03e1fbc7c - cp32/cp33 lower storage size to 45g (#3360) [05:11:43] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [05:12:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.23, 7.18, 5.64 [05:13:16] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.30, 6.73, 5.37 [05:14:28] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:17:14] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.07, 7.71, 6.08 [05:18:28] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:18:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.11, 6.96, 6.09 [05:19:13] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.00, 7.56, 6.23 [05:20:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.55, 7.88, 6.52 [05:21:12] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.01, 8.11, 6.57 [05:22:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.01, 7.31, 6.46 [05:23:11] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.68, 7.17, 6.41 [05:24:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.12, 7.75, 6.73 [05:26:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.89, 7.71, 6.86 [05:27:09] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 3.68, 6.15, 6.25 [05:28:33] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.20, 6.53, 6.54 [05:34:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ba23bc87b773 [05:34:31] [02miraheze/puppet] 07paladox 03ba23bc8 - varnish; Set MALLOC_CONF to thp:nver [05:34:33] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [05:34:34] [02puppet] 07paladox opened pull request 03#3361: varnish; Set MALLOC_CONF to thp:nver - 13https://github.com/miraheze/puppet/pull/3361 [05:34:37] [02puppet] 07paladox edited pull request 03#3361: varnish: Set MALLOC_CONF to thp:nver - 13https://github.com/miraheze/puppet/pull/3361 [05:35:36] [02puppet] 07paladox closed pull request 03#3361: varnish: Set MALLOC_CONF to thp:nver - 13https://github.com/miraheze/puppet/pull/3361 [05:35:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e1fbc7c3f95a...1db9cf98d7a3 [05:35:40] [02miraheze/puppet] 07paladox 031db9cf9 - varnish: Set MALLOC_CONF to thp:nver (#3361) [05:35:43] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [05:35:44] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [05:42:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 12.44, 7.06, 6.00 [05:45:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.87, 8.16, 6.98 [05:46:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.24, 7.39, 6.40 [05:47:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.09, 7.98, 7.07 [05:50:55] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.50, 6.62, 6.34 [05:53:12] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.20, 6.29, 6.63 [06:07:49] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.82, 3.83, 3.43 [07:32:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.80, 3.16, 4.00 [07:34:01] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [07:35:01] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:36:18] PROBLEM - staff.polandballwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for staff.polandballwiki.com could not be found [07:36:58] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:41:54] PROBLEM - corru.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'corru.wiki' expires in 15 day(s) (Wed 23 Aug 2023 07:13:09 GMT +0000). [07:44:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.60, 2.88, 3.40 [07:46:17] PROBLEM - commons.polandballwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commons.polandballwiki.com could not be found [07:46:48] PROBLEM - pt.polandballwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pt.polandballwiki.com could not be found [07:48:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bbf1aa63836b...afc272ba6517 [07:48:36] [02miraheze/ssl] 07MirahezeSSLBot 03afc272b - Bot: Update SSL cert for corru.wiki [08:03:32] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [08:04:02] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [08:05:55] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [08:10:52] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.60, 3.32, 3.16 [08:14:46] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.79, 3.11, 3.12 [08:18:37] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.86, 3.39, 3.22 [08:20:34] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.26, 3.68, 3.34 [08:22:31] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.90, 3.73, 3.40 [08:24:28] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.47, 3.98, 3.53 [08:28:22] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.74, 3.97, 3.64 [08:36:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.11, 3.99, 3.77 [08:40:43] RECOVERY - corru.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'corru.wiki' will expire on Sun 05 Nov 2023 06:48:26 GMT +0000. [08:52:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.76, 3.65, 3.98 [09:06:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.09, 3.10, 3.40 [09:12:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.53, 3.90, 3.62 [09:16:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.06, 3.70, 3.63 [09:20:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.58, 3.08, 3.39 [09:27:38] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [09:34:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.34, 3.51, 3.29 [09:38:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.62, 3.87, 3.52 [09:46:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.18, 3.28, 3.40 [10:27:37] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:34:01] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [11:13:21] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ImportDump/compare/bb3a96c821ab...848800330eec [11:13:23] [02miraheze/ImportDump] 07translatewiki 038488003 - Localisation updates from https://translatewiki.net. [11:13:24] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/a99f0f983e5c...f9465774e1e3 [11:13:27] [02miraheze/MirahezeMagic] 07translatewiki 03f946577 - Localisation updates from https://translatewiki.net. [11:25:22] miraheze/ImportDump - translatewiki the build has errored. [11:34:01] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [12:08:34] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.13, 3.67, 3.26 [12:34:01] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [12:59:29] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [13:04:01] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [13:16:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.15, 3.28, 3.94 [13:29:28] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [13:31:14] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2068118453 secs [13:34:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.52, 2.82, 3.30 [13:40:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.31, 3.48, 3.40 [13:42:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.79, 3.49, 3.40 [13:48:13] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.01824402809 secs [13:50:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.01, 3.31, 3.39 [13:54:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.48, 3.49, 3.45 [14:04:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.58, 3.13, 3.35 [14:12:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.96, 3.47, 3.40 [14:14:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.04, 3.65, 3.48 [14:16:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.57, 3.34, 3.39 [14:51:43] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.12, 3.58, 3.36 [14:55:37] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.78, 3.96, 3.59 [14:59:31] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.63, 3.25, 3.38 [15:04:21] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:24:12] PROBLEM - perpustaxaan.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - perpustaxaan.wiki All nameservers failed to answer the query. [15:28:43] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.34, 3.56, 3.20 [15:30:40] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.42, 3.47, 3.21 [15:34:20] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [15:34:35] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.79, 3.26, 3.20 [15:41:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.92, 3.74, 3.43 [15:47:12] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.24, 3.87, 3.56 [15:49:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.53, 3.83, 3.59 [15:53:54] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://github.com/miraheze/dns/commit/3d036a07bd47 [15:53:55] [02miraheze/dns] 07paladox 033d036a0 - Add wikigenius.org to dns [15:53:57] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/dns [15:53:59] [02dns] 07paladox opened pull request 03#429: Add wikigenius.org to dns - 13https://github.com/miraheze/dns/pull/429 [15:54:02] [02dns] 07paladox closed pull request 03#429: Add wikigenius.org to dns - 13https://github.com/miraheze/dns/pull/429 [15:54:04] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/3f21db1dac57...748080f23837 [15:54:07] [02miraheze/dns] 07paladox 03748080f - Add wikigenius.org to dns (#429) [15:54:08] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2 [15:54:09] PROBLEM - perpustaxaan.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for perpustaxaan.wiki could not be found [15:54:10] [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/dns [15:56:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/afc272ba6517...8ff6dbd95945 [15:56:22] [02miraheze/ssl] 07MirahezeSSLBot 038ff6dbd - Bot: Add SSL cert for wikigenius.org [15:58:54] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.13, 4.14, 3.75 [16:06:31] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.51, 2.93, 2.20 [16:06:43] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.31, 3.98, 3.87 [16:08:31] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.34, 3.43, 2.47 [16:08:40] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.51, 4.16, 3.95 [16:10:31] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.11, 3.27, 2.53 [16:12:34] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.53, 3.95, 3.93 [16:12:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/8ff6dbd95945...04e36159a4b7 [16:12:39] [02miraheze/ssl] 07MirahezeSSLBot 0304e3615 - Bot: Add SSL cert for wikiyri.org [16:13:09] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 6.18, 8.15, 4.83 [16:14:31] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.12, 3.98, 3.94 [16:15:10] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 7.97, 7.80, 5.08 [16:15:33] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/748080f23837...b3e5ff86fb1a [16:15:34] [02miraheze/dns] 07paladox 03b3e5ff8 - Add bn subdomain to wikiyri.org [16:17:09] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 6.34, 8.05, 5.56 [16:19:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/04e36159a4b7...f0e25981949d [16:19:04] [02miraheze/ssl] 07MirahezeSSLBot 03f0e2598 - Bot: Add SSL cert for bn.wikiyri.org [16:19:09] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 2.42, 6.04, 5.13 [16:19:15] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.71, 3.52, 2.95 [16:22:36] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.91, 3.33, 2.99 [16:26:25] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/f30b4b46b939 [16:26:28] [02miraheze/ssl] 07paladox 03f30b4b4 - Redirect wikiyri.org to en.wikiyri.org [16:26:31] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [16:26:32] [02ssl] 07paladox opened pull request 03#666: Redirect wikiyri.org to en.wikiyri.org - 13https://github.com/miraheze/ssl/pull/666 [16:27:33] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f30b4b46b939...ba94025b33b9 [16:27:36] [02miraheze/ssl] 07paladox 03ba94025 - Update redirects.yaml [16:27:37] [02ssl] 07paladox synchronize pull request 03#666: Redirect wikiyri.org to en.wikiyri.org - 13https://github.com/miraheze/ssl/pull/666 [16:27:45] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ba94025b33b9...06bb4ab67db0 [16:27:48] [02miraheze/ssl] 07paladox 0306bb4ab - Update redirects.yaml [16:27:49] [02ssl] 07paladox synchronize pull request 03#666: Redirect wikiyri.org to en.wikiyri.org - 13https://github.com/miraheze/ssl/pull/666 [16:28:00] [02ssl] 07paladox synchronize pull request 03#666: Redirect wikiyri.org to en.wikiyri.org - 13https://github.com/miraheze/ssl/pull/666 [16:28:02] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/06bb4ab67db0...d04be3a76e11 [16:28:05] [02miraheze/ssl] 07paladox 03d04be3a - Update redirects.yaml [16:28:29] [02ssl] 07paladox edited pull request 03#666: Redirect wikiyri.org to en.wikiyri.org - 13https://github.com/miraheze/ssl/pull/666 [16:28:48] [02ssl] 07paladox edited pull request 03#666: Redirect wikiyri.org to en.wikiyri.org - 13https://github.com/miraheze/ssl/pull/666 [16:28:51] [02ssl] 07paladox closed pull request 03#666: Redirect wikiyri.org to en.wikiyri.org - 13https://github.com/miraheze/ssl/pull/666 [16:28:54] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ssl/compare/f0e25981949d...a74a9297b1d2 [16:28:56] [02miraheze/ssl] 07paladox 03a74a929 - Redirect wikiyri.org to en.wikiyri.org (#666) [16:28:58] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [16:28:59] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [16:31:52] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/b3e5ff86fb1a...f7ade822ecf5 [16:31:53] [02miraheze/dns] 07paladox 03f7ade82 - Add esolibrary.com to dns [16:32:37] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 9.19, 7.48, 5.47 [16:34:33] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 3.31, 6.12, 5.23 [16:36:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/a74a9297b1d2...031371e7082a [16:36:09] [02miraheze/ssl] 07MirahezeSSLBot 03031371e - Bot: Add SSL cert for www.esolibrary.com [16:38:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/031371e7082a...141affbb51f8 [16:38:17] [02miraheze/ssl] 07MirahezeSSLBot 03141affb - Bot: Add SSL cert for wiki.esufranchise.com [16:40:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/141affbb51f8...4dac8b8b892a [16:40:03] [02miraheze/ssl] 07MirahezeSSLBot 034dac8b8 - Bot: Add SSL cert for wiki.theatlas.live [16:41:52] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/25512530f465 [16:41:55] [02miraheze/ssl] 07paladox 032551253 - Remove spcodex.wiki [16:41:58] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [16:41:59] [02ssl] 07paladox opened pull request 03#667: Remove spcodex.wiki - 13https://github.com/miraheze/ssl/pull/667 [16:42:06] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/25512530f465...5a78f0fe5875 [16:42:09] [02miraheze/ssl] 07paladox 035a78f0f - Delete spcodex.wiki.crt [16:42:11] [02ssl] 07paladox synchronize pull request 03#667: Remove spcodex.wiki - 13https://github.com/miraheze/ssl/pull/667 [16:42:17] [02ssl] 07paladox closed pull request 03#667: Remove spcodex.wiki - 13https://github.com/miraheze/ssl/pull/667 [16:42:18] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [16:42:21] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/4dac8b8b892a...cdda8bb67f8c [16:42:22] [02miraheze/ssl] 07paladox 03cdda8bb - Remove spcodex.wiki (#667) [16:42:24] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [16:42:29] PROBLEM - db142 Puppet on db142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:42:32] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:42:32] PROBLEM - cp23 Puppet on cp23 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:42:33] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:42:34] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:42:39] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:42:40] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:42:52] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:43:05] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:43:09] PROBLEM - mw133 Puppet on mw133 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:43:19] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:43:22] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:43:37] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:43:37] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:43:56] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 3 minutes ago with 26 failures. Failed resources (up to 3 shown): File[/etc/postgresql/13/main/ssl/wildcard.miraheze.org.key],File[/etc/puppetlabs/puppetdb/logback.xml],File[/etc/puppetlabs/puppet/routes.yaml],File[/etc/puppetlabs/puppetserver/logback.xml] [16:44:03] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:44:32] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.85, 3.51, 3.29 [16:44:32] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:46:16] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.39, 3.06, 3.81 [16:46:32] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.72, 3.20, 3.21 [16:47:14] !log increase puppet141 ram to 8g [16:47:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:11] PROBLEM - puppet141 NTP time on puppet141 is CRITICAL: NTP CRITICAL: Offset 0.9183512628 secs [16:49:44] [02ssl] 07MacFan4000 commented on pull request 03#662: Redirect recaptimewiki to squad.lorebooks.eu.org - 13https://github.com/miraheze/ssl/pull/662#issuecomment-1668248611 [16:49:51] [02ssl] 07MacFan4000 closed pull request 03#662: Redirect recaptimewiki to squad.lorebooks.eu.org - 13https://github.com/miraheze/ssl/pull/662 [16:50:14] RECOVERY - puppet141 NTP time on puppet141 is OK: NTP OK: Offset 0.0005989074707 secs [16:50:48] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:51:47] RECOVERY - puppet141 Puppet on puppet141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:56:34] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:57:02] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.53, 3.45, 3.28 [16:58:30] RECOVERY - cp23 Puppet on cp23 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:58:58] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.80, 3.25, 3.23 [17:01:51] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.05, 3.64, 3.64 [17:01:54] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:02:30] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:05:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.66, 3.54, 3.63 [17:06:07] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:06:21] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:06:38] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.64, 3.42, 3.31 [17:06:53] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:07:32] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [17:08:15] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:08:33] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.91, 3.13, 3.21 [17:09:59]