[00:01:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.42, 3.95, 3.61 [00:01:28] RECOVERY - test131 Disk Space on test131 is OK: DISK OK - free space: / 4163 MB (14% inode=79%); [00:02:21] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.63, 9.96, 8.61 [00:02:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.22, 4.08, 3.38 [00:03:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.79, 4.29, 3.77 [00:04:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.32, 9.06, 8.46 [00:09:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.07, 3.60, 3.75 [00:16:39] PROBLEM - dcmultiversewiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dcmultiversewiki.com could not be found [00:16:58] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.06, 3.53, 3.58 [00:17:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.75, 7.89, 5.10 [00:18:57] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.81, 3.18, 3.44 [00:20:55] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.54, 2.56, 3.18 [00:21:00] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.73, 7.22, 5.52 [00:23:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.02, 5.45, 5.08 [00:26:49] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.03, 4.34, 3.79 [00:30:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.48, 3.25, 3.92 [00:32:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.51, 3.91, 4.06 [00:32:50] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.00, 3.95, 3.85 [00:33:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 20.24, 10.87, 7.16 [00:33:15] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 5.81, 4.20, 2.90 [00:34:44] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.17, 3.79, 3.78 [00:36:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.38, 3.38, 3.83 [00:36:34] PROBLEM - www.dcmultiversewiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.dcmultiversewiki.com could not be found [00:36:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.79, 3.03, 3.50 [00:37:14] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1412089467 secs [00:38:41] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.53, 2.54, 3.26 [00:39:15] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 2.48, 3.77, 3.19 [00:41:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.79, 6.41, 6.77 [00:41:15] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 2.39, 3.22, 3.06 [00:44:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.02, 3.11, 3.42 [00:48:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.37, 3.20, 3.40 [00:53:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.51, 7.43, 6.49 [00:57:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.40, 5.04, 3.96 [00:59:00] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.77, 7.83, 7.03 [00:59:29] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.14, 2.97, 1.76 [01:01:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.98, 8.12, 7.22 [01:01:14] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.08887159824 secs [01:01:28] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.73, 2.99, 1.92 [01:05:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.82, 5.82, 6.59 [01:07:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.16, 3.89, 4.00 [01:18:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.59, 2.82, 3.30 [01:20:48] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [01:22:42] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.79, 3.45, 3.46 [01:24:39] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.90, 3.28, 3.40 [01:29:20] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.13, 5.43, 2.74 [01:33:18] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.12, 3.44, 2.57 [01:35:17] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.56, 2.44, 2.31 [01:36:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.67, 3.76, 3.48 [01:40:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.00, 3.48, 3.51 [01:44:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.87, 2.99, 3.34 [01:45:14] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.4172464609 secs [01:49:27] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:52:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.90, 3.60, 3.39 [01:53:17] PROBLEM - www.dcmultiversewiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.dcmultiversewiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:54:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.23, 3.76, 3.45 [01:56:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.67, 3.30, 3.32 [01:59:51] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.10, 3.57, 2.99 [02:01:50] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.08, 3.22, 2.92 [02:02:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.24, 3.47, 3.36 [02:04:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.19, 4.49, 3.75 [02:07:14] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.02187880874 secs [02:10:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.16, 3.72, 3.76 [02:16:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.24, 4.41, 3.86 [02:16:44] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1077833772 secs [02:18:09] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:18:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.63, 3.66, 3.67 [02:23:00] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.77, 5.47, 2.96 [02:24:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.76, 2.73, 3.27 [02:25:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.04, 4.63, 2.97 [02:26:28] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.38, 4.28, 3.07 [02:28:27] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.33, 3.78, 3.03 [02:30:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.91, 3.10, 2.87 [02:30:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.58, 3.19, 3.31 [02:32:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.19, 3.04, 3.23 [02:32:44] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.01838830113 secs [02:41:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.80, 4.32, 3.69 [02:42:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.30, 3.46, 3.02 [02:44:23] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.93, 2.90, 2.87 [02:49:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.23, 3.66, 3.64 [02:51:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.67, 2.99, 3.40 [03:00:51] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.77, 4.70, 2.34 [03:02:50] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.50, 3.95, 2.38 [03:04:49] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.81, 2.85, 2.16 [03:28:41] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.83, 2.46, 1.68 [03:29:14] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.3103561699 secs [03:30:41] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.13, 3.22, 2.06 [03:32:41] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.98, 2.30, 1.86 [03:34:33] PROBLEM - www.dcmultiversewiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.dcmultiversewiki.com All nameservers failed to answer the query. [03:37:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.73, 4.20, 3.51 [03:39:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.74, 3.91, 3.50 [03:41:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.63, 3.12, 3.26 [03:44:39] PROBLEM - dcmultiversewiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dcmultiversewiki.com All nameservers failed to answer the query. [03:49:14] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.009372800589 secs [04:00:28] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.54, 3.75, 3.41 [04:02:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.48, 3.98, 3.51 [04:06:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.10, 3.22, 3.33 [04:24:27] RECOVERY - dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dcmultiversewiki.com' will expire on Wed 10 May 2023 21:06:25 GMT +0000. [04:34:33] PROBLEM - www.dcmultiversewiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.dcmultiversewiki.com could not be found [04:44:39] PROBLEM - dcmultiversewiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dcmultiversewiki.com could not be found [04:56:27] PROBLEM - dcmultiversewiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address dcmultiversewiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:56:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.29, 3.55, 2.51 [04:58:36] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.42, 3.08, 2.47 [05:03:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 145.94, 51.13, 19.99 [05:07:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.01, 3.83, 3.22 [05:09:36] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.02, 3.28, 3.09 [05:20:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.19, 4.53, 3.60 [05:25:24] RECOVERY - dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dcmultiversewiki.com' will expire on Wed 10 May 2023 21:06:25 GMT +0000. [05:26:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.21, 3.60, 3.58 [05:30:04] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.40, 2.65, 3.21 [05:30:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.53, 4.06, 2.30 [05:31:18] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:32:41] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.72, 3.22, 2.21 [05:33:18] RECOVERY - cp23 APT on cp23 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [05:34:33] PROBLEM - www.dcmultiversewiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.dcmultiversewiki.com All nameservers failed to answer the query. [05:36:36] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.06, 1.34, 3.98 [05:40:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.90, 1.08, 3.26 [05:47:07] [02puppet] 07Reception123 closed pull request 03#3171: add alex (orangestar) to mw-admins - 13https://github.com/miraheze/puppet/pull/3171 [05:47:08] [02miraheze/puppet] 07Reception123 deleted branch 03Reception123-patch-2 [05:47:10] [02puppet] 07Reception123 deleted branch 03Reception123-patch-2 - 13https://github.com/miraheze/puppet [05:47:13] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8b57281791ec...c9224a2dfbe3 [05:47:15] [02miraheze/puppet] 07Reception123 03c9224a2 - add alex (orangestar) to mw-admins (#3171) [05:51:17] RECOVERY - www.dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dcmultiversewiki.com' will expire on Wed 10 May 2023 21:06:25 GMT +0000. [06:26:27] PROBLEM - dcmultiversewiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address dcmultiversewiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:50:10] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:55:24] RECOVERY - dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dcmultiversewiki.com' will expire on Wed 10 May 2023 21:06:25 GMT +0000. [07:04:33] PROBLEM - www.dcmultiversewiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.dcmultiversewiki.com could not be found [07:14:39] PROBLEM - dcmultiversewiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dcmultiversewiki.com All nameservers failed to answer the query. [07:18:49] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [07:26:27] PROBLEM - dcmultiversewiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address dcmultiversewiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:29:02] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.47, 3.41, 2.09 [07:31:01] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.56, 2.63, 1.96 [07:34:33] PROBLEM - www.dcmultiversewiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.dcmultiversewiki.com All nameservers failed to answer the query. [07:48:10] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [07:55:14] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2042197883 secs [07:56:10] PROBLEM - db142 Backups SQL on db142 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864206 seconds old and 1753 bytes [08:04:30] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c9224a2dfbe3...96268caa8ab4 [08:04:32] [02miraheze/puppet] 07Reception123 0396268ca - also add alex (orangestar) to bastion group [08:04:34] PROBLEM - www.dcmultiversewiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.dcmultiversewiki.com could not be found [08:11:14] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.07749396563 secs [08:14:39] PROBLEM - dcmultiversewiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dcmultiversewiki.com could not be found [08:45:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 10.20, 4.75, 1.99 [08:49:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 15.85, 10.00, 4.84 [08:50:54] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.28, 3.18, 2.06 [08:53:16] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.015 second response time [08:53:53] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:59] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:54:08] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:54:21] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.015 second response time [08:55:15] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:57] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 3.58, 3.35, 2.18 [08:56:23] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.20, 3.35, 1.93 [08:56:40] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:57:04] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 5 backends are down. mw121 mw131 mw132 mw141 mw142 [08:57:16] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.540 second response time [08:57:20] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.315 second response time [08:57:37] PROBLEM - prometheus131 Puppet on prometheus131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:57:44] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 4 backends are down. mw121 mw131 mw132 mw141 [08:57:57] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 2 backends are down. mw141 mw142 [08:57:58] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 2.86, 3.09, 2.22 [08:58:06] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.316 second response time [08:58:16] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 2 backends are down. mw122 mw141 [08:58:23] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.43, 2.57, 1.81 [08:58:24] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.222 second response time [08:58:54] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 3.61, 3.93, 2.98 [08:58:57] PROBLEM - swiftac111 Puppet on swiftac111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:01:04] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [09:01:44] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [09:01:57] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [09:03:16] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.015 second response time [09:03:21] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.015 second response time [09:04:04] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 6.08, 4.58, 3.09 [09:04:54] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.02, 3.82, 3.23 [09:05:04] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw132 mw141 [09:05:16] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:05:18] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.481 second response time [09:05:20] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:05:43] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw132 mw142 [09:05:57] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [09:06:17] PROBLEM - ldap141 Puppet on ldap141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:06:18] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.123 second response time [09:06:24] PROBLEM - bast141 Current Load on bast141 is CRITICAL: CRITICAL - load average: 2.32, 1.62, 0.91 [09:06:29] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.423 second response time [09:06:30] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:06:41] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:06:54] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 3.47, 3.83, 3.31 [09:07:12] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.228 second response time [09:08:24] RECOVERY - bast141 Current Load on bast141 is OK: OK - load average: 0.50, 1.22, 0.86 [09:09:04] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [09:09:14] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.354 second response time [09:09:44] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [09:09:54] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 1.01, 3.37, 3.16 [09:09:57] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [09:10:15] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [09:10:54] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 1.97, 2.99, 3.10 [09:11:01] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [09:15:00] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 0.33, 3.48, 7.51 [09:17:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 0.42, 2.45, 6.64 [09:21:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 0.04, 0.74, 3.72 [09:22:39] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:22:57] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [09:23:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 0.05, 0.52, 3.27 [09:23:17] PROBLEM - www.dcmultiversewiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.dcmultiversewiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:23:36] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [09:30:39] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [09:32:30] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:34:16] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:34:42] PROBLEM - wiki.wisrail.de - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wisrail.de All nameservers failed to answer the query. [10:59:42] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.59, 2.84, 1.89 [11:01:41] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.41, 2.24, 1.78 [11:34:07] RECOVERY - wiki.wisrail.de - reverse DNS on sslhost is OK: SSL OK - wiki.wisrail.de reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:54:59] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.29, 3.44, 2.37 [11:56:57] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.53, 3.89, 2.67 [11:58:55] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.83, 3.60, 2.71 [12:00:54] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.41, 2.88, 2.56 [12:13:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.38, 9.97, 8.19 [12:15:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.65, 9.73, 8.31 [12:20:06] [02puppet] 07MacFan4000 opened pull request 03#3172: add OrangeStar to icinga - 13https://github.com/miraheze/puppet/pull/3172 [12:28:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.04, 10.01, 9.05 [12:29:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 10.19, 9.97, 9.14 [12:43:00] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.81, 5.32, 2.95 [12:45:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.02, 5.69, 3.45 [12:48:05] Reception123: ^ (see my PR when you have a chance) [12:55:46] [02puppet] 07Reception123 closed pull request 03#3172: add OrangeStar to icinga - 13https://github.com/miraheze/puppet/pull/3172 [12:55:48] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/96268caa8ab4...4111d6b6ca31 [12:55:50] [02miraheze/puppet] 07MacFan4000 034111d6b - add OrangeStar to icinga (#3172) [13:06:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.16, 2.90, 2.09 [13:08:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.30, 3.29, 2.34 [13:16:02] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 10.88, 8.16, 6.16 [13:16:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b1f214747d95 [13:16:18] [02miraheze/puppet] 07paladox 03b1f2147 - swift: Increase proxy workers to 72 [13:16:21] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [13:16:23] [02puppet] 07paladox opened pull request 03#3173: swift: Increase proxy workers to 72 - 13https://github.com/miraheze/puppet/pull/3173 [13:16:35] [02puppet] 07paladox synchronize pull request 03#3173: swift: Increase proxy workers to 72 - 13https://github.com/miraheze/puppet/pull/3173 [13:16:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b1f214747d95...4564643a857e [13:16:39] [02miraheze/puppet] 07paladox 034564643 - Update swiftproxy111.yaml [13:16:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4564643a857e...7feebb5f45f1 [13:16:53] [02miraheze/puppet] 07paladox 037feebb5 - Update swiftproxy131.yaml [13:16:56] [02puppet] 07paladox synchronize pull request 03#3173: swift: Increase proxy workers to 72 - 13https://github.com/miraheze/puppet/pull/3173 [13:17:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 26.12, 13.20, 6.11 [13:17:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 11.66, 6.21, 2.90 [13:17:18] [02puppet] 07paladox closed pull request 03#3173: swift: Increase proxy workers to 72 - 13https://github.com/miraheze/puppet/pull/3173 [13:17:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/4111d6b6ca31...67833b287771 [13:17:24] [02miraheze/puppet] 07paladox 0367833b2 - swift: Increase proxy workers to 72 (#3173) [13:17:25] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [13:17:26] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [13:18:02] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 9.30, 8.63, 6.58 [13:22:18] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:47] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:08] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:35] PROBLEM - swiftobject121 Puppet on swiftobject121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:45] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.23, 3.48, 2.63 [13:25:30] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.41, 3.72, 2.40 [13:25:55] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.013 second response time [13:26:17] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.022 second response time [13:26:33] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.95, 2.80, 2.48 [13:27:05] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 2 backends are down. mw121 mw122 [13:27:12] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.267 second response time [13:27:24] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 3.35, 3.58, 2.51 [13:27:49] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.383 second response time [13:28:15] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.361 second response time [13:28:21] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.966 second response time [13:28:27] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.212 second response time [13:29:04] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [13:29:18] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 1.55, 2.90, 2.39 [13:30:24] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[icinga2] [13:30:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.66, 4.24, 3.15 [13:34:36] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.48, 3.88, 3.29 [13:35:15] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 4.07, 3.15, 2.29 [13:36:34] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.79, 3.17, 3.10 [13:37:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.11, 2.18, 3.77 [13:39:16] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 3.10, 3.31, 2.58 [13:41:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.56, 1.81, 3.24 [13:44:56] !log mon141: upgrade grafana [13:45:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:47:30] !log mon141: upgrade icinga-php-library [13:47:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:48:14] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 25.27, 13.03, 7.70 [13:50:18] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:21] PROBLEM - mw131 Disk Space on mw131 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=82%); [13:50:47] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:51:35] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:52:14] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.291 second response time [13:52:21] RECOVERY - mw131 Disk Space on mw131 is OK: DISK OK - free space: / 9684 MB (40% inode=83%); [13:52:23] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:53:00] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 0.39, 2.34, 7.22 [13:54:14] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 5.22, 10.75, 8.95 [13:54:15] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 22173 MB (16% inode=98%); [13:55:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 0.41, 1.69, 6.39 [13:55:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/52f049331370 [13:55:09] [02miraheze/puppet] 07paladox 0352f0493 - matomo: upgrade to 4.14.1 [13:55:12] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [13:55:14] [02puppet] 07paladox opened pull request 03#3174: matomo: upgrade to 4.14.1 - 13https://github.com/miraheze/puppet/pull/3174 [13:56:03] [02puppet] 07paladox closed pull request 03#3174: matomo: upgrade to 4.14.1 - 13https://github.com/miraheze/puppet/pull/3174 [13:56:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [13:56:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/67833b287771...b433562db9e9 [13:56:11] [02miraheze/puppet] 07paladox 03b433562 - matomo: upgrade to 4.14.1 (#3174) [13:56:13] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [13:56:14] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 4.82, 8.72, 8.42 [13:59:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.55, 10.70, 9.18 [13:59:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.00, 10.92, 9.76 [14:01:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.63, 10.09, 9.16 [14:03:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.62, 5.43, 5.97 [14:03:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.08, 11.20, 10.11 [14:05:00] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 6.01, 3.98, 3.06 [14:05:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.39, 5.20, 5.85 [14:05:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.70, 12.02, 10.12 [14:06:59] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.98, 3.99, 3.18 [14:07:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.25, 10.96, 9.96 [14:08:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.08, 4.11, 3.30 [14:08:59] RECOVERY - matomo131 Current Load on matomo131 is OK: OK - load average: 2.20, 3.40, 3.07 [14:10:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.59, 3.20, 3.06 [14:13:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.95, 9.73, 9.75 [14:17:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.21, 10.99, 10.36 [14:17:34] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [14:20:10] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:21:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.20, 9.64, 9.95 [14:23:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.00, 11.24, 11.70 [14:26:55] !log sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=adventurezonewikiwiki Miraheze_maintenance_script [14:27:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 21.69, 14.66, 12.67 [14:34:54] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 5.05, 3.49, 2.65 [14:35:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.00, 11.53, 10.45 [14:37:07] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.3049947321 secs [14:37:12] PROBLEM - mw121 Disk Space on mw121 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=80%); [14:37:39] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php /home/alex/adventurezonewikiwiki-images --wiki=adventurezonewikiwiki --summary="Imported from theadventurezone.fandom.com" [14:37:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:37:58] PROBLEM - mw121 MediaWiki Rendering on mw121 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6664 bytes in 0.061 second response time [14:38:43] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php /home/alex/adventurezonewikiwiki-images --wiki=adventurezonewikiwiki --user=Miraheze_maintenance_script --summary="Imported from theadventurezone.fandom.com" [14:38:54] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.22, 3.67, 2.93 [14:39:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.06, 10.99, 10.49 [14:39:37] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.27, 5.90, 4.85 [14:40:14] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.20, 1.76, 1.23 [14:40:54] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.28, 3.16, 2.84 [14:41:00] PROBLEM - mw121 APT on mw121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [14:41:37] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.14, 6.69, 5.26 [14:42:14] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.39, 1.27, 1.11 [14:43:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 2.51, 7.81, 11.18 [14:43:37] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.65, 6.43, 5.36 [14:45:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 2.02, 5.86, 10.05 [14:45:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.27, 9.40, 9.99 [14:45:55] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.349 second response time [14:47:01] PROBLEM - mw121 APT on mw121 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (3 critical updates). [14:47:12] RECOVERY - mw121 Disk Space on mw121 is OK: DISK OK - free space: / 6157 MB (27% inode=81%); [14:47:20] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:48:48] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [14:51:57] [02mw-config] 07thelevking opened pull request 03#5171: Fix Kartographer in isvwiki - 13https://github.com/miraheze/mw-config/pull/5171 [14:53:06] miraheze/mw-config - thelevking the build passed. [14:55:07] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.02310681343 secs [14:59:29] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.80, 3.54, 2.08 [15:01:37] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:03:37] PROBLEM - wiki.denby.tech - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.40588903427124 seconds [15:04:07] RECOVERY - cp23 APT on cp23 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [15:05:09] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.66, 4.24, 3.25 [15:05:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.68, 11.11, 10.45 [15:09:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.67, 9.94, 10.13 [15:09:34] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.26, 6.61, 4.15 [15:10:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.71, 3.58, 3.37 [15:11:25] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.02, 3.46, 3.55 [15:12:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.91, 2.72, 3.08 [15:13:22] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.88, 6.50, 4.70 [15:13:24] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.71, 2.53, 3.19 [15:33:36] RECOVERY - wiki.denby.tech - reverse DNS on sslhost is OK: SSL OK - wiki.denby.tech reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:35:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.49, 9.28, 8.79 [15:37:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.72, 9.06, 8.78 [15:41:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.00, 10.63, 9.30 [15:42:32] PROBLEM - fid.koyomi-ya.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query koyomi-ya.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:45:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.40, 11.53, 10.05 [15:46:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.04, 4.75, 3.31 [15:46:50] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:47:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.84, 9.74, 9.56 [15:48:10] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [15:48:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.23, 3.74, 3.11 [15:52:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.37, 3.14, 3.01 [15:58:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.40, 3.82, 3.35 [16:00:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.54, 3.37, 3.23 [16:01:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.26, 3.20, 2.59 [16:03:07] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.31, 2.42, 2.38 [16:04:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.39, 9.33, 8.68 [16:06:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.87, 3.57, 3.35 [16:06:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.35, 9.24, 8.72 [16:12:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.52, 3.03, 3.20 [16:12:29] PROBLEM - fid.koyomi-ya.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fid.koyomi-ya.net All nameservers failed to answer the query. [16:13:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.65, 9.83, 9.10 [16:15:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.63, 8.89, 8.84 [16:26:47] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.52, 3.44, 2.59 [16:28:45] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.21, 2.93, 2.51 [16:29:50] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.68, 3.85, 2.38 [16:31:31] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:33:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.34, 3.77, 3.14 [16:33:30] RECOVERY - cp23 APT on cp23 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:33:49] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.49, 3.81, 2.75 [16:35:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.47, 3.89, 3.28 [16:37:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.99, 4.24, 3.45 [16:37:47] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.02, 2.62, 2.53 [16:39:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.26, 3.95, 3.45 [16:43:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.43, 3.26, 3.27 [16:56:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.98, 3.60, 2.83 [17:00:23] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.98, 2.96, 2.76 [17:08:54] PROBLEM - test2.specialchat.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'test2.specialchat.org' expires in 15 day(s) (Sun 16 Apr 2023 16:51:59 GMT +0000). [17:12:27] RECOVERY - fid.koyomi-ya.net - reverse DNS on sslhost is OK: SSL OK - fid.koyomi-ya.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [17:14:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9606432dba16...51da57847c0c [17:14:49] [02miraheze/ssl] 07MirahezeSSLBot 0351da578 - Bot: Update SSL cert for test2.specialchat.org [17:16:21] PROBLEM - db121 Backups SQL on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:16:36] @Site Reliability Engineers I'm receiving 502 Bad Gateway errors [17:16:55] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.016 second response time [17:17:26] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:30] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.019 second response time [17:17:44] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:04] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:13] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:25] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:50] PROBLEM - db121 Backups SQL on db121 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 1066861 seconds old and 1753 bytes [17:18:53] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.351 second response time [17:18:54] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.51, 2.91, 2.23 [17:19:04] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [17:19:11] PROBLEM - db121 MariaDB Connections on db121 is CRITICAL: PHP Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Connection failed: Connection refused [17:19:20] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.257 second response time [17:19:30] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.242 second response time [17:19:42] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.325 second response time [17:19:46] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [17:19:50] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw132 mw141 [17:19:54] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 57.72, 76.28, 36.97 [17:19:57] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [17:19:58] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.363 second response time [17:20:02] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [17:20:09] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.338 second response time [17:20:11] Now I'm getting 503 errors [17:20:20] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.704 second response time [17:20:46] Reception123, paladox: ^ [17:21:07] RECOVERY - db121 MariaDB Connections on db121 is OK: OK connection usage: 0.2%Current connections: 1 [17:21:43] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 166 Threads: 4 Questions: 16134 Slow queries: 1362 Opens: 1671 Open tables: 1665 Queries per second avg: 97.192 [17:22:02] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [17:22:17] I’m mobile [17:22:34] we're back but it seems we keep going down periodically [17:22:54] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.75, 3.03, 2.49 [17:23:04] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [17:23:50] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [17:23:57] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [17:27:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.42, 3.29, 2.85 [17:27:59] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query zulaclub.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:28:50] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.37, 4.48, 3.12 [17:29:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.97, 3.30, 2.91 [17:38:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.45, 3.93, 3.74 [17:44:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.61, 4.09, 3.86 [17:45:54] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 0.79, 1.28, 7.60 [17:47:54] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.52, 1.01, 6.73 [17:48:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.89, 3.47, 3.70 [17:52:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.15, 2.34, 3.19 [17:56:59] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [17:59:12] !log db121: MariaDB [(none)]> set global read_only = 0; [17:59:12] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.70, 4.10, 2.28 [17:59:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:11] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.86, 3.38, 2.23 [18:07:41] RECOVERY - test2.specialchat.org - LetsEncrypt on sslhost is OK: OK - Certificate 'test2.specialchat.org' will expire on Thu 29 Jun 2023 16:14:39 GMT +0000. [18:31:23] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.25, 4.16, 2.93 [18:33:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.17, 3.41, 2.81 [18:35:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.23, 2.70, 2.63 [18:41:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.90, 10.03, 8.13 [18:43:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.45, 9.66, 8.22 [18:47:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.41, 10.25, 8.80 [18:47:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.11, 9.89, 8.34 [18:49:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.20, 10.12, 8.61 [18:51:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.38, 10.06, 9.06 [18:51:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.55, 9.10, 8.42 [19:03:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/51da57847c0c...0eda6787192c [19:03:25] [02miraheze/ssl] 07MirahezeSSLBot 030eda678 - Bot: Add SSL cert for wiki.trfurs.net [19:04:37] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0eda6787192c...7451b58c209a [19:04:39] [02miraheze/ssl] 07Reception123 037451b58 - add redirect for wiki.trfurs.net [19:05:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/7451b58c209a...08ad1596557b [19:05:38] [02miraheze/ssl] 07MirahezeSSLBot 0308ad159 - Bot: Add SSL cert for wiki.orvyn.ca [19:10:51] !log [salt-user@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=dmlwikiwiki (END - exit=65280) [19:10:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:11:44] !log [salt-user@mw141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=dmlwikiwiki (END - exit=0) [19:11:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:11:58] !log [salt-user@mw132] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=dmlwikiwiki (END - exit=0) [19:12:02] !log [salt-user@mw131] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=dmlwikiwiki (END - exit=0) [19:12:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:04] !log [salt-user@mw142] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=dmlwikiwiki (END - exit=0) [19:12:05] !log [salt-user@mw121] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=dmlwikiwiki (END - exit=0) [19:12:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:07] !log [salt-user@mw122] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=dmlwikiwiki (END - exit=0) [19:12:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:13:23] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.59, 3.43, 2.90 [19:15:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.97, 3.31, 2.92 [19:26:25] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[icinga2] [19:28:32] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.65, 3.59, 2.77 [19:30:30] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.31, 2.80, 2.58 [19:35:07] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1380154192 secs [19:41:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.10, 4.41, 3.51 [19:43:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.32, 3.71, 3.37 [19:45:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.16, 3.15, 3.19 [19:49:07] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.06707361341 secs [19:52:24] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [19:57:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.49, 4.01, 3.18 [19:58:11] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:00:13] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:01:07] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.15, 3.67, 3.28 [20:03:05] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.24, 2.76, 2.99 [20:27:18] PROBLEM - decyclopedia.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'decyclopedia.tk' expires in 15 day(s) (Sun 16 Apr 2023 20:15:52 GMT +0000). [20:27:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.59, 3.22, 2.78 [20:28:23] PROBLEM - www.amusementparkarchives.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.amusementparkarchives.com' expires in 15 day(s) (Sun 16 Apr 2023 20:14:48 GMT +0000). [20:28:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.43, 3.01, 1.80 [20:29:41] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.87, 2.86, 2.69 [20:31:12] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:31:58] PROBLEM - cp23 PowerDNS Recursor on cp23 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:32:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/08ad1596557b...85ee91f83eba [20:32:04] [02miraheze/ssl] 07MirahezeSSLBot 0385ee91f - Bot: Update SSL cert for decyclopedia.tk [20:33:12] RECOVERY - cp23 APT on cp23 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:33:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/85ee91f83eba...1a5a85659e53 [20:33:29] [02miraheze/ssl] 07MirahezeSSLBot 031a5a856 - Bot: Update SSL cert for www.amusementparkarchives.com [20:33:58] RECOVERY - cp23 PowerDNS Recursor on cp23 is OK: DNS OK: 0.179 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [20:41:39] PROBLEM - comprehensibleinputwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'comprehensibleinputwiki.org' expires in 15 day(s) (Sun 16 Apr 2023 20:14:15 GMT +0000). [20:43:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.06, 3.36, 2.84 [20:45:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.09, 2.87, 2.72 [20:46:41] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.71, 1.74, 3.61 [20:48:41] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.66, 1.37, 3.24 [20:49:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1a5a85659e53...2da17d2257e3 [20:49:40] [02miraheze/ssl] 07MirahezeSSLBot 032da17d2 - Bot: Update SSL cert for comprehensibleinputwiki.org [20:53:28] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2da17d2257e3...008432276b4c [20:53:31] [02miraheze/ssl] 07MacFan4000 030084322 - remove nenawiki.org from certs.yaml [20:55:23] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.33, 5.22, 3.65 [20:55:28] RECOVERY - mon141 Check correctness of the icinga configuration on mon141 is OK: Icinga configuration is correct [20:57:46] RECOVERY - www.amusementparkarchives.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.amusementparkarchives.com' will expire on Thu 29 Jun 2023 19:33:18 GMT +0000. [20:59:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.03, 3.82, 2.95 [20:59:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.24, 3.91, 3.45 [20:59:59] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:00:37] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.02, 4.67, 3.57 [21:02:47] RECOVERY - cp23 APT on cp23 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:03:04] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.16, 4.56, 3.81 [21:04:31] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.29, 3.05, 3.16 [21:05:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.67, 3.46, 3.49 [21:07:02] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.70, 2.85, 3.26 [21:10:50] RECOVERY - comprehensibleinputwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'comprehensibleinputwiki.org' will expire on Thu 29 Jun 2023 19:49:30 GMT +0000. [21:12:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.84, 3.91, 3.85 [21:14:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.09, 3.67, 3.74 [21:16:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.07, 3.57, 3.70 [21:18:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.92, 3.91, 3.80 [21:20:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.09, 3.48, 3.65 [21:24:06] [02puppet] 07RhinosF1 opened pull request 03#3175: Update workflow.yml - 13https://github.com/miraheze/puppet/pull/3175 [21:25:02] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.98, 4.58, 3.52 [21:26:07] RECOVERY - decyclopedia.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'decyclopedia.tk' will expire on Thu 29 Jun 2023 19:31:55 GMT +0000. [21:26:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.00, 2.86, 3.32 [21:29:02] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.38, 3.82, 3.49 [21:29:36] [02puppet] 07RhinosF1 synchronize pull request 03#3175: Update workflow.yml - 13https://github.com/miraheze/puppet/pull/3175 [21:30:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.71, 3.78, 3.59 [21:31:02] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.55, 3.11, 3.27 [21:34:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.76, 3.51, 3.55 [21:35:35] [02puppet] 07RhinosF1 synchronize pull request 03#3175: Update workflow.yml - 13https://github.com/miraheze/puppet/pull/3175 [21:36:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.62, 3.77, 3.62 [21:38:39] [02puppet] 07RhinosF1 synchronize pull request 03#3175: Update workflow.yml - 13https://github.com/miraheze/puppet/pull/3175 [21:40:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.44, 3.34, 3.46 [21:42:38] [02puppet] 07RhinosF1 synchronize pull request 03#3175: Update workflow.yml - 13https://github.com/miraheze/puppet/pull/3175 [21:42:45] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.62, 2.72, 3.21 [21:44:59] [02puppet] 07RhinosF1 synchronize pull request 03#3175: Update workflow.yml - 13https://github.com/miraheze/puppet/pull/3175 [21:46:36] [02puppet] 07RhinosF1 synchronize pull request 03#3175: Update workflow.yml - 13https://github.com/miraheze/puppet/pull/3175 [21:47:44] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:48:12] [02puppet] 07RhinosF1 edited pull request 03#3175: python: upgrade CI, support bookworm - 13https://github.com/miraheze/puppet/pull/3175 [21:55:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.54, 3.76, 3.51 [21:59:37] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.30, 3.10, 3.31 [22:08:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.09, 3.68, 3.39 [22:10:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.73, 3.31, 3.27 [22:12:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.52, 3.19, 3.24 [22:15:02] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.40, 3.56, 2.96 [22:17:02] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.46, 3.17, 2.89 [22:17:24] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [22:20:36] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:21:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.83, 10.16, 8.89 [22:23:06] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.46, 11.87, 9.65 [22:25:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.82, 11.19, 9.66 [22:25:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.41, 4.16, 3.23 [22:25:24] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.28, 3.76, 3.39 [22:27:23] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.99, 3.79, 3.44 [22:29:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.20, 3.20, 3.26 [22:30:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.69, 9.62, 9.43 [22:31:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.43, 4.00, 3.56 [22:33:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.83, 3.30, 3.36 [22:34:47] Can someone please merge https://github.com/miraheze/ssl/pull/644 [22:36:40] MacFan4000: ^ [22:41:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.36, 4.55, 3.75 [22:41:30] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.90, 11.60, 10.18 [22:43:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.33, 10.74, 10.02 [22:47:47] [02ssl] 07MacFan4000 closed pull request 03#644: Remove dcmultiversewiki redirects - 13https://github.com/miraheze/ssl/pull/644 [22:47:48] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/008432276b4c...255b4c552ec7 [22:47:51] [02miraheze/ssl] 07Universal-Omega 03255b4c5 - Remove dcmultiversewiki redirects (#644) [22:47:54] Done [22:49:06] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.59, 9.38, 9.73 [22:50:27] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [22:55:23] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.18, 3.63, 3.17 [22:57:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.36, 3.63, 3.23 [22:57:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.19, 3.95, 3.96 [22:59:24] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.53, 4.28, 3.54 [23:01:24] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.52, 3.68, 3.42 [23:03:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.52, 2.90, 3.16 [23:03:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.17, 3.48, 3.68 [23:05:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.70, 3.76, 3.78 [23:07:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.72, 5.45, 4.41 [23:11:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.67, 3.89, 4.00 [23:17:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.22, 4.53, 4.06 [23:19:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.76, 3.68, 3.80 [23:20:27] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [23:23:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 11.01, 5.25, 4.25 [23:27:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.82, 3.62, 3.35 [23:27:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.67, 3.84, 3.92 [23:29:26] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.21, 3.25, 3.25 [23:29:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.61, 4.27, 4.07 [23:31:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.33, 3.53, 3.83 [23:43:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.58, 2.58, 3.31 [23:47:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.89, 3.10, 3.44 [23:49:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.70, 2.95, 3.34 [23:55:28] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.74, 3.41, 2.93 [23:57:28] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.87, 4.49, 3.42 [23:59:28] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.11, 3.96, 3.36