[00:06:34] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:35:42] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [01:05:09] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [01:37:10] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:41:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.49, 3.74, 2.69 [01:43:39] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.87, 3.08, 2.56 [02:06:19] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:08:46] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [02:18:08] PROBLEM - drowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'drowiki.com' expires in 11 day(s) (Tue 25 Jul 2023 15:53:31 GMT +0000). [02:37:31] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1014423966 secs [02:37:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.95, 3.30, 2.46 [02:38:01] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:39:47] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.62, 2.82, 2.40 [02:48:07] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:59:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.13, 4.14, 2.76 [02:59:31] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09202313423 secs [03:01:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.24, 3.33, 2.63 [03:36:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.45, 4.92, 2.90 [03:37:15] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [03:38:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.30, 3.99, 2.79 [03:40:14] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14601 MB (10% inode=99%); [03:40:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.61, 3.16, 2.64 [04:00:26] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:01:31] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1163378358 secs [04:06:24] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:11:31] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.08800020814 secs [04:27:58] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [04:30:15] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:37:14] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.05, 3.17, 2.10 [04:45:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.85, 3.56, 2.98 [04:47:06] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.78, 2.57, 2.68 [04:57:17] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [05:05:13] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [05:09:31] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1010906696 secs [05:11:31] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09987995028 secs [05:15:09] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.294595331 secs [05:35:06] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.01423180103 secs [05:48:07] PROBLEM - drowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'drowiki.com' expires in 11 day(s) (Tue 25 Jul 2023 15:53:31 GMT +0000). [05:49:31] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1060847342 secs [05:51:31] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.08713448048 secs [06:01:02] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1338553131 secs [06:05:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.72, 4.29, 2.51 [06:10:35] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:11:27] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.34, 3.38, 2.52 [06:11:50] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.16, 3.32, 2.79 [06:13:01] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.0914799273 secs [06:13:27] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.59, 3.42, 2.64 [06:15:27] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.55, 3.09, 2.62 [06:18:04] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1517058611 secs [06:18:08] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:23:27] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.04, 3.53, 2.97 [06:25:27] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.64, 3.51, 3.03 [06:27:27] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.78, 3.28, 3.01 [06:30:04] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.0590352118 secs [06:34:16] PROBLEM - db131 Backups SQL on db131 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864206 seconds old and 1753 bytes [06:39:51] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:41:27] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.33, 3.78, 3.33 [06:43:27] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.60, 3.71, 3.36 [06:49:27] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.04, 3.35, 3.31 [07:00:14] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.32, 3.52, 3.40 [07:02:11] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.12, 3.69, 3.47 [07:04:08] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.25, 3.52, 3.44 [07:06:05] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.26, 3.04, 3.27 [07:53:46] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.012 second response time [07:55:46] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw131 [07:56:22] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw131 [07:56:28] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw131 [07:56:55] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw131 [08:35:34] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:46:57] We've seen php get badly stuck before [09:08:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.77, 3.84, 2.21 [09:12:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.56, 3.43, 2.45 [09:13:33] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.405317544937134 seconds [09:14:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.29, 4.16, 2.84 [09:20:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.53, 3.10, 3.02 [09:30:04] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1060756743 secs [09:35:12] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [09:42:49] RECOVERY - wiki.altilunium.my.id - reverse DNS on sslhost is OK: SSL OK - wiki.altilunium.my.id reverse DNS resolves to cp23.miraheze.org - CNAME OK [09:51:31] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1093306243 secs [09:54:04] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07964706421 secs [10:23:31] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09667482972 secs [10:48:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.60, 3.92, 2.62 [10:50:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.23, 2.95, 2.42 [10:59:00] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.264352262 secs [11:03:00] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.08985829353 secs [11:06:36] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [11:18:07] PROBLEM - drowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'drowiki.com' expires in 11 day(s) (Tue 25 Jul 2023 15:53:31 GMT +0000). [11:36:26] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [11:36:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.11, 2.66, 1.74 [11:42:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.08, 3.82, 2.74 [11:46:43] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 19.56, 8.84, 4.75 [11:56:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.31, 2.95, 3.82 [12:00:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.99, 1.78, 3.13 [12:01:16] PROBLEM - nijigen.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nijigen.wiki' expires in 15 day(s) (Sun 30 Jul 2023 11:45:45 GMT +0000). [12:02:47] PROBLEM - mercy.theking1543.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mercy.theking1543.eu.org All nameservers failed to answer the query. [12:03:27] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 25.16, 19.23, 10.02 [12:04:12] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/86febf124e25...a04ee611c65d [12:04:14] [02miraheze/ssl] 07MirahezeSSLBot 03a04ee61 - Bot: Update SSL cert for nijigen.wiki [12:07:22] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 3.12, 10.23, 8.38 [12:08:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.60, 3.12, 3.13 [12:09:19] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 3.42, 8.05, 7.80 [12:10:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.96, 2.31, 2.83 [12:11:27] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.16, 3.25, 2.38 [12:13:27] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.10, 3.21, 2.48 [12:13:31] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1054714918 secs [12:18:07] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:25:08] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.68, 3.45, 3.00 [12:29:02] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.33, 3.28, 3.02 [12:30:04] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1043229699 secs [12:30:33] RECOVERY - nijigen.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nijigen.wiki' will expire on Thu 12 Oct 2023 11:04:04 GMT +0000. [12:36:43] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.81, 3.78, 3.36 [12:38:40] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.77, 4.06, 3.51 [12:39:31] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09848695993 secs [12:43:32] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 41.62, 24.48, 11.82 [12:44:00] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 24.59, 13.62, 7.74 [12:44:04] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08220297098 secs [12:44:30] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.77, 3.88, 3.67 [12:44:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 14.51, 7.56, 4.13 [12:47:34] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 11.69, 10.08, 6.09 [12:50:04] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1246424019 secs [12:50:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.26, 3.28, 3.31 [12:51:09] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.67, 11.26, 8.00 [12:51:20] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 5.60, 11.14, 10.44 [12:51:34] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 8.27, 10.20, 7.14 [12:52:50] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 26.10, 20.46, 11.66 [12:53:09] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.35, 9.47, 7.74 [12:53:17] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 24.79, 14.64, 11.67 [12:53:49] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 6.03, 10.91, 10.44 [12:55:49] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.28, 8.65, 9.67 [12:56:04] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09364384413 secs [12:57:09] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 17.70, 14.62, 10.13 [12:58:08] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.36, 2.98, 3.35 [12:59:09] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 7.16, 11.67, 9.59 [12:59:49] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.81, 15.50, 12.58 [13:00:39] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 6.20, 10.64, 10.42 [13:01:09] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.92, 9.95, 9.20 [13:02:47] PROBLEM - mercy.theking1543.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mercy.theking1543.eu.org could not be found [13:03:49] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 5.07, 10.75, 11.37 [13:04:34] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 5.37, 8.85, 9.81 [13:04:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.54, 2.93, 2.74 [13:05:39] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 23.54, 16.32, 10.37 [13:06:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.74, 4.41, 3.33 [13:07:49] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 2.67, 6.73, 9.60 [13:08:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.22, 3.75, 3.21 [13:09:34] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 3.93, 9.30, 8.81 [13:10:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.93, 2.99, 2.99 [13:11:09] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 2.70, 6.66, 11.48 [13:15:09] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 2.33, 4.26, 9.39 [14:07:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.35, 3.99, 2.53 [14:09:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.02, 3.92, 2.66 [14:09:41] [02ssl] 07Pix1234 opened pull request 03#661: Remove nercy.theking1543.eu.org - no longer points - 13https://github.com/miraheze/ssl/pull/661 [14:09:49] [02ssl] 07Pix1234 edited pull request 03#661: Remove mercy.theking1543.eu.org - no longer points - 13https://github.com/miraheze/ssl/pull/661 [14:10:21] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [14:13:06] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.20, 3.20, 2.76 [14:28:19] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.94, 3.35, 1.89 [14:30:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.24, 2.49, 1.75 [14:32:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.89, 3.83, 2.98 [14:34:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.54, 6.81, 4.21 [14:40:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.52, 3.28, 3.42 [14:42:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.10, 2.52, 3.12 [15:36:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.63, 3.16, 2.03 [15:39:48] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [15:52:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.17, 3.24, 3.86 [15:56:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.65, 1.85, 3.14 [16:08:57] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:22:04] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1132533848 secs [16:38:13] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [16:44:04] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08180576563 secs [17:09:49] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.16, 10.33, 6.38 [17:11:49] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.94, 8.25, 6.10 [17:36:04] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1154346466 secs [17:38:46] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:50:47] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [17:52:04] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07386016846 secs [17:52:38] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/commit/fe87441d403c [17:52:39] [02miraheze/ManageWiki] 07dependabot[bot] 03fe87441 - Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 [17:52:41] [02ManageWiki] 07dependabot[bot] opened pull request 03#412: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/ManageWiki/pull/412 [17:52:43] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 - 13https://github.com/miraheze/ManageWiki [17:52:44] [02ManageWiki] 07dependabot[bot] labeled pull request 03#412: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/ManageWiki/pull/412 [17:52:45] [02ManageWiki] 07dependabot[bot] labeled pull request 03#412: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/ManageWiki/pull/412 [17:58:33] miraheze/ManageWiki - dependabot[bot] the build has errored. [18:08:01] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:20:46] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [18:24:14] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.30, 3.76, 3.16 [18:25:35] PROBLEM - wiki.controlistes.fr - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.controlistes.fr. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:26:11] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.57, 3.69, 3.20 [18:33:59] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.98, 3.27, 3.22 [18:35:51] PROBLEM - wiki.site97.fr - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.site97.fr. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:39:12] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [18:41:43] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.44, 3.51, 3.33 [18:47:33] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.13, 3.64, 3.44 [18:49:30] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.15, 3.47, 3.40 [18:50:46] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [18:53:27] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.01, 3.49, 3.41 [18:55:09] RECOVERY - wiki.controlistes.fr - reverse DNS on sslhost is OK: SSL OK - wiki.controlistes.fr reverse DNS resolves to cp22.miraheze.org - CNAME OK [18:55:27] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.16, 2.96, 3.22 [19:05:17] RECOVERY - wiki.site97.fr - reverse DNS on sslhost is OK: SSL OK - wiki.site97.fr reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:08:28] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:20:46] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [19:23:50] RECOVERY - www.obeymewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'obeymewiki.com' will expire on Fri 01 Sep 2023 14:14:34 GMT +0000. [19:34:43] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.27, 4.34, 2.70 [19:37:43] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [19:42:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.90, 3.99, 3.22 [19:43:00] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1594704092 secs [19:46:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 16.53, 7.28, 4.51 [19:47:57] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7967 MB (5% inode=99%); [19:54:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.16, 3.03, 3.77 [19:55:44] PROBLEM - www.obeymewiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.obeymewiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:58:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.98, 2.47, 3.36 [20:09:09] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09350317717 secs [20:10:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.17, 4.41, 3.67 [20:19:34] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:19:55] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [20:20:18] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:30] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 0.248 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [20:21:55] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:22:13] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:29:37] PROBLEM - redkrayolawiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - redkrayolawiki.org All nameservers failed to answer the query. [20:46:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.04, 1.35, 3.71 [20:48:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.53, 1.27, 3.38 [20:59:28] PROBLEM - redkrayolawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for redkrayolawiki.org could not be found [21:06:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.55, 2.69, 2.43 [21:07:26] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2161984742 secs [21:08:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.26, 4.39, 3.09 [21:16:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.72, 2.96, 3.07 [21:21:24] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.009878367186 secs [21:26:37] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1228741705 secs [21:52:04] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1013123989 secs [21:56:37] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.07975947857 secs [22:07:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.11, 3.94, 2.69 [22:22:20] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2144221067 secs [22:22:47] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query zulaclub.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:22:56] [02miraheze/IncidentReporting] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 [+0/-0/±2] 13https://github.com/miraheze/IncidentReporting/commit/d77053feacfa [22:22:57] [02miraheze/IncidentReporting] 07dependabot[bot] 03d77053f - Bump stylelint-config-wikimedia from 0.14.0 to 0.16.0 [22:22:58] [02IncidentReporting] 07dependabot[bot] opened pull request 03#65: Bump stylelint-config-wikimedia from 0.14.0 to 0.16.0 - 13https://github.com/miraheze/IncidentReporting/pull/65 [22:23:00] [02IncidentReporting] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 - 13https://github.com/miraheze/IncidentReporting [22:23:02] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#65: Bump stylelint-config-wikimedia from 0.14.0 to 0.16.0 - 13https://github.com/miraheze/IncidentReporting/pull/65 [22:23:03] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#65: Bump stylelint-config-wikimedia from 0.14.0 to 0.16.0 - 13https://github.com/miraheze/IncidentReporting/pull/65 [22:23:04] [02IncidentReporting] 07dependabot[bot] commented on pull request 03#62: Bump stylelint-config-wikimedia from 0.14.0 to 0.15.0 - 13https://github.com/miraheze/IncidentReporting/pull/62#issuecomment-1636508244 [22:23:06] [02IncidentReporting] 07dependabot[bot] closed pull request 03#62: Bump stylelint-config-wikimedia from 0.14.0 to 0.15.0 - 13https://github.com/miraheze/IncidentReporting/pull/62 [22:23:09] [02miraheze/IncidentReporting] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.15.0 [22:23:10] [02IncidentReporting] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.15.0 - 13https://github.com/miraheze/IncidentReporting [22:24:04] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08901482821 secs [22:24:20] PROBLEM - cp32 NTP time on cp32 is CRITICAL: NTP CRITICAL: Offset 0.5116044283 secs [22:27:00] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.75, 2.48, 3.69 [22:27:43] miraheze/IncidentReporting - dependabot[bot] the build has errored. [22:28:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.88, 2.03, 3.38 [22:32:35] [02miraheze/CreateWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/commit/aaa563a685cf [22:32:37] [02miraheze/CreateWiki] 07dependabot[bot] 03aaa563a - Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 [22:32:39] [02CreateWiki] 07dependabot[bot] opened pull request 03#425: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/CreateWiki/pull/425 [22:32:40] [02CreateWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 - 13https://github.com/miraheze/CreateWiki [22:32:43] [02CreateWiki] 07dependabot[bot] labeled pull request 03#425: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/CreateWiki/pull/425 [22:32:44] [02CreateWiki] 07dependabot[bot] labeled pull request 03#425: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/CreateWiki/pull/425 [22:38:18] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.4971056879 secs [22:40:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.06, 3.74, 3.30 [22:42:18] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.000158727169 secs [22:42:47] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.60, 3.48, 3.28 [22:42:54] miraheze/CreateWiki - dependabot[bot] the build has errored. [22:44:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.08, 3.09, 3.17 [22:48:07] PROBLEM - drowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'drowiki.com' expires in 10 day(s) (Tue 25 Jul 2023 15:53:31 GMT +0000). [22:52:32] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [23:06:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.35, 3.44, 2.70 [23:08:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.04, 2.87, 2.58 [23:17:15] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.95, 11.69, 7.64 [23:18:07] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [23:19:13] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.13, 8.96, 7.11 [23:22:18] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [23:37:18] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.72, 3.17, 2.44 [23:39:18] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 12.43, 6.67, 3.80 [23:47:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.47, 3.39, 3.56 [23:49:09] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.28, 2.70, 3.28