[00:00:10] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.15, 4.80, 3.72 [00:04:10] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.18, 3.66, 3.54 [00:06:10] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.09, 3.64, 3.53 [00:08:10] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.67, 3.83, 3.62 [00:12:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.80, 2.89, 3.30 [00:30:37] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:30:38] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 22.20, 14.97, 9.40 [00:33:04] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.99, 3.02, 3.93 [00:34:41] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 8.06, 10.48, 8.80 [00:35:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.03, 4.65, 4.48 [00:35:20] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 18.94, 14.71, 9.26 [00:36:38] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 7.87, 9.68, 8.71 [00:39:20] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 6.13, 10.37, 8.75 [00:40:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.42, 3.15, 3.86 [00:41:20] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 5.58, 8.58, 8.28 [00:46:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.29, 3.17, 3.55 [00:48:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.27, 2.78, 3.36 [01:05:45] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.54, 4.13, 3.56 [01:07:43] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.53, 4.00, 3.59 [01:13:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.46, 2.71, 3.18 [01:26:10] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.13, 4.26, 3.20 [01:28:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.09, 3.81, 3.16 [01:30:10] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.66, 4.09, 3.34 [01:32:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 3.92, 3.37 [01:34:11] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.77, 3.40, 3.24 [01:38:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.89, 3.74, 3.40 [01:38:25] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.49, 4.04, 3.37 [01:40:11] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.48, 2.86, 3.12 [01:40:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.86, 3.27, 3.17 [01:46:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.94, 3.64, 3.41 [01:48:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.66, 2.82, 3.13 [01:58:17] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:07:36] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:16:30] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1426894665 secs [02:30:30] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07652583718 secs [02:36:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.51, 4.04, 2.92 [02:38:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.67, 3.94, 3.02 [02:44:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.66, 3.36, 3.05 [02:58:43] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:03:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.33, 4.33, 3.15 [03:06:51] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:07:12] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.42, 3.62, 3.12 [03:09:12] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.88, 3.21, 3.02 [03:10:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.66, 3.84, 3.33 [03:12:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.06, 3.18, 3.15 [03:14:09] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:19:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.54, 3.97, 3.36 [03:21:12] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.91, 3.31, 3.20 [03:27:37] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 8.87, 11.69, 8.01 [03:27:57] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [03:29:20] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 22.34, 21.09, 12.14 [03:29:37] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.83, 9.16, 7.52 [03:29:57] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 10.45, 12.65, 7.60 [03:31:55] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 3.62, 9.30, 6.96 [03:33:20] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 4.35, 11.67, 10.29 [03:35:20] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 5.46, 9.58, 9.69 [03:38:26] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [03:39:18] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.85, 4.19, 3.39 [03:41:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.24, 3.48, 3.22 [03:42:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.62, 3.11, 2.96 [03:43:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.45, 3.05, 3.09 [03:44:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.48, 3.49, 3.09 [03:46:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.46, 2.99, 2.95 [03:57:10] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:07:01] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.20, 3.93, 3.30 [04:10:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.47, 3.61, 3.33 [04:12:09] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:12:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.13, 3.15, 3.20 [04:15:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.32, 3.45, 2.84 [04:17:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.47, 3.12, 2.80 [04:20:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.30, 3.52, 3.23 [04:22:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.64, 3.29, 3.18 [04:28:20] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [04:32:45] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.39, 3.84, 3.34 [04:37:41] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:48:31] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.48, 13.70, 7.99 [04:49:36] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 13.69, 10.98, 5.91 [04:50:25] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 5.23, 10.51, 7.48 [04:51:31] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 5.39, 8.69, 5.66 [04:52:20] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 3.78, 8.35, 7.05 [04:57:34] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:00:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.12, 2.16, 3.84 [05:03:02] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [05:04:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.83, 2.98, 3.73 [05:06:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.98, 3.31, 3.76 [05:10:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.75, 4.77, 4.20 [05:16:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.54, 3.61, 3.97 [05:24:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.68, 2.22, 3.15 [05:32:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.15, 3.55, 3.34 [05:32:45] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:39:09] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.01, 3.43, 1.86 [05:40:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.06, 2.94, 3.18 [05:41:08] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.28, 3.10, 1.95 [06:02:25] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:06:22] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 8.70, 9.60, 6.08 [06:06:33] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:18:22] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.88, 6.99, 7.03 [06:20:22] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.20, 6.07, 6.68 [06:22:37] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:25:14] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:28:22] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 20.15, 15.19, 10.11 [06:37:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.37, 4.19, 3.00 [06:39:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.79, 3.34, 2.83 [06:50:37] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:03:42] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:06:22] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 0.82, 4.18, 7.52 [07:08:22] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.28, 2.89, 6.64 [07:33:22] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [07:37:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.04, 2.60, 1.74 [07:39:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.82, 2.60, 1.85 [08:07:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.70, 3.01, 2.05 [08:17:42] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.79, 3.64, 3.93 [08:21:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.22, 2.40, 3.35 [08:47:46] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 11.80, 9.53, 5.74 [08:49:45] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 3.73, 7.26, 5.37 [09:04:06] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1520896554 secs [09:08:12] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [09:24:07] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.05595073104 secs [09:37:50] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [09:55:35] PROBLEM - wiki.andreijiroh.uk.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.andreijiroh.uk.eu.org' expires in 15 day(s) (Sat 15 Jul 2023 09:42:22 GMT +0000). [10:05:34] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [10:35:15] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:04:56] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [11:14:06] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1146512628 secs [11:14:26] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/4d7b3216005f...d0949855655b [11:14:29] [02miraheze/CreateWiki] 07translatewiki 03d094985 - Localisation updates from https://translatewiki.net. [11:14:30] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/b5ce0c9fb9bf...70cc3e8cc54d [11:14:31] [02miraheze/landing] 07translatewiki 0370cc3e8 - Localisation updates from https://translatewiki.net. [11:14:32] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/387bc748324d...53661ce1a7df [11:14:34] [02miraheze/MirahezeMagic] 07translatewiki 0353661ce - Localisation updates from https://translatewiki.net. [11:15:27] miraheze/landing - translatewiki the build passed. [11:22:07] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.0894485414 secs [11:27:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2831f4da5afc [11:28:01] [02miraheze/puppet] 07paladox 032831f4d - matomo: Set proxy_client_headers [11:28:03] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [11:28:05] [02puppet] 07paladox opened pull request 03#3256: matomo: Set proxy_client_headers - 13https://github.com/miraheze/puppet/pull/3256 [11:28:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [11:28:08] [02puppet] 07paladox closed pull request 03#3256: matomo: Set proxy_client_headers - 13https://github.com/miraheze/puppet/pull/3256 [11:28:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/72c1b21ff95a...91f5c6f8b8cd [11:28:12] [02miraheze/puppet] 07paladox 0391f5c6f - matomo: Set proxy_client_headers (#3256) [11:28:15] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [11:36:32] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [11:37:29] miraheze/MirahezeMagic - translatewiki the build passed. [11:37:44] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.96, 3.19, 1.89 [11:39:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.33, 3.25, 2.07 [11:44:08] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [11:49:53] PROBLEM - db121 Backups SQL on db121 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864239 seconds old and 1879 bytes [12:12:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.79, 3.87, 2.77 [12:13:45] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:14:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.18, 2.97, 2.57 [12:35:51] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:36:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.71, 3.61, 2.47 [12:40:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.54, 3.01, 2.54 [12:43:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [12:44:08] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:49:37] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 19.13, 13.23, 7.81 [12:53:37] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.03, 9.46, 7.59 [13:05:32] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [13:12:47] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:13:00] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [13:44:34] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:50:43] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [14:05:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.44, 3.93, 2.42 [14:13:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.92, 3.46, 3.05 [14:14:13] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [14:14:38] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.51, 3.27, 1.84 [14:15:05] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.16, 2.75, 2.85 [14:16:37] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.15, 2.44, 1.71 [14:18:40] [02mw-config] 07thelevking commented on pull request 03#5259: Update isvwiki to CC BY-SA 4.0 - 13https://github.com/miraheze/mw-config/pull/5259#issuecomment-1613265679 [14:19:57] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:38:39] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 16.04, 12.05, 7.23 [14:39:38] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 18.58, 13.33, 7.66 [14:40:38] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 4.65, 9.20, 6.78 [14:41:37] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.39, 10.01, 7.13 [14:41:47] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.49, 3.81, 2.77 [14:43:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.47, 2.96, 2.58 [14:45:47] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:07:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.95, 3.11, 2.31 [15:09:30] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.73, 4.73, 3.01 [15:11:29] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.70, 3.77, 2.86 [15:13:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.92, 5.81, 3.80 [15:15:25] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [15:17:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.64, 3.32, 3.21 [15:24:59] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [15:45:02] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:48:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/fdfe707aacd7 [15:48:03] PROBLEM - bast121 NTP time on bast121 is WARNING: NTP WARNING: Offset 0.4368842244 secs [15:48:04] [02miraheze/puppet] 07paladox 03fdfe707 - mariadb: increase table cache size [15:48:05] [02puppet] 07paladox opened pull request 03#3257: mariadb: increase table cache size - 13https://github.com/miraheze/puppet/pull/3257 [15:48:08] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [15:53:11] [02puppet] 07paladox closed pull request 03#3257: mariadb: increase table cache size - 13https://github.com/miraheze/puppet/pull/3257 [15:53:12] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [15:53:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/91f5c6f8b8cd...d94831f28dfd [15:53:16] [02miraheze/puppet] 07paladox 03d94831f - mariadb: increase table cache size (#3257) [15:53:17] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [15:54:01] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:06:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.82, 3.28, 2.24 [16:12:53] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.36, 3.27, 2.85 [16:20:03] RECOVERY - bast121 NTP time on bast121 is OK: NTP OK: Offset -0.00865021348 secs [16:23:03] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [16:29:48] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 16.46, 13.35, 8.13 [16:31:38] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 15.44, 14.30, 8.25 [16:31:45] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 5.91, 10.53, 7.72 [16:33:37] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 4.81, 10.65, 7.64 [16:33:42] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 3.82, 8.33, 7.24 [20:07:11] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.59, 5.76, 5.53 [20:10:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.55, 3.77, 3.10 [20:14:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.25, 2.69, 2.83 [20:19:29] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:22:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.85, 3.53, 3.04 [20:24:06] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.123875618 secs [20:26:07] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.51, 3.67, 3.31 [20:28:30] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1043328047 secs [20:30:05] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 3.07, 3.16 [20:30:06] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07526922226 secs [20:36:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.17, 3.90, 3.37 [20:38:06] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1224080324 secs [20:40:30] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08219009638 secs [20:43:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.38, 3.53, 3.76 [20:44:06] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09346595407 secs [20:45:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.79, 2.63, 3.40 [20:55:11] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.404593467712402 seconds [21:05:17] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.3219400644 secs [21:07:17] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.02406811714 secs [21:07:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 11.62, 6.70, 4.33 [21:15:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.77, 3.16, 3.73 [21:19:37] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.35, 2.63, 3.38 [21:22:17] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:26:34] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [21:30:33] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [21:41:24] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.10, 3.61, 2.91 [21:42:08] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.08, 17.26, 10.38 [21:42:11] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 23.12, 22.08, 11.48 [21:43:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.15, 3.11, 2.82 [21:44:38] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 9.42, 12.19, 7.85 [21:45:59] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 4.35, 10.21, 8.99 [21:46:08] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 3.34, 11.79, 9.64 [21:46:38] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.91, 9.79, 7.48 [21:46:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c2f8b2296c00 [21:47:00] [02miraheze/puppet] 07paladox 03c2f8b22 - mariadb: increase table_definition_cache to 80k [21:47:01] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [21:47:03] [02puppet] 07paladox opened pull request 03#3259: mariadb: increase table_definition_cache to 80k - 13https://github.com/miraheze/puppet/pull/3259 [21:47:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [21:47:08] [02puppet] 07paladox closed pull request 03#3259: mariadb: increase table_definition_cache to 80k - 13https://github.com/miraheze/puppet/pull/3259 [21:47:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d94831f28dfd...2f357e2843ea [21:47:14] [02miraheze/puppet] 07paladox 032f357e2 - mariadb: increase table_definition_cache to 80k (#3259) [21:47:15] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [21:47:54] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.59, 8.43, 8.48 [21:48:06] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 3.28, 9.03, 8.89 [21:49:01] [02miraheze/puppet] 07paladox pushed 032 commits to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/80a4bc163346...d5882657264d [21:49:03] [02miraheze/puppet] 07paladox 03d588265 - Merge branch 'master' into paladox-patch-2 [21:49:06] [02puppet] 07paladox synchronize pull request 03#3258: mariadb: increase table cache to 100k - 13https://github.com/miraheze/puppet/pull/3258 [21:51:30] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:56:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.12, 3.75, 2.98 [21:58:11] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.11, 3.64, 3.01 [22:02:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.62, 3.44, 3.09 [22:04:11] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.85, 2.84, 2.91 [22:07:07] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.20, 5.14, 3.57 [22:13:04] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.72, 3.44, 3.41 [22:15:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.39, 2.76, 3.17 [22:28:56] RECOVERY - wiki.simorgh.me - reverse DNS on sslhost is OK: SSL OK - wiki.simorgh.me reverse DNS resolves to cp23.miraheze.org - CNAME OK [22:30:11] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.58, 3.20, 2.73 [22:32:11] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.32, 3.32, 2.84 [22:42:27] RECOVERY - wiki.simorgh.me - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.simorgh.me' will expire on Mon 24 Jul 2023 15:02:22 GMT +0000. [23:07:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.27, 5.70, 3.71 [23:13:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.53, 3.94, 3.57 [23:15:25] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.64, 3.12, 3.32 [23:22:46] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [23:23:02] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [23:37:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.80, 7.38, 4.48 [23:51:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.29, 3.49, 4.00 [23:52:27] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [23:52:46] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.50, 3.27, 2.75 [23:55:12] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.59, 3.87, 3.06 [23:56:27] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [23:57:12] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.48, 3.00, 2.84 [23:59:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.89, 2.46, 3.29