[00:04:18] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 4.26, 2.71, 1.46 [00:05:15] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [00:06:18] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 3.97, 3.23, 1.81 [00:08:20] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 2.71, 3.05, 1.92 [00:08:34] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8648 MB (6% inode=99%); [00:15:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.2654885054 secs [00:22:00] [02CreateWiki] 07Universal-Omega opened pull request 03#393: Add a hook to change the cw_wikis table from being used - 13https://github.com/miraheze/CreateWiki/pull/393 [00:23:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.01488274336 secs [00:28:16] [02CreateWiki] 07Universal-Omega synchronize pull request 03#393: Add a hook to change the cw_wikis table from being used - 13https://github.com/miraheze/CreateWiki/pull/393 [00:29:59] miraheze/CreateWiki - Universal-Omega the build passed. [00:30:18] [02CreateWiki] 07Universal-Omega synchronize pull request 03#393: Add a hook to change the cw_wikis table from being used - 13https://github.com/miraheze/CreateWiki/pull/393 [00:32:03] miraheze/CreateWiki - Universal-Omega the build has errored. [00:32:29] [02CreateWiki] 07Universal-Omega synchronize pull request 03#393: Add a hook to change the cw_wikis table from being used - 13https://github.com/miraheze/CreateWiki/pull/393 [00:33:57] [02CreateWiki] 07Universal-Omega synchronize pull request 03#393: Add a hook to change the cw_wikis table from being used - 13https://github.com/miraheze/CreateWiki/pull/393 [00:34:03] miraheze/CreateWiki - Universal-Omega the build has errored. [00:34:56] miraheze/CreateWiki - Universal-Omega the build has errored. [00:35:53] [02CreateWiki] 07Universal-Omega synchronize pull request 03#393: Add a hook to change the cw_wikis table from being used - 13https://github.com/miraheze/CreateWiki/pull/393 [00:36:11] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.45, 6.48, 5.09 [00:36:38] miraheze/CreateWiki - Universal-Omega the build has errored. [00:38:11] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.57, 6.88, 5.41 [00:40:10] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.16, 6.77, 5.56 [00:42:21] [02CreateWiki] 07Universal-Omega opened pull request 03#394: Add config to disable the ManageInactiveWikis script - 13https://github.com/miraheze/CreateWiki/pull/394 [00:45:07] [02CreateWiki] 07Universal-Omega synchronize pull request 03#394: Add config to disable the ManageInactiveWikis script - 13https://github.com/miraheze/CreateWiki/pull/394 [00:46:06] miraheze/CreateWiki - Universal-Omega the build passed. [00:47:49] [02CreateWiki] 07Universal-Omega synchronize pull request 03#394: Add config to disable the ManageInactiveWikis script - 13https://github.com/miraheze/CreateWiki/pull/394 [00:48:06] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.33, 7.22, 6.05 [00:50:05] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.69, 7.75, 6.41 [00:50:08] miraheze/CreateWiki - Universal-Omega the build passed. [00:51:57] [02CreateWiki] 07Universal-Omega synchronize pull request 03#393: Add a hook to change the cw_wikis table from being used - 13https://github.com/miraheze/CreateWiki/pull/393 [00:52:04] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.60, 5.92, 5.91 [00:54:06] miraheze/CreateWiki - Universal-Omega the build passed. [00:56:33] miraheze/CreateWiki - Universal-Omega the build passed. [01:21:53] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.81, 6.05, 5.22 [01:23:52] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.39, 6.06, 5.34 [02:10:34] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7961 MB (5% inode=99%); [02:35:22] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.403826475143433 seconds [02:37:05] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [02:53:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/95c3122acf7c...ba69ddb1ef4a [02:54:00] [02miraheze/ssl] 07MirahezeSSLBot 03ba69ddb - Bot: Update SSL cert for readerswhoknow.paneidoverse.com [03:04:05] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gs.sidem.wiki All nameservers failed to answer the query. [03:06:40] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:16:44] RECOVERY - readerswhoknow.paneidoverse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'readerswhoknow.paneidoverse.com' will expire on Wed 12 Jul 2023 01:53:50 GMT +0000. [03:22:17] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.42, 5.95, 4.65 [03:24:18] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.85, 5.49, 4.66 [03:28:34] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 9597 MB (7% inode=99%); [03:31:02] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [03:33:14] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query sidem.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:36:15] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:41:30] PROBLEM - www.sidem.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query sidem.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:52:15] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1239976883 secs [03:59:01] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:03:10] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gs.sidem.wiki All nameservers failed to answer the query. [04:07:47] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:10:34] RECOVERY - www.sidem.wiki - reverse DNS on sslhost is OK: SSL OK - www.sidem.wiki reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:24:15] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.08004543185 secs [04:33:16] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.40646767616272 seconds [04:37:22] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [05:01:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1006375551 secs [05:03:12] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gs.sidem.wiki All nameservers failed to answer the query. [05:10:44] PROBLEM - www.sidem.wiki - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.405160903930664 seconds [05:17:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.0911988318 secs [05:33:14] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406825542449951 seconds [05:38:29] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:48:16] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1398005784 secs [05:50:15] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.02330610156 secs [06:00:16] PROBLEM - cp33 NTP time on cp33 is CRITICAL: NTP CRITICAL: Offset 0.5402417481 secs [06:02:15] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.007494926453 secs [06:08:04] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:08:47] PROBLEM - www.sidem.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.sidem.wiki All nameservers failed to answer the query. [06:38:05] PROBLEM - www.sidem.wiki - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.405871152877808 seconds [06:58:17] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.24, 5.07, 2.79 [07:00:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.33, 3.55, 2.50 [07:16:16] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.3544408083 secs [07:19:05] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [07:20:19] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1588892639 secs [07:44:15] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.02628755569 secs [07:48:09] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [07:52:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09747347236 secs [08:03:13] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gs.sidem.wiki All nameservers failed to answer the query. [08:08:07] RECOVERY - www.sidem.wiki - reverse DNS on sslhost is OK: SSL OK - www.sidem.wiki reverse DNS resolves to cp22.miraheze.org - CNAME OK [08:28:43] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.484233886 secs [08:33:12] RECOVERY - gs.sidem.wiki - reverse DNS on sslhost is OK: SSL OK - gs.sidem.wiki reverse DNS resolves to cp22.miraheze.org - CNAME OK [08:38:29] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.002947390079 secs [08:51:50] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [09:20:41] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [09:44:17] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.153446734 secs [09:49:03] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:10:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.05862504244 secs [10:47:07] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [10:56:15] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1292491853 secs [11:13:43] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/MirahezeMagic/compare/5a59909a890e...31bd6b671e93 [11:13:44] [02miraheze/MirahezeMagic] 07translatewiki 0331bd6b6 - Localisation updates from https://translatewiki.net. [11:14:15] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.07596579194 secs [11:17:02] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:21:50] miraheze/MirahezeMagic - translatewiki the build passed. [12:37:12] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:06:23] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:48:16] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1618884802 secs [14:20:16] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.07532161474 secs [14:34:17] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2446024716 secs [14:42:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.01906105876 secs [16:24:17] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.109282136 secs [16:28:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.09593483806 secs [16:38:55] PROBLEM - gametheorywiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [16:49:03] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [16:50:17] PROBLEM - gametheorywiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gametheorywiki.org reverse DNS resolves to any-in-2001-4860-4802-34--15.1e100.net [17:18:05] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [17:51:50] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [18:20:10] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [18:20:40] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:49:12] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:49:50] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:47:20] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:06:15] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1592158675 secs [20:16:55] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:19:21] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [20:20:15] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.05583509803 secs [20:24:15] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1311065257 secs [20:28:15] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09554114938 secs [20:30:07] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 0.55 ms [20:36:13] PROBLEM - Host cloud11 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:200) [20:54:28] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 1.39 ms [21:30:17] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.22, 4.46, 2.22 [21:32:18] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.91, 4.78, 2.61 [21:47:37] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [22:17:02] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:17:12] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [22:47:04] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [22:48:44] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:18:19] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [23:19:02] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [23:47:54] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [23:48:06] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:55:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1