[00:02:26] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 8.92, 6.91, 3.10 [00:02:35] [02MatomoAnalytics] 07paladox synchronize pull request 03#106: Set a 24H cache expiry - 13https://github.com/miraheze/MatomoAnalytics/pull/106 [00:02:38] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/88b368943dd1...61deb1d51069 [00:02:41] [02miraheze/MatomoAnalytics] 07paladox 0361deb1d - Update MatomoAnalytics.php [00:04:57] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/61deb1d51069...5bd9a9d0648e [00:05:00] [02miraheze/MatomoAnalytics] 07paladox 035bd9a9d - Update MatomoAnalyticsHooks.php [00:05:01] [02MatomoAnalytics] 07paladox synchronize pull request 03#106: Set a 24H cache expiry - 13https://github.com/miraheze/MatomoAnalytics/pull/106 [00:05:20] miraheze/CreateWiki - paladox the build has errored. [00:07:40] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.03424459696 secs [00:08:26] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 2.31, 3.81, 2.89 [00:10:26] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 1.14, 2.83, 2.64 [00:11:44] miraheze/MatomoAnalytics - paladox the build passed. [00:13:11] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 7.32, 5.36, 3.15 [00:16:31] RECOVERY - ndgkb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ndgkb.nena.org' will expire on Tue 07 Nov 2023 20:47:37 GMT +0000. [00:21:05] RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp22.miraheze.org - CNAME OK [00:22:00] [02CreateWiki] 07paladox closed pull request 03#431: Set a 24H cache expiry - 13https://github.com/miraheze/CreateWiki/pull/431 [00:22:03] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-2 [00:22:04] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/251ae85e0144...925edbb2dec8 [00:22:07] [02miraheze/CreateWiki] 07paladox 03925edbb - Set a 24H cache expiry (#431) [00:22:10] [02CreateWiki] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/CreateWiki [00:23:53] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/925edbb2dec8...14dba174f5b4 [00:23:55] [02miraheze/CreateWiki] 07paladox 0314dba17 - Revert "Set a 24H cache expiry (#431)" [00:24:07] [02MatomoAnalytics] 07paladox closed pull request 03#106: Set a 24H cache expiry - 13https://github.com/miraheze/MatomoAnalytics/pull/106 [00:24:10] [02miraheze/MatomoAnalytics] 07paladox deleted branch 03paladox-patch-1 [00:24:11] [02MatomoAnalytics] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/MatomoAnalytics [00:30:06] PROBLEM - arcanewaterswiki.org - reverse DNS on sslhost is WARNING: NXDOMAIN: The DNS query name does not exist: arcanewaterswiki.org. [00:30:29] miraheze/CreateWiki - paladox the build has errored. [00:31:11] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.45, 1.51, 3.95 [00:31:40] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2191987038 secs [00:35:11] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.35, 0.84, 3.11 [00:49:43] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.82, 3.50, 3.08 [00:51:42] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.46, 3.13, 3.00 [00:55:00] PROBLEM - kb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:37:16 GMT +0000). [01:01:43] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.61, 3.19, 2.91 [01:02:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e358a6000c26...b617ebc5c871 [01:02:56] [02miraheze/ssl] 07MirahezeSSLBot 03b617ebc - Bot: Update SSL cert for kb.nena.org [01:03:39] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.91, 3.09, 2.91 [01:05:34] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.41, 3.35, 3.07 [01:07:33] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.67, 3.77, 3.26 [01:09:32] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.35, 3.64, 3.27 [01:11:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.17, 3.41, 3.15 [01:13:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.85, 3.27, 3.13 [01:17:29] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.29, 3.13, 3.21 [01:24:02] RECOVERY - kb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kb.nena.org' will expire on Tue 07 Nov 2023 20:31:38 GMT +0000. [01:41:47] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.73, 3.37, 3.10 [01:43:43] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.57, 3.05, 3.01 [01:55:00] PROBLEM - kb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:37:16 GMT +0000). [01:56:31] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.55, 3.37, 3.14 [01:56:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b617ebc5c871...394ad626eedf [01:56:58] [02miraheze/ssl] 07MirahezeSSLBot 03394ad62 - Bot: Update SSL cert for kb.nena.org [01:58:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.16, 3.25, 3.11 [02:19:42] PROBLEM - cp33 NTP time on cp33 is CRITICAL: NTP CRITICAL: Offset 0.5003714561 secs [03:01:51] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.73, 3.44, 3.31 [03:09:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.08, 3.59, 3.43 [03:11:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.00, 3.30, 3.34 [03:21:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.30, 3.42, 3.37 [03:31:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.88, 3.28, 3.36 [03:46:27] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.53, 3.42, 3.31 [04:01:52] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.09, 3.27, 3.37 [04:04:22] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13849 MB (10% inode=99%); [04:06:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ee5dddb8d984...8d1aaefbd848 [04:06:57] [02miraheze/mw-config] 07paladox 038d1aaef - Set wgWatchlistExpiry to true [04:08:02] miraheze/mw-config - paladox the build passed. [04:08:46] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 29.12, 12.84, 6.65 [04:09:06] !log [@mwtask141] starting deploy of {'config': True} to all [04:09:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:09:16] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [04:09:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:16:24] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.53, 3.44, 3.36 [04:18:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.99, 3.29, 3.31 [04:19:42] PROBLEM - ndgkb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ndgkb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:32:28 GMT +0000). [04:20:22] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7828 MB (5% inode=99%); [04:23:02] RECOVERY - kb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kb.nena.org' will expire on Tue 07 Nov 2023 20:31:38 GMT +0000. [04:23:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/bbc09fa35815 [04:23:38] [02miraheze/mw-config] 07paladox 03bbc09fa - Temporarily disable StopForumSpam [04:23:40] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [04:23:42] [02mw-config] 07paladox opened pull request 03#5295: Temporarily disable StopForumSpam - 13https://github.com/miraheze/mw-config/pull/5295 [04:24:06] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.57, 6.63, 7.00 [04:24:47] miraheze/mw-config - paladox the build passed. [04:26:01] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.43, 5.27, 6.46 [04:26:18] !log [@test131] starting deploy of {'config': True} to all [04:26:19] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [04:26:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:26:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:26:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/394ad626eedf...c6dd99d48efd [04:26:57] [02miraheze/ssl] 07MirahezeSSLBot 03c6dd99d - Bot: Update SSL cert for ndgkb.nena.org [04:27:59] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.64, 3.55, 3.29 [04:29:55] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.03, 3.46, 3.30 [04:31:51] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.66, 3.20, 3.22 [04:36:22] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 17203 MB (12% inode=99%); [04:39:55] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.90, 7.96, 6.92 [04:41:55] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 13.98, 8.49, 7.14 [04:43:55] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.49, 7.14, 6.81 [04:45:55] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 6.78, 6.61, 6.63 [04:46:22] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13789 MB (10% inode=99%); [04:46:58] [02mw-config] 07paladox closed pull request 03#5295: Temporarily disable StopForumSpam - 13https://github.com/miraheze/mw-config/pull/5295 [04:46:59] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query zulaclub.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:46:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8d1aaefbd848...588615700101 [04:47:01] [02miraheze/mw-config] 07paladox 035886157 - Temporarily disable StopForumSpam (#5295) [04:47:02] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [04:47:04] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [04:47:30] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [04:47:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:47:39] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [04:47:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:48:03] miraheze/mw-config - paladox the build passed. [04:56:22] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 18232 MB (13% inode=99%); [04:56:25] !log [@test131] starting deploy of {'config': True} to all [04:56:26] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [04:56:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:56:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:11:29] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.63, 3.50, 3.31 [05:13:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.70, 2.92, 3.13 [05:16:51] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [05:19:00] RECOVERY - ndgkb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ndgkb.nena.org' will expire on Tue 07 Nov 2023 20:47:37 GMT +0000. [05:29:29] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [05:59:30] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [06:09:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.97, 3.18, 2.75 [06:11:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.34, 3.15, 2.79 [06:15:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.50, 3.55, 3.06 [06:19:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.01, 3.27, 3.06 [06:23:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.33, 3.81, 3.33 [06:27:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.39, 3.65, 3.38 [06:28:26] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 2.83, 3.46, 1.82 [06:30:26] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.90, 2.54, 1.68 [06:31:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.30, 3.83, 3.50 [06:45:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.80, 3.98, 3.92 [06:47:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.55, 4.17, 4.00 [06:53:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.36, 3.84, 3.93 [06:55:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.42, 4.05, 4.00 [07:01:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.40, 3.72, 3.93 [07:07:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.68, 2.29, 3.20 [07:21:31] PROBLEM - ndgkb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ndgkb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:32:28 GMT +0000). [07:22:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c6dd99d48efd...1790846e0e3c [07:22:08] [02miraheze/ssl] 07MirahezeSSLBot 031790846 - Bot: Update SSL cert for ndgkb.nena.org [07:55:00] PROBLEM - kb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:37:16 GMT +0000). [08:01:05] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1790846e0e3c...ac6655d6e6c0 [08:01:08] [02miraheze/ssl] 07MirahezeSSLBot 03ac6655d - Bot: Update SSL cert for kb.nena.org [08:07:05] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.95, 2.74, 1.85 [08:09:05] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.99, 2.82, 1.99 [08:23:55] PROBLEM - elephants.ggreig.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406881332397461 seconds [08:39:25] RECOVERY - wiki.altilunium.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.altilunium.my.id' will expire on Wed 11 Oct 2023 05:49:31 GMT +0000. [08:47:08] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.67, 3.58, 2.98 [08:50:26] RECOVERY - ndgkb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ndgkb.nena.org' will expire on Tue 07 Nov 2023 20:47:37 GMT +0000. [08:50:59] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.57, 3.31, 3.03 [08:53:55] RECOVERY - elephants.ggreig.com - reverse DNS on sslhost is OK: SSL OK - elephants.ggreig.com reverse DNS resolves to cp23.miraheze.org - CNAME OK [09:11:27] PROBLEM - wiki.altilunium.my.id - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.altilunium.my.id and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:53:00] RECOVERY - kb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kb.nena.org' will expire on Tue 07 Nov 2023 20:31:38 GMT +0000. [10:06:38] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [10:35:17] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:41:01] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:49:40] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.0998467505 secs [10:56:00] PROBLEM - kb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:37:16 GMT +0000). [11:20:55] [02miraheze/PDFEmbed] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/PDFEmbed/compare/2ade94e28516...b753ff75ed96 [11:20:57] [02miraheze/PDFEmbed] 07translatewiki 03b753ff7 - Localisation updates from https://translatewiki.net. [11:21:01] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/f9465774e1e3...bbdda5e1ef58 [11:21:04] [02miraheze/MirahezeMagic] 07translatewiki 03bbdda5e - Localisation updates from https://translatewiki.net. [11:25:00] RECOVERY - kb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kb.nena.org' will expire on Tue 07 Nov 2023 20:31:38 GMT +0000. [11:27:08] miraheze/MirahezeMagic - translatewiki the build has errored. [11:28:20] miraheze/PDFEmbed - translatewiki the build passed. [11:36:38] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [11:39:01] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:05:17] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:12:31] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.62, 3.34, 2.70 [12:18:18] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.20, 3.23, 2.86 [12:24:07] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.36, 3.77, 3.19 [12:26:02] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.72, 3.71, 3.24 [12:27:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.08, 3.73, 3.29 [12:31:49] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.78, 3.75, 3.39 [12:33:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.09, 3.88, 3.48 [12:34:39] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:35:41] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.57, 3.78, 3.50 [12:37:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.49, 4.07, 3.64 [12:41:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.95, 3.68, 3.60 [12:47:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.12, 3.72, 3.63 [12:49:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.81, 3.76, 3.65 [12:53:30] PROBLEM - ndgkb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ndgkb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:32:28 GMT +0000). [12:53:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.65, 3.96, 3.74 [12:55:00] PROBLEM - kb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:37:16 GMT +0000). [12:59:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.04, 3.42, 3.65 [13:03:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.89, 2.58, 3.25 [13:24:00] RECOVERY - kb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kb.nena.org' will expire on Tue 07 Nov 2023 20:31:38 GMT +0000. [14:22:29] RECOVERY - ndgkb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ndgkb.nena.org' will expire on Tue 07 Nov 2023 20:47:37 GMT +0000. [14:29:30] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [14:41:02] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:51:11] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.altilunium.my.id All nameservers failed to answer the query. [14:59:30] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:09:33] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [15:21:13] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.altilunium.my.id could not be found [15:39:01] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:59:30] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [16:07:05] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.38, 3.07, 2.04 [16:11:05] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.63, 3.07, 2.29 [16:17:05] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.74, 3.58, 2.76 [16:18:30] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/49394ed54d5a [16:18:32] [02miraheze/CreateWiki] 07paladox 0349394ed - Use DeferredUpdates to generate wiki/database list [16:18:33] [02CreateWiki] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/CreateWiki [16:18:34] [02CreateWiki] 07paladox opened pull request 03#432: Use DeferredUpdates to generate wiki/database list - 13https://github.com/miraheze/CreateWiki/pull/432 [16:19:11] [02CreateWiki] 07paladox synchronize pull request 03#432: Use DeferredUpdates to generate wiki/database list - 13https://github.com/miraheze/CreateWiki/pull/432 [16:19:12] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/49394ed54d5a...ee4520700ced [16:19:15] [02miraheze/CreateWiki] 07paladox 03ee45207 - Update CreateWikiJson.php [16:19:25] [02CreateWiki] 07paladox synchronize pull request 03#432: Use DeferredUpdates to generate wiki/database list - 13https://github.com/miraheze/CreateWiki/pull/432 [16:19:27] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ee4520700ced...6ab0d18dba4d [16:19:29] [02miraheze/CreateWiki] 07paladox 036ab0d18 - Update CreateWikiJson.php [16:21:11] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.altilunium.my.id All nameservers failed to answer the query. [16:21:16] [02miraheze/CreateWiki] 07github-actions[bot] pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/6ab0d18dba4d...71dc592fcbcf [16:21:18] [02miraheze/CreateWiki] 07github-actions 0371dc592 - CI: lint code to MediaWiki standards [16:21:20] [02CreateWiki] 07github-actions[bot] synchronize pull request 03#432: Use DeferredUpdates to generate wiki/database list - 13https://github.com/miraheze/CreateWiki/pull/432 [16:22:52] miraheze/CreateWiki - paladox the build has errored. [16:23:05] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.87, 3.18, 2.86 [16:27:05] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.57, 3.87, 3.20 [16:29:05] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.01, 3.47, 3.13 [16:29:30] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [16:31:05] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.71, 3.98, 3.36 [16:33:05] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.10, 3.64, 3.31 [16:35:05] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.10, 3.92, 3.45 [16:36:59] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 788 bytes in 0.166 second response time [16:37:05] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.53, 4.00, 3.56 [16:38:40] [02CreateWiki] 07paladox synchronize pull request 03#432: Use DeferredUpdates to generate wiki/database list - 13https://github.com/miraheze/CreateWiki/pull/432 [16:38:42] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/71dc592fcbcf...18eb9f98e930 [16:38:45] [02miraheze/CreateWiki] 07paladox 0318eb9f9 - Update CreateWikiJson.php [16:39:00] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.705 second response time [16:44:25] miraheze/CreateWiki - paladox the build has errored. [16:45:05] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.87, 3.21, 3.36 [16:48:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/b36250044879 [16:48:28] [02miraheze/mw-config] 07paladox 03b362500 - Unset wgUseLocalMessageCache [16:48:30] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [16:48:31] [02mw-config] 07paladox opened pull request 03#5296: Unset wgUseLocalMessageCache - 13https://github.com/miraheze/mw-config/pull/5296 [16:49:33] miraheze/mw-config - paladox the build passed. [16:49:59] [02mw-config] 07paladox closed pull request 03#5296: Unset wgUseLocalMessageCache - 13https://github.com/miraheze/mw-config/pull/5296 [16:50:02] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [16:50:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/588615700101...007f98428d83 [16:50:07] [02miraheze/mw-config] 07paladox 03007f984 - Unset wgUseLocalMessageCache (#5296) [16:50:09] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [16:50:34] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [16:50:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:50:51] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 17s [16:50:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:51:03] miraheze/mw-config - paladox the build passed. [16:51:12] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.altilunium.my.id could not be found [16:56:39] !log [@test131] starting deploy of {'config': True} to all [16:56:40] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [16:56:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:05:05] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.18, 3.50, 3.32 [17:07:05] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.87, 3.15, 3.20 [17:21:26] PROBLEM - cp23 NTP time on cp23 is CRITICAL: NTP CRITICAL: Offset 0.5004669428 secs [17:41:01] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:57:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.50, 3.06, 2.39 [18:01:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.57, 2.57, 2.37 [18:07:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.60, 3.30, 2.68 [18:09:36] PROBLEM - wiki.susqu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.susqu.org' expires in 15 day(s) (Sat 26 Aug 2023 18:07:21 GMT +0000). [18:11:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.87, 3.38, 2.88 [18:13:28] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ac6655d6e6c0...1bc2fc048b09 [18:13:30] [02miraheze/ssl] 07MirahezeSSLBot 031bc2fc0 - Bot: Update SSL cert for wiki.susqu.org [18:18:44] PROBLEM - wiki.mc.marochuru.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mc.marochuru.net' expires in 15 day(s) (Sat 26 Aug 2023 18:15:12 GMT +0000). [18:23:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.68, 3.64, 3.26 [18:25:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1bc2fc048b09...1cf9c92685bb [18:25:05] [02miraheze/ssl] 07MirahezeSSLBot 031cf9c92 - Bot: Update SSL cert for wiki.mc.marochuru.net [18:25:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.90, 3.37, 3.20 [18:32:29] PROBLEM - wiki.rsf.world - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.rsf.world' expires in 15 day(s) (Sat 26 Aug 2023 18:07:37 GMT +0000). [18:39:02] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [18:40:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d128c0b65738...e3329e9885f7 [18:40:30] [02miraheze/puppet] 07paladox 03e3329e9 - Update dns.pp [18:41:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1cf9c92685bb...48ee836057b6 [18:41:23] [02miraheze/ssl] 07MirahezeSSLBot 0348ee836 - Bot: Update SSL cert for wiki.rsf.world [18:41:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/e3329e9885f7...34bdab270647 [18:41:42] [02miraheze/puppet] 07paladox 0334bdab2 - Rename recursor.conf.erb to recursor.conf.erb [18:41:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+1/-1/±0] 13https://github.com/miraheze/puppet/commit/d128c0b65738 [18:42:00] [02miraheze/puppet] 07paladox 03d128c0b - dns: set threads variable to how many cores there are [18:42:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.06, 3.49, 3.37 [18:44:23] [02puppet] 07paladox synchronize pull request 03#3373: dns: set threads variable to how many cores there are - 13https://github.com/miraheze/puppet/pull/3373 [18:44:27] [02puppet] 07paladox synchronize pull request 03#3373: dns: set threads variable to how many cores there are - 13https://github.com/miraheze/puppet/pull/3373 [18:45:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/48ee836057b6...1f301a18480d [18:45:01] [02miraheze/ssl] 07MirahezeSSLBot 031f301a1 - Bot: Add SSL cert for sonicpedia.org [18:45:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/34bdab270647...5ee3d01e0663 [18:45:05] [02miraheze/puppet] 07paladox 035ee3d01 - Update dns.pp [18:45:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/385878641cfc...8f3479dcc1c2 [18:45:57] [02miraheze/puppet] 07paladox 038f3479d - Fix [18:46:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://github.com/miraheze/puppet/compare/dbc12c40d41a...385878641cfc [18:46:15] [02miraheze/puppet] 07paladox 033858786 - dns: set threads variable to how many cores there are (#3373) [18:46:37] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.79, 3.34, 3.34 [18:46:45] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [18:47:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [18:47:35] [02puppet] 07paladox synchronize pull request 03#3373: dns: set threads variable to how many cores there are - 13https://github.com/miraheze/puppet/pull/3373 [18:49:05] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [18:49:11] PROBLEM - swiftobject112 Puppet on swiftobject112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:49:37] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:53:04] [02puppet] 07paladox opened pull request 03#3373: dns: set threads variable to how many cores there are - 13https://github.com/miraheze/puppet/pull/3373 [18:54:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8f3479dcc1c2...5bc59e6a1d19 [18:55:01] [02miraheze/puppet] 07paladox 035bc59e6 - Fix [18:55:13] [02puppet] 07paladox closed pull request 03#3373: dns: set threads variable to how many cores there are - 13https://github.com/miraheze/puppet/pull/3373 [18:58:04] PROBLEM - ndgkb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ndgkb.nena.org' expires in 14 day(s) (Fri 25 Aug 2023 18:32:28 GMT +0000). [19:01:27] RECOVERY - wiki.rsf.world - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rsf.world' will expire on Wed 08 Nov 2023 17:39:39 GMT +0000. [19:03:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/320e7a65bd86 [19:04:01] [02miraheze/puppet] 07paladox 03320e7a6 - dns: lower remotes-ringbuffer-entries to 1000 [19:04:02] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [19:04:20] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [19:04:45] [02puppet] 07paladox closed pull request 03#3374: dns: lower remotes-ringbuffer-entries to 1000 - 13https://github.com/miraheze/puppet/pull/3374 [19:05:28] [02puppet] 07paladox opened pull request 03#3374: dns: lower remotes-ringbuffer-entries to 1000 - 13https://github.com/miraheze/puppet/pull/3374 [19:06:44] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [19:06:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5bc59e6a1d19...13cc81275a7f [19:06:49] [02miraheze/puppet] 07paladox 0313cc812 - dns: lower remotes-ringbuffer-entries to 1000 (#3374) [19:08:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/13cc81275a7f...db34464b0442 [19:08:30] [02miraheze/puppet] 07paladox 03db34464 - dns: increase max-cache-ttl to 600 (10 mins) [19:11:02] PROBLEM - cloud12 Puppet on cloud12 is UNKNOWN: NRPE: Unable to read output [19:11:48] PROBLEM - cloud10 Puppet on cloud10 is UNKNOWN: NRPE: Unable to read output [19:11:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.13, 3.32, 3.00 [19:12:00] PROBLEM - mwtask141 PowerDNS Recursor on mwtask141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [19:12:05] PROBLEM - bast121 Puppet on bast121 is UNKNOWN: NRPE: Unable to read output [19:12:07] PROBLEM - bast121 PowerDNS Recursor on bast121 is CRITICAL: No response from DNS 127.0.0.1 [19:12:15] PROBLEM - cloud10 PowerDNS Recursor on cloud10 is CRITICAL: CRITICAL - Plugin timed out while executing system call [19:12:22] PROBLEM - cloud12 PowerDNS Recursor on cloud12 is CRITICAL: CRITICAL - Plugin timed out while executing system call [19:12:59] PROBLEM - mwtask141 Puppet on mwtask141 is UNKNOWN: NRPE: Unable to read output [19:13:56] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.32, 3.33, 3.04 [19:16:44] RECOVERY - wiki.mc.marochuru.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mc.marochuru.net' will expire on Wed 08 Nov 2023 17:21:20 GMT +0000. [19:17:02] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:03] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:17:49] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.19, 3.91, 3.35 [19:19:47] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.71, 3.80, 3.37 [19:21:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.96, 4.29, 3.61 [19:27:31] RECOVERY - ndgkb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ndgkb.nena.org' will expire on Wed 08 Nov 2023 06:21:59 GMT +0000. [19:27:42] [02puppet] 07paladox opened pull request 03#3375: dns: load ns2 first in forward-zones - 13https://github.com/miraheze/puppet/pull/3375 [19:27:46] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [19:28:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [19:29:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e8aeeb53c0ca [19:29:37] [02miraheze/puppet] 07paladox 03e8aeeb5 - dns: load ns2 first in forward-zones [19:31:14] [02puppet] 07paladox closed pull request 03#3375: dns: load ns2 first in forward-zones - 13https://github.com/miraheze/puppet/pull/3375 [19:31:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [19:32:09] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [19:33:32] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.67, 3.97, 3.84 [19:37:27] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.75, 4.28, 3.99 [19:38:14] RECOVERY - wiki.susqu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.susqu.org' will expire on Wed 08 Nov 2023 17:10:13 GMT +0000. [19:38:40] RECOVERY - mwtask141 PowerDNS Recursor on mwtask141 is OK: DNS OK: 0.160 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [19:38:59] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:39:13] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:39:17] RECOVERY - cloud10 PowerDNS Recursor on cloud10 is OK: DNS OK: 0.130 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [19:39:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.46, 4.00, 3.93 [19:39:27] RECOVERY - cloud12 PowerDNS Recursor on cloud12 is OK: DNS OK: 0.233 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [19:41:02] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:41:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0740f1b05c1b...5169e4b9d567 [19:41:06] [02miraheze/puppet] 07paladox 035169e4b - matomo: update to 4.15.1 [19:41:27] RECOVERY - bast121 PowerDNS Recursor on bast121 is OK: DNS OK: 0.047 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [19:41:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/db34464b0442...0740f1b05c1b [19:41:38] [02miraheze/puppet] 07paladox 030740f1b - Fix [19:41:42] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:52:25] [02puppet] 07paladox opened pull request 03#3376: matomo: run systemd timer for queuedtracking - 13https://github.com/miraheze/puppet/pull/3376 [19:52:55] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [19:52:56] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [19:54:07] PROBLEM - swiftproxy111 NTP time on swiftproxy111 is WARNING: NTP WARNING: Offset -0.1206197739 secs [19:54:22] [02puppet] 07paladox closed pull request 03#3376: matomo: run systemd timer for queuedtracking - 13https://github.com/miraheze/puppet/pull/3376 [19:56:59] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [19:57:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a4b1c3481790 [19:57:10] [02miraheze/puppet] 07paladox 03a4b1c34 - matomo: run systemd timer for queuedtracking [19:57:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5169e4b9d567...7c2479b67604 [19:57:51] [02miraheze/puppet] 07paladox 037c2479b - matomo: run systemd timer for queuedtracking (#3376) [20:01:35] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [20:04:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.09, 3.56, 3.54 [20:06:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.85, 3.61, 3.56 [20:08:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.50, 3.88, 3.66 [20:10:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.84, 3.49, 3.55 [20:16:58] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.59, 3.15, 3.39 [20:19:04] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.altilunium.my.id All nameservers failed to answer the query. [20:24:23] PROBLEM - graylog131 NTP time on graylog131 is WARNING: NTP WARNING: Offset -0.1174511015 secs [20:42:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.64, 3.45, 3.15 [20:43:10] PROBLEM - bast121 APT on bast121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:43:23] PROBLEM - cp23 APT on cp23 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:44:38] PROBLEM - cp22 APT on cp22 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:44:58] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.20, 3.36, 3.16 [20:48:00] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.altilunium.my.id could not be found [20:50:01] PROBLEM - cp32 APT on cp32 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:50:51] PROBLEM - bast141 APT on bast141 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:51:04] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:52:49] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:55:08] PROBLEM - cp33 APT on cp33 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:55:10] PROBLEM - mon141 APT on mon141 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (1 critical updates). [21:08:12] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:16:55] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.altilunium.my.id All nameservers failed to answer the query. [21:45:51] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.altilunium.my.id could not be found [21:58:23] RECOVERY - graylog131 NTP time on graylog131 is OK: NTP OK: Offset -0.07825490832 secs [22:06:54] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [22:34:46] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=riversidewiki --no-updates --username-prefix=wikia:tremontaine /home/macfan/ImportDump/riversidewiki-20230806220708.xml (START) [22:34:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:36:13] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:36:48] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=riversidewiki --no-updates --username-prefix=wikia:tremontaine /home/macfan/ImportDump/riversidewiki-20230806220708.xml (END - exit=0) [22:36:49] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=riversidewiki (START) [22:36:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:36:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:38:29] PROBLEM - graylog131 NTP time on graylog131 is WARNING: NTP WARNING: Offset -0.1349353492 secs [22:38:41] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=riversidewiki (END - exit=0) [22:38:42] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=riversidewiki --active --update (END - exit=0) [22:38:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:38:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:41:26] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=sonicpediawiki --no-updates --username-prefix=w /home/macfan/ImportDump/sonicpediawiki-20230807170823.xml (START) [22:41:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:42:03] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=sonicpediawiki --no-updates --username-prefix=w /home/macfan/ImportDump/sonicpediawiki-20230807170823.xml (END - exit=0) [22:42:04] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=sonicpediawiki (START) [22:42:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:42:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:47:41] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.78, 3.80, 2.85 [22:49:38]