[00:02:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 46.64, 28.51, 11.59 [00:03:14] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 18.54, 18.41, 8.30 [00:28:10] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.58, 3.55, 2.77 [00:32:03] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.63, 3.66, 2.99 [00:33:59] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.13, 3.77, 3.11 [00:35:55] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.96, 3.79, 3.19 [00:37:19] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.67, 3.44, 2.93 [00:39:14] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.46, 0.63, 3.87 [00:39:16] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.35, 3.30, 2.93 [00:41:44] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.28, 3.97, 3.45 [00:42:14] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 0.26, 0.75, 3.65 [00:43:14] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.87, 0.78, 3.18 [00:44:14] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.44, 0.69, 3.27 [00:47:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.45, 3.94, 3.64 [01:09:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.92, 4.02, 3.76 [01:15:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.69, 3.86, 3.80 [01:25:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.23, 2.89, 3.37 [01:27:31] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:56:02] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:00:28] RECOVERY - db121 Backups SQL on db121 is OK: FILE_AGE OK: /var/log/sql-backup.log is 27 seconds old and 0 bytes [03:04:16] PROBLEM - cp32 Current Load on cp32 is WARNING: LOAD WARNING - total load average: 2.48, 3.52, 2.39 [03:06:15] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 1.29, 2.80, 2.26 [03:23:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:52:21] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query with.cpt-ra.bid. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:52:32] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [04:03:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:21:33] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:33:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [04:50:40] RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp22.miraheze.org - CNAME OK [05:12:33] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.69, 3.28, 2.63 [05:14:29] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.78, 3.08, 2.64 [05:34:53] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.36, 3.62, 3.15 [05:36:49] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.35, 3.44, 3.13 [05:52:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.38, 3.38, 3.33 [05:58:10] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.03, 4.15, 3.65 [06:28:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 5.29, 5.74, 2.76 [06:32:14] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 1.35, 3.16, 2.35 [07:17:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.75, 2.57, 3.97 [07:21:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.52, 1.95, 3.39 [07:23:23] PROBLEM - pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'pyramidgames.wiki' expires in 7 day(s) (Sun 27 Aug 2023 07:04:54 GMT +0000). [07:24:02] PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'pyramidgames.wiki' expires in 7 day(s) (Sun 27 Aug 2023 07:04:54 GMT +0000). [07:33:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [07:37:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.10, 3.27, 3.05 [07:53:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.75, 3.79, 3.86 [08:05:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.46, 3.00, 3.37 [09:03:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [09:53:02] PROBLEM - cloud13 Puppet on cloud13 is UNKNOWN: NRPE: Unable to read output [10:21:02] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:33:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [11:46:33] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.42, 3.46, 3.08 [11:48:29] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.13, 3.39, 3.10 [12:03:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:06:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.97, 4.06, 3.52 [12:16:02] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/864862451d8e...07d09e19b28c [12:16:04] [02miraheze/mw-config] 07paladox 0307d09e1 - Update LocalSettings.php [12:17:04] miraheze/mw-config - paladox the build passed. [12:18:34] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.05, 3.80, 3.87 [12:26:02] !log [@test131] starting deploy of {'config': True} to all [12:26:03] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [12:26:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:26:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.85, 3.78, 3.58 [12:39:53] !log [@mwtask141] starting deploy of {'config': True} to all [12:39:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:40:07] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 13s [12:40:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:41:49] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.96, 3.73, 3.64 [12:42:18] !log [paladox@mwtask141] sudo -u www-data php refreshLinks.php --wiki constantnoblewiki [12:42:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:43:46] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.37, 4.06, 3.78 [13:31:27] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_39 [+0/-0/±2] 13https://github.com/miraheze/mediawiki/compare/cd3465460bc6...885cd94f3c7f [13:31:28] [02miraheze/mediawiki] 07paladox 03885cd94 - Update DataTransfer [13:31:47] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [13:31:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:32:26] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±2] 13https://github.com/miraheze/mediawiki/compare/dbfb45035e04...2cb916fa1ba8 [13:32:27] [02miraheze/mediawiki] 07paladox 032cb916f - Update DataTransfer [13:32:37] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [13:32:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:36:20] !log [paladox@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [13:36:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:37:14] !log [paladox@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [13:37:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:37:46] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [13:37:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:38:00] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [13:38:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:55:18] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1038s [13:55:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:03:19] PROBLEM - os131 PowerDNS Recursor on os131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:04:33] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 10.42, 8.83, 6.29 [14:04:42] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 5.95, 4.26, 3.00 [14:05:18] PROBLEM - os131 Current Load on os131 is CRITICAL: CRITICAL - load average: 5.55, 3.81, 2.08 [14:05:18] RECOVERY - os131 PowerDNS Recursor on os131 is OK: DNS OK: 1.216 second response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [14:05:30] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.48, 9.58, 7.24 [14:06:33] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 8.82, 9.17, 6.75 [14:07:19] PROBLEM - os131 Current Load on os131 is WARNING: WARNING - load average: 3.38, 3.60, 2.21 [14:07:27] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.59, 9.68, 7.61 [14:09:19] RECOVERY - os131 Current Load on os131 is OK: OK - load average: 2.09, 3.05, 2.18 [14:10:42] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.24, 3.88, 3.28 [14:12:42] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 2.52, 3.40, 3.18 [14:23:14] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 2728s [14:23:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:53:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [15:02:03] PROBLEM - 100acg.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '100acg.wiki' expires in 15 day(s) (Mon 04 Sep 2023 14:36:17 GMT +0000). [15:07:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6a0e7915479b...df2be97acd1e [15:07:01] [02miraheze/ssl] 07MirahezeSSLBot 03df2be97 - Bot: Update SSL cert for 100acg.wiki [15:22:32] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:23:09] PROBLEM - wiki.esnmilanostatale.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.esnmilanostatale.it' expires in 15 day(s) (Mon 04 Sep 2023 15:02:17 GMT +0000). [15:26:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/df2be97acd1e...98ad667dcf6b [15:26:41] [02miraheze/ssl] 07MirahezeSSLBot 0398ad667 - Bot: Update SSL cert for wiki.esnmilanostatale.it [20:38:26] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/2cb916fa1ba8...161587a5f927 [20:38:29] [02miraheze/mediawiki] 07paladox 03161587a - Update Lingo [20:39:09] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [20:39:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:04] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 475s [20:47:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:03:12] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [21:33:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [21:37:39] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/161587a5f927...2833d4a61a26 [21:37:42] [02miraheze/mediawiki] 07paladox 032833d4a - Update SimpleTooltip [21:40:02] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/885cd94f3c7f...7d8365b2b393 [21:40:05] [02miraheze/mediawiki] 07paladox 037d8365b - Update SimpleTooltip [21:40:15] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [21:40:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:24] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 428s [21:47:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:50] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [21:47:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:06] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 436s [21:55:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:28:22] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/7d8365b2b393...dd64574677e8 [22:28:23] [02miraheze/mediawiki] 07paladox 03dd64574 - Update SocialProfile [22:29:06] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/2833d4a61a26...26c935976e38 [22:29:09] [02miraheze/mediawiki] 07paladox 0326c9359 - Update SocialProfile [22:29:23] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [22:29:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:29:43] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [22:29:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:36:52] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 428s [22:36:54] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 450s [22:36:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:37:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:30] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.15, 9.01, 4.14 [23:08:30] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.17, 7.02, 4.57 [23:10:30] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.58, 4.76, 4.04