[00:00:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.23, 4.73, 3.34 [00:02:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.79, 7.01, 6.38 [00:03:25] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 15.50, 7.23, 3.23 [00:04:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.44, 6.12, 6.14 [00:05:43] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wizardia.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:06:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.60, 3.97, 3.48 [00:07:31] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.47, 3.52, 1.96 [00:10:49] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.70, 3.39, 3.36 [00:11:01] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.62, 9.79, 9.58 [00:12:30] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.05, 3.74, 2.82 [00:14:51] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 6.01, 8.46, 9.16 [00:14:55] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 9 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [00:15:25] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.96, 7.90, 7.03 [00:16:18] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.18, 3.68, 3.02 [00:16:46] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.70, 3.51, 2.59 [00:17:25] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.82, 6.16, 6.49 [00:17:27] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.61, 3.92, 3.12 [00:18:42] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 1.81, 2.95, 2.50 [00:19:31] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.24, 3.28, 2.98 [00:20:06] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.56, 3.13, 2.96 [00:24:33] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.71, 3.31, 2.76 [00:26:29] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.31, 3.70, 2.97 [00:26:48] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.03, 10.38, 9.63 [00:27:37] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.51, 4.07, 3.56 [00:28:26] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.70, 3.61, 3.02 [00:28:43] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 7.30, 9.25, 9.31 [00:33:21] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.24, 3.82, 3.66 [00:34:14] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.89, 3.11, 2.98 [00:35:18] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.05, 3.96, 3.73 [00:41:44] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.98, 4.58, 3.46 [00:42:34] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.33, 9.72, 7.23 [00:43:34] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.28, 4.23, 3.33 [00:44:39] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:46:33] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.47, 9.55, 7.81 [00:48:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.02, 3.60, 3.99 [00:54:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.92, 6.59, 6.21 [00:56:11] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.40, 6.24, 6.14 [00:57:56] PROBLEM - graylog131 SSH on graylog131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:58:31] PROBLEM - mem131 PowerDNS Recursor on mem131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:58:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.02, 2.64, 3.29 [01:00:00] RECOVERY - graylog131 SSH on graylog131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [01:00:29] RECOVERY - mem131 PowerDNS Recursor on mem131 is OK: DNS OK: 0.043 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:07:13] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [01:08:23] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:11:13] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [01:11:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.55, 3.14, 3.87 [01:15:53] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.43, 3.25, 3.98 [01:18:51] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.12, 3.59, 3.33 [01:19:30] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.36, 2.55, 3.29 [01:20:57] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.86, 3.52, 3.33 [01:22:52] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.05, 3.75, 3.44 [01:23:44] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.11, 2.56, 3.39 [01:42:30] !log [@mwtask141] starting deploy of {'config': True} to all [01:42:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:44:10] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 102s [01:44:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:46:50] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.81, 3.39, 3.90 [01:50:48] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.28, 3.74, 3.93 [01:52:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.11, 3.45, 3.80 [01:54:51] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.57, 3.75, 3.86 [01:56:20] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.qadrishattari.xyz. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:56:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.17, 3.21, 3.65 [02:00:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.59, 3.54, 3.67 [02:03:43] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Sat 30 Dec 2023 06:22:18 GMT +0000. [02:06:23] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [02:08:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.87, 9.14, 7.20 [02:09:08] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [02:10:09] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.55, 6.17, 5.38 [02:10:10] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.73, 8.43, 7.17 [02:10:46] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.01, 9.50, 8.03 [02:11:48] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [02:12:06] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.28, 5.98, 5.39 [02:12:41] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.90, 9.58, 8.23 [02:16:41] PROBLEM - alphabirdswiki.gq - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - alphabirdswiki.gq All nameservers failed to answer the query. [02:22:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.29, 9.84, 8.76 [02:25:01] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.qadrishattari.xyz All nameservers failed to answer the query. [02:26:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 7.75, 9.66, 9.00 [02:33:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.06, 6.23, 5.67 [02:34:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.11, 3.44, 3.97 [02:35:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.65, 6.40, 5.80 [02:35:44] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wizardia.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:36:54] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.95, 3.88, 4.06 [02:38:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.90, 3.52, 3.91 [02:41:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:42:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.44, 1.84, 3.13 [02:46:37] PROBLEM - alphabirdswiki.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for alphabirdswiki.gq could not be found [02:50:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.26, 10.14, 9.36 [02:54:12] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query qadrishattari.xyz. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:00:24] RECOVERY - mon141 Backups Grafana on mon141 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 21 seconds old and 785 bytes [03:00:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 5.85, 9.17, 9.52 [03:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:11:16] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.40, 9.91, 9.32 [03:17:01] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.14, 9.81, 9.51 [03:24:42] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.69, 10.17, 9.73 [03:27:07] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikipedia.yogfront.ooo could not be found [03:34:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 10.02, 10.15, 9.95 [03:43:02] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 9 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [03:48:40] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [03:48:41] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/d44df6ee0277...d973fa2c2e57 [03:48:43] [02miraheze/RottenLinks] 07paladox 03d973fa2 - Update RottenLinksJob.php [03:51:01] miraheze/RottenLinks - paladox the build has errored. [04:03:43] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Sat 30 Dec 2023 06:22:18 GMT +0000. [04:11:00] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/d973fa2c2e57...f1ab350eeee8 [04:11:02] [02miraheze/RottenLinks] 07paladox 03f1ab350 - Update RottenLinksPager.php [04:11:03] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:12:01] PROBLEM - wiki.slimelab.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.slimelab.net' expires in 7 day(s) (Tue 19 Dec 2023 03:56:52 GMT +0000). [04:13:19] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/f1ab350eeee8...9b34ebe51f0e [04:13:22] [02miraheze/RottenLinks] 07paladox 039b34ebe - Update rottenlinks.sql [04:13:24] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:13:26] miraheze/RottenLinks - paladox the build has errored. [04:13:35] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.78, 10.23, 9.83 [04:14:46] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:14:47] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/9b34ebe51f0e...436c9b7fefa0 [04:14:50] [02miraheze/RottenLinks] 07paladox 03436c9b7 - Update CHANGELOG.md [04:15:30] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 10.06, 10.14, 9.85 [04:16:15] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:16:18] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/436c9b7fefa0...856ac8a75f3c [04:16:21] [02miraheze/RottenLinks] 07paladox 03856ac8a - Update extension.json [04:18:39] miraheze/RottenLinks - paladox the build has errored. [04:20:26] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:20:28] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/856ac8a75f3c...186c80db4045 [04:20:31] [02miraheze/RottenLinks] 07paladox 03186c80d - Update RottenLinksPager.php [04:22:46] miraheze/RottenLinks - paladox the build has errored. [04:23:08] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:23:11] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/186c80db4045...6191bb3b9549 [04:23:12] [02miraheze/RottenLinks] 07paladox 036191bb3 - Update RottenLinksPager.php [04:24:10] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.qadrishattari.xyz reverse DNS resolves to cp24.miraheze.org - CNAME OK [04:25:37] miraheze/RottenLinks - paladox the build has errored. [04:26:22] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:26:24] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/6191bb3b9549...e64da216cedc [04:26:26] [02miraheze/RottenLinks] 07paladox 03e64da21 - Update RottenLinksJob.php [04:28:02] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:28:05] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/e64da216cedc...07d4aabfd761 [04:28:08] [02miraheze/RottenLinks] 07paladox 0307d4aab - Update RottenLinksHooks.php [04:28:39] miraheze/RottenLinks - paladox the build has errored. [04:30:27] miraheze/RottenLinks - paladox the build has errored. [04:30:52] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.07, 10.09, 9.50 [04:32:47] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 10.20, 10.04, 9.55 [04:41:41] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:41:42] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/07d4aabfd761...5a1cd5739746 [04:41:44] [02miraheze/RottenLinks] 07paladox 035a1cd57 - Update RottenLinksPager.php [04:42:19] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/5a1cd5739746...dc428d7c70b3 [04:42:22] [02miraheze/RottenLinks] 07paladox 03dc428d7 - Update SpecialRottenLinks.php [04:42:24] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:43:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.84, 6.72, 6.32 [04:43:55] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:43:57] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/dc428d7c70b3...5c290b6a10ce [04:43:58] [02miraheze/RottenLinks] 07paladox 035c290b6 - Update RottenLinksHooks.php [04:44:46] miraheze/RottenLinks - paladox the build has errored. [04:45:15] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.03, 10.43, 9.96 [04:45:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.81, 6.52, 6.30 [04:46:33] miraheze/RottenLinks - paladox the build has errored. [04:47:51] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:47:54] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/5c290b6a10ce...2201caed1bb9 [04:47:56] [02miraheze/RottenLinks] 07paladox 032201cae - Update RottenLinksJob.php [04:48:55] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:48:57] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/2201caed1bb9...69e3806a2de9 [04:48:59] [02miraheze/RottenLinks] 07paladox 0369e3806 - Update RottenLinksJob.php [04:50:02] [02miraheze/RottenLinks] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/69e3806a2de9...d2d296fcfd1a [04:50:03] [02miraheze/RottenLinks] 07github-actions 03d2d296f - CI: lint code to MediaWiki standards [04:50:05] [02RottenLinks] 07github-actions[bot] synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:51:30] miraheze/RottenLinks - paladox the build has errored. [04:52:09] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [04:52:11] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/d2d296fcfd1a...1eae9f527000 [04:52:13] [02miraheze/RottenLinks] 07paladox 031eae9f5 - Update RottenLinksJob.php [04:57:12] miraheze/RottenLinks - paladox the build has errored. [04:58:40] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 6.26, 9.06, 9.90 [05:05:51] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wizardia.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:08:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.61, 9.60, 9.37 [05:12:13] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:12:44] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 9 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [05:15:11] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/d6ad4a3666bf [05:15:13] [02miraheze/mw-config] 07paladox 03d6ad4a3 - Fix $wgLockManagers for beta but also fix configuration for prod [05:15:14] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [05:15:15] [02mw-config] 07paladox opened pull request 03#5400: Fix $wgLockManagers for beta but also fix configuration for prod - 13https://github.com/miraheze/mw-config/pull/5400 [05:15:44] [02mw-config] 07paladox synchronize pull request 03#5400: Fix $wgLockManagers for beta but also fix configuration for prod - 13https://github.com/miraheze/mw-config/pull/5400 [05:15:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d6ad4a3666bf...a111498d8177 [05:15:46] [02miraheze/mw-config] 07paladox 03a111498 - Update FileBackend.php [05:16:10] miraheze/mw-config - paladox the build passed. [05:16:13] [02mw-config] 07paladox closed pull request 03#5400: Fix $wgLockManagers for beta but also fix configuration for prod - 13https://github.com/miraheze/mw-config/pull/5400 [05:16:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b503750cc79b...ea4a1fc30293 [05:16:19] [02miraheze/mw-config] 07paladox 03ea4a1fc - Fix $wgLockManagers for beta but also fix configuration for prod (#5400) [05:16:20] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [05:16:21] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [05:16:25] !log [paladox@test131] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@=al.miraheze.org [05:16:28] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [05:16:29] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [05:16:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:16:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:16:37] !log [paladox@test131] starting deploy of {'config': True} to all [05:16:38] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:16:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:16:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:16:48] miraheze/mw-config - paladox the build passed. [05:16:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:16:56] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [05:17:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:17:07] miraheze/mw-config - paladox the build passed. [05:17:17] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 20s [05:17:19] !log [paladox@mwtask141] starting deploy of {'config': True} to all [05:17:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:17:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:17:31] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 11s [05:17:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:18:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.12, 10.08, 9.87 [05:21:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.51, 6.69, 6.28 [05:22:20] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [05:22:23] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/1eae9f527000...448fb20ca738 [05:22:26] [02miraheze/RottenLinks] 07paladox 03448fb20 - Update RottenLinksHooks.php [05:22:27] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.82, 10.71, 10.16 [05:23:02] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.20, 6.06, 6.09 [05:25:43] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [05:25:44] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/448fb20ca738...088ef20802d9 [05:25:47] [02miraheze/RottenLinks] 07paladox 03088ef20 - Update RottenLinksJob.php [05:26:16] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query qadrishattari.xyz. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:26:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 13.09, 12.03, 10.81 [05:28:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.08, 11.64, 10.82 [05:31:19] miraheze/RottenLinks - paladox the build passed. [05:39:08] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [05:42:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 8.33, 9.22, 10.04 [05:48:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.31, 9.87, 9.93 [05:55:03] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.qadrishattari.xyz reverse DNS resolves to cp25.miraheze.org - CNAME OK [06:04:15] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.18, 3.76, 2.30 [06:10:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 13.52, 12.03, 11.07 [06:11:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:11:49] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [06:12:44] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:14:00] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.11, 3.86, 3.14 [06:15:13] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [06:17:51] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.11, 3.57, 3.16 [06:18:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.50, 11.95, 11.48 [06:19:47] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.70, 3.60, 3.21 [06:21:41] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.79, 3.27, 3.14 [06:24:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 5.94, 8.10, 9.91 [06:24:43] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [06:24:44] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/088ef20802d9...75d2b4df0bc3 [06:24:45] [02miraheze/RottenLinks] 07paladox 0375d2b4d - Update RottenLinksJob.php [06:30:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.45, 10.00, 10.12 [06:31:58] miraheze/RottenLinks - paladox the build passed. [06:33:56] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [06:41:24] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 9 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [06:45:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [06:50:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.09, 6.53, 6.08 [06:52:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.28, 5.94, 5.91 [06:55:05] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/75d2b4df0bc3...6eae1c544b35 [06:55:07] [02miraheze/RottenLinks] 07paladox 036eae1c5 - Update RottenLinksPager.php [06:55:09] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [06:55:48] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/6eae1c544b35...a670a5d0b42c [06:55:51] [02miraheze/RottenLinks] 07paladox 03a670a5d - Update RottenLinksJob.php [06:55:54] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [06:58:13] miraheze/RottenLinks - paladox the build has errored. [07:00:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 4.53, 8.27, 9.64 [07:00:37] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [07:00:38] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/a670a5d0b42c...c30858d0ac56 [07:00:39] [02miraheze/RottenLinks] 07paladox 03c30858d - Update RottenLinksPager.php [07:02:07] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [07:02:08] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/c30858d0ac56...e726d9c4a33b [07:02:11] [02miraheze/RottenLinks] 07paladox 03e726d9c - Update RottenLinksJob.php [07:02:53] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp25.miraheze.org - CNAME OK [07:03:13] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [07:03:14] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/e726d9c4a33b...59c4b4cad7aa [07:03:16] [02miraheze/RottenLinks] 07paladox 0359c4b4c - Update RottenLinksJob.php [07:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:08:22] miraheze/RottenLinks - paladox the build passed. [07:10:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.67, 9.43, 9.29 [07:12:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.98, 9.57, 9.36 [07:14:37] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [07:14:39] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/59c4b4cad7aa...f7ee68417662 [07:14:40] [02miraheze/RottenLinks] 07paladox 03f7ee684 - Update CHANGELOG.md [07:14:48] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [07:14:51] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/f7ee68417662...87bc6f98ff39 [07:14:54] [02miraheze/RottenLinks] 07paladox 0387bc6f9 - Update extension.json [07:15:33] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [07:15:36] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/87bc6f98ff39...653ea3ccd88a [07:15:38] [02miraheze/RottenLinks] 07paladox 03653ea3c - Update CHANGELOG.md [07:15:46] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [07:15:49] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/653ea3ccd88a...a9c48f1395cf [07:15:52] [02miraheze/RottenLinks] 07paladox 03a9c48f1 - Update extension.json [07:18:19] miraheze/RottenLinks - paladox the build has errored. [07:20:31] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/a9c48f1395cf...a83536b6d033 [07:20:34] [02miraheze/RottenLinks] 07paladox 03a83536b - Update mediawiki-tests.yml [07:20:35] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [07:23:58] miraheze/RottenLinks - paladox the build has errored. [07:25:11] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.27, 10.88, 10.03 [07:27:06] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.52, 10.67, 10.05 [07:33:44] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Sat 30 Dec 2023 06:22:18 GMT +0000. [07:34:39] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/a83536b6d033...a4f07cab4f95 [07:34:41] [02miraheze/RottenLinks] 07paladox 03a4f07ca - Update mediawiki-tests.yml [07:34:43] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [07:35:10] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [07:35:11] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/a4f07cab4f95...5dcdf46431c7 [07:35:12] [02miraheze/RottenLinks] 07paladox 035dcdf46 - Update mediawiki-tests.yml [07:38:01] miraheze/RottenLinks - paladox the build has errored. [07:39:08] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [07:43:41] PROBLEM - mw131 NTP time on mw131 is CRITICAL: connect to address 2a10:6740::6:403 port 5666: No route to hostconnect to host 2a10:6740::6:403 port 5666: No route to host [07:44:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 6.68, 2.11, 0.74 [07:45:43] RECOVERY - mw131 NTP time on mw131 is OK: NTP OK: Offset -0.006808489561 secs [07:46:52] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw134 [07:47:12] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw134 [07:47:24] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw134 [07:47:32] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw134 [07:48:27] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: connect to address 2a10:6740::6:410 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [07:49:07] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [07:49:18] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [07:49:24] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:49:29] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [07:50:26] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.377 second response time [07:50:44] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [07:51:20] PROBLEM - mw141 NTP time on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: Connection refusedconnect to host 2a10:6740::6:502 port 5666: Connection refused [07:51:24] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:53:24] RECOVERY - mw141 NTP time on mw141 is OK: NTP OK: Offset -0.004345685244 secs [07:54:41] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw141 [07:54:55] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw141 [07:54:55] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: connect to address 2a10:6740::6:502 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [07:55:00] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw141 [07:55:05] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw141.miraheze.org port 443 after 0 ms: Couldn't connect to server [07:55:23] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw141 [07:58:52] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:58:55] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.832 second response time [07:59:05] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/2 301 - 345 bytes in 0.009 second response time [08:00:30] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [08:00:41] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [08:00:41] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [08:00:48] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.267 second response time [08:01:15] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [08:05:04] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/5dcdf46431c7...044b0e1ac6ba [08:05:06] [02miraheze/RottenLinks] 07paladox 03044b0e1 - Update config.php [08:05:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.53, 10.16, 7.05 [08:05:08] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [08:06:00] [02RottenLinks] 07paladox edited pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [08:10:25] miraheze/RottenLinks - paladox the build passed. [08:10:35] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:11:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:12:47] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.78, 10.18, 8.34 [08:32:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.30, 10.17, 9.48 [08:34:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.68, 10.06, 9.53 [08:35:51] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wizardia.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:38:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.50, 7.22, 6.22 [08:40:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.95, 6.64, 6.13 [08:47:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.52, 7.03, 6.46 [08:49:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.38, 7.70, 6.75 [08:51:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.21, 7.24, 6.68 [08:53:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.31, 6.44, 6.44 [09:39:08] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [09:39:46] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 9 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [10:09:30] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:11:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:11:48] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [10:34:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.10, 9.38, 8.78 [11:00:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.54, 9.93, 10.06 [11:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:09:08] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [11:10:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.41, 10.09, 9.84 [11:26:19] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [11:38:41] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 9 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [11:40:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 7.32, 9.57, 10.15 [11:41:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [11:52:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.00, 9.66, 9.37 [11:55:04] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.qadrishattari.xyz reverse DNS resolves to cp25.miraheze.org - CNAME OK [12:02:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.29, 11.52, 10.46 [12:08:25] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:10:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 7.79, 10.42, 10.58 [12:10:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 6.37, 5.75 [12:12:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 7.74, 9.44, 10.19 [12:12:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.76, 6.11, 5.73 [12:15:13] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [12:15:15] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/landing/compare/f78e1d35ef6e...b45b88564ce8 [12:15:16] [02miraheze/landing] 07translatewiki 03b45b885 - Localisation updates from https://translatewiki.net. [12:16:05] miraheze/landing - translatewiki the build passed. [12:18:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.06, 10.12, 10.18 [12:30:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.19, 11.62, 10.98 [12:32:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.96, 11.70, 11.08 [12:36:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.27, 11.76, 11.23 [12:38:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 12.00, 11.84, 11.32 [12:38:55] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 9.78, 5.31, 2.76 [12:44:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.39, 3.12, 2.75 [12:45:14] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [12:45:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:48:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 5.72, 7.91, 9.77 [12:49:13] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [12:54:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.47, 10.26, 10.20 [12:56:59] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.qadrishattari.xyz All nameservers failed to answer the query. [13:07:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.43, 6.69, 6.34 [13:09:08] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [13:09:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.73, 6.12, 6.16 [13:25:39] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.qadrishattari.xyz reverse DNS resolves to cp25.miraheze.org - CNAME OK [13:26:14] [02miraheze/ImportDump] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-10.0.0 [+0/-0/±2] 13https://github.com/miraheze/ImportDump/commit/88cb4c89cd2b [13:26:16] [02miraheze/ImportDump] 07dependabot[bot] 0388cb4c8 - Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 [13:26:17] [02ImportDump] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-10.0.0 - 13https://github.com/miraheze/ImportDump [13:26:19] [02ImportDump] 07dependabot[bot] opened pull request 03#57: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/ImportDump/pull/57 [13:26:22] [02ImportDump] 07dependabot[bot] labeled pull request 03#57: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/ImportDump/pull/57 [13:26:23] [02ImportDump] 07dependabot[bot] labeled pull request 03#57: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/ImportDump/pull/57 [13:32:01] miraheze/ImportDump - dependabot[bot] the build has errored. [13:40:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.07, 11.44, 10.97 [13:41:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.76, 11.42, 11.02 [13:52:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.02, 11.37, 11.11 [13:54:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.67, 11.48, 11.18 [13:54:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.61, 6.86, 6.35 [13:56:22] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.89, 6.63, 6.33 [13:56:39] RECOVERY - wizardia.wiki - reverse DNS on sslhost is OK: SSL OK - wizardia.wiki reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [14:00:50] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.39, 3.07, 1.69 [14:02:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 6.50, 8.22, 9.86 [14:02:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.54, 2.64, 1.72 [14:03:43] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Sat 30 Dec 2023 06:22:18 GMT +0000. [14:07:20] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 9 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [14:10:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.10, 9.91, 9.86 [14:24:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.23, 11.53, 10.86 [14:26:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.72, 11.59, 10.96 [14:37:04] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:40:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 7.47, 4.37, 2.42 [14:42:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.99, 3.53, 2.36 [14:44:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.51, 2.46, 2.11 [14:53:32] PROBLEM - kirisame-kissaten.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kirisame-kissaten.cf' expires in 15 day(s) (Wed 27 Dec 2023 14:45:38 GMT +0000). [14:53:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/01dc698e2f6b...bc42146b1369 [14:53:53] [02miraheze/ssl] 07MirahezeSSLBot 03bc42146 - Bot: Update SSL cert for kirisame-kissaten.cf [14:56:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.35, 11.80, 11.20 [14:58:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.75, 11.75, 11.26 [15:06:37] PROBLEM - wiki.mxlinuxusers.de - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mxlinuxusers.de All nameservers failed to answer the query. [15:06:48] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 9 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [15:08:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 6.59, 7.86, 9.75 [15:12:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.20, 10.08, 10.27 [15:20:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.49, 9.86, 10.20 [15:24:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.18, 10.55, 10.39 [15:28:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.91, 8.35, 6.70 [15:30:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.46, 7.50, 6.59 [15:34:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.94, 6.54, 6.39 [15:36:33] RECOVERY - wiki.mxlinuxusers.de - reverse DNS on sslhost is OK: SSL OK - wiki.mxlinuxusers.de reverse DNS resolves to cp25.miraheze.org - CNAME OK [15:38:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.26, 11.75, 11.15 [15:40:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.85, 11.56, 11.17 [15:44:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 5.72, 8.32, 9.92 [15:50:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.79, 9.59, 9.80 [15:51:41] RECOVERY - kirisame-kissaten.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'kirisame-kissaten.cf' will expire on Sun 10 Mar 2024 13:53:38 GMT +0000. [15:54:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.28, 7.40, 6.65 [15:56:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.25, 6.85, 5.87 [15:58:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.82, 7.45, 6.95 [16:00:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.89, 6.79, 6.76 [16:02:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.16, 11.72, 10.91 [16:02:57] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 3.48, 5.61, 5.73 [16:06:15] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:06:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.36, 11.84, 11.16 [16:22:02] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.77, 3.45, 2.69 [16:22:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.54, 6.45, 5.79 [16:24:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 21.18, 13.28, 11.82 [16:24:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 7.12, 4.11, 2.65 [16:24:57] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.86, 6.18, 5.77 [16:25:52] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [16:26:02] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.30, 3.04, 2.73 [16:28:01] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 9.961 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [16:28:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.70, 3.63, 2.83 [16:30:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.97, 2.66, 2.57 [16:38:02] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.20, 3.47, 3.08 [16:38:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.61, 11.39, 11.96 [16:39:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.30, 7.08, 6.67 [16:40:02] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.20, 3.09, 2.99 [16:40:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.48, 11.63, 11.96 [16:41:13] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [16:43:09] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.98, 6.46, 6.51 [16:43:18] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:07] PROBLEM - mwtask141 SSH on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:33] PROBLEM - mwtask141 PowerDNS Recursor on mwtask141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [16:44:48] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.16, 4.11, 3.13 [16:45:13] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [16:46:49] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:46:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.89, 3.42, 2.98 [16:47:07] PROBLEM - mwtask141 APT on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:47:16] PROBLEM - mwtask141 HTTPS on mwtask141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [16:48:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.09, 3.08, 2.92 [16:49:04] PROBLEM - mwtask141 NTP time on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:34] PROBLEM - mwtask141 ferm_active on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:35] PROBLEM - mwtask141 JobRunner Service on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:39] PROBLEM - mwtask141 php-fpm on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:40] PROBLEM - mwtask141 Disk Space on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:50:15] PROBLEM - mwtask141 conntrack_table_size on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:52:37] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 33858MiB (47% inode=90%); [16:52:40] RECOVERY - mwtask141 SSH on mwtask141 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [16:52:40] RECOVERY - mwtask141 php-fpm on mwtask141 is OK: PROCS OK: 7 processes with command name 'php-fpm8.2' [16:52:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.41, 9.43, 7.67 [16:52:53] RECOVERY - mwtask141 conntrack_table_size on mwtask141 is OK: OK: nf_conntrack is 0 % full [16:53:26] RECOVERY - mwtask141 APT on mwtask141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:53:32] RECOVERY - mwtask141 HTTPS on mwtask141 is OK: HTTP OK: HTTP/2 301 - 353 bytes in 3.181 second response time [16:54:20] [02miraheze/MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-10.0.0 [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/commit/de583f8f32d6 [16:54:23] [02miraheze/MatomoAnalytics] 07dependabot[bot] 03de583f8 - Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 [16:54:26] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-10.0.0 - 13https://github.com/miraheze/MatomoAnalytics [16:54:29] RECOVERY - mwtask141 NTP time on mwtask141 is OK: NTP OK: Offset -0.00112041831 secs [16:54:29] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#114: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/114 [16:54:31] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#114: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/114 [16:54:34] RECOVERY - mwtask141 JobRunner Service on mwtask141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [16:54:34] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#114: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/114 [16:54:46] RECOVERY - mwtask141 ferm_active on mwtask141 is OK: OK ferm input default policy is set [16:54:48] RECOVERY - mwtask141 PowerDNS Recursor on mwtask141 is OK: DNS OK: 0.107 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [16:54:48] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.88, 3.75, 3.22 [16:55:31] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.91, 7.12, 6.34 [16:56:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.28, 7.83, 7.41 [16:57:27] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.81, 6.66, 6.26 [16:58:57] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 18 minutes ago with 0 failures [16:59:42] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.110 second response time [16:59:45] miraheze/MatomoAnalytics - dependabot[bot] the build passed. [17:00:51] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.35, 3.42, 3.31 [17:04:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.36, 6.12, 6.79 [17:06:49] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.78, 2.77, 3.11 [17:08:02] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 5.22, 3.41, 2.93 [17:10:02] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.60, 3.39, 2.98 [17:10:31] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/044b0e1ac6ba...23cb98782005 [17:10:32] [02miraheze/RottenLinks] 07paladox 0323cb987 - Update updateExternalLinks.php [17:10:35] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [17:11:48] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [17:11:49] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/23cb98782005...353581a3fcb0 [17:11:50] [02miraheze/RottenLinks] 07paladox 03353581a - Update CHANGELOG.md [17:12:02] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.66, 3.02, 2.89 [17:14:17] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-10.0.0 [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/commit/e8b76a6833a1 [17:14:18] [02miraheze/ManageWiki] 07dependabot[bot] 03e8b76a6 - Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 [17:14:19] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-10.0.0 - 13https://github.com/miraheze/ManageWiki [17:14:22] [02ManageWiki] 07dependabot[bot] opened pull request 03#435: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/ManageWiki/pull/435 [17:14:24] [02ManageWiki] 07dependabot[bot] labeled pull request 03#435: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/ManageWiki/pull/435 [17:14:25] [02ManageWiki] 07dependabot[bot] labeled pull request 03#435: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/ManageWiki/pull/435 [17:17:01] miraheze/RottenLinks - paladox the build passed. [17:17:31] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/353581a3fcb0...a43c1c7a53a7 [17:17:32] [02miraheze/RottenLinks] 07paladox 03a43c1c7 - Update RottenLinksHooks.php [17:17:35] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [17:19:42] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/RottenLinks/compare/a43c1c7a53a7...76d44097ebf9 [17:19:43] [02miraheze/RottenLinks] 07paladox 0376d4409 - Create patch-drop-rl_pageusage.sql [17:19:44] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [17:19:46] miraheze/ManageWiki - dependabot[bot] the build passed. [17:25:12] miraheze/RottenLinks - paladox the build passed. [17:27:36] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:19] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw133 [17:28:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.81, 10.29, 7.92 [17:28:39] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.08, 10.45, 8.29 [17:30:19] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [17:30:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.29, 9.78, 8.01 [17:30:37] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.69, 9.58, 8.22 [17:31:38] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.026 second response time [17:35:34] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 9 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [17:38:23] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [17:39:08] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [17:42:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.09, 6.85, 6.34 [17:44:07] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.33, 6.35, 6.22 [17:46:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 2.27, 0.53, 0.18 [17:49:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.18, 7.69, 6.70 [17:51:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.30, 7.07, 6.59 [17:59:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.86, 6.58, 6.62 [18:05:02] PROBLEM - mwtask141 Puppet on mwtask141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [18:05:18] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:05:34] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.76, 9.92, 7.47 [18:07:32] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.23, 9.29, 7.54 [18:11:49] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [18:12:44] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 9 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [18:14:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.52, 6.84, 6.51 [18:14:54] PROBLEM - wiki.queenscourt.games - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.queenscourt.games' expires in 15 day(s) (Wed 27 Dec 2023 18:03:14 GMT +0000). [18:15:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bc42146b1369...436411db4748 [18:15:09] [02miraheze/ssl] 07MirahezeSSLBot 03436411d - Bot: Update SSL cert for wiki.queenscourt.games [18:16:59] !log increase mwtask141 ram by 2g (6g in total) [18:17:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:18:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.65, 6.67, 6.52 [18:22:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.23, 7.41, 6.94 [18:23:52] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:24:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f067e79bfad7 [18:24:58] [02miraheze/puppet] 07paladox 03f067e79 - jobrunner-hi: tune [18:25:14] [02puppet] 07paladox opened pull request 03#3563: jobrunner-hi: tune - 13https://github.com/miraheze/puppet/pull/3563 [18:25:18] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:23] PROBLEM - mwtask141 mathoid on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 and port 10044: Connection refused [18:25:23] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.36, 2.84, 2.15 [18:25:41] PROBLEM - mwtask141 PowerDNS Recursor on mwtask141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [18:25:46] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 5.78, 3.11, 1.76 [18:25:47] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [18:26:33] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw142 [18:26:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw142 mw143 [18:26:40] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 26.38, 16.86, 10.31 [18:26:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.46, 6.15, 6.55 [18:27:06] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.36, 10.24, 8.29 [18:27:09] PROBLEM - mwtask141 APT on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:27:19] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.85, 3.23, 2.38 [18:27:26] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.68, 9.69, 8.21 [18:27:46] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 1.89, 2.48, 1.69 [18:28:30] PROBLEM - mwtask141 HTTPS on mwtask141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [18:28:33] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [18:28:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [18:29:05] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.42, 8.62, 7.92 [18:29:13] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.30, 3.03, 2.42 [18:29:17] PROBLEM - mwtask141 SSH on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:29:22] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 5.62, 8.21, 7.85 [18:29:37] RECOVERY - mwtask141 APT on mwtask141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:29:38] PROBLEM - mwtask141 Puppet on mwtask141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 41 minutes ago with 0 failures [18:30:28] RECOVERY - mwtask141 HTTPS on mwtask141 is OK: HTTP OK: HTTP/2 301 - 353 bytes in 0.020 second response time [18:30:39] RECOVERY - mwtask141 PowerDNS Recursor on mwtask141 is OK: DNS OK: 0.077 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [18:31:15] RECOVERY - mwtask141 SSH on mwtask141 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [18:31:21] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.796 second response time [18:31:23] RECOVERY - mwtask141 mathoid on mwtask141 is OK: TCP OK - 0.003 second response time on 2a10:6740::6:504 port 10044 [18:35:01] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 9 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [18:35:54] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.50, 6.66, 5.83 [18:36:23] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [18:36:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.51, 6.92, 6.59 [18:37:14] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 3.62, 9.24, 11.23 [18:37:51] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.91, 6.37, 5.82 [18:38:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.23, 6.65, 6.52 [18:41:45] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:42:44] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:42:59] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 8.80, 8.35, 10.08 [18:44:45] RECOVERY - wiki.queenscourt.games - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.queenscourt.games' will expire on Sun 10 Mar 2024 17:15:00 GMT +0000. [18:48:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.71, 7.18, 6.70 [18:51:33] [02puppet] 07paladox closed pull request 03#3563: jobrunner-hi: tune - 13https://github.com/miraheze/puppet/pull/3563 [18:51:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0247dd209916...fa1e8bd34483 [18:51:36] [02miraheze/puppet] 07paladox 03fa1e8bd - jobrunner-hi: tune (#3563) [18:51:37] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [18:51:40] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [18:52:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.69, 6.48, 6.53 [19:00:58] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:04:45] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:10:23] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [19:21:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.97, 6.54, 6.21 [19:21:55] PROBLEM - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.rosestulipsandliberty.com' expires in 7 day(s) (Tue 19 Dec 2023 19:19:10 GMT +0000). [19:24:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.60, 6.14, 6.12 [19:34:29] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 9 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [19:35:42] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/90cdb6e535de [19:35:43] [02miraheze/mw-config] 07paladox 0390cdb6e - Don't set X-Wiki-Visibility header if using CLI [19:35:44] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [19:35:45] [02mw-config] 07paladox opened pull request 03#5401: Don't set X-Wiki-Visibility header if using CLI - 13https://github.com/miraheze/mw-config/pull/5401 [19:35:51] [02mw-config] 07paladox closed pull request 03#5401: Don't set X-Wiki-Visibility header if using CLI - 13https://github.com/miraheze/mw-config/pull/5401 [19:35:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ea4a1fc30293...35c52e32e4a9 [19:35:55] [02miraheze/mw-config] 07paladox 0335c52e3 - Don't set X-Wiki-Visibility header if using CLI (#5401) [19:35:57] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [19:35:59] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [19:36:30] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [19:36:41] miraheze/mw-config - paladox the build passed. [19:36:54] miraheze/mw-config - paladox the build passed. [19:36:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:38:00] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 90s [19:38:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:38:56] !log [paladox@mwtask141] starting deploy of {'config': True} to all [19:39:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:05] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [19:39:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:49:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.19, 7.05, 6.53 [19:51:09] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.15, 6.36, 6.34 [19:56:40] !log [@test131] starting deploy of {'config': True} to all [19:56:42] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [19:56:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.72, 7.01, 6.64 [20:01:39] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.83, 6.14, 6.39 [20:09:46] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [20:20:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.52, 7.00, 6.69 [20:24:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.99, 6.71, 6.67 [20:28:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.61, 7.95, 7.18 [20:28:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.03, 6.73, 6.04 [20:30:57] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.18, 6.04, 5.86 [20:32:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.32, 7.06, 7.05 [20:34:14] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [20:36:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.68, 6.37, 6.76 [20:41:10] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [20:56:49] !log [paladox@mwtask141] starting deploy of {'world': True} to all [20:57:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:31] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.26, 3.47, 2.83 [21:00:04] !log [paladox@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 195s [21:00:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:01:30] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.63, 3.30, 2.85 [21:04:14] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 9 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [21:05:32] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.28, 4.11, 3.28 [21:06:51] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.45, 4.02, 2.66 [21:07:28] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.53, 3.29, 2.36 [21:08:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.86, 3.52, 2.64 [21:09:28] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.02, 3.28, 2.48 [21:09:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.65, 3.61, 3.29 [21:09:48] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [21:11:31] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.50, 4.27, 3.56 [21:11:45] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.33, 10.09, 7.64 [21:12:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.26, 2.82, 2.60 [21:13:32] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.39, 4.22, 3.03 [21:13:43] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 5.39, 8.26, 7.26 [21:17:32] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.25, 4.00, 3.69 [21:21:28] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.49, 3.52, 3.29 [21:23:28] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.18, 3.10, 3.16 [21:27:30] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.01, 2.72, 3.23 [21:29:27] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.07, 3.47, 3.32 [21:33:36] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.92, 3.34, 3.32 [21:34:14] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [21:35:28] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.14, 3.15, 3.26 [21:35:30] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.62, 3.08, 3.23 [21:36:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.99, 6.99, 6.54 [21:38:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.12, 7.11, 6.62 [21:40:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.60, 7.25, 6.73 [21:41:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [21:44:01] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.66, 6.40, 6.50 [21:58:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.96, 6.78, 6.52 [22:00:12] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.52, 6.71, 6.53 [22:11:49] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [22:32:34] [02miraheze/CreateWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-10.0.0 [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/commit/7fc500b52cac [22:32:35] [02miraheze/CreateWiki] 07dependabot[bot] 037fc500b - Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 [22:32:37] [02CreateWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-10.0.0 - 13https://github.com/miraheze/CreateWiki [22:32:39] [02CreateWiki] 07dependabot[bot] opened pull request 03#454: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/CreateWiki/pull/454 [22:32:42] [02CreateWiki] 07dependabot[bot] labeled pull request 03#454: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/CreateWiki/pull/454 [22:32:43] [02CreateWiki] 07dependabot[bot] labeled pull request 03#454: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/CreateWiki/pull/454 [22:42:22] miraheze/CreateWiki - dependabot[bot] the build passed. [22:46:30] PROBLEM - lebork.info.pl - LetsEncrypt on sslhost is CRITICAL: connect to address lebork.info.pl and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [22:52:50] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.63, 4.02, 2.28 [22:56:49] [02miraheze/IncidentReporting] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-10.0.0 [+0/-0/±2] 13https://github.com/miraheze/IncidentReporting/commit/5e4f5ba87edc [22:56:51] [02miraheze/IncidentReporting] 07dependabot[bot] 035e4f5ba - Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 [22:56:54] [02IncidentReporting] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-10.0.0 - 13https://github.com/miraheze/IncidentReporting [22:56:56] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#72: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/IncidentReporting/pull/72 [22:56:59] [02IncidentReporting] 07dependabot[bot] opened pull request 03#72: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/IncidentReporting/pull/72 [22:57:01] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#72: Bump stylelint-config-idiomatic-order from 9.0.0 to 10.0.0 - 13https://github.com/miraheze/IncidentReporting/pull/72 [23:00:50] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.47, 3.95, 3.08 [23:02:30] miraheze/IncidentReporting - dependabot[bot] the build passed. [23:02:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.42, 4.40, 3.37 [23:04:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.51, 3.81, 3.28 [23:06:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.89, 3.05, 3.06 [23:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:11:15] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:11:33] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 14.95, 9.90, 6.92 [23:13:14] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.343 second response time [23:13:31] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.22, 8.78, 6.88 [23:15:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [23:16:01] RECOVERY - lebork.info.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'lebork.info.pl' will expire on Thu 04 Jan 2024 12:06:26 GMT +0000. [23:38:25] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:45:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'}