[00:01:10] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [00:02:31] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.95, 9.42, 5.79 [00:03:07] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 13.46, 6.98, 3.22 [00:04:28] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 5.98, 7.72, 5.58 [00:04:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 14 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [00:13:07] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.59, 6.63, 5.90 [00:34:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:29:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [02:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 14 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [02:59:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [03:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:32:25] PROBLEM - test131 MediaWiki Rendering on test131 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [03:34:25] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.766 second response time [03:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 14 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [04:06:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/30b23cc778a1 [04:06:30] [02miraheze/mw-config] 07paladox 0330b23cc - Database: set load to 0 to indicate it's a master [04:06:32] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [04:06:33] [02mw-config] 07paladox opened pull request 03#5396: Database: set load to 0 to indicate it's a master - 13https://github.com/miraheze/mw-config/pull/5396 [04:07:02] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03paladox-patch-4 [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/30b23cc778a1...bca27a9c2ce0 [04:07:03] [02miraheze/mw-config] 07github-actions 03bca27a9 - CI: lint code to MediaWiki standards [04:07:06] [02mw-config] 07github-actions[bot] synchronize pull request 03#5396: Database: set load to 0 to indicate it's a master - 13https://github.com/miraheze/mw-config/pull/5396 [04:07:36] miraheze/mw-config - paladox the build passed. [04:09:07] [02mw-config] 07paladox closed pull request 03#5396: Database: set load to 0 to indicate it's a master - 13https://github.com/miraheze/mw-config/pull/5396 [04:09:10] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [04:09:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/405f5299a421...81ca2c3a1616 [04:09:13] [02miraheze/mw-config] 07paladox 0381ca2c3 - Database: set load to 0 to indicate it's a master (#5396) [04:09:15] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [04:09:21] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [04:09:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:09:34] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 12s [04:09:35] !log [paladox@mwtask141] starting deploy of {'config': True} to all [04:09:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:09:44] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [04:09:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:09:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:10:23] miraheze/mw-config - paladox the build passed. [04:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:26:07] !log [@test131] starting deploy of {'config': True} to all [04:26:08] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [04:26:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:26:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:26:40] PROBLEM - wikipwaadia.works - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikipwaadia.works' expires in 15 day(s) (Fri 22 Dec 2023 04:11:53 GMT +0000). [04:26:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8f19f32853a9...4f67a2c1790e [04:27:01] [02miraheze/ssl] 07MirahezeSSLBot 034f67a2c - Bot: Update SSL cert for wikipwaadia.works [04:34:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 14 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [04:45:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/3661ea9c1c62 [04:45:50] [02miraheze/mw-config] 07paladox 033661ea9 - Send "Cache-control: no-cache" header by default [04:45:53] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [04:45:54] [02mw-config] 07paladox opened pull request 03#5397: Send "Cache-control: no-cache" header by default - 13https://github.com/miraheze/mw-config/pull/5397 [04:45:57] [02mw-config] 07paladox closed pull request 03#5397: Send "Cache-control: no-cache" header by default - 13https://github.com/miraheze/mw-config/pull/5397 [04:45:58] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [04:46:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/81ca2c3a1616...943883fb20ba [04:46:01] [02miraheze/mw-config] 07paladox 03943883f - Send "Cache-control: no-cache" header by default (#5397) [04:46:02] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [04:46:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:46:05] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [04:46:09] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [04:46:10] !log [paladox@mwtask141] starting deploy of {'config': True} to all [04:46:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:46:16] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [04:46:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:46:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:46:48] miraheze/mw-config - paladox the build passed. [04:46:56] miraheze/mw-config - paladox the build passed. [04:55:51] RECOVERY - wikipwaadia.works - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipwaadia.works' will expire on Tue 05 Mar 2024 03:26:50 GMT +0000. [04:56:44] !log [@test131] starting deploy of {'config': True} to all [04:56:45] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [04:56:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:56:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:04:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:13:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/943883fb20ba...baae971771a6 [05:13:31] [02miraheze/mw-config] 07paladox 03baae971 - Set $wgSQLMode to null [05:13:35] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [05:13:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:13:49] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 14s [05:13:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:13:57] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [05:13:59] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [05:14:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:14:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:14:09] !log [paladox@test131] starting deploy of {'config': True} to all [05:14:10] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:14:11] !log [paladox@mwtask141] starting deploy of {'config': True} to all [05:14:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:14:18] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [05:14:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:14:23] miraheze/mw-config - paladox the build passed. [05:14:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:14:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:23:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/439a29ce44b1 [05:23:22] [02miraheze/mw-config] 07paladox 03439a29c - Disable LoadMonitor in CLI [05:23:25] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [05:23:27] [02mw-config] 07paladox opened pull request 03#5398: Disable LoadMonitor in CLI - 13https://github.com/miraheze/mw-config/pull/5398 [05:23:31] [02mw-config] 07paladox closed pull request 03#5398: Disable LoadMonitor in CLI - 13https://github.com/miraheze/mw-config/pull/5398 [05:23:33] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [05:23:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/baae971771a6...23090b2d55cf [05:23:35] [02miraheze/mw-config] 07paladox 0323090b2 - Disable LoadMonitor in CLI (#5398) [05:23:35] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [05:23:37] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [05:23:37] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [05:23:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:23:38] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [05:23:38] !log [paladox@test131] starting deploy of {'config': True} to all [05:23:39] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:23:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:23:43] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [05:23:46] !log [paladox@mwtask141] starting deploy of {'config': True} to all [05:23:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:23:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:23:54] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [05:23:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:24:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:24:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:24:25] miraheze/mw-config - paladox the build passed. [05:24:30] miraheze/mw-config - paladox the build passed. [05:47:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/23090b2d55cf...6c3bf8916f31 [05:47:27] [02miraheze/mw-config] 07paladox 036c3bf89 - Set $wgDLPMaxCacheTime to 7 days [05:47:27] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [05:47:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:46] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 18s [05:47:50] !log [paladox@mwtask141] starting deploy of {'config': True} to all [05:47:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:57] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [05:48:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:48:26] miraheze/mw-config - paladox the build passed. [05:56:11] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [05:56:12] !log [@test131] starting deploy of {'config': True} to all [05:56:13] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [05:56:14] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:56:15] !log [paladox@test131] starting deploy of {'config': True} to all [05:56:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:56:16] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:56:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:56:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:56:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:56:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:56:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:09:50] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [06:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 14 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [06:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:07:50] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:05:03] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 12.04, 10.11, 6.34 [09:07:03] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 5.21, 8.13, 6.07 [09:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 14 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [10:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:18:19] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.000343978405 secs [10:53:00] PROBLEM - wiki.bloxotel.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.bloxotel.tk' expires in 15 day(s) (Fri 22 Dec 2023 10:32:47 GMT +0000). [11:03:02] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.94, 10.09, 6.55 [11:05:02] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.31, 8.28, 6.33 [11:19:10] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [11:23:10] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [12:32:53] PROBLEM - db101 Backups SQL on db101 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864136 seconds old and 135805 bytes [12:34:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 14 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [13:04:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:00:19] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1015763879 secs [14:09:50] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [14:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 14 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [14:29:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [14:42:55] PROBLEM - swiftobject113 APT on swiftobject113 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [14:44:56] PROBLEM - swiftobject113 APT on swiftobject113 is CRITICAL: APT CRITICAL: 86 packages available for upgrade (32 critical updates). [14:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:59:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [15:07:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.81, 6.46, 5.69 [15:07:50] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:17] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.87, 6.02, 5.61 [15:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 14 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [15:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:00:44] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.35, 10.66, 7.04 [16:02:38] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.57, 8.91, 6.84 [16:04:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 14 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [16:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 14 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [16:34:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:34:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.56, 7.54, 6.12 [16:36:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 7.36, 6.67 [16:36:59] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.39, 6.76, 5.99 [16:37:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.60, 7.61, 6.83 [16:39:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.63, 7.20, 6.77 [16:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:49:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.12, 7.11, 6.84 [16:51:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.44, 6.99, 6.82 [17:29:23] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [17:35:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://github.com/miraheze/puppet/compare/c9e2246c3373...fafaa63ea1c5 [17:35:37] [02miraheze/puppet] 07paladox 03fafaa63 - Update es exporter to prometheus-es-exporter_0.11.1-2_all [17:38:34] PROBLEM - os131 APT on os131 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [17:41:17] PROBLEM - os131 Puppet on os131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[prometheus-es-exporter] [17:48:34] PROBLEM - os131 APT on os131 is CRITICAL: APT CRITICAL: 11 packages available for upgrade (9 critical updates). [17:52:35] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [17:52:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/fafaa63ea1c5...5aab58d91657 [17:52:40] [02miraheze/puppet] 07paladox 035aab58d - prometheus::exporter::elasticsearch: Make sure python3-opensearch is installed [17:53:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5aab58d91657...f7ba8cd210cb [17:53:43] [02miraheze/puppet] 07paladox 03f7ba8cd - Fix [17:55:18] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:59:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:18:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/294d4249a836 [18:18:34] [02miraheze/puppet] 07paladox 03294d424 - prometheus-es-exporter: use --addr to support ipv6 [18:18:37] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [18:18:41] [02puppet] 07paladox opened pull request 03#3550: prometheus-es-exporter: use --addr to support ipv6 - 13https://github.com/miraheze/puppet/pull/3550 [18:18:45] [02puppet] 07paladox closed pull request 03#3550: prometheus-es-exporter: use --addr to support ipv6 - 13https://github.com/miraheze/puppet/pull/3550 [18:18:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f7ba8cd210cb...ca52f2e21159 [18:18:49] [02miraheze/puppet] 07paladox 03ca52f2e - prometheus-es-exporter: use --addr to support ipv6 (#3550) [18:18:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [18:18:55] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [18:20:35] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:34:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 14 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [19:04:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:16:41] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 3 backends are down. mw141 mw142 mw133 [19:16:51] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw141 mw133 [19:17:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 16.83, 13.31, 8.23 [19:17:07] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 14.09, 11.49, 6.99 [19:17:07] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 25.53, 17.40, 9.75 [19:17:13] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw141 [19:17:17] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.67, 11.94, 7.91 [19:17:38] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 12.22, 11.69, 6.83 [19:17:46] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw141 [19:18:00] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 16.30, 14.26, 8.47 [19:18:01] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:18:11] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [19:18:13] PROBLEM - mw141 SSH on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:18:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.20, 6.41, 5.93 [19:19:02] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 7.10, 10.89, 7.95 [19:19:07] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 4.54, 8.61, 6.47 [19:19:14] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.63, 9.34, 7.43 [19:19:43] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 8.89, 10.23, 6.88 [19:20:06] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/2 301 - 345 bytes in 0.028 second response time [19:20:09] RECOVERY - mw141 SSH on mw141 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u2 (protocol 2.0) [19:20:17] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 55.68, 44.87, 21.89 [19:20:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.01, 7.30, 6.31 [19:20:52] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.51, 3.50, 2.43 [19:21:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.59, 11.50, 8.50 [19:21:39] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 7.68, 9.72, 7.13 [19:21:46] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [19:22:08] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.321 second response time [19:22:42] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [19:22:50] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.40, 3.49, 2.57 [19:22:51] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [19:23:02] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.80, 9.43, 8.12 [19:23:13] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [19:23:54] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 7.83, 12.00, 9.54 [19:24:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.38, 6.22, 6.11 [19:24:50] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.69, 3.31, 2.62 [19:25:03] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 5.63, 11.85, 10.52 [19:25:54] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.77, 9.52, 8.94 [19:27:03] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 3.37, 8.98, 9.64 [19:28:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.34, 6.79, 6.35 [19:33:49] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 2.80, 6.23, 11.69 [19:35:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.42, 6.46, 6.48 [19:37:49] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 3.67, 4.89, 9.89 [20:32:04] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.81, 6.64, 5.95 [20:34:03] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.27, 6.97, 6.11 [20:36:03] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.67, 6.77, 6.12 [20:38:02] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.45, 6.04, 5.92 [20:51:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.40, 6.55, 6.15 [20:53:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.67, 6.14, 6.04 [21:16:28] [02miraheze/MirahezeMagic] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/setup-python-5 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/85a7c1c17317 [21:16:31] [02miraheze/MirahezeMagic] 07dependabot[bot] 0385a7c1c - Bump actions/setup-python from 4 to 5 [21:16:34] [02MirahezeMagic] 07dependabot[bot] created branch 03dependabot/github_actions/actions/setup-python-5 - 13https://github.com/miraheze/MirahezeMagic [21:16:37] [02MirahezeMagic] 07dependabot[bot] opened pull request 03#445: Bump actions/setup-python from 4 to 5 - 13https://github.com/miraheze/MirahezeMagic/pull/445 [21:16:39] [02MirahezeMagic] 07dependabot[bot] labeled pull request 03#445: Bump actions/setup-python from 4 to 5 - 13https://github.com/miraheze/MirahezeMagic/pull/445 [21:16:40] [02MirahezeMagic] 07dependabot[bot] labeled pull request 03#445: Bump actions/setup-python from 4 to 5 - 13https://github.com/miraheze/MirahezeMagic/pull/445 [21:25:56] miraheze/MirahezeMagic - dependabot[bot] the build passed. [21:47:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.29, 6.98, 6.33 [21:51:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.76, 6.73, 6.40 [22:03:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.99, 6.34, 6.16 [22:05:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.23, 6.02, 6.07 [22:09:50] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [22:13:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.78, 7.04, 6.50 [22:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 13 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [22:15:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.01, 6.69, 6.44 [22:36:15] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.83, 6.67, 5.83 [22:38:14] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.00, 5.54, 5.51 [22:45:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.81, 6.88, 6.48 [22:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [22:47:00] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.13, 6.56, 6.40 [22:59:23] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [23:04:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 13 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [23:07:50] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:20:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.73, 8.15, 6.85 [23:22:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.46, 7.46, 6.74 [23:24:09] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.72, 6.77, 6.57 [23:29:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [23:34:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket