[00:03:28] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 13.59, 7.22, 3.27 [00:09:20] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [00:09:57] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.81, 3.24, 2.26 [00:11:55] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.27, 2.80, 2.22 [00:15:28] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.07, 7.11, 6.73 [00:17:28] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.21, 6.26, 6.49 [00:19:17] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.07, 3.40, 2.49 [00:21:17] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.23, 3.02, 2.46 [00:24:06] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 11.66, 7.66, 5.83 [00:25:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.87, 4.00, 2.87 [00:25:08] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:25:21] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.32, 9.50, 6.72 [00:25:25] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.68, 4.16, 2.69 [00:25:37] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 13.13, 10.83, 8.08 [00:26:06] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 12.22, 10.39, 7.12 [00:27:01] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.79, 4.00, 3.02 [00:27:13] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 3.257 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:27:37] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.65, 11.25, 8.59 [00:28:06] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 7.72, 9.67, 7.28 [00:29:16] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.21, 3.96, 2.98 [00:29:21] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.65, 8.52, 7.00 [00:29:37] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.71, 9.79, 8.37 [00:30:57] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.16, 3.05, 2.86 [00:32:21] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.08, 6.71, 6.05 [00:34:19] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.64, 5.96, 5.86 [00:35:04] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.17, 3.35, 3.01 [00:38:54] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [00:39:37] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.13, 3.43, 2.73 [00:39:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 6.68, 6.26 [00:41:36] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.29, 3.38, 2.80 [00:41:43] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.44, 6.38, 6.19 [01:08:02] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:08:12] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.15, 3.86, 3.35 [01:08:27] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [01:08:51] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 20.67, 18.57, 11.69 [01:11:13] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [01:12:08] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.422 second response time [01:12:10] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.97, 3.18, 3.22 [01:14:50] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 6.67, 10.65, 10.30 [01:16:11] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.11, 3.84, 3.48 [01:16:50] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 6.66, 9.34, 9.86 [01:17:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5870a391ce57...da35ac951900 [01:17:13] [02miraheze/mw-config] 07paladox 03da35ac9 - Unset readTimeout [01:18:10] miraheze/mw-config - paladox the build passed. [01:18:11] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.32, 3.50, 3.42 [01:20:10] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.59, 2.50, 3.06 [01:20:37] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [01:20:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:20:56] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 19s [01:21:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:02] !log [paladox@mwtask141] starting deploy of {'config': True} to all [01:21:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:09] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [01:21:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.19, 6.34, 5.88 [01:25:58] !log [@test131] starting deploy of {'config': True} to all [01:25:59] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [01:26:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.76, 6.43, 6.11 [01:38:02] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:38:12] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.93, 3.42, 2.61 [01:39:59] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:41:13] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:41:53] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.083 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:42:10] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.89, 2.95, 2.70 [02:07:17] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.62, 3.54, 3.03 [02:09:06] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [02:09:17] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.71, 3.28, 3.00 [02:14:11] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.17, 6.61, 5.81 [02:16:09] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.25, 6.59, 5.91 [02:40:47] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:53:20] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 13.29, 10.15, 6.23 [02:55:17] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 5.24, 8.23, 5.99 [03:00:36] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:21:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.07, 6.04, 5.80 [03:23:41] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.23, 6.26, 5.93 [03:43:34] PROBLEM - elsterclopedie.marithaime.nl - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - elsterclopedie.marithaime.nl All nameservers failed to answer the query. [03:50:36] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [04:07:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:09:51] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [04:12:53] RECOVERY - elsterclopedie.marithaime.nl - reverse DNS on sslhost is OK: SSL OK - elsterclopedie.marithaime.nl reverse DNS resolves to cp25.miraheze.org - CNAME OK [04:37:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:40:13] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.37, 9.67, 6.85 [04:42:07] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.37, 8.17, 6.63 [05:11:12] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:11:18] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 2.72, 3.46, 3.24 [05:15:17] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.05, 3.08, 3.12 [06:07:04] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [06:10:36] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [06:23:58] PROBLEM - wiki.themarkside.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.themarkside.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:40:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.51, 7.37, 6.39 [06:44:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.95, 6.30, 6.19 [06:53:44] RECOVERY - wiki.themarkside.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.themarkside.com' will expire on Fri 16 Feb 2024 19:22:49 GMT +0000. [07:07:04] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [07:22:46] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [07:28:32] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 6951 MB (5% inode=99%); [07:29:28] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 9.12, 10.43, 5.98 [07:45:28] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.43, 7.53, 7.47 [07:49:28] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.92, 5.44, 6.67 [07:54:46] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:02:32] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 11731 MB (8% inode=99%); [08:03:28] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.89, 10.80, 8.33 [08:14:33] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7680 MB (5% inode=99%); [08:24:32] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13108 MB (9% inode=99%); [08:29:28] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.95, 5.93, 7.55 [08:33:28] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.38, 4.48, 6.62 [08:35:53] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 5.06, 3.47, 2.31 [08:37:52] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.35, 3.10, 2.32 [08:41:02] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:52:16] [1/2] I’ll teach 15 individuals to earn $50k or more within 72 hours but you will pay me 10% of your profit when you receive it. Note only interested people should apply, drop a message let's get started by asking (HOW) [08:52:16] [2/2] ‪WhatsApp: +1 (203) 935‑8366 [09:10:43] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [09:37:04] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [09:42:23] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:53:29] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [10:07:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [10:21:29] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:52:46] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [11:11:27] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [11:24:45] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [11:43:06] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:12:47] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [12:13:35] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/dfdba17d0191...a2633d6eec45 [12:13:38] [02miraheze/CreateWiki] 07translatewiki 03a2633d6 - Localisation updates from https://translatewiki.net. [12:23:23] miraheze/CreateWiki - translatewiki the build passed. [12:27:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.45, 6.63, 6.05 [12:29:10] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.53, 5.92, 5.87 [12:44:28] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:23:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.20, 6.81, 6.18 [13:25:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.55, 6.49, 6.12 [13:36:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.00, 6.90, 6.46 [13:38:37] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.73, 6.34, 6.30 [13:51:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.75, 7.10, 6.74 [13:53:29] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.59, 6.68, 6.64 [14:11:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.08, 6.85, 6.53 [14:15:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.17, 6.04, 6.28 [14:43:14] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [14:45:03] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 14.82, 13.32, 8.72 [14:47:00] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 4.78, 10.15, 8.11 [15:14:53] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:21:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.06, 6.89, 6.32 [15:23:00] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.54, 6.58, 6.29 [15:44:34] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [16:16:13] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:24:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.40, 6.53, 5.87 [16:28:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.87, 6.09, 5.86 [16:35:16] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.81, 6.87, 5.93 [16:36:08] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 5.39, 4.10, 3.35 [16:37:16] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.03, 6.28, 5.83 [16:38:06] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.26, 3.63, 3.27 [16:40:05] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.54, 3.14, 3.13 [17:08:45] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:37:23] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [18:22:51] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/da35ac951900...b72d63537505 [18:22:54] [02miraheze/mw-config] 07paladox 03b72d635 - Remove duplicate DeferredUpdates [18:23:46] miraheze/mw-config - paladox the build passed. [18:26:22] !log [@test131] starting deploy of {'config': True} to all [18:26:23] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [18:26:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:39:07] !log [@mwtask141] starting deploy of {'config': True} to all [18:39:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:39:21] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 13s [18:39:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:16:41] PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.410 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [19:22:47] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [19:46:10] RECOVERY - storytime.jdstroy.cf - reverse DNS on sslhost is OK: SSL OK - storytime.jdstroy.cf reverse DNS resolves to cp25.miraheze.org - CNAME OK [19:54:46] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [20:13:38] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.71, 6.77, 5.99 [20:15:36] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.93, 6.61, 6.04 [20:23:29] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [20:43:30] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [20:46:15] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=fiworldwiki --no-updates --username-prefix=wikia:dingxingpedia /home/paladox/ImportDump/fiworldwiki-20231115054854.xml (START) [20:46:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:32] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=fiworldwiki --no-updates --username-prefix=wikia:dingxingpedia /home/paladox/ImportDump/fiworldwiki-20231115054854.xml (END - exit=0) [20:47:33] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=fiworldwiki (START) [20:47:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:06] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=fiworldwiki (END - exit=0) [20:48:07] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=fiworldwiki --active --update (END - exit=0) [20:48:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:29] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=transformationwiki --no-updates --username-prefix=wikia:transformation /home/paladox/ImportDump/transformationwiki-20231115232746.xml (START) [20:48:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:37] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=dappledflightwiki --no-updates --username-prefix=wikia:dappledflight /home/paladox/ImportDump/dappledflightwiki-20231116005718.xml (START) [20:48:38] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=dappledflightwiki --no-updates --username-prefix=wikia:dappledflight /home/paladox/ImportDump/dappledflightwiki-20231116005718.xml (END - exit=0) [20:48:39] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=dappledflightwiki (START) [20:48:40] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=dappledflightwiki (END - exit=0) [20:48:41] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=dappledflightwiki --active --update (END - exit=0) [20:48:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:56] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=gamingmemorialswiki --no-updates --username-prefix=w /home/paladox/ImportDump/gamingmemorialswiki-20231116040236.xml (START) [20:48:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:29] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [21:08:34] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.51, 3.96, 2.54 [21:10:00] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=uottawampspwiki --no-updates --username-prefix=wikia:uottawa-mpsp /home/paladox/ImportDump/uottawampspwiki-20231117040223.xml (END - exit=0) [21:10:01] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=uottawampspwiki (START) [21:10:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:32] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.22, 3.88, 2.69 [21:10:32] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=zcrpwwiki --no-updates --username-prefix=mh:zcrp /home/paladox/ImportDump/zcrpwwiki-20231122182434.xml (END - exit=0) [21:10:33] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=zcrpwwiki (START) [21:10:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:19] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=gamingmemorialswiki (END - exit=0) [21:11:20] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=gamingmemorialswiki --active --update (END - exit=0) [21:11:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:29] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.59, 2.71, 2.41 [21:12:42] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=heavensarenawiki (END - exit=0) [21:12:43] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=heavensarenawiki --active --update (END - exit=0) [21:12:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:50] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=zcrpwwiki (END - exit=0) [21:12:51] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=zcrpwwiki --active --update (END - exit=0) [21:12:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:13:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:13:40] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=phightingwiki --no-updates --username-prefix=wikia:phighting /home/paladox/ImportDump/phightingwiki-20231123201326.xml (START) [21:13:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:13:54] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=animatedinanimatebattlewiki --no-updates --username-prefix=wikia:animatedinanimatebattle /home/paladox/ImportDump/animatedinanimatebattlewiki-20231125212536.xml (START) [21:13:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:20] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=animatedinanimatebattlewiki --no-updates --username-prefix=wikia:animatedinanimatebattle /home/paladox/ImportDump/animatedinanimatebattlewiki-20231125212536.xml (END - exit=0) [21:14:22] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=animatedinanimatebattlewiki (START) [21:14:23] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=robloxddaywiki --no-updates --username-prefix=wikia:d-day-roblox /home/paladox/ImportDump/robloxddaywiki-20231122211423.xml (END - exit=0) [21:14:23] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=robloxddaywiki (START) [21:14:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:32] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=uottawampspwiki (END - exit=0) [21:14:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:33] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=uottawampspwiki --active --update (END - exit=0) [21:14:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:37] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=animatedinanimatebattlewiki (END - exit=0) [21:14:38] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=animatedinanimatebattlewiki --active --update (END - exit=0) [21:14:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:12] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [21:15:29] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=pcohopwiki --no-updates --username-prefix=wikia:pcohop /home/paladox/ImportDump/pcohopwiki-20231123034709.xml (END - exit=0) [21:15:30] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=pcohopwiki (START) [21:15:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:47] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=robloxddaywiki (END - exit=0) [21:15:48] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=robloxddaywiki --active --update (END - exit=0) [21:15:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:48] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=pcohopwiki (END - exit=0) [21:18:49] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=pcohopwiki --active --update (END - exit=0) [21:18:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:28:19] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=phightingwiki --no-updates --username-prefix=wikia:phighting /home/paladox/ImportDump/phightingwiki-20231123201326.xml (END - exit=0) [21:28:20] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=phightingwiki (START) [21:28:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:28:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:34:49] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=phightingwiki (END - exit=0) [21:34:50] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=phightingwiki --active --update (END - exit=0) [21:34:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:34:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:37:04] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [21:41:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.27, 6.84, 6.23 [21:45:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.98, 6.04, 6.08 [21:54:46] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [21:59:01] !log install perccli_007.1910.0000.0000_all.deb (alternative to hpacucli) on cloud14 (installed from https://www.dell.com/support/home/en-ee/drivers/driversdetails?driverid=36g6n) [21:59:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:40] !log install perccli_007.1910.0000.0000_all.deb (alternative to hpacucli) on cloud13 (installed from https://www.dell.com/support/home/en-ee/drivers/driversdetails?driverid=36g6n) [22:06:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:24:37] PROBLEM - wiki.anglish.info - reverse DNS on sslhost is UNKNOWN: [22:31:57] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 17.10, 13.45, 5.67 [22:39:31] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 0.77, 3.51, 3.78 [22:43:25] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 1.20, 2.19, 3.17 [22:49:22] PROBLEM - mw131 Disk Space on mw131 is WARNING: DISK WARNING - free space: / 2448 MB (10% inode=79%); [22:49:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.07, 6.60, 6.23 [22:51:22] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.68, 6.31, 6.17 [23:34:28] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query.