[00:00:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.93, 7.92, 7.22 [00:02:14] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=urgwiki --no-updates /home/macfan/ImportDump/urgwiki-20231105105104.xml (START) [00:02:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:35] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=urgwiki --no-updates /home/macfan/ImportDump/urgwiki-20231105105104.xml (END - exit=0) [00:03:36] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=urgwiki (START) [00:03:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:48] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 17.14, 8.23, 3.69 [00:03:53] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.17, 5.84, 4.70 [00:03:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.71, 7.63, 7.12 [00:05:06] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.86, 3.61, 3.17 [00:05:53] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 5.57, 5.67, 4.77 [00:05:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.37, 7.48, 7.13 [00:06:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.75, 7.85, 7.47 [00:07:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.05, 4.03, 3.37 [00:09:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.57, 3.49, 3.25 [00:09:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.94, 6.52, 6.79 [00:10:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.43, 7.47, 7.33 [00:11:05] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.68, 3.28, 3.21 [00:12:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.35, 7.01, 7.18 [00:13:48] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.20, 7.93, 6.83 [00:13:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.55, 7.04, 6.92 [00:15:20] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=urgwiki (END - exit=0) [00:15:21] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=urgwiki --active --update (END - exit=0) [00:15:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.46, 6.51, 6.75 [00:17:48] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.51, 5.62, 6.20 [00:22:02] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.45, 3.09, 3.07 [00:24:01] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.08, 2.65, 2.90 [00:32:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.83, 6.27, 6.76 [00:55:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.49, 7.12, 6.60 [00:57:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.89, 6.54, 6.45 [01:00:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.44, 6.74, 6.39 [01:02:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.00, 7.83, 6.81 [01:04:34] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.66, 6.60, 6.49 [01:14:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.59, 7.12, 6.64 [01:18:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 7.57, 6.92 [01:20:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.58, 7.77, 7.08 [01:24:07] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.55, 6.43, 6.67 [01:41:06] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:42:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.60, 6.90, 6.62 [01:46:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.12, 6.70, 6.61 [01:48:17] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 126.14 ms [01:54:14] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 119.87 ms [01:57:13] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.59, 3.57, 2.84 [01:58:17] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 125.31 ms [01:59:12] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.58, 3.11, 2.76 [02:09:40] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [02:29:59] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 118.54 ms [02:39:06] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [02:44:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.38, 6.73, 6.33 [02:46:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.86, 6.36, 6.24 [02:48:25] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=carlyraejepsenwiki --no-updates --username-prefix=wikia:carlyraejepsen /home/macfan/ImportDump/carlyraejepsenwiki-20231105131914.xml (START) [02:48:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:58] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 121.35 ms [02:54:57] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 116.75 ms [02:59:00] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 121.44 ms [02:59:15] PROBLEM - wiki.albrusmusic.ru - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [03:06:55] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 117.46 ms [03:19:18] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=carlyraejepsenwiki --no-updates --username-prefix=wikia:carlyraejepsen /home/macfan/ImportDump/carlyraejepsenwiki-20231105131914.xml (END - exit=0) [03:19:19] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=carlyraejepsenwiki (START) [03:19:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:19:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:22:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.34, 7.32, 6.68 [03:24:39] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.36, 6.53, 6.47 [03:28:05] RECOVERY - wiki.albrusmusic.ru - reverse DNS on sslhost is OK: SSL OK - wiki.albrusmusic.ru reverse DNS resolves to cp25.miraheze.org - CNAME OK [03:37:49] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=carlyraejepsenwiki (END - exit=0) [03:37:50] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=carlyraejepsenwiki --active --update (END - exit=0) [03:37:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:43:09] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=n01rfixationwiki --no-updates --username-prefix=wikia:n01r-fiction /home/macfan/ImportDump/n01rfixationwiki-20231106074018.xml (START) [03:43:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:44:37] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=n01rfixationwiki --no-updates --username-prefix=wikia:n01r-fiction /home/macfan/ImportDump/n01rfixationwiki-20231106074018.xml (END - exit=0) [03:44:38] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=n01rfixationwiki (START) [03:44:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:44:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:45:13] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=n01rfixationwiki (END - exit=0) [03:45:14] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=n01rfixationwiki --active --update (END - exit=0) [03:45:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:45:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:59:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.39, 6.73, 5.97 [04:01:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.03, 6.04, 5.81 [04:17:49] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=ewwiki --no-updates --username-prefix=wikia:era-warfare-official /home/macfan/ImportDump/ewwiki-20231106172313.xml (START) [04:17:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:20:01] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=ewwiki --no-updates --username-prefix=wikia:era-warfare-official /home/macfan/ImportDump/ewwiki-20231106172313.xml (END - exit=0) [04:20:02] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ewwiki (START) [04:20:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:20:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:25:47] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:28:45] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ewwiki (END - exit=0) [04:28:46] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=ewwiki --active --update (END - exit=0) [04:28:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:28:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:30:53] PROBLEM - mail121 Current Load on mail121 is CRITICAL: LOAD CRITICAL - total load average: 4.32, 3.02, 1.43 [04:34:53] RECOVERY - mail121 Current Load on mail121 is OK: LOAD OK - total load average: 0.18, 1.45, 1.14 [04:36:21] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [04:36:43] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=projectexonautwiki --no-updates --username-prefix=wikia:projectexonaut /home/macfan/ImportDump/projectexonautwiki-20231106210557.xml (START) [04:36:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:40:21] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [04:55:48] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:57:17] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=projectexonautwiki --no-updates --username-prefix=wikia:projectexonaut /home/macfan/ImportDump/projectexonautwiki-20231106210557.xml (END - exit=0) [04:57:18] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=projectexonautwiki (START) [04:57:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:57:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:09:42] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=projectexonautwiki (END - exit=0) [05:09:43] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=projectexonautwiki --active --update (END - exit=0) [05:09:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:09:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:19:36] PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.pyramidgames.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:27:42] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.48, 3.20, 2.68 [05:29:41] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.01, 2.74, 2.57 [05:49:36] PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pyramidgames.wiki' expires in 13 day(s) (Wed 22 Nov 2023 14:28:59 GMT +0000). [05:52:25] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 22.60, 15.31, 7.47 [06:01:21] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7615 MB (5% inode=99%); [06:02:23] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.82, 6.87, 6.97 [06:04:22] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.04, 5.52, 6.45 [06:11:05] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [06:23:16] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.96, 7.00, 6.55 [06:25:15] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.86, 5.52, 6.06 [06:27:53] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5101375282 secs [06:31:21] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13787 MB (10% inode=99%); [06:35:12] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.75, 9.24, 7.48 [06:53:21] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15281 MB (11% inode=99%); [06:57:09] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.01, 5.73, 7.43 [06:59:08] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.37, 4.32, 6.72 [07:09:05] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [07:41:05] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [07:47:40] PROBLEM - wiki.helsinginpoytaroolipelaajat.fi - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.second-ns.de.', 'ns1.your-server.de.', 'ns.second-ns.com.'], 'CNAME': None} [07:49:36] PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.pyramidgames.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:52:23] PROBLEM - wiki.albrusmusic.ru - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [07:55:47] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:09:41] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [08:16:49] PROBLEM - wiki.helsinginpoytaroolipelaajat.fi - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.helsinginpoytaroolipelaajat.fi reverse DNS resolves to www381.your-server.de [08:19:36] PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pyramidgames.wiki' expires in 13 day(s) (Wed 22 Nov 2023 14:28:59 GMT +0000). [08:21:12] PROBLEM - wiki.albrusmusic.ru - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.albrusmusic.ru All nameservers failed to answer the query. [08:25:47] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [08:49:58] RECOVERY - wiki.albrusmusic.ru - reverse DNS on sslhost is OK: SSL OK - wiki.albrusmusic.ru reverse DNS resolves to cp24.miraheze.org - CNAME OK [09:56:11] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.13, 2.98, 2.65 [09:58:10] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.37, 2.79, 2.63 [10:18:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.11, 6.76, 6.00 [10:20:04] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.35, 6.47, 6.01 [10:20:04] PROBLEM - wiki.albrusmusic.ru - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.407 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [11:46:38] RECOVERY - wiki.albrusmusic.ru - reverse DNS on sslhost is OK: SSL OK - wiki.albrusmusic.ru reverse DNS resolves to cp25.miraheze.org - CNAME OK [12:16:35] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/4fb5fd1a365b...06c4e0d6bf25 [12:16:36] [02miraheze/ManageWiki] 07translatewiki 0306c4e0d - Localisation updates from https://translatewiki.net. [12:23:21] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14604 MB (10% inode=99%); [12:23:56] miraheze/ManageWiki - translatewiki the build passed. [12:28:47] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.44, 2.68, 1.37 [12:30:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 6.54, 5.99 [12:32:36] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.96, 6.30, 5.97 [12:33:01] PROBLEM - mwtask141 HTTPS on mwtask141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mwtask141.miraheze.org port 443 after 3381 ms: Couldn't connect to server [12:33:11] PROBLEM - mwtask141 Disk Space on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 port 5666: No route to hostconnect to host 2a10:6740::6:504 port 5666: No route to host [12:33:11] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [12:34:02] PROBLEM - mwtask141 JobRunner Service on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 port 5666: No route to hostconnect to host 2a10:6740::6:504 port 5666: No route to host [12:34:07] PROBLEM - mwtask141 PowerDNS Recursor on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 port 5666: No route to hostconnect to host 2a10:6740::6:504 port 5666: No route to host [12:34:17] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 port 5666: No route to hostconnect to host 2a10:6740::6:504 port 5666: No route to host [12:34:17] PROBLEM - mwtask141 ferm_active on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 port 5666: No route to hostconnect to host 2a10:6740::6:504 port 5666: No route to host [12:34:27] PROBLEM - mwtask141 php-fpm on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 port 5666: No route to hostconnect to host 2a10:6740::6:504 port 5666: No route to host [12:34:40] PROBLEM - mwtask141 SSH on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 and port 22: No route to host [12:34:54] PROBLEM - ping6 on mwtask141 is CRITICAL: CRITICAL - Host Unreachable (2a10:6740::6:504) [12:35:02] PROBLEM - mwtask141 nutcracker process on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 port 5666: No route to hostconnect to host 2a10:6740::6:504 port 5666: No route to host [12:35:02] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 port 5666: No route to hostconnect to host 2a10:6740::6:504 port 5666: No route to host [12:35:15] PROBLEM - mwtask141 conntrack_table_size on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 port 5666: No route to hostconnect to host 2a10:6740::6:504 port 5666: No route to host [12:35:19] PROBLEM - mwtask141 NTP time on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 port 5666: No route to hostconnect to host 2a10:6740::6:504 port 5666: No route to host [12:35:59] RECOVERY - mwtask141 JobRunner Service on mwtask141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [12:36:06] RECOVERY - mwtask141 PowerDNS Recursor on mwtask141 is OK: DNS OK: 0.269 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [12:36:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.25, 0.59, 0.20 [12:36:14] RECOVERY - mwtask141 ferm_active on mwtask141 is OK: OK ferm input default policy is set [12:36:22] RECOVERY - mwtask141 php-fpm on mwtask141 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [12:36:38] RECOVERY - mwtask141 SSH on mwtask141 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u2 (protocol 2.0) [12:36:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.13, 3.60, 2.49 [12:36:54] RECOVERY - ping6 on mwtask141 is OK: PING OK - Packet loss = 0%, RTA = 1.37 ms [12:36:57] !log [paladox@mwtask141] starting deploy of {'world': True} to all [12:36:57] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [12:36:59] RECOVERY - mwtask141 HTTPS on mwtask141 is OK: HTTP OK: HTTP/2 301 - 353 bytes in 0.014 second response time [12:37:00] RECOVERY - mwtask141 nutcracker process on mwtask141 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [12:37:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:08] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 34643 MB (48% inode=90%); [12:37:11] RECOVERY - mwtask141 conntrack_table_size on mwtask141 is OK: OK: nf_conntrack is 1 % full [12:37:13] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.854 second response time [12:37:24] RECOVERY - mwtask141 NTP time on mwtask141 is OK: NTP OK: Offset 0.003028124571 secs [12:39:22] !log [paladox@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 145s [12:39:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:40:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.79, 2.77, 2.41 [12:42:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.47, 6.52, 6.06 [12:43:53] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.39, 3.42, 3.13 [12:44:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.49, 6.34, 6.04 [12:45:52] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.87, 3.25, 3.10 [12:55:47] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:20:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:25:47] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [13:30:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:36:19] PROBLEM - wiki.anglish.info - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.anglish.info All nameservers failed to answer the query. [13:42:32] PROBLEM - db101 Backups SQL on db101 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209644 seconds old and 121568 bytes [13:46:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.25, 7.14, 6.50 [13:48:08] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.94, 9.81, 8.67 [13:48:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.93, 6.98, 6.52 [13:50:08] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 9.00, 9.90, 8.87 [13:50:34] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.96, 6.23, 6.30 [13:56:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.04, 7.14, 6.68 [13:58:08] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.85, 10.88, 9.71 [14:00:08] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.22, 10.04, 9.56 [14:04:08] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.41, 10.55, 9.90 [14:04:14] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.04, 6.78, 6.76 [14:05:44] PROBLEM - wiki.anglish.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.anglish.info could not be found [14:06:08] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 9.04, 9.96, 9.76 [14:09:37] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:11:53] PROBLEM - www.twistwoodtaleswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.twistwoodtaleswiki.com could not be found [14:33:54] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.94, 3.46, 3.00 [14:38:13] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.72, 9.68, 8.25 [14:40:13] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.77, 9.03, 8.20 [14:41:05] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [14:41:51] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.25, 3.31, 3.11 [14:49:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 6.64, 6.24 [14:51:07] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.01, 6.30, 6.16 [15:02:39] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.65, 3.45, 3.24 [15:04:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.96, 2.94, 3.08 [15:09:40] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [15:18:05] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.26, 10.07, 8.78 [15:20:04] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.46, 9.31, 8.65 [15:28:08] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.48, 10.42, 9.20 [15:32:08] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.21, 9.29, 9.07 [15:40:08] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.03, 10.62, 9.62 [15:42:08] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.70, 10.38, 9.65 [15:46:08] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.25, 10.02, 9.72 [15:52:08] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.12, 10.87, 10.15 [15:52:12] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.94, 3.41, 3.16 [15:54:08] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.31, 10.06, 9.94 [15:54:11] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.11, 3.38, 3.18 [16:00:08] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.97, 10.69, 10.19 [16:11:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.34, 3.58, 3.30 [16:13:06] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.24, 3.23, 3.22 [16:14:08] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.36, 9.26, 9.80 [16:16:52] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.71, 9.11, 8.10 [16:18:52] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.29, 8.88, 8.14 [16:19:11] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.20, 3.32, 1.93 [16:28:49] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.33, 10.21, 9.05 [16:30:49] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.34, 9.72, 9.02 [16:33:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.79, 3.39, 3.20 [16:34:48] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.28, 11.14, 9.69 [16:38:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/b8f7a4e921cc [16:38:15] [02miraheze/puppet] 07paladox 03b8f7a4e - Add poolcounter module [16:38:18] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [16:38:21] [02puppet] 07paladox opened pull request 03#3489: Add poolcounter module - 13https://github.com/miraheze/puppet/pull/3489 [16:38:47] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.14, 11.89, 10.42 [16:39:05] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.94, 3.18, 3.20 [16:39:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/b8f7a4e921cc...4ccc97604953 [16:39:19] [02miraheze/puppet] 07paladox 034ccc976 - Create poolcounter.systemd.erb [16:39:22] [02puppet] 07paladox synchronize pull request 03#3489: Add poolcounter module - 13https://github.com/miraheze/puppet/pull/3489 [16:40:47] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.16, 12.24, 10.72 [16:44:27] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.49, 9.92, 9.17 [16:45:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4ccc97604953...4f4f0b541500 [16:45:14] [02miraheze/puppet] 07paladox 034f4f0b5 - Update init.pp [16:45:15] [02puppet] 07paladox synchronize pull request 03#3489: Add poolcounter module - 13https://github.com/miraheze/puppet/pull/3489 [16:51:10] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.47, 3.47, 3.93 [16:51:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/4f4f0b541500...1c12594b0b1b [16:51:15] [02miraheze/puppet] 07paladox 031c12594 - Create poolcounter.pp [16:51:17] [02puppet] 07paladox synchronize pull request 03#3489: Add poolcounter module - 13https://github.com/miraheze/puppet/pull/3489 [16:51:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1c12594b0b1b...40035d670005 [16:51:40] [02miraheze/puppet] 07paladox 0340035d6 - Update poolcounter.pp [16:51:41] [02puppet] 07paladox synchronize pull request 03#3489: Add poolcounter module - 13https://github.com/miraheze/puppet/pull/3489 [16:52:03] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.82, 3.44, 3.24 [16:52:12] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.68, 9.35, 9.40 [16:52:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/40035d670005...8f59847e103b [16:52:24] [02miraheze/puppet] 07paladox 038f59847 - Update site.pp [16:52:27] [02puppet] 07paladox synchronize pull request 03#3489: Add poolcounter module - 13https://github.com/miraheze/puppet/pull/3489 [16:52:44] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.10, 11.23, 11.33 [16:54:02] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.07, 3.28, 3.21 [16:54:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/8f59847e103b...e14025ed77a8 [16:54:43] [02miraheze/puppet] 07paladox 03e14025e - Update site.pp [16:54:46] [02puppet] 07paladox synchronize pull request 03#3489: Add poolcounter module - 13https://github.com/miraheze/puppet/pull/3489 [16:55:10] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 0.54, 1.86, 3.16 [16:55:39] [02puppet] 07paladox synchronize pull request 03#3489: Add poolcounter module - 13https://github.com/miraheze/puppet/pull/3489 [16:55:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e14025ed77a8...061be95d4be6 [16:55:45] [02miraheze/puppet] 07paladox 03061be95 - Update poolcounter.pp [16:57:17] [02puppet] 07paladox closed pull request 03#3489: Add poolcounter module - 13https://github.com/miraheze/puppet/pull/3489 [16:57:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+4/-0/±1] 13https://github.com/miraheze/puppet/compare/f9bc706ec587...7a29c9f5884c [16:57:20] [02miraheze/puppet] 07paladox 037a29c9f - Add poolcounter module (#3489) [16:57:22] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [16:57:24] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [16:57:59] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.78, 3.48, 3.29 [16:59:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7a29c9f5884c...7a6257abe4fb [16:59:54] [02miraheze/puppet] 07paladox 037a6257a - poolcounter: install libevent-dev [16:59:58] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.67, 3.14, 3.19 [17:02:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7a6257abe4fb...f33f5d0e6c59 [17:02:37] [02miraheze/puppet] 07paladox 03f33f5d0 - poolcounter: add some monitoring [17:08:41] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.19, 9.08, 10.05 [17:31:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.57, 6.65 [17:33:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.52, 6.85, 6.50 [17:35:03] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.30, 6.47, 6.41 [17:41:07] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [17:48:44] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/5483cd28e305 [17:48:45] [02miraheze/mw-config] 07paladox 035483cd2 - Set $wgPoolCountClientConf [17:48:47] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [17:48:49] [02mw-config] 07paladox opened pull request 03#5370: Set $wgPoolCountClientConf - 13https://github.com/miraheze/mw-config/pull/5370 [17:49:52] miraheze/mw-config - paladox the build passed. [18:14:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.78, 6.87, 6.53 [18:16:01] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.73, 6.39, 6.39 [18:42:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.05, 6.81, 6.44 [18:47:55] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.70, 6.78, 6.54 [19:04:20] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [19:10:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:20:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:34:23] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c6d9b15ed033...8d68b7c532a3 [19:34:24] [02miraheze/mediawiki] 07paladox 038d68b7c - PoolCounterConnectionManager: Add support for ipv6 [19:35:17] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a56bb1fd76a5...783fd496b2db [19:35:18] [02miraheze/mediawiki] 07paladox 03783fd49 - PoolCounterConnectionManager: Add support for ipv6 [19:35:56] !log [paladox@mwtask141] starting deploy of {'world': True} to all [19:36:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:36:31] !log [paladox@test131] starting deploy of {'world': True} to all [19:36:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:05] !log [paladox@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 68s [19:37:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:35] !log [paladox@test131] finished deploy of {'world': True} to all - SUCCESS in 63s [19:37:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:29] [02mw-config] 07paladox synchronize pull request 03#5370: Set $wgPoolCountClientConf - 13https://github.com/miraheze/mw-config/pull/5370 [19:39:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5483cd28e305...e96d168066f0 [19:39:32] [02miraheze/mw-config] 07paladox 03e96d168 - Update GlobalSettings.php [19:39:44] [02mw-config] 07paladox closed pull request 03#5370: Set $wgPoolCountClientConf - 13https://github.com/miraheze/mw-config/pull/5370 [19:39:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/448b11c269b3...b29744360b1f [19:39:46] [02miraheze/mw-config] 07paladox 03b297443 - Set $wgPoolCountClientConf (#5370) [19:39:49] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [19:39:50] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [19:39:52] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [19:39:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:59] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [19:40:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:07] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [19:40:09] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [19:40:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:15] !log [paladox@mwtask141] starting deploy of {'config': True} to all [19:40:16] !log [paladox@test131] starting deploy of {'config': True} to all [19:40:17] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [19:40:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:23] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [19:40:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:30] miraheze/mw-config - paladox the build passed. [19:40:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:39] miraheze/mw-config - paladox the build passed. [19:40:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b29744360b1f...780b3428ffc5 [19:48:08] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [19:48:08] [02miraheze/mw-config] 07paladox 03780b342 - Increase job redis timeout to 5 [19:48:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:15] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [19:48:19] !log [paladox@mwtask141] starting deploy of {'config': True} to all [19:48:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:28] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [19:48:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:03] miraheze/mw-config - paladox the build passed. [19:49:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.09, 6.54 [19:49:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/780b3428ffc5...76acdbc6bd99 [19:49:46] [02miraheze/mw-config] 07paladox 0376acdbc - Testing with timeout [19:49:47] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [19:49:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:55] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [19:49:57] !log [paladox@mwtask141] starting deploy of {'config': True} to all [19:50:08] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 11s [19:50:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:38] miraheze/mw-config - paladox the build passed. [19:51:14] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.81, 6.61, 6.43 [19:52:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/76acdbc6bd99...05888e4cb5a9 [19:52:13] [02miraheze/mw-config] 07paladox 0305888e4 - Testing with timeout [19:52:15] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [19:52:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:25] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [19:52:29] !log [paladox@mwtask141] starting deploy of {'config': True} to all [19:52:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:37] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [19:52:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:13] miraheze/mw-config - paladox the build passed. [19:56:21] !log [@test131] starting deploy of {'config': True} to all [19:56:22] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [19:56:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.57, 3.23, 3.05 [20:02:14] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.55, 3.09, 3.03 [20:04:43] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [20:08:58] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:11:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.32, 7.64, 6.86 [20:13:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.57, 7.12, 6.76 [20:15:28] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.24, 6.67, 6.63 [20:19:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.57, 7.09, 6.77 [20:21:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.04, 6.96, 6.76 [20:23:14] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.28, 6.31, 6.54 [20:24:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/05888e4cb5a9...e93df77c4977 [20:24:09] [02miraheze/mw-config] 07paladox 03e93df77 - Experiment with Increasing timeout to 15 for job [20:24:59] miraheze/mw-config - paladox the build passed. [20:25:11] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [20:25:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:25:23] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 12s [20:25:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:25:29] !log [paladox@mwtask141] starting deploy of {'config': True} to all [20:25:36] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [20:25:37]