[00:00:01] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [00:00:05] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 0.069 second response time [00:00:21] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 5% [00:00:21] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [00:00:40] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [00:02:39] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/da9f0e52486b714b52c5a9e64d253e7686d4fa3a [00:02:47] 02statichelp/03main 07WikiTideBot 03da9f0e5 Bot: Auto-update Tech namespace pages 2025-09-08 00:01:51 [00:05:24] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [00:07:18] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56748MiB (12% inode=99%); [00:07:29] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 56985MiB (12% inode=99%); [00:08:03] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57431MiB (12% inode=99%); [00:09:17] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [00:15:10] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [00:20:57] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 0.62, 1.65, 11.27 [00:22:55] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.74, 1.38, 10.00 [01:01:04] !log [universalomega@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'TitleKey'} to all [01:01:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:01:27] !log [universalomega@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'TitleKey'} to all - SUCCESS in 22s [01:01:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:42:28] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:42:34] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 20.57, 10.13, 5.21 [01:44:25] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.062 second response time [01:44:34] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.29, 6.98, 4.65 [01:46:45] !log [universalomega@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'TitleKey'} to all [01:46:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:47:09] !log [universalomega@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'TitleKey'} to all - SUCCESS in 23s [01:47:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:07] > [08/09/2025 11:44] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.062 second response time [01:48:10] wdym 410 is okay [01:50:40] expected [01:52:30] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=fnaffangamewiki dump.xml --no-updates (START) [01:52:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:16] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/e4a4bcab196921d642f3cbd68263278b7c611698 [01:56:16] 02mw-config/03main 07CosmicAlpha 03e4a4bca getDatabaseClusters: always return array [01:57:13] miraheze/mw-config - Universal-Omega the build passed. [02:12:46] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/cda457705a9b9a3b359e4a8099a89c9afc6f7bc9 [02:12:46] 02mw-config/03main 07CosmicAlpha 03cda4577 Fix closed and inactive database lists [02:12:57] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [02:13:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:13:23] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [02:13:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:13:38] miraheze/mw-config - Universal-Omega the build passed. [02:15:59] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/closed.php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:ChangeMediaWikiVersion --mwversion=1.44 (START) [02:16:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:16:51] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/closed.php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:ChangeMediaWikiVersion --mwversion=1.44 (END - exit=2) [02:17:00] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 57.47, 19.40, 7.97 [02:17:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:23:55] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.52, 9.38, 7.64 [02:29:18] [02MirahezeMagic] 07Universal-Omega created 03ChangeMediaWikiVersion-states (+1 new commit) 13https://github.com/miraheze/MirahezeMagic/commit/9be4a4628060 [02:29:18] 02MirahezeMagic/03ChangeMediaWikiVersion-states 07CosmicAlpha 039be4a46 ChangeMediaWikiVersion: add state options [02:29:23] [02MirahezeMagic] 07Universal-Omega opened pull request #595: ChangeMediaWikiVersion: add state options (03main...03ChangeMediaWikiVersion-states) 13https://github.com/miraheze/MirahezeMagic/pull/595 [02:29:30] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request #595:
[…] 13https://github.com/miraheze/MirahezeMagic/pull/595#issuecomment-3264367203 [02:30:53] [02MirahezeMagic] 07Universal-Omega merged pull request #595: ChangeMediaWikiVersion: add state options (03main...03ChangeMediaWikiVersion-states) 13https://github.com/miraheze/MirahezeMagic/pull/595 [02:30:53] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/4eab05d877c365bbb0f2b439c262d159a2d6de2b [02:30:53] 02MirahezeMagic/03main 07CosmicAlpha 034eab05d ChangeMediaWikiVersion: add state options [02:30:54] [02MirahezeMagic] 07Universal-Omega 04deleted 03ChangeMediaWikiVersion-states at 039be4a46 13https://api.github.com/repos/miraheze/MirahezeMagic/commit/9be4a46 [02:31:07] !log [universalomega@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all [02:31:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:52] !log [universalomega@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 44s [02:31:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:43] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:ChangeMediaWikiVersion --wiki=metawiki --closed --inactive --mwversion=1.44 (END - exit=2) [02:36:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:22] miraheze/MirahezeMagic - Universal-Omega the build passed. [02:43:10] miraheze/MirahezeMagic - Universal-Omega the build passed. [02:49:57] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1 [02:56:07] !log [universalomega@mwtask181] starting deploy of {'extension_list': True, 'versions': ['1.43', '1.44']} to all [02:56:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:58] !log [universalomega@mwtask181] finished deploy of {'extension_list': True, 'versions': ['1.43', '1.44']} to all - SUCCESS in 51s [02:57:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:57] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=fnaffangamewiki dump.xml --no-updates (END - exit=0) [03:14:58] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=fnaffangamewiki (START) [03:15:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:15:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:26] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:ChangeMediaWikiVersion --wiki=metawiki --closed --mwversion=1.44 (END - exit=0) [03:33:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:16:02] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=fnaffangamewiki (END - exit=0) [04:16:04] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/initSiteStats.php --wiki=fnaffangamewiki --update (END - exit=0) [04:16:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:16:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:18:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [05:26:34] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [07:08:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [07:16:34] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [07:22:50] !log [universalomega@mwtask181] Kill my old screen sessions, some going all the way back to May [07:22:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:30:18] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:ChangeMediaWikiVersion --wiki=metawiki --inactive --mwversion=1.44 (END - exit=0) [07:30:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:01:57] RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1 [08:25:10] !log [blankeclair@mwtask171] Starting import for spidermineswiki (XML: None; Images: images) (START) [08:25:11] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=spidermineswiki --sleep=0 '--comment=Importing images from https://infestedroblox.fandom.com ([[phorge:T14240|T14240]])' -- images (START) [08:25:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:25:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:25:40] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=spidermineswiki --sleep=0 '--comment=Importing images from https://infestedroblox.fandom.com ([[phorge:T14240|T14240]])' -- images (END - exit=0) [08:25:41] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php initSiteStats --wiki=spidermineswiki --update (START) [08:25:42] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php initSiteStats --wiki=spidermineswiki --update (END - exit=0) [08:25:43] !log [blankeclair@mwtask171] Finished import for spidermineswiki (XML: None; Images: images) (END - exit=0) [08:25:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:25:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:25:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:25:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:38:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [08:46:34] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [09:51:18] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49835MiB (10% inode=99%); [10:31:29] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49875MiB (10% inode=99%); [10:46:03] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49935MiB (10% inode=99%); [12:10:03] [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/ece6d52fc9cc4267913d694e4f11f746456785c2 [12:10:03] 02DiscordNotifications/03main 07translatewiki.net 03ece6d52 Localisation updates from https://translatewiki.net. [12:10:05] [02landing] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/2ad15278fb7a7e55cd07e07d050698dfe9592c52 [12:10:05] 02landing/03main 07translatewiki.net 032ad1527 Localisation updates from https://translatewiki.net. [12:10:06] [02MatomoAnalytics] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MatomoAnalytics/commit/fac454ad578ef7d1f0b84df77b99fd479d15e0b2 [12:10:08] 02MatomoAnalytics/03main 07translatewiki.net 03fac454a Localisation updates from https://translatewiki.net. [12:10:10] [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/0a75b58b06886dd2161fa6c93cfdab1b103f6ffb [12:10:12] [02RequestCustomDomain] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/RequestCustomDomain/commit/36fe73d6a56e8a3c090b997b0865afd1b4b98a06 [12:10:14] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/d4dc54362450bdb6be428953a0fbbb9f904eb9e6 [12:10:16] [02WikiDiscover] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/WikiDiscover/commit/5b9c091571339ec2a5299772f77f25fb10e2bf2a [12:10:17] 02ManageWiki/03main 07translatewiki.net 030a75b58 Localisation updates from https://translatewiki.net. [12:10:19] 02RequestCustomDomain/03main 07translatewiki.net 0336fe73d Localisation updates from https://translatewiki.net. [12:10:21] 02MirahezeMagic/03main 07translatewiki.net 03d4dc543 Localisation updates from https://translatewiki.net. [12:10:22] 02WikiDiscover/03main 07translatewiki.net 035b9c091 Localisation updates from https://translatewiki.net. [12:14:30] miraheze/landing - translatewiki the build passed. [12:16:29] miraheze/ManageWiki - translatewiki the build passed. [12:17:04] miraheze/MatomoAnalytics - translatewiki the build passed. [12:28:39] miraheze/MirahezeMagic - translatewiki the build passed. [12:31:22] miraheze/WikiDiscover - translatewiki the build passed. [12:34:26] miraheze/DiscordNotifications - translatewiki the build passed. [12:34:47] miraheze/RequestCustomDomain - translatewiki the build passed. [14:56:19] !log curl -X DELETE https://opensearch-mw.wikitide.net/adrenwiki_{content,general} [14:56:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:57:03] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=adrenwiki --startOver (END - exit=0) [14:57:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:57:53] oops should've probably used mwtask171 [14:57:54] oh well [14:58:44] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=adrenwiki (END - exit=0) [14:58:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:19:30] [02ImportDump] 07Universal-Omega created 03revert-147-InterwikiCentralDB (+1 new commit) 13https://github.com/miraheze/ImportDump/commit/cd3a6560179c [16:19:30] 02ImportDump/03revert-147-InterwikiCentralDB 07CosmicAlpha 03cd3a656 Revert "Still use InterwikiCentralDB for now"… [16:19:52] [02ImportDump] 07Universal-Omega opened pull request #150: Use virtual-interwiki (03main...03revert-147-InterwikiCentralDB) 13https://github.com/miraheze/ImportDump/pull/150 [16:20:01] [02ImportDump] 07coderabbitai[bot] commented on pull request #150:
[…] 13https://github.com/miraheze/ImportDump/pull/150#issuecomment-3267044322 [16:20:26] [02ImportDump] 07Universal-Omega pushed 1 new commit to 03revert-147-InterwikiCentralDB 13https://github.com/miraheze/ImportDump/commit/a7edbd2333110c7d559bdf2f535dce2fb228b6d0 [16:20:26] 02ImportDump/03revert-147-InterwikiCentralDB 07CosmicAlpha 03a7edbd2 Bump requirement to 1.44 [16:22:10] [02ImportDump] 07Universal-Omega pushed 1 new commit to 03revert-147-InterwikiCentralDB 13https://github.com/miraheze/ImportDump/commit/cc740f73211e4f69695adca7ce3814de3a14d2ca [16:22:10] 02ImportDump/03revert-147-InterwikiCentralDB 07CosmicAlpha 03cc740f7 Remove REL1_43 [16:22:32] [02ImportDump] 07Universal-Omega retitled PR #150: "Use virtual-interwiki" ➜ "Use virtual-interwiki and update for MediaWiki 1.44" 13https://github.com/miraheze/ImportDump/pull/150 [16:24:07] [02ImportDump] 07Universal-Omega pushed 1 new commit to 03revert-147-InterwikiCentralDB 13https://github.com/miraheze/ImportDump/commit/d6d63a82c0b3a082762ca8cfcc7a2379d76c4569 [16:24:07] 02ImportDump/03revert-147-InterwikiCentralDB 07CosmicAlpha 03d6d63a8 Disable class alias support [16:28:22] miraheze/ImportDump - Universal-Omega the build has errored. [16:41:41] PROBLEM - mw151 PowerDNS Recursor on mw151 is CRITICAL: Domain 'mw151.fsslc.wtnet' was not found by the server [16:41:42] PROBLEM - mw152 PowerDNS Recursor on mw152 is CRITICAL: No response from DNS ::1 [16:41:43] PROBLEM - mw192 PowerDNS Recursor on mw192 is CRITICAL: No response from DNS ::1 [16:41:43] PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: No response from DNS 2001:41d0:801:2000::4089 [16:42:24] PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 96.14 ms [16:42:30] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:33] PROBLEM - rdb151 PowerDNS Recursor on rdb151 is CRITICAL: Domain 'rdb151.fsslc.wtnet' was not found by the server [16:42:38] PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 96.05 ms [16:42:43] PROBLEM - graylog161 PowerDNS Recursor on graylog161 is CRITICAL: Domain 'graylog161.fsslc.wtnet' was not found by the server [16:42:56] PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 96.05 ms [16:42:59] PROBLEM - mem151 PowerDNS Recursor on mem151 is CRITICAL: No response from DNS ::1 [16:43:46] PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 48.53 ms [16:43:46] PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 96.03 ms [16:43:46] PROBLEM - www.mwcosmos.com - Cloudflare on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:43:46] PROBLEM - cp171 Puppet on cp171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:43:46] RECOVERY - mw192 PowerDNS Recursor on mw192 is OK: DNS OK: 0.127 seconds response time. mw192.fsslc.wtnet returns 10.0.19.161 [16:43:46] RECOVERY - mw152 PowerDNS Recursor on mw152 is OK: DNS OK: 0.036 seconds response time. mw152.fsslc.wtnet returns 10.0.15.115 [16:43:46] RECOVERY - mw151 PowerDNS Recursor on mw151 is OK: DNS OK: 0.033 seconds response time. mw151.fsslc.wtnet returns 10.0.15.114 [16:43:47] RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 0.326 seconds response time. ns2.wikitide.net returns 2001:41d0:801:2000::4089,51.75.170.66 [16:43:47] PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 57.81 ms [16:44:13] PROBLEM - www.marakyhbu.org - Cloudflare on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:46] PROBLEM - ns2 ferm_active on ns2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:44:52] PROBLEM - mon181 PowerDNS Recursor on mon181 is CRITICAL: Domain 'mon181.fsslc.wtnet' was not found by the server [16:46:21] PROBLEM - puppet181 PowerDNS Recursor on puppet181 is CRITICAL: Domain 'puppet181.wikitide.net' was not found by the server [16:46:21] PROBLEM - ns2 conntrack_table_size on ns2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:46:21] RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 96.43 ms [16:46:21] PROBLEM - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift-container-stats_mw-media [16:46:21] PROBLEM - kafka181 PowerDNS Recursor on kafka181 is CRITICAL: Domain 'kafka181.fsslc.wtnet' was not found by the server [16:46:21] PROBLEM - ns2 APT on ns2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:46:21] PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:46:22] PROBLEM - mem201 PowerDNS Recursor on mem201 is CRITICAL: Domain 'mem201.fsslc.wtnet' was not found by the server [16:46:22] PROBLEM - mem161 Puppet on mem161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:46:23] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:46:28] RECOVERY - rdb151 PowerDNS Recursor on rdb151 is OK: DNS OK: 0.032 seconds response time. rdb151.fsslc.wtnet returns 10.0.15.142 [16:46:31] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 30 Oct 2025 08:11:02 AM GMT +0000. [16:46:33] RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.31 ms [16:46:42] RECOVERY - graylog161 PowerDNS Recursor on graylog161 is OK: DNS OK: 0.033 seconds response time. graylog161.fsslc.wtnet returns 10.0.16.130 [16:46:44] RECOVERY - ns2 ferm_active on ns2 is OK: OK ferm input default policy is set [16:46:46] RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.30 ms [16:46:49] RECOVERY - mon181 PowerDNS Recursor on mon181 is OK: DNS OK: 0.037 seconds response time. mon181.fsslc.wtnet returns 10.0.18.109 [16:46:55] RECOVERY - ns2 conntrack_table_size on ns2 is OK: OK: nf_conntrack is 0 % full [16:46:56] RECOVERY - mem151 PowerDNS Recursor on mem151 is OK: DNS OK: 0.031 seconds response time. mem151.fsslc.wtnet returns 10.0.15.113 [16:46:57] RECOVERY - puppet181 PowerDNS Recursor on puppet181 is OK: DNS OK: 0.045 seconds response time. puppet181.wikitide.net returns 10.0.18.100 [16:46:58] RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.30 ms [16:47:08] RECOVERY - ping6 on cloud18 is OK: PING OK - Packet loss = 0%, RTA = 0.27 ms [16:47:18] RECOVERY - kafka181 PowerDNS Recursor on kafka181 is OK: DNS OK: 0.035 seconds response time. kafka181.fsslc.wtnet returns 10.0.18.159 [16:47:21] RECOVERY - ns2 APT on ns2 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [16:47:23] RECOVERY - mem201 PowerDNS Recursor on mem201 is OK: DNS OK: 0.025 seconds response time. mem201.fsslc.wtnet returns 10.0.20.148 [16:47:50] RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms [16:48:19] PROBLEM - db161 Puppet on db161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:51:05] RECOVERY - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is OK: OK: Status of the systemd unit swift-container-stats_mw-media [16:52:06] [02ImportDump] 07Universal-Omega pushed 1 new commit to 03revert-147-InterwikiCentralDB 13https://github.com/miraheze/ImportDump/commit/4d33766a19327a726b90e0a2dd313310b1cf5839 [16:52:08] 02ImportDump/03revert-147-InterwikiCentralDB 07CosmicAlpha 034d33766 Update [16:54:42] PROBLEM - graylog161 PowerDNS Recursor on graylog161 is CRITICAL: Domain 'graylog161.fsslc.wtnet' was not found by the server [16:54:59] PROBLEM - cp191 PowerDNS Recursor on cp191 is CRITICAL: Domain 'cp191.wikitide.net' was not found by the server [16:55:42] miraheze/ImportDump - Universal-Omega the build has errored. [16:56:23] @cosmicalpha the servers are not doing too great [16:56:42] RECOVERY - graylog161 PowerDNS Recursor on graylog161 is OK: DNS OK: 2.127 seconds response time. graylog161.fsslc.wtnet returns 10.0.16.130 [16:56:58] RECOVERY - cp191 PowerDNS Recursor on cp191 is OK: DNS OK: 2.131 seconds response time. cp191.wikitide.net returns 2602:294:0:b33::102 [16:56:59] seems networking died? [16:57:00] meta takes 20s to load for me [16:57:05] apparently [16:58:08] PROBLEM - swiftobject161 PowerDNS Recursor on swiftobject161 is CRITICAL: Domain 'swiftobject161.fsslc.wtnet' was not found by the server [16:58:19] PROBLEM - swiftproxy171 PowerDNS Recursor on swiftproxy171 is CRITICAL: Domain 'swiftproxy171.fsslc.wtnet' was not found by the server [16:58:56] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:59:18] I am looking [16:59:18] thx [16:59:24] PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 96.10 ms [16:59:24] PROBLEM - www.wikimoma.art - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [16:59:24] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:59:24] PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 96.15 ms [16:59:47] PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 96.13 ms [16:59:47] PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 96.11 ms [16:59:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2602:294:0:b13::110/cpweb [16:59:55] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:00:07] RECOVERY - swiftobject161 PowerDNS Recursor on swiftobject161 is OK: DNS OK: 0.461 seconds response time. swiftobject161.fsslc.wtnet returns 10.0.16.134 [17:00:19] RECOVERY - swiftproxy171 PowerDNS Recursor on swiftproxy171 is OK: DNS OK: 0.043 seconds response time. swiftproxy171.fsslc.wtnet returns 10.0.17.108 [17:00:23] PROBLEM - mw162 PowerDNS Recursor on mw162 is CRITICAL: Domain 'mw162.fsslc.wtnet' was not found by the server [17:00:24] RECOVERY - mw162 PowerDNS Recursor on mw162 is OK: DNS OK: 0.031 seconds response time. mw162.fsslc.wtnet returns 10.0.16.133 [17:00:39] RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.29 ms [17:00:46] RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.31 ms [17:00:54] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 30 Oct 2025 08:11:02 AM GMT +0000. [17:00:57] RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.28 ms [17:01:00] RECOVERY - www.wikimoma.art - Cloudflare on sslhost is OK: OK - Certificate 'www.wikimoma.art' will expire on Fri 07 Nov 2025 08:24:26 AM GMT +0000. [17:01:04] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 04 Nov 2025 06:36:56 AM GMT +0000. [17:01:10] RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.28 ms [17:01:35] RECOVERY - www.mwcosmos.com - Cloudflare on sslhost is OK: OK - Certificate 'www.mwcosmos.com' will expire on Mon 13 Oct 2025 02:50:05 AM GMT +0000. [17:01:52] PROBLEM - changeprop201 Puppet on changeprop201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:02:11] ns2 seems to have died. [17:02:27] oh [17:02:50] Should be recovering let's see if it stays. [17:03:22] alr [17:04:10] RECOVERY - www.marakyhbu.org - Cloudflare on sslhost is OK: OK - Certificate 'www.marakyhbu.org' will expire on Tue 18 Nov 2025 03:29:39 AM GMT +0000. [17:05:22] [02ImportDump] 07Universal-Omega pushed 1 new commit to 03revert-147-InterwikiCentralDB 13https://github.com/miraheze/ImportDump/commit/b50003ee8b243638626f3f96080ccf33d9758a61 [17:05:22] 02ImportDump/03revert-147-InterwikiCentralDB 07CosmicAlpha 03b50003e Update [17:05:43] [02ImportDump] 07coderabbitai[bot] left a review on pull request #150: **Actionable comments posted: 0** […] 13https://github.com/miraheze/ImportDump/pull/150#pullrequestreview-3197382580 [17:10:03] RECOVERY - cp171 Puppet on cp171 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:12:10] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:12:58] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:13:03] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:14:43] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [17:25:49] miraheze/ImportDump - Universal-Omega the build passed. [17:27:29] RECOVERY - changeprop201 Puppet on changeprop201 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [17:33:03] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 4.55, 10.47, 6.60 [17:33:11] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.08, 10.17, 6.54 [17:41:44] [02mw-config] 07Universal-Omega created 03virtual-globaljsonlinks (+1 new commit) 13https://github.com/miraheze/mw-config/commit/7508559cefa1 [17:41:45] 02mw-config/03virtual-globaljsonlinks 07CosmicAlpha 037508559 Add virtual-globaljsonlinks [17:41:48] [02mw-config] 07Universal-Omega opened pull request #6093: Add virtual-globaljsonlinks (03main...03virtual-globaljsonlinks) 13https://github.com/miraheze/mw-config/pull/6093 [17:42:45] miraheze/mw-config - Universal-Omega the build passed. [17:44:19] [02mw-config] 07Universal-Omega pushed 1 new commit to 03virtual-globaljsonlinks 13https://github.com/miraheze/mw-config/commit/d7def4cbfcf809778e9fb4a98e4a99fde0d25177 [17:44:20] 02mw-config/03virtual-globaljsonlinks 07CosmicAlpha 03d7def4c Order [17:45:09] miraheze/mw-config - Universal-Omega the build passed. [17:47:48] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:ChangeMediaWikiVersion --wiki=commonswiki --mwversion=1.44 (END - exit=0) [17:50:51] [02mw-config] 07Universal-Omega merged pull request #6093: Add virtual-globaljsonlinks (03main...03virtual-globaljsonlinks) 13https://github.com/miraheze/mw-config/pull/6093 [17:50:52] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/f8ea52901b9dc855a251cf58d09aa88020d3c964 [17:50:52] 02mw-config/03main 07CosmicAlpha 03f8ea529 Add virtual-globaljsonlinks [17:50:53] [02mw-config] 07Universal-Omega 04deleted 03virtual-globaljsonlinks at 03d7def4c 13https://api.github.com/repos/miraheze/mw-config/commit/d7def4c [17:50:59] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [17:51:22] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [17:51:48] miraheze/mw-config - Universal-Omega the build passed. [18:58:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [19:06:34] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [19:27:57] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1 [19:29:18] PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2 [20:45:56] PROBLEM - matomo151 Disk Space on matomo151 is WARNING: DISK WARNING - free space: / 1968MiB (10% inode=89%); [22:01:56] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 6604MiB (36% inode=89%); [23:21:44] [02MirahezeMagic] 07Universal-Omega created 03all-wikis (+1 new commit) 13https://github.com/miraheze/MirahezeMagic/commit/fab4ba034e44 [23:21:44] 02MirahezeMagic/03all-wikis 07CosmicAlpha 03fab4ba0 ChangeMediaWikiVersion: add all-wikis option [23:22:24] [02MirahezeMagic] 07Universal-Omega opened pull request #596: ChangeMediaWikiVersion: add all-wikis option (03main...03all-wikis) 13https://github.com/miraheze/MirahezeMagic/pull/596 [23:22:30] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request #596:
[…] 13https://github.com/miraheze/MirahezeMagic/pull/596#issuecomment-3268331827 [23:27:41] !log improve performance for cf -> cp (switch from smart tiered cache to generic cache tired and switch on regional tired). [23:28:33] !log improve performance for cf -> cp (switch from smart tiered cache to generic cache tired and switch on regional tired). [23:28:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:19] [02MirahezeMagic] 07Universal-Omega merged pull request #596: ChangeMediaWikiVersion: add all-wikis option (03main...03all-wikis) 13https://github.com/miraheze/MirahezeMagic/pull/596 [23:29:19] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/d49a30f6ede0a621396a2dcf3e39d2c7b98754fa [23:29:19] 02MirahezeMagic/03main 07CosmicAlpha 03d49a30f ChangeMediaWikiVersion: add all-wikis option [23:29:20] [02MirahezeMagic] 07Universal-Omega 04deleted 03all-wikis at 03fab4ba0 13https://api.github.com/repos/miraheze/MirahezeMagic/commit/fab4ba0 [23:29:44] !log [universalomega@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'MirahezeMagic'} to all [23:29:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:06] !log [universalomega@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 22s [23:30:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:23] miraheze/MirahezeMagic - Universal-Omega the build passed. [23:41:37] miraheze/MirahezeMagic - Universal-Omega the build passed.