[00:07:18] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 57157MiB (12% inode=99%); [00:07:29] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57319MiB (12% inode=99%); [00:08:03] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57888MiB (12% inode=99%); [00:40:52] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 45% [00:46:43] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 24% [01:57:40] !log [blankeclair@mwtask181] Starting import for botsandmusketswiki (XML: None; Images: images) (START) [01:57:41] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=botsandmusketswiki --sleep=0 '--comment=Importing images from https://bots-muskets.fandom.com ([[phorge:T14228T14228]])' -- images (START) [01:57:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:58:48] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=botsandmusketswiki --sleep=0 '--comment=Importing images from https://bots-muskets.fandom.com ([[phorge:T14228T14228]])' -- images (END - exit=0) [01:58:49] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initSiteStats --wiki=botsandmusketswiki --update (START) [01:58:50] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initSiteStats --wiki=botsandmusketswiki --update (END - exit=0) [01:58:51] !log [blankeclair@mwtask181] Finished import for botsandmusketswiki (XML: None; Images: images) (END - exit=0) [01:58:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:58:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:58:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:59:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:43:18] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=falcomwiki dump.xml --no-updates (START) [02:43:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:46:08] PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3 [02:54:41] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=falcomwiki dump.xml --no-updates (END - exit=0) [02:54:42] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=falcomwiki (START) [02:54:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:44:21] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 40% [03:46:21] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [03:48:21] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 54% [03:52:21] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 38% [04:40:34] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 32.59, 11.99, 6.04 [04:50:34] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 1.20, 10.55, 11.44 [04:52:34] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.04, 7.37, 10.16 [06:52:40] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=falcomwiki (END - exit=0) [06:52:43] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/initSiteStats.php --wiki=falcomwiki --update (END - exit=0) [06:52:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:52:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:02:08] RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3 [08:22:21] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [08:24:21] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 52% [08:32:21] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [08:38:21] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 59% [08:40:21] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 26% [09:13:29] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49882MiB (10% inode=99%); [09:19:33] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.71, 17.82, 12.73 [09:29:19] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.85, 23.81, 18.49 [09:31:18] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49826MiB (10% inode=99%); [09:31:18] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.14, 24.58, 19.42 [09:34:34] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 22.17, 13.00, 6.18 [09:36:34] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.74, 11.07, 6.24 [09:37:19] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.99, 23.03, 20.48 [09:38:34] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.50, 7.74, 5.60 [09:45:19] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 11.96, 17.77, 19.31 [10:20:03] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49933MiB (10% inode=99%); [11:12:36] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [11:12:36] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [11:14:20] RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [11:14:23] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:15:22] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:17:38] RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:23:37] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.03, 9.27, 4.64 [13:25:35] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.22, 6.33, 4.12 [14:06:12] miraheze/mw-config - pskyechology the build passed. [14:09:03] miraheze/mw-config - pskyechology the build passed. [14:09:11] [02mw-config] 07pskyechology readied pull request #6026: T13996: Add wmgCodeMirrorEnableDefault (07miraheze:03main...07pskyechology:03T13996) 13https://github.com/miraheze/mw-config/pull/6026 [14:09:47] [02mw-config] 07pskyechology merged pull request #6026: T13996: Add wmgCodeMirrorEnableDefault (07miraheze:03main...07pskyechology:03T13996) 13https://github.com/miraheze/mw-config/pull/6026 [14:09:47] [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/b99152119d4379575cfe48b4cef2090306bd309d [14:09:47] 02mw-config/03main 07Skye 03b991521 T13996: Add wmgCodeMirrorEnableDefault (#6026)… [14:10:42] miraheze/mw-config - pskyechology the build passed. [14:12:54] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:12:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:13:20] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [14:13:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:15:29] [1/2] that didn't age too well lol [14:15:30] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1413890353135747104/Screenshot_20250906_161509_GitHub.jpg?ex=68bd9381&is=68bc4201&hm=7a8f5b7a89587980c8cd7b94dd03804124fd8adeda792b220c9514b34ddfe6fd& [14:43:53] would've been worse if I actually made all the default preferences changeable [15:01:16] true [15:21:19] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.53, 18.63, 11.60 [15:25:19] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.59, 21.75, 14.48 [15:27:18] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.04, 22.13, 15.51 [15:29:19] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.42, 19.21, 15.25 [15:57:39] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.34, 10.06, 5.44 [15:59:37] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.81, 6.88, 4.83 [16:09:18] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.33, 19.48, 14.12 [16:11:19] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.36, 20.96, 15.33 [16:19:18] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.01, 22.77, 18.38 [16:21:19] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.69, 23.05, 19.02 [16:27:19] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 23.66, 20.54 [16:35:19] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.53, 22.84, 21.65 [16:37:18] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 8.01, 17.38, 19.80 [17:16:56] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757175380000&orgId=1&to=1757179016683 [17:26:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757176010000&orgId=1&to=1757179616684[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757175380000&orgId=1&to=1757179616684 [17:27:06] [02puppet] 07Universal-Omega created 03mediaModerationScanFileJob (+1 new commit) 13https://github.com/miraheze/puppet/commit/12350dee84e7 [17:27:06] 02puppet/03mediaModerationScanFileJob 07CosmicAlpha 0312350de changeprop: add mediaModerationScanFileJob as own queue [17:27:11] [02puppet] 07Universal-Omega opened pull request #4509: changeprop: add mediaModerationScanFileJob as own queue (03main...03mediaModerationScanFileJob) 13https://github.com/miraheze/puppet/pull/4509 [17:27:18] [02puppet] 07coderabbitai[bot] commented on pull request #4509:

[…] 13https://github.com/miraheze/puppet/pull/4509#issuecomment-3262699670 [17:36:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757176010000&orgId=1&to=1757180216686[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757176400000&orgId=1&to=1757180000000 [17:38:22] [02puppet] 07Universal-Omega pushed 1 new commit to 03mediaModerationScanFileJob 13https://github.com/miraheze/puppet/commit/6c48132996ca74f5a93bdf70b8dbb9b0bae4c005 [17:38:22] 02puppet/03mediaModerationScanFileJob 07CosmicAlpha 036c48132 Use 15 [17:39:35] [02puppet] 07Universal-Omega merged pull request #4509: changeprop: add mediaModerationScanFileJob as own queue (03main...03mediaModerationScanFileJob) 13https://github.com/miraheze/puppet/pull/4509 [17:39:35] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e27b7096bfd4ce0ce34d2c006df412b93f572aa4 [17:39:37] 02puppet/03main 07CosmicAlpha 03e27b709 changeprop: add mediaModerationScanFileJob as own queue [17:39:39] [02puppet] 07Universal-Omega 04deleted 03mediaModerationScanFileJob at 036c48132 13https://api.github.com/repos/miraheze/puppet/commit/6c48132 [18:18:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [18:24:26] PROBLEM - matomo151 Disk Space on matomo151 is WARNING: DISK WARNING - free space: / 1954MiB (10% inode=89%); [18:26:34] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [19:01:56] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 6472MiB (35% inode=89%); [19:04:57] [02dns] 07Universal-Omega created 03editorconfig (+1 new commit) 13https://github.com/miraheze/dns/commit/c3142be041ba [19:04:58] 02dns/03editorconfig 07CosmicAlpha 03c3142be Add .editorconfig [19:05:02] [02dns] 07Universal-Omega opened pull request #694: Add .editorconfig (03main...03editorconfig) 13https://github.com/miraheze/dns/pull/694 [19:06:27] [02dns] 07Universal-Omega merged pull request #694: Add .editorconfig (03main...03editorconfig) 13https://github.com/miraheze/dns/pull/694 [19:06:29] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/1b6e2e0d2645dfb5548c56810b7fef73db398014 [19:06:31] [02dns] 07Universal-Omega 04deleted 03editorconfig at 03c3142be 13https://api.github.com/repos/miraheze/dns/commit/c3142be [19:06:32] 02dns/03main 07CosmicAlpha 031b6e2e0 Add .editorconfig [19:06:42] miraheze/dns - Universal-Omega the build passed. [19:07:37] miraheze/dns - Universal-Omega the build passed. [19:07:38] [02dns] 07Universal-Omega created 03wtnet-ntp (+1 new commit) 13https://github.com/miraheze/dns/commit/07004872fd8f [19:07:38] 02dns/03wtnet-ntp 07CosmicAlpha 030700487 wtnet: add ntp.fsslc.wtnet [19:07:45] [02dns] 07Universal-Omega opened pull request #695: wtnet: add ntp.fsslc.wtnet (03main...03wtnet-ntp) 13https://github.com/miraheze/dns/pull/695 [19:09:51] miraheze/dns - Universal-Omega the build passed. [19:14:30] miraheze/dns - Universal-Omega the build passed. [19:49:37] !log [skye@test151] starting deploy of {'config': True} to test151 [19:49:38] !log [skye@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [19:49:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:14] !log [skye@test151] starting deploy of {'config': True} to test151 [19:53:15] !log [skye@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [19:53:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:31] gonna do a crazy and commit straight to main [20:39:50] miraheze/mw-config - pskyechology the build passed. [20:40:09] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [20:40:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:35] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [20:40:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:52:49] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php updateSpecialPages --wiki=falcomwiki (END - exit=0) [20:52:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:06] !log [universalomega@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_skins': 'Cosmos'} to all [20:56:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:33] !log [universalomega@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_skins': 'Cosmos'} to all - SUCCESS in 26s [20:56:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:56] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757176010000&orgId=1&to=1757193110000 [21:25:51] miraheze/mw-config - Universal-Omega the build passed. [21:26:39] miraheze/mw-config - Universal-Omega the build passed. [21:32:01] miraheze/mw-config - Universal-Omega the build passed. [21:38:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [21:46:34] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [22:17:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757193410000&orgId=1&to=1757197040013 [22:47:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757193410000&orgId=1&to=1757198810000