[00:01:21] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/40c1d3ababe23fc8ef44afb3b9bb9968b769145e [00:01:22] 02statichelp/03main 07WikiTideBot 0340c1d3a Bot: Auto-update Tech namespace pages 2025-09-13 00:01:19 [00:01:26] [02ManageWiki] 07Universal-Omega created 03installer-indexes (+1 new commit) 13https://github.com/miraheze/ManageWiki/commit/05a117685310 [00:01:27] 02ManageWiki/03installer-indexes 07CosmicAlpha 0305a1176 Installer: add support for index patches [00:01:30] [02ManageWiki] 07Universal-Omega opened pull request #745: Installer: add support for index patches (03main...03installer-indexes) 13https://github.com/miraheze/ManageWiki/pull/745 [00:01:37] [02ManageWiki] 07coderabbitai[bot] commented on pull request #745:

[…] 13https://github.com/miraheze/ManageWiki/pull/745#issuecomment-3287189085 [00:01:59] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03installer-indexes 13https://github.com/miraheze/ManageWiki/commit/fe55b7b423fa368d28f67788260f9ea78a83d67b [00:01:59] 02ManageWiki/03installer-indexes 07CosmicAlpha 03fe55b7b Update [00:07:55] miraheze/ManageWiki - Universal-Omega the build passed. [00:08:12] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 56461MiB (12% inode=99%); [00:08:37] !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'CookieConsent'} to test151 [00:08:38] !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'CookieConsent'} to test151 - SUCCESS in 1s [00:08:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:08:41] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 56674MiB (12% inode=99%); [00:08:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:09:16] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 55984MiB (12% inode=99%); [00:42:49] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03installer-indexes 13https://github.com/miraheze/ManageWiki/commit/6542dcfb2414a7ef5be230837f94897a5fc304ee [00:42:49] 02ManageWiki/03installer-indexes 07CosmicAlpha 036542dcf Update [00:43:56] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03installer-indexes 13https://github.com/miraheze/ManageWiki/commit/20c952f26e0d1b098037f7f3f95b1c42750889aa [00:43:56] 02ManageWiki/03installer-indexes 07CosmicAlpha 0320c952f Update [00:49:43] miraheze/ManageWiki - Universal-Omega the build passed. [02:06:50] [02mw-config] 07pskyechology opened pull request #6106: Change footer icons for thechurchofthestatuewiki (07miraheze:03main...07pskyechology:03T14269) 13https://github.com/miraheze/mw-config/pull/6106 [02:07:47] miraheze/mw-config - pskyechology the build passed. [02:08:14] [02mw-config] 07pskyechology merged pull request #6106: Change footer icons for thechurchofthestatuewiki (07miraheze:03main...07pskyechology:03T14269) 13https://github.com/miraheze/mw-config/pull/6106 [02:08:15] [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/239fad9cdd42cd62d4583bebf6af47684e050b85 [02:08:15] 02mw-config/03main 07Skye 03239fad9 Change footer icons for thechurchofthestatuewiki (#6106)… [02:08:31] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [02:08:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:08:55] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [02:08:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:09:08] miraheze/mw-config - pskyechology the build passed. [02:49:00] PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3 [05:21:16] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49824MiB (10% inode=99%); [05:50:40] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49909MiB (10% inode=99%); [05:58:12] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49863MiB (10% inode=99%); [08:03:00] RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3 [11:46:45] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 56.90, 23.13, 9.18 [11:50:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.44, 20.58, 16.32 [11:50:45] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 2.39, 11.15, 7.41 [11:52:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.48, 21.59, 17.21 [11:52:45] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.05, 7.79, 6.63 [11:54:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.72, 22.63, 18.12 [11:58:06] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.33, 23.08, 19.35 [12:00:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.68, 23.97, 20.13 [12:02:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.93, 23.76, 20.54 [12:04:45] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 15.93, 12.49, 8.30 [12:06:45] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.88, 8.62, 7.39 [12:12:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.82, 23.74, 21.92 [12:12:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757761910000&orgId=1&to=1757765540013 [12:20:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.15, 23.99, 22.91 [12:22:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 24.06, 23.06 [12:22:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757761910000&orgId=1&to=1757766140014[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757762240000&orgId=1&to=1757766140014 [12:32:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757761910000&orgId=1&to=1757766740017[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757762900000&orgId=1&to=1757766500000 [13:46:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.67, 23.24, 23.90 [13:52:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757761910000&orgId=1&to=1757771540026[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757767700000&orgId=1&to=1757771540027 [14:08:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.80, 23.68, 23.48 [14:10:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.28, 23.18, 23.33 [14:14:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.49, 23.62, 23.46 [14:16:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.98, 23.22, 23.35 [14:18:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.19, 24.04, 23.63 [14:20:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.26, 23.73, 23.58 [14:22:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757761910000&orgId=1&to=1757773340030[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757769440000&orgId=1&to=1757773040000 [14:24:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.21, 23.58, 23.53 [14:30:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.69, 22.62, 23.28 [15:16:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.81, 23.55, 23.02 [15:22:14] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'EmbedVideo'} to test151 [15:22:16] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'EmbedVideo'} to test151 - SUCCESS in 2s [15:22:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:22:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:30:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.67, 23.83, 23.68 [15:34:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.26, 24.27, 23.87 [15:40:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.16, 23.67, 23.79 [15:47:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757761910000&orgId=1&to=1757778440040[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757774660000&orgId=1&to=1757778440040 [15:52:04] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importDump --wiki=--wiki=utaitewiki --no-updates --username-prefix=fandom:utaite -- ./utaite1.xml (START) [15:52:05] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importDump --wiki=--wiki=utaitewiki --no-updates --username-prefix=fandom:utaite -- ./utaite1.xml (END - exit=0) [15:52:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:52:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:52:40] do i have an stupid [15:52:49] LMAO YEAH I DO [15:53:18] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importDump --wiki=utaitewiki --no-updates --username-prefix=fandom:utaite -- ./utaite1.xml (START) [15:53:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.49, 23.71, 23.06 [16:00:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.69, 23.77, 23.25 [16:02:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.58, 23.97, 23.38 [16:12:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.25, 23.77, 23.66 [16:17:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757761910000&orgId=1&to=1757780240046[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757776400000&orgId=1&to=1757780000000 [16:17:31] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importDump --wiki=utaitewiki --no-updates --username-prefix=fandom:utaite -- ./utaite2.xml (START) [16:17:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:06] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.28, 23.85, 23.71 [16:19:48] i do be bullying mwtask181 rn [16:20:16] (to be fair it's the jobs mainly) [16:20:20] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importDump --wiki=utaitewiki --no-updates --username-prefix=fandom:utaite -- ./utaite1.xml (END - exit=256) [16:20:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:24:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.22, 23.84, 23.82 [16:24:09] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importDump --wiki=utaitewiki --no-updates --username-prefix=fandom:utaite -- ./utaite2.xml (END - exit=0) [16:24:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:27:02] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'EmbedVideo'} to all [16:27:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:29:51] i guess i should go curse them some more for multiple xml files [16:30:18] [1/6] but just in case someone smarter than me knows better: [16:30:18] [2/6] ``` [16:30:19] [3/6] Wikimedia\Rdbms\DBQueryError from line 1232 of /srv/mediawiki/1.44/includes/libs/rdbms/database/Database.php: Error 1062: Duplicate entry '141235' for key 'PRIMARY' [16:30:19] [4/6] Function: MediaWiki\Revision\RevisionStore::insertRevisionRowOn [16:30:19] [5/6] Query: INSERT INTO `revision` (rev_page,rev_parent_id,rev_actor,rev_minor_edit,rev_timestamp,rev_deleted,rev_len,rev_sha1,rev_comment_id,rev_id) VALUES (4991,141227,7071,0,'20140921032850',0,14536,'31amiz3kwglqok7r5ng8cp4a18qe8ad',39515,141235) [16:30:20] [6/6] ``` [16:30:43] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'EmbedVideo'} to test151 [16:30:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:30:55] [1/12] ``` [16:30:55] [2/12] #0 /srv/mediawiki/1.44/includes/libs/rdbms/database/Database.php(1216): Wikimedia\Rdbms\Database->getQueryException('Duplicate entry...', 1062, 'INSERT INTO `re...', 'MediaWiki\\Revis...') [16:30:55] [3/12] #1 /srv/mediawiki/1.44/includes/libs/rdbms/database/Database.php(1190): Wikimedia\Rdbms\Database->getQueryExceptionAndLog('Duplicate entry...', 1062, 'INSERT INTO `re...', 'MediaWiki\\Revis...') [16:30:56] [4/12] #2 /srv/mediawiki/1.44/includes/libs/rdbms/database/Database.php(647): Wikimedia\Rdbms\Database->reportQueryError('Duplicate entry...', 1062, 'INSERT INTO `re...', 'MediaWiki\\Revis...', false) [16:30:56] [5/12] #3 /srv/mediawiki/1.44/includes/libs/rdbms/database/Database.php(1498): Wikimedia\Rdbms\Database->query(Object(Wikimedia\Rdbms\Query), 'MediaWiki\\Revis...') [16:30:56] [6/12] #4 /srv/mediawiki/1.44/includes/libs/rdbms/querybuilder/InsertQueryBuilder.php(340): Wikimedia\Rdbms\Database->insert('revision', Array, 'MediaWiki\\Revis...', Array) [16:30:57] [7/12] #5 /srv/mediawiki/1.44/includes/Revision/RevisionStore.php(863): Wikimedia\Rdbms\InsertQueryBuilder->execute() [16:30:57] [8/12] #6 /srv/mediawiki/1.44/includes/Revision/RevisionStore.php(632): MediaWiki\Revision\RevisionStore->insertRevisionRowOn(Object(Wikimedia\Rdbms\DatabaseMySQL), Object(MediaWiki\Revision\MutableRevisionRecord), 141227) [16:30:57] [9/12] #7 /srv/mediawiki/1.44/includes/Revision/RevisionStore.php(483): MediaWiki\Revision\RevisionStore->insertRevisionInternal(Object(MediaWiki\Revision\MutableRevisionRecord), Object(Wikimedia\Rdbms\DatabaseMySQL), Object(MediaWiki\User\User), Object(MediaWiki\CommentStore\CommentStoreComment), Object(MediaWiki\Page\PageIdentityValue), 4991, 141227) [16:30:57] [10/12] #8 /srv/mediawiki/1.44/includes/libs/rdbms/database/Database.php(2293): MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Object(Wikimedia\Rdbms\DatabaseMySQL), 'MediaWiki\\Revis...') [16:30:58] [11/12] #9 /srv/mediawiki/1.44/includes/libs/rdbms/database/DBConnRef.php(127): Wikimedia\Rdbms\Database->doAtomicSection('MediaWiki\\Revis...', Object(Closure)) [16:30:58] [12/12] ``` [16:31:11] [1/19] ``` [16:31:11] [2/19] #10 /srv/mediawiki/1.44/includes/libs/rdbms/database/DBConnRef.php(661): Wikimedia\Rdbms\DBConnRef->__call('doAtomicSection', Array) [16:31:11] [3/19] #11 /srv/mediawiki/1.44/includes/Revision/RevisionStore.php(474): Wikimedia\Rdbms\DBConnRef->doAtomicSection('MediaWiki\\Revis...', Object(Closure)) [16:31:12] [4/19] #12 /srv/mediawiki/1.44/includes/import/ImportableOldRevisionImporter.php(175): MediaWiki\Revision\RevisionStore->insertRevisionOn(Object(MediaWiki\Revision\MutableRevisionRecord), Object(Wikimedia\Rdbms\DBConnRef)) [16:31:12] [5/19] #13 /srv/mediawiki/1.44/includes/import/WikiRevision.php(640): ImportableOldRevisionImporter->import(Object(WikiRevision)) [16:31:12] [6/19] #14 /srv/mediawiki/1.44/includes/import/WikiImporter.php(490): WikiRevision->importOldRevision() [16:31:13] [7/19] #15 /srv/mediawiki/1.44/maintenance/importDump.php(210): WikiImporter->importRevision(Object(WikiRevision)) [16:31:13] [8/19] #16 /srv/mediawiki/1.44/includes/import/WikiImporter.php(612): BackupReader->handleRevision(Object(WikiRevision), Object(WikiImporter)) [16:31:13] [9/19] #17 /srv/mediawiki/1.44/includes/import/WikiImporter.php(1110): WikiImporter->revisionCallback(Object(WikiRevision)) [16:31:14] [10/19] #18 /srv/mediawiki/1.44/includes/import/WikiImporter.php(977): WikiImporter->processRevision(Array, Array) [16:31:14] [11/19] #19 /srv/mediawiki/1.44/includes/import/WikiImporter.php(909): WikiImporter->handleRevision(Array) [16:31:14] [12/19] #20 /srv/mediawiki/1.44/includes/import/WikiImporter.php(727): WikiImporter->handlePage() [16:31:14] [13/19] #21 /srv/mediawiki/1.44/maintenance/importDump.php(367): WikiImporter->doImport() [16:31:15] [14/19] #22 /srv/mediawiki/1.44/maintenance/importDump.php(296): BackupReader->importFromHandle(Resource id #1661) [16:31:15] [15/19] #23 /srv/mediawiki/1.44/maintenance/importDump.php(143): BackupReader->importFromFile('./utaite1.xml') [16:31:16] [16/19] #24 /srv/mediawiki/1.44/maintenance/includes/MaintenanceRunner.php(694): BackupReader->execute() [16:31:16] [17/19] #25 /srv/mediawiki/1.44/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run() [16:31:17] [18/19] #26 {main} [16:31:17] [19/19] ``` [16:33:37] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importDump --wiki=utaitewiki --no-updates --username-prefix=fandom:utaite -- ./utaite1.xml (START) [16:33:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:33:54] what could go wrong if i do it again :3 [16:34:16] Could kill the farm [16:34:20] But minor detail [16:34:49] nah i kill the farm when i accidentally add a tiny little * when deleting a swift container [16:35:04] i sure hope it doesn't actually work like that [16:35:31] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'EmbedVideo'} to all - SUCCESS in 508s [16:35:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:40] well it made it further way faster than last time so ig this is a good sign [16:39:56] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'EmbedVideo'} to test151 - SUCCESS in 553s [16:40:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:42:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757761910000&orgId=1&to=1757781740048[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757778080000&orgId=1&to=1757781740048 [16:44:17] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importDump --wiki=utaitewiki --no-updates --username-prefix=fandom:utaite -- ./utaite1.xml (END - exit=0) [16:44:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:47:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757761910000&orgId=1&to=1757782040049[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757778200000&orgId=1&to=1757781800000 [16:47:28] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importDump --wiki=utaitewiki --no-updates --username-prefix=fandom:utaite -- ./utaite3.xml (START) [16:47:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:49:54] miraheze/ManageWiki - Universal-Omega the build passed. [16:53:04] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importDump --wiki=utaitewiki --no-updates --username-prefix=fandom:utaite -- ./utaite3.xml (END - exit=0) [16:53:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:40] !log [skye@mwtask171] Starting import for utaitewiki (XML: ./utaite4.xml; Images: None) (START) [16:56:41] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importDump --wiki=utaitewiki --no-updates --username-prefix=fandom:utaite -- ./utaite4.xml (START) [16:56:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:48] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importDump --wiki=utaitewiki --no-updates --username-prefix=fandom:utaite -- ./utaite4.xml (END - exit=0) [16:56:49] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php rebuildall --wiki=utaitewiki (START) [16:56:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:12:22] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.39, 20.37, 14.94 [17:14:18] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 20.06, 20.21, 15.51 [17:17:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757761910000&orgId=1&to=1757783840055[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1757779940000&orgId=1&to=1757783840055 [17:18:13] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 28.01, 23.84, 17.97 [17:18:55] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [17:18:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:22] miraheze/mw-config - pskyechology the build passed. [17:19:27] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 32s [17:19:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:20:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.16, 23.40, 22.92 [17:20:09] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.23, 22.91, 18.32 [17:21:38] !log silence job queue grafana alerts [17:21:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:22:05] damn rhinos really said stfu [17:24:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.11, 23.56, 23.12 [17:29:49] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 26.21, 22.40, 19.81 [17:30:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 24.05, 23.47 [17:31:45] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.86, 22.21, 20.03 [17:32:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.76, 23.38, 23.31 [17:35:12] Yes [17:35:19] It's not like we're going to action it [17:35:23] It's going down [17:35:26] Just very slowly [17:35:32] Cause loginwiki [17:35:52] We need more capacity. We can't make it run quicker. [17:36:18] Unless someone writes a runJobs script that replicates the web version of runSingleJob and runJobs @pskyechology [17:36:18] indeed [17:36:40] if i did that it would probably be more like runExplosion [17:37:13] I mean we should probably write it at some point [17:37:28] calling it runExplosion would be fun to be honest [17:37:35] You're exploding the queue [17:39:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 27.99, 24.19, 21.57 [17:40:05] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.42, 14.61, 19.90 [17:41:58] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 19.65, 15.92 [17:43:58] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 22.19, 20.37, 16.63 [17:49:58] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 13.15, 17.70, 16.92 [17:51:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 19.12, 23.60, 23.43 [18:03:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 28.77, 22.52, 21.93 [18:20:58] miraheze/ManageWiki - Universal-Omega the build passed. [18:36:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.11, 15.05, 10.14 [18:38:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.93, 18.36, 11.95 [18:42:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.68, 21.03, 14.65 [18:47:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 13.61, 18.47, 22.79 [18:48:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.09, 22.65, 17.45 [18:50:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.97, 22.93, 18.19 [18:55:32] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 10.58, 14.34, 19.62 [19:08:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.77, 23.16, 21.55 [19:10:02] miraheze/ManageWiki - Universal-Omega the build passed. [19:10:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.99, 22.93, 21.67 [19:12:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.54, 23.70, 22.09 [19:14:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.19, 23.69, 22.30 [19:17:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 25.28, 19.95, 16.88 [19:18:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.69, 24.05, 22.75 [19:21:07] miraheze/mw-config - pskyechology the build passed. [19:24:05] !log [skye@test151] starting deploy of {'config': True, 'folders': '1.45/extensions/ManageWiki'} to test151 [19:24:06] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.52, 23.94, 23.20 [19:24:06] !log [skye@test151] finished deploy of {'config': True, 'folders': '1.45/extensions/ManageWiki'} to test151 - SUCCESS in 0s [19:24:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.93, 23.28, 20.34 [19:29:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 24.17, 23.84, 20.91 [19:31:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 20.58, 23.15, 21.06 [19:36:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.34, 23.64, 23.31 [19:37:20] miraheze/mw-config - pskyechology the build passed. [19:37:22] PROBLEM - test151 Check unit status of mediawiki_job_mediamoderation-scan on test151 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_mediamoderation-scan [19:38:56] !log [skye@test151] starting deploy of {'config': True} to test151 [19:38:57] !log [skye@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [19:39:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 25.18, 23.71, 22.19 [19:41:39] miraheze/mw-config - Universal-Omega the build passed. [19:49:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.73, 23.16, 22.62 [19:50:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.90, 23.81, 23.77 [19:50:17] miraheze/mw-config - pskyechology the build passed. [19:51:00] !log [skye@test151] starting deploy of {'config': True} to test151 [19:51:01] !log [skye@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [19:51:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 24.58, 23.90, 22.96 [19:52:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.44, 23.89, 23.80 [19:53:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 23.47, 23.80, 23.04 [19:54:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.06, 23.58, 23.70 [19:55:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 25.49, 24.59, 23.43 [19:57:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.06, 23.83, 23.32 [19:59:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 27.46, 25.32, 23.93 [20:00:06] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.14, 24.19, 23.90 [20:01:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.33, 23.70, 23.50 [20:07:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 24.12, 22.23, 22.68 [20:09:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 19.80, 21.46, 22.35 [20:12:54] miraheze/mw-config - pskyechology the build passed. [20:13:46] miraheze/mw-config - pskyechology the build passed. [20:17:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 24.04, 21.88, 22.01 [20:18:18] !log [skye@test151] starting deploy of {'config': True} to test151 [20:18:19] !log [skye@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [20:18:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:23:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 23.84, 23.24, 22.66 [20:25:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 28.03, 24.88, 23.31 [20:27:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.18, 23.51, 23.03 [20:31:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 24.83, 23.19, 22.95 [20:33:42] !log [skye@test151] starting deploy of {'folders': '1.45/extensions/ManageWiki'} to test151 [20:33:43] !log [skye@test151] finished deploy of {'folders': '1.45/extensions/ManageWiki'} to test151 - SUCCESS in 0s [20:33:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:33:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:35:22] RECOVERY - test151 Check unit status of mediawiki_job_mediamoderation-scan on test151 is OK: OK: Status of the systemd unit mediawiki_job_mediamoderation-scan [20:39:28] miraheze/mw-config - pskyechology the build passed. [20:39:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.18, 23.13, 23.06 [20:40:04] !log [skye@test151] starting deploy of {'config': True} to test151 [20:40:05] !log [skye@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [20:40:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:45:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 26.93, 23.78, 23.26 [20:48:16] !log [universalomega@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'ManageWiki'} to all [20:48:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:49] !log [universalomega@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'ManageWiki'} to all - SUCCESS in 33s [20:48:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:50:00] miraheze/mw-config - Universal-Omega the build passed. [20:53:39] miraheze/ManageWiki - Universal-Omega the build passed. [20:55:10] miraheze/mw-config - Universal-Omega the build passed. [20:56:47] miraheze/mw-config - pskyechology the build passed. [20:57:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 17.41, 23.15, 23.90 [20:58:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 17.51, 22.55, 23.96 [21:02:18] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'versions': '1.44', 'upgrade_extensions': ['ManageWiki', 'SearchDigest']} to all [21:02:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:02:30] miraheze/mw-config - pskyechology the build passed. [21:02:51] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'versions': '1.44', 'upgrade_extensions': ['ManageWiki', 'SearchDigest']} to all - SUCCESS in 32s [21:02:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:49] miraheze/mw-config - Universal-Omega the build passed. [21:05:19] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [21:05:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:05:29] miraheze/mw-config - Universal-Omega the build passed. [21:05:47] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s [21:05:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:06:53] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [21:06:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:07:22] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s [21:07:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:07:29] miraheze/mw-config - Universal-Omega the build passed. [21:07:45] miraheze/mw-config - Universal-Omega the build passed. [21:09:21] @pskyechology It seems that the SearchDigest special page might be cached? I changed the number of misses to 1 but still don't see anything on the special page. [21:09:23] miraheze/mw-config - Universal-Omega the build passed. [21:09:41] miraheze/mw-config - Universal-Omega the build passed. [21:09:54] i believe it is based on jobs and our job queue is filled to the brim with GUP [21:10:04] Oh that makes a lot of sense [21:10:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.13, 23.14, 23.50 [21:10:35] > When a failed search is made, a job is added to the job queue (why?) to update the count in the database. You will need to periodically run the runJobs.php maintenance script or set a value for $wgJobRunRate to process them. [21:11:32] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 27.14, 22.20, 21.93 [21:11:52] The GUP has a like 15 hour backlog [21:12:05] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.03, 22.61, 23.25 [21:12:23] 794 minutes [21:13:05] CA told me he's gonna beat two mwtasks into existence tomorrowish [21:13:32] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 23.03, 22.34, 22.03 [21:13:42] may also be called mwjobrunners [21:15:39] @pskyechology yes [21:15:51] The job queue isn't like a 1 in, 1 out system though [21:16:25] So a backlog taking 794 minutes to clear doesn't mean that a job sent now won't run for 794 minutes [21:16:55] miraheze/mw-config - Universal-Omega the build passed. [21:17:06] It should be a lot less than that [21:17:19]