[00:05:42] miraheze/DiscordNotifications - SomeMWDev the build passed. [00:06:43] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57290MiB (12% inode=99%); [00:07:50] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56911MiB (12% inode=99%); [00:07:55] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57590MiB (12% inode=99%); [00:39:41] !log [blankeclair@mwtask181] Starting import for skibiditoiletwikiwiki (XML: skibiditoiletwiki.xml; Images: images/) (START) [00:39:42] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importDump --wiki=skibiditoiletwikiwiki --no-updates --username-prefix=fandom:skibidi-toilet -- skibiditoiletwiki.xml (START) [00:39:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:26] PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3 [04:51:51] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importDump --wiki=skibiditoiletwikiwiki --no-updates --username-prefix=fandom:skibidi-toilet -- skibiditoiletwiki.xml (END - exit=0) [04:51:52] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=skibiditoiletwikiwiki '--comment=Importing images from https://skibidi-toilet.fandom.com ([[phorge:T13960|T13960]])' -- images/ (START) [04:51:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:51:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:04] lol i forgot about that [05:25:56] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1752467120000&orgId=1&to=1752470756679 [05:35:08] ^ is BlankEclair's image import [05:35:17] ah oki [05:35:51] 10+ GB of images go brrrrr [05:35:56] [Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1752467540000&orgId=1&to=1752471140000 [05:37:19] BlankEclair: in future, add --sleep=1 but don't bother restarting it [05:37:29] It'll recover itself [05:37:30] wait, that's an option? [05:37:37] BlankEclair: ye [05:37:57] oh. [05:38:05] should i set it as the default for the mwimport tool then? [05:39:09] BlankEclair: nah cause it's discretion [05:39:24] If you're only importing a couple of hundred it's fine [05:39:40] so only trigger on file count >= 1000? [05:40:33] That's sensible [06:20:33] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=skibiditoiletwikiwiki '--comment=Importing images from https://skibidi-toilet.fandom.com ([[phorge:T13960|T13960]])' -- images/ (END - exit=0) [06:20:34] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php rebuildall --wiki=skibiditoiletwikiwiki (START) [06:20:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:20:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:31:46] BlankEclair: job queue is showing about 50 minutes to clear [06:31:51] bleh [06:31:56] currently writing the impl right now [06:32:00] should i stop the rebuildall? [06:32:01] or nah [06:32:04] BlankEclair: nah [06:32:07] ah okay [06:32:11] 50 minutes is pretty good [06:32:18] ah ic [06:35:56] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php rebuildall --wiki=skibiditoiletwikiwiki (END - exit=-2) [06:35:57] !log [blankeclair@mwtask181] Finished import for skibiditoiletwikiwiki (XML: skibiditoiletwiki.xml; Images: images/) (END - exit=-2) [06:35:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:36:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:36:13] [14/07/2025 16:25] @blankeclair What's an images_misatch> [06:36:15] [14/07/2025 16:26] bitbyte10: images that are either not the same size as initially uploaded, or not the same file contents as initially uploaded [06:36:16] [14/07/2025 16:35] Oh, we're fine with those getting imported [06:36:18] the context [06:43:17] miraheze/python-functions - BlankEclair the build has errored. [06:52:17] miraheze/python-functions - BlankEclair the build passed. [06:52:57] !log [blankeclair@mwtask181] Starting import for skibiditoiletwikiwiki (XML: None; Images: images_mismatch/) (START) [06:52:58] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=skibiditoiletwikiwiki --sleep=1 '--comment=Importing images from https://skibidi-toilet.fandom.com ([[phorge:T13960|T13960]])' -- images_mismatch/ (START) [06:53:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:53:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:53:57] RhinosF1: up to review and ? [06:53:57] [GitHub] [miraheze/python-functions #156] open PR by BlankEclair, created 11 minutes, 56 seconds ago: Add support for --sleep in importImages | * **New Features** […] [06:53:57] [GitHub] [miraheze/python-functions #157] open PR by BlankEclair, created 6 minutes, 8 seconds ago: Fix test_check_up_debug_fail | * **Tests** […] [06:55:54] !log [blankeclair@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php ManageWiki:PopulateNamespaces --wiki=exttestwikibeta (END - exit=256) [06:55:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:56:20] !log [blankeclair@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php ManageWiki:PopulateNamespaces --wiki=exttestwikibeta --force (END - exit=0) [06:56:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:23:02] miraheze/python-functions - Universal-Omega the build passed. [07:48:15] miraheze/python-functions - BlankEclair the build passed. [07:56:03] miraheze/python-functions - Universal-Omega the build passed. [08:02:26] RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3 [10:11:00] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=skibiditoiletwikiwiki --sleep=1 '--comment=Importing images from https://skibidi-toilet.fandom.com ([[phorge:T13960|T13960]])' -- images_mismatch/ (END - exit=0) [10:11:01] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initSiteStats --wiki=skibiditoiletwikiwiki --update (START) [10:11:02] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initSiteStats --wiki=skibiditoiletwikiwiki --update (END - exit=0) [10:11:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:11:03] !log [blankeclair@mwtask181] Finished import for skibiditoiletwikiwiki (XML: None; Images: images_mismatch/) (END - exit=0) [10:11:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:11:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:11:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:11:50] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49835MiB (10% inode=99%); [11:03:55] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49927MiB (10% inode=99%); [11:16:43] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49885MiB (10% inode=99%); [12:11:15] [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/5cbff11a7e311ada8559b42f1140f14469f7e528 [12:11:15] 02DiscordNotifications/03main 07translatewiki.net 035cbff11 Localisation updates from https://translatewiki.net. [12:11:15] [02CreateWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/1ef69175e4173f96d75a3b7259a1aa56d2dd7703 [12:11:15] 02CreateWiki/03main 07translatewiki.net 031ef6917 Localisation updates from https://translatewiki.net. [12:11:16] [02ErrorPages] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ErrorPages/commit/f9f69c6ef06492ee0fbe0aad40f1df2623f673ef [12:11:18] 02ErrorPages/03main 07translatewiki.net 03f9f69c6 Localisation updates from https://translatewiki.net. [12:11:20] [02ImportDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ImportDump/commit/77e5d41f140247eca63c681765c33af44294d47a [12:11:22] [02DataDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DataDump/commit/8a88b2cbcc38625069aa4ba885a7c0d502d4a9c2 [12:11:23] 02DataDump/03main 07translatewiki.net 038a88b2c Localisation updates from https://translatewiki.net. [12:11:25] 02ImportDump/03main 07translatewiki.net 0377e5d41 Localisation updates from https://translatewiki.net. [12:11:27] [02PDFEmbed] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/PDFEmbed/commit/7c9ffdf79d7d2fb891927da8b5ba68eacc45d399 [12:11:29] [02MatomoAnalytics] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MatomoAnalytics/commit/058e7aa4173833605c5fdbbf9b898b6f8e9edeec [12:11:31] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/fb869905c7f1a08307e14743112e6f2e3c8ecb7c [12:11:33] [02RemovePII] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/RemovePII/commit/d06198549cf34d16e84e097d20a0bb498ffea3b2 [12:11:35] [02IncidentReporting] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/IncidentReporting/commit/690fc3ca27a93c2476ab1ce4c42ab45225b9ec2a [12:11:37] [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/b4f8a0323a9b83241638c0e761f921a187dc877a [12:11:39] [02RequestSSL] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/RequestSSL/commit/5ad2439b91c4e8fdf6cf4e806eeba3af28f7961e [12:11:41] [02RottenLinks] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/RottenLinks/commit/5f05d8e0dc6564b1f3df09f668bb746274ff8b0d [12:11:43] [02WikiDiscover] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/WikiDiscover/commit/add9b58f539f37dde1bd75dec0191e4d846a22c0 [12:11:44] 02PDFEmbed/03main 07translatewiki.net 037c9ffdf Localisation updates from https://translatewiki.net. [12:11:46] 02MatomoAnalytics/03main 07translatewiki.net 03058e7aa Localisation updates from https://translatewiki.net. [12:11:48] 02MirahezeMagic/03main 07translatewiki.net 03fb86990 Localisation updates from https://translatewiki.net. [12:11:49] 02RemovePII/03main 07translatewiki.net 03d061985 Localisation updates from https://translatewiki.net. [12:11:51] 02IncidentReporting/03main 07translatewiki.net 03690fc3c Localisation updates from https://translatewiki.net. [12:11:52] 02ManageWiki/03main 07translatewiki.net 03b4f8a03 Localisation updates from https://translatewiki.net. [12:11:54] 02RequestSSL/03main 07translatewiki.net 035ad2439 Localisation updates from https://translatewiki.net. [12:11:56] 02RottenLinks/03main 07translatewiki.net 035f05d8e Localisation updates from https://translatewiki.net. [12:11:57] 02WikiDiscover/03main 07translatewiki.net 03add9b58 Localisation updates from https://translatewiki.net. [12:15:35] miraheze/ErrorPages - translatewiki the build passed. [12:15:49] miraheze/IncidentReporting - translatewiki the build passed. [12:21:12] miraheze/DataDump - translatewiki the build passed. [12:21:14] miraheze/MatomoAnalytics - translatewiki the build passed. [12:21:42] miraheze/RottenLinks - translatewiki the build passed. [12:21:47] miraheze/PDFEmbed - translatewiki the build passed. [12:22:11] miraheze/ManageWiki - translatewiki the build passed. [12:23:32] miraheze/RemovePII - translatewiki the build passed. [12:24:36] miraheze/WikiDiscover - translatewiki the build passed. [12:29:40] miraheze/ImportDump - translatewiki the build passed. [12:29:46] miraheze/MirahezeMagic - translatewiki the build passed. [12:30:57] miraheze/CreateWiki - translatewiki the build passed. [12:32:27] miraheze/DiscordNotifications - translatewiki the build passed. [12:37:43] miraheze/RequestSSL - translatewiki the build passed. [13:50:01] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 54% [13:54:01] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 29% [16:10:20] !log [skye@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'DynamicPageList3'} to test151 [16:10:21] !log [skye@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [16:10:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:11:16] ...128? [16:35:50] Thats a non zero number [16:36:37] that much I figured but I've no idea as to the implications of it [16:37:00] What's the error on the cli? [16:37:27] you're gonna like this one [16:37:45] [1/7] ``` [16:37:46] [2/7] [skye@test151:/srv/mediawiki-staging]$ mwdeploy --upgrade-extensions=DynamicPageList3 --servers=all [16:37:46] [3/7] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:1053: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings [16:37:46] [4/7] warnings.warn( [16:37:47] [5/7] Failed to upgrade DynamicPageList3 (exit code: 128). [16:37:47] [6/7] Exiting due to non-zero status. [16:37:47] [7/7] ``` [16:37:58] Fuck deploy tool [16:38:17] I have no idea why that isn't using the wrapper [16:39:05] Anything calling a command that's tracked should call run_command [16:41:28] It's calling git pull [16:43:20] Try running sudo -u www-data git -C /srv/mediawiki-staging/1.44/DynamicPageList3 pull --recurse-submodules [16:43:28] Thats because of merge conflict btw since beta DPL3 isnt running main. But its fixed now. [16:43:40] ah [16:43:52] I think @cosmicalpha might have deployed something broken which he's just said ye [16:44:08] I need to fix that code to tell you what it ran [16:45:44] Every shell out in that script used to give debug lines [16:45:51] That said Execute: X [16:46:13] I have DPL3 pre release pulled to beta yeah so when main has changes it causes merge conflicts with git pull (and main had i18n changes not on pre release) [16:46:52] There's a reason I want to be able to change mediawiki-repos so we can have a different branch on beta to prod too [16:47:08] Yeah its something I'll work on very soon. [16:47:58] ill be merging branch, commit, and alpha_branch into one option that can be specified per MW version and accept any commits also. [18:12:56] !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'ArticleRatings'} to test151 [18:12:57] !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'ArticleRatings'} to test151 - SUCCESS in 1s [18:12:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:13:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:08:39] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 13.36, 13.78, 7.27 [20:12:39] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 7.85, 10.96, 7.67 [20:14:39] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 1.98, 7.75, 6.90 [21:57:40] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o