[00:03:50] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 55947MiB (12% inode=99%); [00:04:13] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 57888MiB (12% inode=99%); [00:59:02] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03add-hook 13https://github.com/miraheze/CreateWiki/commit/4f9331cc53b26c61745de45917a2f691d548bc02 [00:59:02] 02CreateWiki/03add-hook 07CosmicAlpha 034f9331c Cleanup [01:11:12] miraheze/CreateWiki - Universal-Omega the build passed. [01:25:11] [02RequestSSL] 07Universal-Omega created 03namespaced-classes from 03master (+0 new commit) 13https://github.com/miraheze/RequestSSL/compare/namespaced-classes [01:25:47] [02RequestSSL] 07Universal-Omega pushed 1 new commit to 03namespaced-classes 13https://github.com/miraheze/RequestSSL/commit/0a194bde65759310f3026d2b5989315f100f40f6 [01:25:49] 02RequestSSL/03namespaced-classes 07CosmicAlpha 030a194bd Use namespaced classes [01:26:01] [02RequestSSL] 07Universal-Omega opened pull request #69: Use namespaced classes (03master...03namespaced-classes) 13https://github.com/miraheze/RequestSSL/pull/69 [01:26:11] [02RequestSSL] 07coderabbitai[bot] commented on pull request #69: --- […] 13https://github.com/miraheze/RequestSSL/pull/69#issuecomment-2738781937 [01:27:35] [02RequestSSL] 07github-actions[bot] pushed 1 new commit to 03namespaced-classes 13https://github.com/miraheze/RequestSSL/commit/32501b60173d5e98139e7415798e433a9f2b090a [01:27:35] 02RequestSSL/03namespaced-classes 07github-actions 0332501b6 CI: lint code to MediaWiki standards… [01:33:04] [02RequestSSL] 07Universal-Omega merged pull request #69: Use namespaced classes (03master...03namespaced-classes) 13https://github.com/miraheze/RequestSSL/pull/69 [01:33:04] [02RequestSSL] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/RequestSSL/commit/70e3c663e7cdbda7d8cac371ef7ea2a51a310373 [01:33:04] 02RequestSSL/03master 07CosmicAlpha 0370e3c66 Use namespaced classes [01:33:05] [02RequestSSL] 07Universal-Omega 04deleted 03namespaced-classes at 0332501b6 13https://github.com/miraheze/RequestSSL/commit/32501b6 [01:36:49] [02IncidentReporting] 07Universal-Omega created 03namespace (+1 new commit) 13https://github.com/miraheze/IncidentReporting/commit/3380547f57f7 [01:36:49] 02IncidentReporting/03namespace 07CosmicAlpha 033380547 Namespace extension [01:36:56] [02IncidentReporting] 07Universal-Omega opened pull request #101: Namespace extension (03master...03namespace) 13https://github.com/miraheze/IncidentReporting/pull/101 [01:37:00] [02IncidentReporting] 07coderabbitai[bot] commented on pull request #101: --- […] 13https://github.com/miraheze/IncidentReporting/pull/101#issuecomment-2738808824 [01:37:48] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03namespace 13https://github.com/miraheze/IncidentReporting/commit/2b5084c74b7cde5998dd0a00ae97e921da0a9646 [01:37:48] 02IncidentReporting/03namespace 07CosmicAlpha 032b5084c Update [01:38:10] miraheze/RequestSSL - Universal-Omega the build passed. [01:40:34] miraheze/IncidentReporting - Universal-Omega the build has errored. [01:42:18] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03namespace 13https://github.com/miraheze/IncidentReporting/commit/ffc2bd3c7f6f406fbb6a4148a2d805cfd05b37cc [01:42:18] 02IncidentReporting/03namespace 07CosmicAlpha 03ffc2bd3 Update [01:44:41] miraheze/RequestSSL - Universal-Omega the build passed. [01:44:46] miraheze/IncidentReporting - Universal-Omega the build has errored. [01:45:45] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03namespace 13https://github.com/miraheze/IncidentReporting/commit/cf1d300c874c5c9703479d2241ed5fb7d0899bff [01:45:45] 02IncidentReporting/03namespace 07CosmicAlpha 03cf1d300 Update [01:46:48] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03namespace 13https://github.com/miraheze/IncidentReporting/commit/a3ceb02c55b0eda297937769bbc4d69b9e3d12e5 [01:46:48] 02IncidentReporting/03namespace 07CosmicAlpha 03a3ceb02 Update [01:47:13] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03namespace 13https://github.com/miraheze/IncidentReporting/commit/b6728fb89be4a2decb7a5e72d88df254b4a9d806 [01:47:13] 02IncidentReporting/03namespace 07CosmicAlpha 03b6728fb Fix [01:49:18] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03namespace 13https://github.com/miraheze/IncidentReporting/commit/f1e9ce89ee5045cd1bdbf4a057bcffb0fcdc8953 [01:49:18] 02IncidentReporting/03namespace 07CosmicAlpha 03f1e9ce8 Update [01:49:44] miraheze/IncidentReporting - Universal-Omega the build has errored. [01:51:32] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03namespace 13https://github.com/miraheze/IncidentReporting/commit/b18c5e18b70d1288aac8978cf1aafc7a708b13d1 [01:51:32] 02IncidentReporting/03namespace 07CosmicAlpha 03b18c5e1 Update [01:51:39] miraheze/IncidentReporting - Universal-Omega the build has errored. [01:53:38] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03namespace 13https://github.com/miraheze/IncidentReporting/commit/64a8c9e6149a6656c8815a5136fac8bb8745e8fe [01:53:38] 02IncidentReporting/03namespace 07CosmicAlpha 0364a8c9e Update [01:53:57] miraheze/IncidentReporting - Universal-Omega the build has errored. [01:58:27] miraheze/IncidentReporting - Universal-Omega the build passed. [01:58:59] [02IncidentReporting] 07Universal-Omega merged pull request #101: Namespace extension (03master...03namespace) 13https://github.com/miraheze/IncidentReporting/pull/101 [01:58:59] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/IncidentReporting/commit/48af5097a00574daa6f8f92b7388863c18f3e810 [01:58:59] 02IncidentReporting/03master 07CosmicAlpha 0348af509 Namespace extension (#101) [01:59:00] [02IncidentReporting] 07Universal-Omega 04deleted 03namespace at 0364a8c9e 13https://github.com/miraheze/IncidentReporting/commit/64a8c9e [02:03:20] miraheze/IncidentReporting - Universal-Omega the build passed. [02:13:51] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CreateWiki:SetContainersAccess --wiki=amogusgamingincidentofficalwiki (END - exit=0) [02:13:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:08:37] [02MirahezeMagic] 07Universal-Omega approved pull request #562 13https://github.com/miraheze/MirahezeMagic/pull/562#pullrequestreview-2701065218 [03:08:46] [02MirahezeMagic] 07Universal-Omega merged 07Markus-Rost's pull request #562: add Miraheze key to APIQuerySiteInfoGeneralInfo (07miraheze:03master...07Markus-Rost:03patch-1) 13https://github.com/miraheze/MirahezeMagic/pull/562 [03:10:14] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/MirahezeMagic/commit/75ec2baa2ac646181703d222c03fd2c32bb9954b [03:10:14] 02MirahezeMagic/03master 07MarkusRost 0375ec2ba add Miraheze key to APIQuerySiteInfoGeneralInfo (#562)… [03:21:59] miraheze/MirahezeMagic - Universal-Omega the build passed. [08:36:15] PROBLEM - Host mattermost1 is DOWN: PING CRITICAL - Packet loss = 100% [08:38:15] RECOVERY - Host mattermost1 is UP: PING OK - Packet loss = 0%, RTA = 30.02 ms [09:51:39] miraheze/MirahezeMagic - Universal-Omega the build passed. [10:08:52] PROBLEM - with.cpt-ra.bid - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [10:12:58] Hey, that's epic. [10:38:07] RECOVERY - with.cpt-ra.bid - Cloudflare on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Sat 24 May 2025 07:05:41 PM GMT +0000. [10:55:44] ~log [blankeclair@mwtask181:~/schemafuckfuckfuck/pagelinks]$ sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php ~/BulkSql.php --wiki=metawiki --wikis wikis-with-nonconforming-schema-1.txt --query 'DELETE FROM pagelinks WHERE pl_target_id IS NULL' [10:55:49] oops [10:55:51] !log [blankeclair@mwtask181:~/schemafuckfuckfuck/pagelinks]$ sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php ~/BulkSql.php --wiki=metawiki --wikis wikis-with-nonconforming-schema-1.txt --query 'DELETE FROM pagelinks WHERE pl_target_id IS NULL' [10:55:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:56:54] !log [blankeclair@mwtask181:~/schemafuckfuckfuck/pagelinks]$ sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php ~/BulkSql.php --wiki=metawiki --wikis wikis-with-nonconforming-schema-1.txt --query 'DELETE FROM pagelinks WHERE pl_target_id IS NULL' (END - exit=0) [10:56:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:57:00] !log [blankeclair@mwtask181:~/schemafuckfuckfuck/pagelinks]$ sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php ~/BulkSql.php --wiki=metawiki --wikis wikis-with-nonconforming-schema-1.txt patch-pagelinks-drop-pl_title-CUSTOM-index-if-exists-and-split-alters.sql [10:57:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:57:37] !log [blankeclair@mwtask181:~/schemafuckfuckfuck/pagelinks]$ sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php ~/BulkSql.php --wiki=metawiki --wikis wikis-with-nonconforming-schema-1.txt patch-pagelinks-drop-pl_title-CUSTOM-index-if-exists-and-split-alters.sql (END - exit=1) [10:57:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:01:22] !log japolandballwiki: DELETE FROM pagelinks WHERE pl_from = 30232 AND pl_target_id = 1573 LIMIT 1; -- (Reasoning: Two duplicate entries :p) [11:01:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:01:29] !log [blankeclair@mwtask181:~/schemafuckfuckfuck/pagelinks]$ sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php ~/BulkSql.php --wiki=metawiki --wikis wikis-with-nonconforming-schema-1.txt patch-pagelinks-drop-pl_title-CUSTOM-index-if-exists-and-split-alters.sql [11:01:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:02:29] !log [blankeclair@mwtask181:~/schemafuckfuckfuck/pagelinks]$ sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php ~/BulkSql.php --wiki=metawiki --wikis wikis-with-nonconforming-schema-1.txt patch-pagelinks-drop-pl_title-CUSTOM-index-if-exists-and-split-alters.sql (END - exit=1) [11:02:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:03:41] BlankEclair: what's it dying with? [11:03:50] duplicate primary keys [11:03:51] schemafuckfuck? [11:03:59] schemafuckfuckfuck actually [11:04:05] true [11:04:12] that's going on bash [11:04:14] where i'll be keeping my work re the fucked migration [11:04:17] if i can both to login [11:04:23] bother [11:04:37] i'm going in the db to manually delete the duplicate entries lol [11:04:46] !log makoentertainmentwiki: DELETE FROM pagelinks WHERE pl_from = 3719 AND pl_target_id = 21471 LIMIT 1; [11:04:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:04:54] BlankEclair: i wonder if you can write an SQL to delete the duplicates [11:05:01] i'm hoping that it's not too tedious [11:05:07] BlankEclair: what's the dupe [11:05:19] like which columns [11:05:34] pl_from, pl_target_id, and pl_from_namespace [11:05:36] i.e. all of them [11:05:56] !log [blankeclair@mwtask181:~/schemafuckfuckfuck/pagelinks]$ sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php ~/BulkSql.php --wiki=metawiki --wikis wikis-with-nonconforming-schema-1.2.txt patch-pagelinks-drop-pl_title-CUSTOM-index-if-exists-and-split-alters.sql [11:05:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:05:58] BlankEclair: are all now unique? [11:06:06] working through the list :p [11:06:17] ok fuck this i'll check if i can drop duplicates [11:06:51] !log [blankeclair@mwtask181:~/schemafuckfuckfuck/pagelinks]$ sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php ~/BulkSql.php --wiki=metawiki --wikis wikis-with-nonconforming-schema-1.2.txt patch-pagelinks-drop-pl_title-CUSTOM-index-if-exists-and-split-alters.sql (END - exit=1) [11:06:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:07:00] i can try and come up with an SQL If you give me an example [11:07:12] give me the patch and the error [11:07:39] Wikimedia\Rdbms\DBQueryError from line 1198 of /srv/mediawiki/1.43/includes/libs/rdbms/database/Database.php: Error 1062: Duplicate entry '16434-19254' for key 'PRIMARY' [11:07:52] Query: ALTER TABLE `pagelinks` ADD PRIMARY KEY (pl_from, pl_target_id) [11:10:27] is it a good idea for me to do this manually as you write the sql? [11:14:22] !log makoentertainmentwiki: DELETE FROM pagelinks WHERE (pl_from, pl_target_id) IN (SELECT pl_from, pl_target_id FROM (SELECT pl_from, pl_target_id, ROW_NUMBER() OVER (PARTITION BY pl_from, pl_target_id ORDER BY pl_from) AS rn FROM pagelinks) AS duplicates WHERE rn > 1); -- As suggested by RhinosF1 prompting ChatGPT :p [11:14:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:15:06] yep, it worked lol [11:15:39] !log [blankeclair@mwtask181:~/schemafuckfuckfuck/pagelinks]$ sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php ~/BulkSql.php --wiki=metawiki --wikis wikis-with-nonconforming-schema-1.2.txt patch-pagelinks-drop-pl_title-CUSTOM-index-if-exists-and-split-alters.sql (END - exit=0) [11:15:40] finally \o/ [11:15:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:18:37] tfw i ran two sql queries on 13.3k wikis in like ~15s [11:19:33] BlankEclair: it's fast ye [11:32:29] BlankEclair: is pagelinks fixed then? [11:32:37] double checking schema [11:32:47] i should write a more efficient diff script... [11:42:14] oh wow... they're all identical now [11:42:16] so uh [11:42:26] RhinosF1: pagelinks schema is now good globally [11:42:59] BlankEclair: yey! [11:43:02] now uh [11:43:06] BlankEclair: now refreshLinks [11:43:07] time for the dreaded rebuildLinks [11:43:11] refresh oops [11:43:51] BlankEclair: https://www.mediawiki.org/wiki/Manual:RefreshLinks.php#Chunking_refreshLinks.php_to_refresh_all_links_without_memory_leak [11:44:01] fun fact: the start parameter does not work in my testing [11:44:37] huh [11:44:40] fun [11:44:43] yea [11:44:45] gotta debug _that_ [11:46:57] BlankEclair: you can try running it without that [11:47:02] but it'll OOM [11:47:06] yeeeeah [11:47:13] cause the links stuff is leaky [11:47:18] the script feels like a workaround to a memleak tbh [11:47:22] oh lol [11:47:32] yup [11:47:37] it is [11:47:45] the links stuff is a walking memory leak [11:47:57] BlankEclair: try it without and just run refreshLinks [11:48:04] and see how long for it blow up [11:48:07] on all 13.3k wikis? [11:48:11] huh, really? [11:48:11] BlankEclair: ye [11:48:30] BlankEclair: i mean it'll leak at the first big wiki [11:48:40] and probably never finish [11:48:46] but we can at least start it [11:48:54] we probably need to shard it [11:48:58] by something [11:49:01] and run multiple [11:50:30] should i do the sharding part? [11:51:02] BlankEclair: if you can [11:51:08] should be fairly easy [12:20:31] !log [blankeclair@mwtask181] runRefreshLinks: [0/3329] Handling 100acgwiki [12:20:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:20:46] !log [blankeclair@mwtask151] runRefreshLinks: [0/3330] Handling 07xxwiki [12:20:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:21:00] !log [blankeclair@mwtask161] runRefreshLinks: [0/3330] Handling 0b0twiki [12:21:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:21:07] !log [blankeclair@mwtask171] runRefreshLinks: [0/3329] Handling 1000sunswiki [12:21:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:21:47] a message is output every 100 wikis [12:21:57] actually, may as well send the script i'm using here [12:22:11] https://cdn.discordapp.com/attachments/808001911868489748/1352255901767045201/runRefreshLinks.sh?ex=67dd59f2&is=67dc0872&hm=f8839779bf5f3e5f4ecef687e0bf41c9aedcea7c55e035fb876e52c82b057254& [12:23:35] secret script [12:23:51] top sacred [12:25:03] let's see how long 100 takes [12:25:18] if it's shorter than the IA backup then i'll riot /j [12:27:39] i'd expect quicker tbh [12:27:45] IA has to upload externally too [12:27:49] true [12:28:02] poor mwtask181 cpu being pegged at 100% (according to htop) [12:28:15] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.33, 16.89, 8.58 [12:28:21] oh yep [12:28:42] You said its name, so now it wants attention. [12:31:10] wtf how is php-fpm being used on mwtask181 [12:31:12] that's impressive [12:32:12] !log [blankeclair@mwtask181] runRefreshLinks: [3/3329] 16384wikiwiki exited with 255 [12:32:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:15] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.76, 20.60, 11.95 [12:32:36] 16384wikiwiki no longer exists iirc [12:33:09] !log [blankeclair@mwtask181] runRefreshLinks: [0/3325] Handling 196smpwiki [12:33:10] can we just kill mwtask181 already? [12:33:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:18] the uh [12:33:21] the script or the server itself [12:33:27] everything [12:33:38] sudo -u www-data rm -rf /* [12:33:53] --no-preserve-root or something like that [12:34:02] globbing should do [12:34:10] unless rm is smart enough to detect that [12:34:15] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.68, 21.23, 13.24 [12:34:25] I thought that said loud warning for a second. [12:34:31] Why does 181 keep redlining? [12:34:37] it's running backupwikis [12:34:59] and uh, php-fpm is to blame as well? [12:35:13] I guess increase RAM? Idk. [12:35:20] ram is plenty [12:35:34] RAM is always the solution, just keep throwing it at the problem until something sticks. [12:35:41] https://cdn.discordapp.com/attachments/808001911868489748/1352259298130133015/2025-03-20_23-35.png?ex=67dd5d1c&is=67dc0b9c&hm=4412200920cefaa8418dacee783657f0ffe0712c07212d4882eabd6756f72e54& [12:36:37] I wonder why it's being a douche then. [12:38:15] it's a computer [12:38:23] That's true. [12:38:25] touche [12:38:40] This means you have to take all its RAM away tp threaten it. [12:38:49] that'll just give it trauma [12:39:06] don't take away essential needs to convince it to do your bidding >_> [12:39:09] Can we stack the Trauma to get a negative debuff? [12:40:14] did it really do 3 wikis in 12 minutes [12:40:15] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.29, 22.13, 16.10 [12:40:27] that's 4 minutes a wiki [12:40:33] * 3330 [12:41:10] 222 Hours? [12:41:38] that's only like 10 days [12:41:40] So, nine and a half days, roughly. If every wiki takes 4 mintues. [12:41:41] not too bad [12:42:15] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.93, 22.43, 16.96 [12:44:14] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 12.21, 18.75, 16.28 [12:44:29] woa [12:44:35] php-fpm is being pegged less hard [12:44:52] it was probably having a mini heart attack [12:44:55] and then clamed [12:52:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:07:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:12:28] !log [blankeclair@mwtask161] runRefreshLinks: [100/3330] Handling allaboveheavenwiki [13:12:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:12:33] ohey [13:13:26] That's how long [13:13:42] Like an hour [13:13:57] That's not bad [14:11:57] why have none other finished [14:12:21] the first 100 [14:20:06] !log [blankeclair@mwtask151] runRefreshLinks: [100/3330] Handling allaboutspacewiki [14:20:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:20:44] ok cool [14:20:56] average 90 minutes to do 100 then [14:26:16] apart from mwtask181, they are all on al [14:29:25] okay mwtask181 is on a wiki with 14k articles [14:29:28] make sense [14:40:52] Well that’s in SAL now [14:57:42] tbh its accurate [15:11:01] Not the worst thing that's been logged there. [15:34:15] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 31.00, 18.54, 9.98 [15:39:11] !log [blankeclair@mwtask171] runRefreshLinks: [100/3329] Handling allareonewiki [15:39:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:41:20] So average like 2 hours [15:44:15] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 4.86, 17.55, 15.37 [15:54:13] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49942MiB (10% inode=99%); [16:30:49] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/cleanupTitles.php --wiki=polytopewiki (END - exit=0) [16:30:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:55:44] !log [blankeclair@mwtask151] runRefreshLinks: [200/3330] Handling artlistthedogwikiwiki [16:55:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:58:38] ooh nice [16:58:44] our first reaches 200 [16:58:50] after 4.5 hours [16:59:19] 161 is still stuck on att [17:08:15] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.78, 17.16, 10.47 [17:09:08] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 27.32, 19.04, 10.72 [17:09:41] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.33, 16.72, 9.75 [17:09:42] Is each mwtask running one( [17:10:02] yup [17:11:08] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 12.94, 17.15, 11.10 [17:11:41] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 9.49, 14.34, 9.77 [17:12:14] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.22, 16.33, 11.99 [18:47:35] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49888MiB (10% inode=99%); [18:49:06] !log [blankeclair@mwtask151] runRefreshLinks: [300/3330] Handling berninawiki [18:49:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:49:15] There we go [18:49:21] I was just about to say [18:50:18] That's like 6 hours [18:50:23] So like 50 an hour [18:51:04] That's 66 hours [18:51:12] So like 3 days [18:51:14] Not too bad [18:51:19] I think we need to get a bit of a larger sample size to get an accurate estimate, could be an coincidentally large amount of large wikis in the upper alphabet [19:10:29] PROBLEM - baligawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'baligawiki.org' expires in 14 day(s) (Fri 04 Apr 2025 06:57:58 PM GMT +0000). [19:12:32] !log [blankeclair@mwtask171] runRefreshLinks: [200/3329] Handling artresearchwiki [19:12:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:44:14] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.83, 21.28, 14.62 [19:56:14] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 17.27, 22.13, 19.61 [19:58:14] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.98, 19.92, 19.10 [20:00:40] Oooh, is it doing that thing with run.php where it makes the first non-option argument into the name of the script? [20:02:54] Evening [20:09:28] RECOVERY - baligawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'baligawiki.org' will expire on Wed 18 Jun 2025 06:12:04 PM GMT +0000. [20:14:12] > Well that’s in SAL now [20:14:18] pixldev: exactly :3c [20:16:00] idts no [20:16:16] sql.php and refreshLinks.php both use this->getArg(0) [20:20:49] BlankEclair: good morning [20:21:36] Fair, that was a problem months ago, if not longer. If it hadn't been fixed by now, I would be shocked. Wonder what else might be going on then. [20:26:30] mornyan~ [21:00:56] !log [blankeclair@mwtask171] runRefreshLinks: [300/3329] Handling bertiepediawiki [21:00:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:21:32] !log [blankeclair@mwtask151] runRefreshLinks: [400/3330] Handling busesinnycwiki [21:21:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:21:39] wow, speedy [21:22:01] (cw: lewd) 00,00why did i read busesinnyc as bussiesinnyc [21:41:39] miraheze/dns - BlankEclair the build passed. [21:42:50] miraheze/dns - MacFan4000 the build passed. [21:44:36] byeeee macfanbot [21:45:24] ever since I reinstalled the VM, the GH hook has been less stable [21:51:20] [02mw-config] 07Universal-Omega created 03no-cookie-domain (+1 new commit) 13https://github.com/miraheze/mw-config/commit/09d8f4e168f5 [21:51:20] 02mw-config/03no-cookie-domain 07CosmicAlpha 0309d8f4e Don't set $wgCentralAuthCookieDomain [21:51:24] [02mw-config] 07Universal-Omega opened pull request #5878: Don't set $wgCentralAuthCookieDomain (03master...03no-cookie-domain) 13https://github.com/miraheze/mw-config/pull/5878 [21:52:14] miraheze/mw-config - Universal-Omega the build passed. [21:59:56] [02mw-config] 07BlankEclair left a review on pull request #5878: Why though? 13https://github.com/miraheze/mw-config/pull/5878#pullrequestreview-2704220760 [21:59:58] [02mw-config] 07BlankEclair left a file comment in pull request #5878 0309d8f4e: ```suggestion […] 13https://github.com/miraheze/mw-config/pull/5878#discussion_r2006518232 [22:17:15] [02mw-config] 07Universal-Omega commented on pull request #5878: Because I don't get logged out when visiting new custom domains, I do when visiting new miraheze.org subdomains. So wanted to see what happens removing this which is how custom domains are configured right now. 13https://github.com/miraheze/mw-config/pull/5878#issuecomment-2741788848 [22:22:45] !log [blankeclair@mwtask151] runRefreshLinks: [500/3330] Handling clonevpnwiki [22:22:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:29:41] [02CreateWiki] 07Universal-Omega created 03ListDatabasesTest (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/669310bc9967 [22:29:41] 02CreateWiki/03ListDatabasesTest 07CosmicAlpha 03669310b ListDatabases: add tests [22:29:45] [02CreateWiki] 07Universal-Omega opened pull request #711: ListDatabases: add tests (03master...03ListDatabasesTest) 13https://github.com/miraheze/CreateWiki/pull/711 [22:29:52] [02CreateWiki] 07coderabbitai[bot] commented on pull request #711: --- […] 13https://github.com/miraheze/CreateWiki/pull/711#issuecomment-2741806809 [22:30:31] [02CreateWiki] 07Universal-Omega retitled PR #711: "ListDatabases: add tests" ➜ "ListDatabases: add test" 13https://github.com/miraheze/CreateWiki/pull/711 [22:33:06] miraheze/CreateWiki - Universal-Omega the build has errored. [22:33:19] [02CreateWiki] 07Universal-Omega created 03CheckLastWikiActivity-newline (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/5bace7574770 [22:33:19] 02CreateWiki/03CheckLastWikiActivity-newline 07CosmicAlpha 035bace75 CheckLastWikiActivity: add newline in output [22:33:24] [02CreateWiki] 07Universal-Omega opened pull request #712: CheckLastWikiActivity: add newline in output (03master...03CheckLastWikiActivity-newline) 13https://github.com/miraheze/CreateWiki/pull/712 [22:33:32] [02CreateWiki] 07coderabbitai[bot] commented on pull request #712: --- […] 13https://github.com/miraheze/CreateWiki/pull/712#issuecomment-2741811462 [22:34:04] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ListDatabasesTest 13https://github.com/miraheze/CreateWiki/commit/50f834989c6834d0bd20ca2ac37b2844d555b75f [22:34:04] 02CreateWiki/03ListDatabasesTest 07CosmicAlpha 0350f8349 Fix [22:38:07] [02CreateWiki] 07codecov[bot] commented on pull request #712: All modified and coverable lines are covered by tests :white_check_mark: […] 13https://github.com/miraheze/CreateWiki/pull/712#issuecomment-2741818540 [22:38:17] [02CreateWiki] 07codecov[bot] commented on pull request #711: All modified and coverable lines are covered by tests :white_check_mark: […] 13https://github.com/miraheze/CreateWiki/pull/711#issuecomment-2741818745 [22:39:33] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ListDatabasesTest 13https://github.com/miraheze/CreateWiki/commit/28340c0067e61232502ae0b7651aa1fbe44938c6 [22:39:33] 02CreateWiki/03ListDatabasesTest 07CosmicAlpha 0328340c0 Use $ [22:45:45] miraheze/CreateWiki - Universal-Omega the build passed. [22:47:55] [02CreateWiki] 07Universal-Omega merged pull request #712: CheckLastWikiActivity: add newline in output (03master...03CheckLastWikiActivity-newline) 13https://github.com/miraheze/CreateWiki/pull/712 [22:47:55] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/285312b41a849053b543b593e22f0cc3c7c235d8 [22:47:55] 02CreateWiki/03master 07CosmicAlpha 03285312b CheckLastWikiActivity: add newline in output [22:47:56] [02CreateWiki] 07Universal-Omega 04deleted 03CheckLastWikiActivity-newline at 035bace75 13https://github.com/miraheze/CreateWiki/commit/5bace75 [22:51:41] miraheze/CreateWiki - Universal-Omega the build passed. [22:55:30]