[00:12:32] <MirahezeRelay>	 <zeusdeegoose, replying to zipppee> i do think it jumps the gun a fair bit, but i do like the ideas on present
[00:13:27] <MirahezeRelay>	 <zeusdeegoose> dm28 deffo likes to take huge leaps in his proposals, which i find very endearing
[00:18:47] <MirahezeRelay>	 <blessed.bythewind> [1/4] https://cdn.discordapp.com/attachments/1006789349498699827/1553923893524299776/image.jpg?ex=6abb03e6&is=6ab9b266&hm=cd72d849a6a02185a494badfb0abf5e6b2691b5befe614e7b66b7ff46678b39d&
[00:18:47] <MirahezeRelay>	 <blessed.bythewind> [2/4] https://cdn.discordapp.com/attachments/1006789349498699827/1553923894212169728/image.jpg?ex=6abb03e6&is=6ab9b266&hm=e5b101af9c827f4fb44fee73e3763c0f49d22a2cb057c68b2b7ffed74d752d8e&
[00:18:48] <MirahezeRelay>	 <blessed.bythewind> [3/4] https://cdn.discordapp.com/attachments/1006789349498699827/1553923894673670275/image.jpg?ex=6abb03e7&is=6ab9b267&hm=defb8681466e5d22236c03fa0a45b45263c6ab3f7a6092966d5afcbdd978587b&
[00:18:48] <MirahezeRelay>	 <blessed.bythewind> [4/4] https://cdn.discordapp.com/attachments/1006789349498699827/1553923895130718259/image.jpg?ex=6abb03e7&is=6ab9b267&hm=52e054239ad3a9c3a88baff42ceb05dbeedbf80f59a2c0c89322260eebb7a722&
[00:33:08] <MirahezeRelay>	 <notaracham> Yeah, it's not all a wash, and some of these were things we were already working on
[00:33:32] <MirahezeRelay>	 <notaracham> But I have concerns I will log once I'm out of overwork jail
[00:49:19] <MirahezeRelay>	 <abaddriverlol> [1/2] this indicator on grafana currently displays the mean value over time instead of the last value, which doesn't make a lot of sense IMO; any objections to me changing it?
[00:49:19] <MirahezeRelay>	 <abaddriverlol> [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1553931578747387965/image.png?ex=6abb0b0f&is=6ab9b98f&hm=7e84b609da9f373223c804842090a19d3cc7b6f2265412f910969686a6af5c78&
[01:05:25] <MirahezeRelay>	 <cosmicalpha, replying to abaddriverlol> No objections
[08:37:07] <MirahezeRelay>	 <dgox2> [1/3] err https://forsaken.wiki/File:LoadingScreen_1x4.png
[08:37:07] <MirahezeRelay>	 <dgox2> [2/3] affecting all File: pages i believe on this wiki
[08:37:08] <MirahezeRelay>	 <dgox2> [3/3] https://cdn.discordapp.com/attachments/1006789349498699827/1554049304908529735/image.png?ex=6abb78b3&is=6aba2733&hm=557552d6165bd65f1a8472644357eae01bd095f8ae146f4f57b5fda809dc4409&
[08:42:56] <MirahezeRelay>	 <cosmicalpha, replying to dgox2> Fixed
[08:43:19] <MirahezeRelay>	 <dgox2> cool ty
[08:44:04] <MirahezeRelay>	 <cosmicalpha> No problem, apologies for that
[10:21:15] <MirahezeRelay>	 <.book_> [1/4] https://projecttd.miraheze.org/wiki/
[10:21:15] <MirahezeRelay>	 <.book_> [2/4] [3d0c8c93bae814dbff97021c] 2026-09-28 10:18:49: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"
[10:21:15] <MirahezeRelay>	 <.book_> [3/4] Both me and my friend were doing stuff (one file upload, one page edit) and we both encountered this error
[10:21:16] <MirahezeRelay>	 <.book_> [4/4] https://cdn.discordapp.com/attachments/1006789349498699827/1554075510454222920/image-5.webp?ex=6abb911b&is=6aba3f9b&hm=8e0583d2d19e1e4e22c0dc0b1d1d10a6e58ff7c6172b97df991911faabc48505&
[10:22:13] <MirahezeRelay>	 <cosmicalpha, replying to .book_> Fixed
[10:32:37] <MirahezeRelay>	 <.book_> thx
[11:42:12] <MirahezeRelay>	 <malimar87, replying to .book_> I, also, am getting this error, when trying to save edits to several different pages of  highseasunity.miraheze.org — is it, like, miraheze-wide?
[11:43:59] <MirahezeRelay>	 <cosmicalpha, replying to malimar87> Fixed
[11:58:58] <MirahezeRelay>	 <ems0079_> [1/6] Hi, my wiki is already on MediaWiki 1.46, but I can’t save any edits right now.
[11:58:58] <MirahezeRelay>	 <ems0079_> [2/6] Normal page edits, MediaWiki:Common.css, and MediaWiki:Citizen.css all fail with Wikimedia\Rdbms\DBQueryError.
[11:58:58] <MirahezeRelay>	 <ems0079_> [3/6] Reading pages works normally, but saving does not.
[11:58:58] <MirahezeRelay>	 <ems0079_> [4/6] Wiki: https://naramokorea.miraheze.org/
[11:58:59] <MirahezeRelay>	 <ems0079_> [5/6] Example error ID: [91e44d6c53a8ee55ba446e05]
[11:58:59] <MirahezeRelay>	 <ems0079_> [6/6] https://cdn.discordapp.com/attachments/1006789349498699827/1554100099955560468/image.png?ex=6abba801&is=6aba5681&hm=c1181e56ede07c84b5bb55f75f4ef81b3b73bb8f2c9636819987739ff90a4061&
[12:02:09] <MirahezeRelay>	 <cosmicalpha, replying to ems0079_> Fixed, and lucky because this is probably the last one I fix for now per https://discord.com/channels/407504499280707585/407537962553966603/1554099761617829949
[12:02:44] <MirahezeRelay>	 <ems0079_> Thank You
[12:02:45] <MirahezeRelay>	 <cosmicalpha> Apologies for the issues though.
[12:02:50] <MirahezeRelay>	 <cosmicalpha> No problem
[19:50:16] <MirahezeRelay>	 <jenny_on_wiki> [1/2] @cosmicalpha I can take over for an hour or so
[19:50:16] <MirahezeRelay>	 <jenny_on_wiki> [2/2] is there a better way of doing this than manually upgrading every wiki an error is reported for
[19:50:48] <MirahezeRelay>	 <cosmicalpha> That's what I've been doing. Also sounds good because I need to step away for an hour or so. Thanks!
[20:43:56] <MirahezeRelay>	 <jenny_on_wiki, replying to jenny_on_wiki> hmm what if we obtained a list of all wikis and compared it to a list like the cache keys UpgradeMediaWikiVersion uses or some other indicator of being on the old version
[20:44:16] <MirahezeRelay>	 <jenny_on_wiki> rather than just waiting for reports which are now definitely thinning out
[20:57:26] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> We can also just export logs from graylog.
[20:57:54] <MirahezeRelay>	 <posix_memalign> Though I do think this is a good idea. In fact if we have this list we should be upgrading wikis in parallel to get this done as soon as possible.
[20:58:01] <MirahezeRelay>	 <jenny_on_wiki, replying to jenny_on_wiki> `select * from updatelog where ul_key = "upgrade-wiki-1.46";` for each wiki
[20:58:19] <MirahezeRelay>	 <posix_memalign> I think the main issue is that there is no cheap way to get the list of wikis on 1.45.
[20:58:23] <MirahezeRelay>	 <jenny_on_wiki> although that wouldn't work for wikis where we manually finished the update
[20:58:33] <MirahezeRelay>	 <posix_memalign> So maybe we need to check for DB columns.
[21:02:52] <MirahezeRelay>	 <jenny_on_wiki, replying to posix_memalign> idk what's the easiest way to do it. e.g. the `watchlist_label` table is part of the upgrade
[21:03:49] <MirahezeRelay>	 <jenny_on_wiki> `il_target_id` from `imagelinks` is also part and it's what everything was failing on so that is probably most reliable
[21:06:53] <MirahezeRelay>	 <jenny_on_wiki, replying to jenny_on_wiki> so looks like this would be SHOW COLUMNS FROM imagelinks LIKE "il_target_id";
[21:08:51] <MirahezeRelay>	 <jenny_on_wiki, replying to jenny_on_wiki> yeah just checked this with a reported wiki
[21:09:47] <MirahezeRelay>	 <jenny_on_wiki> so I think you could run that inside a foreachwikiindblist, check that sql query and if it returns nothing upgrade it
[21:09:48] <MirahezeRelay>	 <posix_memalign> [1/2] A list is generated but I'm not too sure about completeness
[21:09:48] <MirahezeRelay>	 <posix_memalign> [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1554238721904345170/message.txt?ex=6abc291b&is=6abad79b&hm=fadeec3bd8e349183b6714635f42feff82289522fae45d711da613dcfd78441b&
[21:10:32] <MirahezeRelay>	 <jenny_on_wiki> and now I need to remember how foreachwikiindblist works
[21:11:08] <MirahezeRelay>	 <posix_memalign> I think CA's script is reaching wikis whose dbname starts with `g`. Because every wiki starting with it does not have the migration.
[21:12:39] <MirahezeRelay>	 <jenny_on_wiki> wouldn't explain this though
[21:14:41] <MirahezeRelay>	 <jenny_on_wiki, replying to jenny_on_wiki> yeah this doesn't work for philosophyballwiki but `SHOW COLUMNS FROM imagelinks LIKE "il_target_id";` does
[21:19:55] <MirahezeRelay>	 <jenny_on_wiki, replying to jenny_on_wiki> I've gtg for a bit now. If it's not done when I get back I'll check on MM how I did that command when upgrading beta
[21:20:29] <MirahezeRelay>	 <posix_memalign> I think I have a list. Will run it on wikis starting with `h` and after in parallel.
[21:21:13] <MirahezeRelay>	 <jenny_on_wiki, replying to posix_memalign> yeah, just be careful to check the output. I imagine the most file-heavy wikis are the ones with problems but the script can still fail
[21:22:01] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> For those that fail I think we can go back to them by looking for signs of incomplete migration in the DB for a second pass. The goal rn is to get as much wikis in 1.46 as possible I think.
[21:22:33] <MirahezeRelay>	 <jenny_on_wiki, replying to posix_memalign> what would that sign be then
[21:22:57] <MirahezeRelay>	 <jenny_on_wiki> I'm currently upgrading sagan4alpha which I know is massive so maybe you could just check that one
[21:23:12] <MirahezeRelay>	 <jenny_on_wiki, replying to jenny_on_wiki> other than script output obvs
[21:23:23] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> Missing updatelog entry?
[21:23:31] <MirahezeRelay>	 <jenny_on_wiki> it'll probably be one of the last 2 patches that fail
[21:23:52] <MirahezeRelay>	 <jenny_on_wiki, replying to posix_memalign> hmm that would also catch cases like philosophyballwiki which were finished manually
[21:24:05] <MirahezeRelay>	 <jenny_on_wiki, replying to jenny_on_wiki> nvm that worked fine
[21:26:36] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> I guess you just run the SQL statement again? I didn't fix any of the wikis so I'm not sure what are the other signs that a wiki needs manual attention.
[21:26:37] <MirahezeRelay>	 <jenny_on_wiki> I'm trying `SHOW COLUMNS FROM imagelinks LIKE "il_to";` but I need an example of an unupgraded wiki
[21:27:00] <MirahezeRelay>	 <jenny_on_wiki, replying to posix_memalign> manually run the failed patches
[21:28:20] <MirahezeRelay>	 <jenny_on_wiki, replying to jenny_on_wiki> okay yeah run this command on all wikis
[21:28:40] <MirahezeRelay>	 <jenny_on_wiki> the very final command of the very final patch is to remove this
[21:29:07] <MirahezeRelay>	 <jenny_on_wiki> [1/20] ```bash
[21:29:08] <MirahezeRelay>	 <jenny_on_wiki> [2/20] [wwr@mwtask171:~]$ sql philosophyballwiki
[21:29:08] <MirahezeRelay>	 <jenny_on_wiki> [3/20] Will execute:
[21:29:08] <MirahezeRelay>	 <jenny_on_wiki> [4/20] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php sql --wiki=philosophyballwiki
[21:29:09] <MirahezeRelay>	 <jenny_on_wiki> [5/20] > show columns from imagelinks like "il_to";
[21:29:09] <MirahezeRelay>	 <jenny_on_wiki> [6/20] Query OK, 0 row(s) affected
[21:29:09] <MirahezeRelay>	 <jenny_on_wiki> [7/20] > Done!
[21:29:09] <MirahezeRelay>	 <jenny_on_wiki> [8/20] [wwr@mwtask171:~]$ sql newkoolkrazydjdivineraverswiki
[21:29:10] <MirahezeRelay>	 <jenny_on_wiki> [9/20] Will execute:
[21:29:10] <MirahezeRelay>	 <jenny_on_wiki> [10/20] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php sql --wiki=newkoolkrazydjdivineraverswiki
[21:29:11] <MirahezeRelay>	 <jenny_on_wiki> [11/20] > SHOW COLUMNS FROM imagelinks LIKE "il_to";
[21:29:11] <MirahezeRelay>	 <jenny_on_wiki> [12/20] stdClass Object
[21:29:12] <MirahezeRelay>	 <jenny_on_wiki> [13/20] (
[21:29:12] <MirahezeRelay>	 <jenny_on_wiki> [14/20]     [Field] => il_to
[21:29:13] <MirahezeRelay>	 <jenny_on_wiki> [15/20]     [Type] => varbinary(255)
[21:29:13] <MirahezeRelay>	 <jenny_on_wiki> [16/20]     [Null] => NO
[21:29:14] <MirahezeRelay>	 <jenny_on_wiki> [17/20]     [Key] => PRI
[21:29:14] <MirahezeRelay>	 <jenny_on_wiki> [18/20]     [Default] =>
[21:29:15] <MirahezeRelay>	 <jenny_on_wiki> [19/20]     [Extra] =>
[21:29:15] <MirahezeRelay>	 <jenny_on_wiki> [20/20] )```
[21:34:42] <MirahezeRelay>	 <posix_memalign, replying to posix_memalign> [1/2] Nvm. I'm upgrading wikis on `c4` starting with `j`. Will do so for `c3` wikis on a different mwtask server.
[21:34:42] <MirahezeRelay>	 <posix_memalign, replying to posix_memalign> [2/2] Rerunning the script on the same wiki is harmless so I think we will be fine. The first to run a sql patch wins while the other aborts.
[21:43:07] <MirahezeRelay>	 <posix_memalign> I'm running 6 of these now. mwtask181 for c4 wikis. mwtask171 for c3 wikis. mwtask161 for c2 wikis. 2 processes on each server.
[21:46:13] <MirahezeRelay>	 <posix_memalign> Got 8 of them running now. The DBs seem to be doing fine, but my script only partitioned the DBs into 2 equal portions so there won't be any more parallelism gained out of this.
[22:03:44] <MirahezeRelay>	 <jenny_on_wiki, replying to posix_memalign> what's the status on these now
[22:09:36] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> Had to stop 4 out of the 8 because the servers are overloaded even though DBs are fine. c4 is on magicuswiki. c3 on leagueofserverswiki. c2 on marblekingdomswiki. c1 on limzhihaowinstonwiki.
[22:11:09] <MirahezeRelay>	 <posix_memalign> The assignment was done in a round-robin fashion so it doesn't mean wikis between `j` and `l` are all upgraded. Will need to do batch 2 after batch 1 completes. Each batch is a little more than 1000 wikis.
[22:24:02] <MirahezeRelay>	 <jenny_on_wiki, replying to posix_memalign> I put up challenges so you could maybe get away with doing a few more
[22:25:36] <MirahezeRelay>	 <posix_memalign> I think it didn't have much to do with our normal traffic. The slowness was probably caused by foreachwikiindblist problems which when bad enough can make the whole farm slow with a single process.
[22:26:19] <MirahezeRelay>	 <posix_memalign> The upgrade is 15% complete for one batch and other batches are probably similar. So if a ll goes well after 3 hours this batch will be complete and after that we just need to do batch 2. After batch 2 I think everything will be done.
[23:59:13] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> [1/12] ```php
[23:59:13] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> [2/12] private function getProblem( IReadableDatabase $dbr ): ?string {
[23:59:13] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> [3/12]         if ( !$dbr->fieldExists( 'imagelinks', 'il_target_id', METHOD ) ) {
[23:59:14] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> [4/12]                 return 'not upgraded';
[23:59:14] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> [5/12]         }
[23:59:14] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> [6/12]         if ( $dbr->fieldExists( 'imagelinks', 'il_to', METHOD ) ) {
[23:59:15] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> [7/12]                 return 'imagelinks migration incomplete';
[23:59:15] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> [8/12]         }
[23:59:15] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> [9/12]         return null;
[23:59:16] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> [10/12] }
[23:59:16] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> [11/12] ```
[23:59:17] <MirahezeRelay>	 <posix_memalign, replying to jenny_on_wiki> [12/12] Something like this can catch most issues I think
[23:59:50] <MirahezeRelay>	 <posix_memalign> The previous attempts had some false positives because new wikis do not have updatelog entires.