[07:59:57] FYI T357943 (see also #wikimedia-tech) [07:59:58] T357943: Please increase the default image size at the English Wikivoyage to 300 pixels - https://phabricator.wikimedia.org/T357943 [08:09:38] We had something related about enwiki, let me see my mail [08:09:54] Ah, yes, T355914 [08:09:55] T355914: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914 [08:11:06] The WebTeam are working on a bump to 250, I'll link to that ticket [08:11:28] OIC Peachey88 already got there [08:25:50] dump of s7 on codfw failed [08:26:43] with what error? [08:26:52] Access denied [08:26:58] :-/ [08:27:05] ah, needs a haircut [08:27:09] [EPERM] :) [08:27:15] I don't think we've touched grants [08:27:16] Amir1: ^ [08:27:40] s7 failed too, with Could not read data from fawiki.pagelinks: Table definition has changed, please retry transaction [08:27:54] jynus: that is related to amir's schema change [08:28:13] But the grants thing is weird [08:28:32] oh, the grant error is missleading, it is old [08:28:38] Ah :) [08:28:39] that is the one [08:28:50] Then I guess it was caught in the middle of the schema change [08:28:54] that's ok [08:29:01] I will ask him and retry when done [08:29:08] I'd not be surprised if it happens on s8 too, the schema change on change_tags takes 10h...so there are chances there [08:29:29] that would explain why it failed even after a retry [08:29:44] as usually it reties before 10 hours [08:30:03] and also the migration makes it so it does it faster than usual [09:04:45] I need to do a quick reboot [09:38:31] btullis: did you see https://phabricator.wikimedia.org/T355571#9794965 ? [09:39:39] marostegui: I did, but thanks for the prompt. I will see to it. [09:40:03] btullis: thanks - just wanted to make sure it is in your radar and future plans [09:40:06] Before it is too late [09:41:06] Agreed. I will upgrade clouddb1021 to bookworm first, then schedule the transfer of data to an-redacteddb1001. [09:41:15] sounds good thank you [09:41:40] I am wondering if we have jobs or monitoring or regex in general based on clouddb* as hostname [09:41:47] We'll find out once it is migrated [09:42:14] I know a place we definitely do, and it is in preseed.yaml [09:42:23] For reusing "/srv/" instead of formatting it [09:42:23] I'm also aware that an-mariadb100[1-2] are running MariaDB 10.4 and I need to upgrade those to 10.6, which could be a bit interesting. [09:42:55] btullis: yes, especially because it EOL 18th June [09:44:47] Thanks. Yes, the main place we need to change the hostname is in refinery: https://github.com/wikimedia/analytics-refinery/blob/4d42877ec62a7cfee9103809bafe1297147845d3/python/refinery/util.py#L35 but the preseed is a good call, too. [09:45:40] Yeah, we have a few places too from what I can see, but it shouldn't be too hard [09:45:58] But the preseed.yaml is a must, otherwise if the host is reimaged there will be data loss [09:46:20] As by default we wipe everything [11:50:39] marostegui: Another "emergency" schema change coming your way... [11:51:16] only 60K rows on wikidata... [11:51:38] So probably a jfdi candiate [11:51:38] Reedy: sure thing, task id? [11:51:58] I've only just made the patch ;P [11:52:12] So slow... [11:53:48] T365465 [11:53:50] T365465: translate_reviews unsigned and revision big int - https://phabricator.wikimedia.org/T365465 [11:54:44] ok, I am ready to do it now [11:54:47] Ok to proceed? [11:55:35] Well, CI passes, no one has reviewed it yet ;P [11:56:06] Amir1: ^ [11:56:09] The only pause I can see is that the previous patch made some other columns bigint, even though they aren't in MW core... But those aren't going to be needed for a looong time [11:56:20] sorry, at meeting [11:56:31] meta is 356K rows... [11:56:44] It is probably doable there too directly [11:56:48] For a table of 3 ints... I'm guessing it might be an easy deploy everywhere [11:56:49] Yeah [11:57:10] Is wikidata broken and I need it to run it now or can it wait until someone reviews this? [11:57:30] It's not urgent :) [11:57:31] https://phabricator.wikimedia.org/T364681#9815719 [11:58:01] Number of rows currently affected is a fraction of the other table... And that wasn't exactly big [11:58:12] Ok, I am going to voluntell Amir1 to review it when he's out of the meeting and deploy when done so, sounds good? [11:59:28] Lucas isn't in here, so just pinged him elsewhere [11:59:34] oki :) [12:01:20] gracias [12:01:30] de nada guapo [12:03:30] I'm hoping there won't be more for translate... But I just filed a task to move translate enabling to a dblist for future ease [12:04:41] thanks :) [12:06:12] at least it's not repeated schema changes to the same table [12:07:37] out of meeting [12:08:00] Lucas was looking for a way to find issues like this [12:08:51] Yeah, he created a task [13:06:35] Reedy: schema change deployed in wikidatawiki [13:13:03] tbh i was actually a bit surprised that rev_id surpassing the signed int max value didn't explode bigger [13:14:06] I am getting the schema change deployed in s3 and once done the whole task is finished [13:16:00] Amir1: do you recall if snapshots were fixed to reload the config or that was never implemented? I cannot remember what was the outcome? [13:16:40] marostegui: it was implemented but it's actually quite slow to pick it up [13:16:49] Amir1: Ah ok [13:16:53] because the batches are quite big [13:17:03] yeah that's fine, I can execute that later manually [13:17:07] I was just wondering [13:17:46] cool [14:41:28] marostegui: I'm working on a checklist of dealing with db issues. Scott brought up that it's missing how to deal with a case of master of secondary dc going down. What do you think would be a better approach there? Depooling the dc or trying to reboot the master? [14:43:27] Amir1: Depends on what the issue is. Both approaches should be fine I think. Even if the masters comes back after a reboot we should check the data and then switch it [14:43:51] I'd prefer to depool the DC so we can deal with it with less pressure [14:44:06] thanks [14:50:58] added, thanks