[07:06:07] All groups have been removed from s8 (wikidata) [07:52:46] note that wmf.19 (train this week) broke db transactions and caused issues, I'm on it https://phabricator.wikimedia.org/T300194 [07:53:18] which transactions? [07:53:33] any write transactions (mostly edits) [07:53:39] wtf???? [07:53:51] any write? [07:53:52] it was live for ten minutes only :D [07:54:01] jeeeez [07:54:24] until mw kicked and caused a fatal for transaction taking more than three seconds [07:54:54] hehe hard to even see it at s1 master https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&from=now-24h&to=now&var-job=All&var-server=db1118&var-port=9104 [07:55:00] Or at https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&refresh=5m&from=now-24h&to=now [07:56:07] only on group1 so it should be visible in commons possibly [08:07:21] I'm backporting the fix for it soon-ish [08:53:30] marostegui: is the switchover at 10 something I should pay attention to, or more a "FYI in case of 🔥"? [08:56:49] Emperor: I belive you don't own any db there, so in your case probably the latter. Wouldn't hurt to be around to being able to attend an emergency, as you would be 4th in "succesion line" :-)))) [09:13:10] marostegui: you're gonna have so much fun deploying this https://gerrit.wikimedia.org/r/c/mediawiki/core/+/757614/1/maintenance/archives/patch-templatelinks-target_id.sql [09:13:39] note, the table is 300GB in commons [09:17:56] Emperor: more fyi [09:18:10] but also what jynus said [09:59:42] Going to start m1 failover, I will do it in -operations channel [10:17:55] jynus: I remember you told me long time ago that if we create a table without some work, it might alert. What is the work? maintain views? let me know if you don't remember it and want more context [10:18:21] Amir1: if it is a private table and we don't filter it, it will alert [10:18:22] Amir1: the check on cloud [10:18:30] (wikireplicas) [10:18:48] although I belive it is also on sanitariums on production, too? [10:19:07] so this would prevent the alert? https://gerrit.wikimedia.org/r/c/operations/puppet/+/757622 [10:19:15] I probably need to run maintain views I guess [10:19:30] no, if the table is meant to be public, nothing will alert [10:19:54] I am not sure about that, will depend on the alerting logic [10:20:08] e.g. if it is marked as public you are right [10:20:15] I remember it alerted when wb_terms got created on s4 [10:20:17] but I am not sure what happens if it is not on the list at all [10:20:20] and that was public [10:20:27] but maybe the logic has changed? [10:20:30] I would expect it to alert, but I don't remember [10:20:42] Amir1: cause wb_terms was supposed to be private [10:21:04] we don't have to discuss, we can check the code :-DD [10:21:12] marostegui: oh it was public (at least on s8), many tool builders (including me) depended on it [10:21:29] or we can create it and see if it alerts :P [10:21:29] then I am not sure how it could've alerted [10:21:33] Amir1: but the addition has to be to the sanitarium check list [10:21:44] that just creates the view on next run [10:21:54] Amir1: let me find it [10:22:18] So if it is a public table we have to do nothing, if it is a private table and gets replicated, then it will alert [10:22:19] so, Amir1, to be clear maintain-views.yaml - controls which tables have a view [10:22:47] there is another list that controls which tables are public and private, and ideally all tables should be added there [10:23:00] aaah [10:23:02] I see [10:23:06] let me find it [10:23:12] it is on realm.pp [10:23:14] Amir1: it is like a 2 layer protection [10:23:26] because just the view thingy wasn't super reliable [10:23:31] and that's what I am mentioning, if it is public there's nothing to do, if it is private and gets replicate it will alert [10:24:02] marostegui: but I need to add it to the list of public tables I think [10:24:15] it is possible it does nothing [10:24:34] but it would be nice to track all tables there for self accounting :-) [10:24:36] Amir1: no you don't have to, you only have to add it if it is private or have some columns that shouldn't not be replicated [10:25:04] aah, that explain why I didn't find anything when I was searching for "change_tag" [10:25:05] if you want it to have a view, then you have to add it to maintain-view but that's a different thing from the filtering/alerting [10:25:29] coooool [10:25:36] I'm going to create the table than [10:25:41] any objection? [10:25:42] ok, I see the thingy, there is private table and private dbs [10:25:49] on realm.py [10:25:55] but what happened to the other list? [10:25:59] we filter either tables entirely (private tables on realm.pp) or columns even at filtered_tables.txt [10:26:08] jynus: that's what I meentioned earlier [10:26:14] that is the one I wanted to search [10:32:13] jynus: logical backups run today? [10:32:25] not today, only on tuesdays [10:32:38] ah excellent so I will get db1117:3322 for a couple of hours [10:32:51] for some reason I thought it was both tuesday and thurs [10:37:49] Amir1: indeed, the check only verifyies F columns: https://github.com/wikimedia/puppet/blob/production/modules/role/files/mariadb/check_private_data.py#L166 [10:42:42] marostegui: okay to proceed to upgrading the codfw master of es5 to bullseye? I don't want to step on your toes [10:43:01] Amir1: yep go for it [10:43:09] cooool