[11:23:13] btullis: can you give this some priority please? https://phabricator.wikimedia.org/T337446#8882660 [11:36:29] btullis: and same for https://phabricator.wikimedia.org/T337446#8882673 [13:29:12] btullis: and one more https://phabricator.wikimedia.org/T337446#8882991 [14:04:26] marostegui: I think btullis is out, maybe we should notify someone else in DE? [14:05:23] No idea who is in charge of doing the views anymore to be honest: arturo dcaro dhinus bd808 ^ [14:06:05] what I mean is I think they have a backup, let me search it [14:06:16] (a backup SRE) [14:08:12] stevemunene (sorry backup sounds bad, I meant a peer that can do everything btullis does) [14:08:43] and he is ooo :-( [14:08:54] so bad day today [14:49:08] marostegui: balloons was working to handoff wiki replica things to Data Engineering folks for quite a while. https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas#Step_6:_setting_views seems to agree with that. [14:50:16] the cookbooks to do the views update should be runnable by any SRE [14:50:18] Yeah, I just wanted to get them done as soon as possible so we can continue with the cloning. Right now s1, s2, s5 and s7 on wikireplicas have old data [14:51:33] I've not personally done it in awhile, let's see if I can unblock [14:52:16] * bd808 was hesitant because he hasn't run the scripts in several years at this point [14:53:04] I would appreciate if it can be looked at, I have been trying to handle this on my own and I seriously have my plate full with all the other things, so I'd be more comfortable if someone who owns (or owned) this can take a first look [14:54:51] marostegui, of course. I'll try to help now [14:54:59] thank you [14:55:54] sorry for the rant, but in practice after the data engineering handover it's been very difficult to get any view updates to be done. for example wiki creations taking several months and https://phabricator.wikimedia.org/T315426 (private) still getting no activity [14:58:24] the cookbook in question is sre.wikireplicas.add-wiki btw [14:58:49] I'm not sure what else is out there; I know some more automation was tried and not completed [15:08:44] starting with s7 [15:10:33] bah, why am I not root on cloudb's [15:26:29] balloons: s7 isn't ready yet [15:26:51] balloons: s1,s2 and s5 are [15:27:07] marostegui, ohh.. I was trying to start with something not s1.. something small [15:28:14] but I'm also blocked because I don't have root on the clouddb's.. do I need to repool s7 or will it be ready soon? [15:28:47] there is no need to repool anything,clouddb1021 is the analytics replica [15:28:53] that one isn't on a LB [15:29:34] so to be clear what needs to be done is just adding the views on clouddn1021 for s1, 2 and 5 [15:29:41] on Monday try to clone s7 [15:30:07] once clouddb1021 is fully done with the replicas I will reclone the other affected hosts from that one [15:30:38] essentially https://phabricator.wikimedia.org/T337446#8882685 [15:37:26] ack. I think my lack of global root is blocking me [15:49:26] it's possible to create an admin group for root on clouddb*. maybe that would make sense for an access request [15:52:39] btullis, can you join #wikimedia-cloud-admin for a bit please? [16:10:36] According to Replag, s3 is 34 hours behind but the ticket only says s1,2,5,7 [16:10:47] https://replag.toolforge.org/ [16:21:15] marostegui, I think you should be sorted? I had Andrew proxy run things manually on just cloudb1021.. Can you check and see if you're unblocked? didn't do anything with meta, grants or indexes. Just views