[09:04:32] taavi: morning, it looks like a8de1038620fbc573bd5051b7719c714e3b08925 triggered puppet failures on acmechief[12]001 [09:05:11] vgutierrez: hey, sorry about that! do you have the full error message handy? [09:05:41] https://www.irccloud.com/pastebin/XYuNPGal/ [09:06:34] other instance using keyholder::agent should be experiencing the same issue [09:06:38] *instances [09:08:03] yeah.. alert1001 is complaining about the same [09:08:36] oh, indeed. https://gerrit.wikimedia.org/r/810294 should fix that specific issue [09:08:39] cumin, deploy, netmon.. [09:09:06] oh nice [13:56:40] apergos: a quick question about the new dumps servers (which you might not have an opinion about): I think that the old servers are a drbd pair with the rsyncs only happening on one of them. I'd like to stop using drbd and just let each box in the new pair maintain itself (mostly because this is our last drbd use case and it would be great to stop thinking about it.) Can you think of any downsides? For instance, is running the actual [13:56:40] dumps twice as often going to be a load problem? [14:09:43] running which actual dumps? [14:10:07] andrewbogott: we run twice a month (for sql/xml) and once a week for the rest, there's no way we can speed that up, nor does that make sense [14:10:39] do you mean the new clouddumps boxes should both do web service and do nfs? [14:11:08] I think I need to understand more what you propose and what you're trying to fix, to see what that would mean for stuff on our end. [14:11:45] apergos: to be clear, I barely know what goes on on the existing boxes. The task in question is https://phabricator.wikimedia.org/T309346 and the title expresses about everything I know on the subject. [14:12:09] My understanding is that labstore100[67] are mirrors of each other and that each is both web server and an nfs server, does that sound wrong to you? [14:12:35] only one is active as the web server at any time and the other one is active as the nfs instance server [14:12:41] that's to the best of my understanding [14:12:55] so both have copies of everything [14:17:04] apergos: ok, that doesn't really break with my understanding. [14:17:28] so what would change? [14:17:29] So what I'm asking about is the part where content (*as if by magic*) actually arrives on those systems. [14:17:35] rsync [14:17:44] it's always rsync [14:17:56] Right. Does that already happen separately on each host? I was assuming it only happens on one and then drbd syncs between them. [14:18:07] there's no syncing between them [14:18:11] it's just boring old tedious rsync [14:18:24] oh! Well great, then I probably don't have to think about this at all :) [14:18:33] there's only one rsync that goes between and it's the downloaded wme dumps [14:18:34] Maybe puppet will do all the work; I'll start from that. [14:18:50] Maybe they don't really use drbd, or maybe they have it installed but it doesn't matter. [14:18:57] maybe they don't :-D [14:20:09] You've had your hands on 100[67] in the past, right? So you might be a good judge of whether the new hosts are working or not? [14:20:50] they seem to want to install hdfs clients which could be another wrinkle... [14:22:58] andrewbogott: I'm trying to build the hdfs client for bullseye right now in an attempt to de-wrinkle that for you. [14:23:31] oh yeah! I'm so mired in a different blocker for those hosts that I forgot that was also a blocker :) Thanks! [14:30:03] I could really only say "yep web downloads work" :-D never involved in that nfs service to the instances... [14:32:36] apergos: ok, that alone would be pretty helpful but I'll ping when those boxes are further along [14:32:46] ok!