[10:21:47] pfischer, dcausse: would you have a bit of time to have a look at https://docs.google.com/presentation/d/1ZdN73LyZ5DVBrl_Zhw1mZ-AVFoy_KIynRSqjW7QNRWc/edit#slide=id.g1067b254598_0_137 and jump on a Meet to discuss (that's my part of the presentation for Selena this evening) [13:09:09] gehel: David is ooo. I’ll have a look. [13:16:00] gehel: LGTM; level of detail on slide 3 (status quo) might be a bit high for Selenas flight level, but as long as you talk her through, we should be fine. [13:16:49] BTW: I might not make it or be late for our triage meeting, today. [13:24:53] pfischer: would you mind jumping on a meet to give me some live feedback? [14:02:46] Will not make triage mtg today (dr appointment) [15:50:03] ryankemper are you going to be at the SRE mtg? I might not be back from the doctor yet. They might want us to say something about the outage [15:54:58] heading to doctor appointment, back in ~1h [16:01:57] ryankemper: triage meeting: https://meet.google.com/eki-rafx-cxi [16:15:16] inflatador: yeah I got the sre meeting [17:02:01] internet's cutting out for me [17:21:26] back [17:23:44] thanks ryankemper! [17:48:50] ryankemper: fyi, the meeting invite for turnilo vs grafana, i think david will still be on vacation (this week and next). [17:49:11] i thought there was some email, but not finding it in 2 min of email searching [17:49:20] ebernhardson: I think he's back for 2 days next week? [17:49:59] ebernhardson: yup https://docs.google.com/spreadsheets/d/1Wn1H-6l-hQgt4XC2S1OTdYVwAHFX35emroKyEG9IBkg/edit#gid=797726897 [17:50:40] ahh, ok [18:49:52] would like to get https://gerrit.wikimedia.org/r/c/operations/puppet/+/860129/ through puppet if we can, should be easy [18:51:55] ebernhardson headed to lunch but will put eyes on it once I get back [18:52:59] kk [19:19:32] back, looking [19:20:15] only issue off the top of my head is it might not clear out the extra instances from running, unclear [19:22:21] What are extra instances in this context? Guessing airflow? [19:23:05] this uses a templated systemd unit to run multiple copies of the mjolnir msearch daemon [19:23:15] i'm reducing the number of instances from 8 to 2 [19:23:32] basically reducing the max load it can put on the cluster [19:24:11] the daemon reads queries from kafka, runs them on the cluster, and pushes the results back into kafka [19:26:48] * inflatador re-reads the LTR wikitech page [19:30:51] the overall change happening here is it used to only query the idle cluster, and hit it with a ton of queries. Now we are going to have it query all clusters in parallel with reduced maximum load. [19:31:01] because we no longer have an idle cluster [19:34:02] Gotcha. None of this is blocking the PR, will merge shortly, just curious [19:38:24] OK, merged [19:42:11] cool, thanks! I'll check in on the loaders in a bit once puppet has run [19:44:31] Do I need to run puppet on `search-loader[12]001.(eqiad|codfw).wmnet` ? Don't think I've touched those guys yet [19:44:52] inflatador: can, those should be the only two effected by the patch [19:47:38] * inflatador needs to re-read the "Search" page as well [19:48:29] OK, I ran puppet on the loaders, LMK if I can do anything else to help [19:49:52] i have root there, should be able to indeed looks like it didn't cleanup the extra templated services, i have root there though will cleanup manually [19:50:10] * ebernhardson apparently can't hold a thought through a single sentence and repeat myself :P [19:50:25] you're in good company for that ;) [20:00:15] Search Deep dive is starting in https://meet.google.com/vps-uxdz-wua [21:22:35] quick break, back in ~15 [21:37:39] back [22:07:10] cleaned up all the extra msearch instances, ran puppet and nothing seems to have complained, going to call it good [22:30:59] awesome [22:43:26] see ya tomorrow!