[07:56:58] volans|off: we're kind of time pressed for this one - https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/745629 - could you spare a few minutes today to see if it's ready to get out there to the world? [08:18:07] good morning [08:46:23] errand, be back in ~1.5h [08:57:30] zpapierski: how urgent is that cookbook? I can merge it if needed. [09:27:16] sigh... guava & old spark strikes again... java.lang.NoSuchMethodError: 'java.util.stream.Collector com.google.common.collect.ImmutableList.toImmutableList()' [09:39:04] zpapierski: we need to discuss the other Elasticsearch plugins to update [10:09:51] sure, let me finish my meal and I'm available [10:22:40] ejoseph: I'm available, are you? [10:24:23] gehel: we need it by tomorrow. Preferably is should be merged today, so that we can also verify this ahead of time. [10:30:31] zpapierski: i'm here [10:30:45] ok, let's do what we did last time, meet + codewithme [10:31:16] https://meet.google.com/stk-ocxd-yni [10:41:37] dcausse: you have a minute? we eliminated most of the test failures, but we're not sure how to handle the last one [10:41:45] (or probably not the last one) [10:41:50] sure [10:41:55] join out meet? [11:06:01] break + lunch [11:08:06] CirrusSearchJVMGCOldPoolFlatlined fired for elastic1049-production-search-psi-eqiad and looking at the graph it looks "real" [11:08:08] (https://grafana-rw.wikimedia.org/d/000000462/elasticsearch-memory?viewPanel=56&orgId=1&from=now-30d&to=now&var-datasource=eqiad%20prometheus%2Fops&var-exported_cluster=production-search-psi-eqiad&var-instance=elastic1049&var-top5_fielddata=All&var-top10_terms=All&var-top5_completion=All) [11:09:42] tempted to just restart elastic there [11:12:08] lunch [11:30:29] Lunch [11:31:28] zpapierski: I'll merge this afternoon. This does not impact anything outside of WDQS / WCQS. We can always iterate later if volans_ finds something to improve [14:04:51] Happy Monday, I guess! [14:05:03] o/ [14:05:08] o/ [14:11:33] ejoseph: let me know when you're ready to continue [14:13:12] I am [14:13:17] Let's go [15:08:21] dcausse: can we discuss that 6.8.18 patch for a sec? [15:09:52] zpapierski: sure [15:10:05] https://meet.google.com/zco-dxnk-rxi [15:44:48] removing all extends ESTestCase fixed the issue on my side, pushing to gerrit to see if it's CI related [15:56:08] seems to pass in CI, looks like it was a non-issue after all [16:02:14] ejoseph: triage meeting: https://meet.google.com/qho-jyqp-qos [16:59:12] did the meeting end while I was away? [16:59:38] inflatador: yup, ended right now [16:59:57] ah OK, moving into the SRE mtg then [16:59:59] inflatador: we've got the SRE team meeting coming right up btw, I think this will be your first one since starting? [17:00:08] correct [17:00:38] inflatador: cool, we'll have you introduce yourself when it's time for our team's update [17:00:44] I'll talk first and then hand off to you [17:01:33] OK, I'll be ready ;) [17:02:13] inflatador: each team puts bullet-point updates in their team's section. ours is here: https://docs.google.com/document/d/1laiqY9Mj3ldAdEX9yhZQeUbYU-JSwdsMUfx961NCbRc/edit#heading=h.wbaq2drqa77m [17:02:26] Changes that should be mentioned out loud should be bolded, otherwise left unbolded [17:02:53] I'm a bit out of the loop on any new happenings from last week but I'll put some bullets for the wcqs discussion as well as the auth work in general [17:04:31] ryankemper: I'm still looking for an onboarding buddy for inflatador. Can you send a request during that SRE weekly meeting? [17:04:44] gehel: Will do [17:04:46] Mark and Faidon are aware of the ask (or at least should be) [17:04:49] thanks! [17:04:51] and dinner! [17:29:28] (mentioned the onboarding buddy request) [18:00:05] i have two one-liner patches to the rdf query service nginx (for wcqs) if anyone can look later: https://gerrit.wikimedia.org/r/739942 and https://gerrit.wikimedia.org/r/744892 [18:07:45] :eyes [18:11:37] OK, I added a +1 to https://gerrit.wikimedia.org/r/c/operations/puppet/+/739942 , somehow +2'd https://gerrit.wikimedia.org/r/c/operations/puppet/+/744892 but both look good to me [18:11:47] lunch/errands, back in ~1 hr [18:14:19] dinner [18:59:30] inflatador: you'll still need to submit and puppet-merge those patches. Check with Ryan for guidance! [19:19:34] back [19:45:22] gehel ACK, I think both you and ryankemper walked me through that process. Will reread my notes and wiki docs and hit up Mr. K if I get stuck [19:46:34] n [19:46:37] lunch, even [20:37:52] back [22:00:25] hmm, not sure how to wrangle grafana+promql :S When selecting wdqs vs wcqs in the dropdown, some graphs need wdqs vs wcqs, others need WDQS vs WCQS, others need rdf vs mediainfo. We can't easily change the values, rdf vs mediainfo for example are parts of a kafka topic name and we can't easily special case external metrics collection [23:13:49] ebernhardson maybe "label_replace()" in PromQL? https://prometheus.io/docs/prometheus/latest/querying/functions/ [23:28:25] maybe there is some way to make that work, it has some of the right pieces at least. Will play with it a bit [23:42:27] OK, I'm out. See ya tomorrow!