[09:45:49] <atsukoito>	 Hiya!
[09:46:49] <dcausse>	 hi!
[10:02:52] <dcausse>	 lunch
[13:19:09] <inflatador>	 <o/
[13:22:04] <dcausse>	 o/
[13:22:43] <dcausse>	 oh cool, was about to upgrade opensearch-semantic-search to 3.7.0 and realized it's already done in codfw
[13:23:33] <dcausse>	 in eqiad data-nodes are on 3.7, all masters still on 3.5
[13:24:43] <dcausse>	 inflatador: any objections to me finishing this upgrade?
[13:25:05] <inflatador>	 dcausse please go ahead. LMK if I can help
[13:25:15] <dcausse>	 doing, thanks!
[13:25:33] <inflatador>	 probably my fault that the versions are mismatched ;(
[13:27:57] <dcausse>	 np!
[13:42:49] <ebernhardson>	 \o
[13:44:26] <dcausse>	 o/
[13:48:40] <ebernhardson>	 kinda indecisive on integrating query routing.  I started the method the ticket suggests, which is essentially adding a configurable voter inside SematnicQueryRoute.  Doable, but felt awkward.  So instead i started making query dispatching be configurable...sees better but it's kinda a lot of code :(
[13:57:21] <dcausse>	 this means the dispatcher no longer takes naively the best matching route by score? 
[14:03:54] <dcausse>	 finished master restarts on opensearch-semantic-search@eqiad to pick-up opensearch 3.7
[14:06:41] <ebernhardson>	 dcausse: the dispatcher would still have a score and choose best route based upon it, but there would be multiple voters that can reject a route and such
[14:08:25] <inflatador>	 dcausse I probably should've mentioned this earlier, but are you familiar with brouberol 's script for roll-restarting OpenSearch pods? https://phabricator.wikimedia.org/P96457
[14:09:39] <dcausse>	 inflatador: oops no never saw this one, thanks for sharing
[14:10:40] <inflatador>	 dcausse NP, are you OK with putting it in cirrus toolbox? I'll make an MR if so
[14:10:50] <dcausse>	 inflatador: sure!
[14:11:24] <dcausse>	 ebernhardson: ok
[15:02:58] <dcausse>	 ebernhardson: meeting?
[15:44:19] <ebernhardson>	 inflatador: new opensearch .deb to be uploaded: https://gitlab.wikimedia.org/repos/search-platform/opensearch-plugins-deb/-/packages/2498
[15:44:32] <ebernhardson>	 probably needs a new os38 component in apt
[15:50:56] <inflatador>	 👀
[16:27:50] <dcausse>	 dinner
[16:50:12] <inflatador>	 ebernhardson CR for adding the opensearch component: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1341948
[17:53:36] <ebernhardson>	 thanks! +1
[22:13:45] <inflatador>	 `RESOLVED: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 25.55556172842935 times in the last 10 minutes due to memory usage (mw@codfw to dnsdisc) -`
[22:13:57] <inflatador>	 Taking a look now even though it's resolved
[22:14:02] <inflatador>	 ryankemper ^^ FYI
[22:17:33] <inflatador>	 Ughh, I hate the new grafana way of showing variables
[22:18:46] <ryankemper>	 👀
[22:24:14] <inflatador>	 ebernhardson heads up that I just published the 3.8 plugins pkg, sorry it took so long
[22:37:23] <ryankemper>	 thus far i've confirmed there was a brief episode of memory pressure on chi, don't have a root cause tho
[22:37:35] <ryankemper>	 logstash has mostly given me red herrings so far
[23:02:17] <ryankemper>	 found a couple crazy queries that hit the nested limit, and just emitting log lines for them consumed at least a couple hundred megs. I don't have anything super concrete tho, throwing in the towel here