[11:00:17] <dcausse>	 lunch
[13:10:04] <ebernhardson1>	 \o
[13:14:54] <ebernhardson1>	 hmm, guess something broke with drop_old_data, will look into it
[13:17:39] <inflatador>	 <o/
[13:18:56] <dcausse>	 o/
[13:20:01] <dcausse>	 ebernhardson1: had a look and some partitions were out of range, I cleaned them manually and it went back to green
[13:20:25] <ebernhardson>	 dcausse: oh! i'm surprised out of range causes errors, it's whole point it to drop out of range?
[13:20:41] <dcausse>	 no clue why that happened tho but the default allowed-interval did prevent some
[13:22:38] <dcausse>	 oh there's a big gap between scheduled__2026-03-09T00 & scheduled__2026-03-18T00
[13:23:03] <dcausse>	 looks like the airflow scheduler kind of skipped this dag for 9 days :/
[13:23:07] <ebernhardson>	 dcausse: yea i paused that dag so it wouldn't drop data while backfilling and forgot to unpause for a week :(
[13:23:13] <dcausse>	 ah ok
[13:23:27] <dcausse>	 makes total sense then :)
[13:24:39] <dcausse>	 the allowed-interval is nice to avoid accidental mistakes on backfills, I guess it did its job
[13:29:52] <ebernhardson>	 yea i suppose
[14:25:23] <dcausse>	 wondering a search keyword to vary the sort options could help as a stop-gap to the fact you can't manually pick up those from Special:Search without AdvancedSearch
[14:26:25] <ebernhardson>	 indeed choosing sorting from Special:Search is tedious, i just type them in.  I guess maybe? Keywords that update meta-configuration seem a bit awkward, but we also already have them
[14:27:58] <dcausse>	 true... not a fan of these keywords that mess up with the global context from the ui, you always have to ponder which one wins (ui context vs syntax)...
[14:30:51] <ebernhardson>	 do we even have a defined precedence? I suppose i would think, from least to most, perhaps global config -> url args -> keyword?
[14:32:32] <dcausse>	 the namespace prefix would be one of them I guess: https://en.wikipedia.org/w/index.php?search=help%3Atest&title=Special%3ASearch&profile=advanced&fulltext=1&ns8=1
[14:33:01] <dcausse>	 you select Mediawiki in the UI but search "help:test" you end up searching the Help ns
[14:33:12] <ebernhardson>	 yea i suppose that's a decent example
[14:33:37] <dcausse>	 prefix is roughly similar, it bypasses the ns selector
[14:38:44] <ebernhardson>	 school run
[14:53:57] <dcausse>	 ebernhardson: quick fix when you have a minute https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2118
[15:00:44] <ebernhardson>	 +2
[15:02:12] <dcausse>	 thanks!
[15:05:15] <inflatador>	 are we doing search stand up? I was in a maintenance and just joined
[15:05:38] <ebernhardson>	 inflatador: still daylight confusion time, the triage/planning meeting is on eu time (so, in 1 hr)
[15:06:42] <inflatador>	 ebernhardson ah, good catch. I was looking on the Meet page instead of the calendar
[15:24:24] <dcausse>	 I think we need to swith semantic search to codfw? (T414484)
[15:24:24] <stashbot>	 T414484: Upgrade DSE clusters to kubernetes 1.31 - https://phabricator.wikimedia.org/T414484
[15:25:04] <ebernhardson>	 hmm, we should be able to but i haven't configured he codfw cluster in prod, will put that together
[17:07:46] <dcausse>	 now it seems to be eqiad failing for the opensearch semantic search cluster
[17:09:16] <dcausse>	 inflatador: ^
[17:11:26] <dcausse>	 a bit random tho, some requests still pass
[17:21:22] <ebernhardson>	 i wonder if this kind of thing (if cleaned up more) would help the insource wish: https://people.wikimedia.org/~ebernhardson/insource-search.html
[17:26:22] <dcausse>	 could be? at least nice to get your regex started
[17:49:29] <ebernhardson>	 hmm, wonder what i did differently..in opensearch mandatory plugins most are analysis-icu, opensearch-extra, etc.  But then esperanto and serbian from the other plugins have their names (with spaces) 
[17:49:44] <ebernhardson>	 it's the separate repo, so something different in pom's i imagine
[17:52:27] <dcausse>	 :/
[18:27:59] <inflatador>	 dcausse inflatador :eyes
[18:28:21] <dcausse>	 thanks!
[18:29:32] <inflatador>	 cc ebernhardson , didn't mean to ping myself . Have y'all done anything yet re: semantic-search failures? I'll start looking now
[18:32:03] <dcausse>	 not really, just seeing CERTIFICATE_VERIFY_FAILED on curl requests for both sites
[18:34:06] <inflatador>	 My best guess is that we'll need `k8s-ingress-dse-aa.discovery.wmnet` certificates on ingress, ref https://gerrit.wikimedia.org/r/c/operations/dns/+/1250063/5/templates/wmnet#1035
[18:34:35] <dcausse>	 seems like a 503 by "server: istio-envoy"
[18:34:45] <inflatador>	 yeah, that must be it, iPoid is not affected because it doesn't have istio in front
[18:35:27] <inflatador>	 err....envoy that is
[18:39:49] <dcausse>	 dinner
[18:52:44] <inflatador>	 ah, nope, not related to the active/active stuff at all
[18:52:55] <inflatador>	 it's because we hid the pod-level certificates behind envoy and they expired
[18:53:21] <inflatador>	 ref T419289
[18:53:21] <stashbot>	 T419289: Ensure OpenSearch on k8s clusters can safely use envoy TLS termination - https://phabricator.wikimedia.org/T419289
[18:53:33] <inflatador>	 Fixing, will update shortly
[19:03:12] <btullis>	 I didn't know that we had enabled the tls termination on the ingressgateway for this cluster yet. 
[19:31:56] <ebernhardson>	 btullis: it was actually done for a secondary reason, i mean yes we need tls for prod services, but opensearch also validates the x-search-id header and the one mediawiki sends doesn't match their expectation (and there is no rfc, so their expectation is arbitrary)
[19:50:20] <ebernhardson>	 and envoy  has a place to strip that
[20:34:18] <inflatador>	 semantic search should be back for everyone now