[06:50:31] <pfischer>	 o/
[07:12:38] <dcausse>	 o/
[10:00:55] <dcausse>	 lunch
[13:05:08] <inflatador>	 <o/
[13:05:14] <dcausse>	 o/
[13:09:39] <ebernhardson1>	 \o
[13:15:08] <dcausse>	 o/
[13:23:52] <ebernhardson>	 hmm, guess i gotta figure out how to adjust the logstash elasticsearch dashboard to query the ECS logs
[13:24:18] <inflatador>	 ebernhardson the index pattern is `ecs-*` if that helps
[13:25:03] <ebernhardson>	 i haven't edited a logstash dashboard in a few years, so i'm going to guess that helps but first i have to figure out how this even works :)  I'm sure it wont be too bad
[13:26:15] <inflatador>	 Sure, if you need help LMK. Although I don't know much about logstash dashboards either
[13:26:39] <ebernhardson>	 in other news, reindexes look done.  Having the reindexer override cirrus's very conservative index slicing made it work quickly, even on cloudelastic
[13:26:49] <ebernhardson>	 so guess i'll fix that in cirrus too
[13:30:10] <dcausse>	 \o/
[13:30:59] <dcausse>	 re logstash we should look into the logspam for "no index mapper found for field: [_type] returning default postings format"
[13:31:17] <ebernhardson>	 yea i saw that on discord and started to look, but then noticed the dashboard i usually use is empty
[13:31:56] <inflatador>	 Hopefully that's just an old client in cloudelastic
[13:32:20] <dcausse>	 oh, it's only cloudelastic?
[13:33:01] <inflatador>	 There's only a single prod host shipping ECS logs at the moment, cirrussearch2079
[13:33:17] <inflatador>	 https://logstash.wikimedia.org/goto/d30f28a8c5e3dbc772180a3f39754ac4
[13:33:44] * inflatador will try filtering only on that host and see if we still get it
[13:39:53] <ebernhardson>	 inflatador: one curiosity,  i would expect the `Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[enwiki_content], ...` log messages to also include an exception, but somehow it's missing here. Not sure entirely though why it would be missing.  I'm guessing a regex like `idolized[^.]{0,80}\[\[[^]]{2,40}\]\][^.]{0,20}and\s\[\[` blows up the determinization limits, but
[13:39:56] <ebernhardson>	 can't say exactly
[13:40:08] <ebernhardson>	 that specific message isn't a big deal (probably), but the missing exception message is curious
[13:42:18] <ebernhardson>	 i would have to play around to see, the code looks like there are a couple basic options that control it (if i found the right code): https://github.com/elastic/ecs-logging-java/blob/main/log4j2-ecs-layout/src/main/java/co/elastic/logging/log4j2/EcsLayout.java#L344
[13:42:37] <inflatador>	 ebernhardson ACK, sounds like we still have some work to do on the log formatting
[13:43:12] <inflatador>	 I think it might be getting split into multiple messages
[13:44:48] <ebernhardson>	 hmm, seems possible. I did find some other messge swith error.stack_trace populated
[14:37:36] <pfischer>	 o/ Today is a DPE staff meeting conflicting with our triage. Would you want to attend that? I am still catching up and cleaning up the board. There are a few bug reports we could look into. Other than that, I am working on the ITC feedback (which is due by the end of this week).
[14:44:36] <inflatador>	 I'll be attending the DPE Staff mtg
[15:11:53] <dcausse>	 ebernhardson, Trey314159 triage?
[15:12:12] <Trey314159>	 I'm in the DPE staff mtg
[15:12:21] <ebernhardson>	 dcausse: in DPE staff meeting, can triage if we have things.  
[16:41:49] <ebernhardson>	 is deploy1003 new?  My bash history is completely empty :(
[16:41:53] * ebernhardson copies from codfw...
[16:49:26] <ebernhardson>	 shipped updated SUP to eqiad/codfw/cloudelastic.  Looks all fine, brings in the per-item retries
[16:49:38] <dcausse>	 nice!
[16:52:15] <dcausse>	 could you ship to staging as well? there's no cluster to update there so the change is not strictly needed/testable but it's generally nice to have staging aligned running the same image as prod
[16:52:26] <ebernhardson>	 oh sure, i forgot that exists
[16:52:31] <dcausse>	 thanks!
[17:12:48] <ebernhardson>	 dcausse: did you cleanup airflow? I got a bunch of fail emails but poking in the UI it all looks normal-ish
[17:24:16] <inflatador>	 Nice!
[17:53:28] <ebernhardson>	 inflatador: random thought, we might want to consider different logging levels in cloudelastic.   Query issues are more meaningful on prod where we control the queries,  in cloudelastic it could be any random tool (although honestly, it's probably global-search.  I'll add a ticket to check it's _type handling)
[17:54:53] <inflatador>	 ebernhardson ACK, that's a good point. I'll check around  in Puppet to see if we have any related vars yet
[17:55:59] <ebernhardson>	 hmm, maybe not global-search.  I ran a quick grep and didn't see anything that suggests it sends _type anywhere.   I asked claude to do a check as well, but expecting it to come back with the same
[17:58:28] <inflatador>	 It might be implicit in older versions of libraries. Like I know vector will send a `_type` when it makes indices unless you explicitly tell it to use a newer version of the Elastic API
[17:59:56] <dcausse>	 dinner
[18:00:45] <dcausse>	 ebernhardson: sorry missed your ping, yes I tried to cleanup all airflow failures
[18:01:52] <ebernhardson>	 dcausse: thanks! just wanted to make sure we didn't miss things.
[18:22:18] * ebernhardson is taking far too long to write a comment that explains why i think # of shards is better than upstream 'auto' ... just just ship it
[18:23:46] <ebernhardson>	 but now i'm waffling..maybe the right option is to leave auto-slice max at 20 but turn it into a config value instead of a constant, and set it appropriately in wmf-config since our clusters are much larger than external cirrus installs...
[21:00:45] <inflatador>	 Just opened a ticket to migrate the apifeatureusage and search-loader hosts to Bookworm or Trixie ( T433890 ) . If you have any suggestions/warnings etc. let me know. I barely ever touch those hosts
[21:00:46] <stashbot>	 T433890: Migrate search-loader and apifeatureusage hosts to Bookworm or later - https://phabricator.wikimedia.org/T433890
[21:03:43] <ebernhardson>	 inflatador: for search-loader not expecting any issues, it downloads and runs a conda environment with everything built in.  Should "just work"
[21:05:33] <inflatador>	 ebernhardson ACK, for search-loader I should be able to take down one at a time with no issues?
[21:05:55] <ebernhardson>	 inflatador: they are independant, and work off of received kafka messages. They can just go down and it's fine
[21:06:23] <ebernhardson>	 it might cause some mjolnir jobs in airflow to stall, but not a big deal. 
[21:06:41] <inflatador>	 ACK
[21:07:06] <inflatador>	 for apifeatureusage, looks like an there is a newer version of logstash that 0lly is running on Bookworm, so maybe we go with that?
[21:13:52] <ebernhardson>	 inflatador: hmm, seems reasonable. Sadly i don't know much about that either :S
[21:15:44] <inflatador>	 no worries, we never got a response in T424248 about the apifeatureusage stuff either. I guess technically we still own it per https://wikitech.wikimedia.org/wiki/Search_Platform/Accountability#Misc
[21:15:44] <stashbot>	 T424248: ☂️Migrate non-cirrus indices from production OpenSearch to OpenSearch on k8s ☂️ - https://phabricator.wikimedia.org/T424248
[21:20:03] <inflatador>	 all right, I'm gonna try to reimage search-loader to Bookworm
[21:20:23] <inflatador>	 excuse me, Trixie. I'll try it first, then fall back to Bookworm if it fails
[21:25:54] <ebernhardson>	 +1
[21:36:41] <inflatador>	 cool, reimage in progress
[22:16:06] <inflatador>	 ebernhardson search-loader2002 is back up. It looks OK to me as far as I can tell, but LMK if you notice anything wrong