[07:39:09] <gehel>	 Oscar is sick and staying home, my availability will be reduced today.
[07:46:17] <dcausse>	 hope he gets better soon, take care!
[08:43:11] <pfischer>	 I wish him a fast recovery!
[08:51:55] <pfischer>	 dcausse: Would you have a moment to review https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/14, please? I extracted common.config.CommonConfig as part of this which would affect two more PRs from Erik. 🙊
[08:54:38] <dcausse>	 pfischer: sure
[08:58:13] <gehel>	 errand, back in a few
[09:05:51] <dcausse>	 pfischer: not sure to understand why some tests have changed? e.g. https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/14/diffs#diff-content-4252edc05e199d685628fb38e0acd91e4fdd2015
[09:07:08] <dcausse>	 perhaps to use java9 features, e.g. ImmutableMap.of -> List.of
[09:08:12] <gehel>	 dcausse: MediawikiPageContentChangeEnrichJobManagerNotRunning, does this ring a bell?
[09:09:05] <dcausse>	 gehel: this is the flink job for the page enrichment, Gabriele should know what to do 
[09:09:10] <gehel>	 that seems to be on the DE side
[09:09:40] <dcausse>	 seems like mostly test methods that were moved to different places 
[09:15:36] <gehel>	 dcausse: thanks! I'm talking about it with Gabriele
[09:16:20] <pfischer>	 dcausse: The new feature implemented in this PR is HTTP routing with pattern matching (hence the changed tests). Since this is needed for both, producer and consumer, I decided to extract CommonConfig. I might have overshot by renaming config properties (another cause for changed tests).
[09:17:34] <pfischer>	 dcausse: let me check why it reordered the methods…
[09:17:40] <dcausse>	 yes just saw that the signature changed from Map<String, String> to List<Entry<Pattern, String>>, but it seems like methods were shuffled a bit in the file
[09:18:08] <dcausse>	 no worries tho, just a bit harder to review
[09:26:19] <pfischer>	 dcausse: I removed the superfluous changes.
[09:26:27] <dcausse>	 thanks!
[09:57:00] <dcausse>	 til lombok SuperBuilder (thanks to Peter), very powerful but slightly confusing 
[09:59:15] <pfischer>	 Yeah, agreed. I’m not perfectly happy with it either. But I like the immutable config that comes out of it. 🤷
[10:42:33] <dcausse>	 lunch
[13:14:48] <inflatador>	 <o/
[13:23:47] <dcausse>	 o/
[14:14:51] <hashar>	 hi, since 14:02 roughly, we get a bunch of ` ApiUsageException: Search is currently too busy. Please try again later.` errors
[14:15:24] <hashar>	 all from commmonswiki (I got hit by it while doing a Special:MediaSearch
[14:16:25] <gehel>	 dcausse, inflatador: any chance you could have a look at the increased Cirrus latency ?
[14:16:43] <dcausse>	 sure
[14:26:44] <claime>	 that's exactly following the switchover btw
[14:26:52] <claime>	 (we flagged it on -sre)
[14:29:39] <dcausse>	 claime: we run only codfw, right?
[14:31:49] <claime>	 yeah
[14:32:43] <dcausse>	 ok, we suspect that the cirrus query cache has to warm up
[14:32:59] <inflatador>	 claime are things looking better? We're looking at the cirrus query cache stuff
[14:34:35] <claime>	 yeah it's trending back down to baseline
[14:34:50] <gehel>	 our previous assumption was that since codfw was already receiving traffic before the switch, the caches should already be warm enough. Sounds like we were wrong?
[14:35:48] <dcausse>	 yes seems like the cache is not the same
[14:36:17] <gehel>	 that's an interesting learning in itself
[14:36:17] <dcausse>	 traffic pattern are perhaps very different between eqiad & codfw?
[14:36:19] <volans>	 claime: do you know from the top of your head if the warmup script does warmup also some search or just wiki pages?
[14:36:24] <inflatador>	 I guess because there's usually much more traffic in eqiad? But that's just a guess
[14:36:31] <claime>	 We didn't run the warmup script
[14:36:37] <volans>	 yes I know
[14:36:40] <volans>	 that's why I'm asking
[14:36:41] <volans>	 :D
[14:36:41] <claime>	 Ah
[14:36:48] <claime>	 Err not off the top of my head no
[14:36:52] <volans>	 if it would have helped or not in this situation
[14:37:01] <volans>	 or we might need a dedicated warmup step for search
[14:37:51] <dcausse>	 we also thought that the cirrus query cache was replicated between the two DC but apparently not
[14:37:55] <gehel>	 I vaguely remember we had a dedicated warmup procedure for Search. But that might have been directing a higher percentage of Search traffic to codfw before the switch. dcausse or ebernhardson probably know better than I do
[14:42:18] <dcausse>	 banning one heavily loaded node did help a lot tho 
[14:42:40] <dcausse>	 could be a poor distribution of the shards that was not detected while we had low traffic on codfw
[14:43:29] <gehel>	 we previously kept more_like queries on eqiad until the codfw cluster had time to warm up: https://wikitech.wikimedia.org/w/index.php?title=Switch_Datacenter&oldid=2056048#Preserving_more_like_query_cache_performance
[14:45:59] <gehel>	 looks like 30 minutes of degraded operations. Not good for a planned switchover, but that gives me confidence that in case of an unplanned one, things are going to be mostly ok
[14:47:22] <inflatador>	 in retrospect I see a few short-lived alerts from earlier this week for `CirrusSearch eqiad 95th percentile latency`
[16:01:49] <inflatador>	 Does anyone have strong feelings on whether or not the search issues should be an incident? I'm thinking yes
[16:03:49] <gehel>	 It had a pretty visible user impact, so that sounds nds like something that deserves communication. We also have work to do so that this does not happen during the next DC switch. 
[16:04:09] <gehel>	 That sounds like it makes sense to have an incident doc
[16:04:27] <inflatador>	 gehel ACK, taking a break but will start when I get back
[16:41:51] <inflatador>	 back
[17:22:57] <dcausse>	 going offline
[17:32:46] <inflatador>	 thanks for your help d-causse!
[17:33:15] <inflatador>	 https://wikitech.wikimedia.org/wiki/Incidents/2023-09-20_Elasticsearch_unavailable incident report here, feel free to add/change anything
[17:42:29] <inflatador>	 lunch/doctor appointment, back in ~3h
[20:23:02] <inflatador>	 back