[10:20:54] <dcausse>	 lunch
[12:21:58] <ebernhardson>	 \o
[12:22:44] <dcausse>	 o/
[13:04:05] <inflatador>	 <o/
[13:07:12] <dcausse>	 o/
[13:31:45] <ebernhardson>	 got distracted improving airflow-dags pytest suite.  Now runs in ~30s locally and ~40s in CI (before CI took 353s)
[13:32:38] <inflatador>	 NICE
[13:33:03] <ebernhardson>	 to be fair claude did a good bit of the initial heavy lifting, running analysis and making minor changes.  Still took a decent bit of hand holding, but very happy with results :)
[13:39:56] <dcausse>	 nice!
[13:40:22] <inflatador>	 I'm starting the reimage of search-loader1002 now BTW, holler if that is not going to work for you
[13:57:09] <inflatador>	 OK, reimage started. The maintenance to enabled ECS logs in EQIAD is happening in ~5
[13:59:33] <dcausse>	 inflatador: will have to skip pairing today, but please let me know if there's anything I should be looking at
[14:00:08] <inflatador>	 dcausse np, if you wanna take a look at https://docs.google.com/document/d/16d5KZe34ldvRNJXUiUgk5Gt215fgWTVyK-x1Kz0gsmY/edit?tab=t.0 and offer any suggestions or feedback, that's all I have
[14:00:16] <dcausse>	 sure
[14:07:40] <dcausse>	 inflatador: we restart only one master capable node at a time?
[14:11:14] <inflatador>	 dcausse sadly yes. It's not just a rolling-operation cookbook bug or something, we had the same problem during the reimages when we didn't use the cookbook
[14:14:56] <dcausse>	 ack, how do see logs in logstash? trying host:cirrus* but only seeing some eqiad hosts but not seeing codfw 
[14:15:46] <inflatador>	 You have to pick the `ecs-*` index pattern, I think it defaults to `logstash-*` or something else in the Discover panel
[14:16:22] <ebernhardson>	 even with ecs-* seeing nothing: https://logstash.wikimedia.org/goto/2f19aabde795397dc77694a91d921751
[14:16:53] <dcausse>	 I thought we restarted codfw yesterday to pick-up the logger config
[14:17:16] <inflatador>	 you have to select with `host.name` ;( https://logstash.wikimedia.org/goto/91efcc4ea1b98b86571f20af3598c741
[14:17:25] * inflatador should really document this
[14:17:58] <ebernhardson>	 ah!
[14:18:45] <dcausse>	 ah thanks! got something with host.name:cirrussearch2* 
[14:23:11] <inflatador>	 quick docs update as well https://w.wiki/TG5Z
[14:26:08] <inflatador>	 OpenSearch 3.8 released today https://opensearch.org/blog/whats-new-in-opensearch-3-8/
[14:47:29] <dcausse>	 there's https://docs.opensearch.org/latest/tuning-your-cluster/availability-and-recovery/remote-store/remote-cluster-state/ unsure if that helps but the cluster state is then fetched from that store instead of being transmitted from the master to everyone else
[14:59:04] <inflatador>	 Nice find, I added that to the doc
[16:55:56] <inflatador>	 The maintenance is over and EQIAD is repooled
[17:30:14] <ebernhardson>	 inflatador: i might have a line on a fix (from claude).  It locally reproduced the problem and thinks if we stop using auto_expand_replicas it will stop happening.  I'm (err, claude) working on a narrower reproduction to verify the finding and will make a cirrus patch
[17:30:22] <ebernhardson>	 the master storms
[17:43:43] <dcausse>	 dinner
[17:48:29] <inflatador>	 ebernhardson nice! I'm out for an hour but can look when I get back
[18:10:18] <ebernhardson>	 full reproduction, using a 10 node cluster with 5 master capables and 1740 indices / 5217 shards.  Each index has a single keyword field.  with auto_expand_replicas:0-2 it has 2k elections and storms for 10m+.  With number_of_replicas: 2  it has 3 elections and takes 170s to come to green from yellow
[18:11:03] <ebernhardson>	 with auto_expand master coordinator term increases by 5k+,  with constant replicas increases by exactly 1
[18:50:10] <inflatador>	 Wow, did you do that with docker-compose?
[18:55:31] <ebernhardson>	 yea
[18:57:01] <inflatador>	 that's cool, if you have a compose file to share or a good way to load indices LMK. I have a crappy bash script that loads stuff from dumps
[18:58:21] <ebernhardson>	 inflatador: hmm, i could make it into a repo and put it on gitlab i guess, claude wrote a script that writes a docker-compose file
[18:59:41] <ebernhardson>	 the thing is this depends on having files like `state.9243.json` which has the production state in it, probably fine to commit publicly
[18:59:41] <inflatador>	 Only if you feel like it, I could probably poke Codex a bit and get something similar
[19:10:26] <ebernhardson>	 hmm, the fix ended up being wider than expected. Before looking i thought it would be like 2 places, i've had to touch a whole bunch of files that mention auto_expand_replicas. Still shouldn't take too long though
[19:15:02] <inflatador>	 Will we need to reindex after the fix is applied?
[19:23:00] <ebernhardson>	 yea
[19:25:21] <ebernhardson>	 so it'll take a patch merge, mediawiki or backport deploy, then running the reindex script
[19:43:21] <ebernhardson>	 hmm, actually maybe i can just apply it. testing
[19:59:51] <ebernhardson>	 looks ok locally, doesn't seem like it does a wonky thing like 2 to 0 and then back to 2.  "should" be a noop 
[20:01:00] <ebernhardson>	 i suppose i could put together a script that applies it to cloudelastic first, perhaps with some intentional slowness at first just to make sure it's really doing as expected. My main worry is just that it will drop live replicas and re-copy the primary when auto_expand_replicas goes to false
[20:08:38] <inflatador>	 yeah, a phased rollout might be better in those circumstances, although I'm not sure how we would gate that. I guess just avoid reindexing prod until we see what happens in cloudelastic?
[20:13:36] <ebernhardson>	 reindexing would be fine, but this looks like we should be able to apply it everywhere without any reindexing
[20:14:33] <ebernhardson>	 it's basically curl -XPUT https://cluster:9243/foo_content,foo_general,blah_content,blah_general/_settings -d '{"index":{"auto_expand_replicas": "false", "number_of_replicas": 2}}' 
[20:14:39] <ebernhardson>	 and some json headers
[20:15:10] <ebernhardson>	 or 1 on cloudelastic i guess
[20:17:40] <inflatador>	 yeah, I was gonna go the lazy way and reindex, but if we're confident this is the problem I'm fine with updating in place. We're basically in an emergency situation now
[20:18:13] <ebernhardson>	 inflatador: sure i'll start on that then.  Also i had claude generalize the testbed: https://gitlab.wikimedia.org/ebernhardson/opensearch-testbed#
[20:18:25] <ebernhardson>	 basically the same thing i used to generate a cluster and evaluate this
[20:40:33] <ebernhardson>	 looks reasonable, starting up on cloudelastic:9643
[20:43:39] <ebernhardson>	 inflatador: its done, how do we feel about restarting a :9643 master?
[20:46:48] <inflatador>	 ebernhardson I'm game. I can depool just PSI, do have a preference on DC?
[20:47:03] <ebernhardson>	 inflatador: i did cloudelastic first, but i can apply it to a regular prod i guess
[20:47:09] <ebernhardson>	 it was surprisingly quick
[20:47:35] <inflatador>	 ebernhardson oh, I thought you meant prod. I'm game to restart anything in cloudelastic
[20:48:26] <ebernhardson>	 just the timing of applying these updates makes it look like it's helping, the first batch of 50 took 1.4s to reply to the request, the last one 0.4s
[20:48:46] <ebernhardson>	 which aligns with the finding that it was doing per-index work on all 
[20:49:13] <ebernhardson>	 maybe 0.7s if more fair, i guess the last request probably didn't have 50
[20:49:34] <ebernhardson>	 inflatador: yea lets restart one of the :9643 masters in cloudelastic, if no election storm i'll start applying to all the prod clusters
[20:51:11] <inflatador>	 OK, restarting 1009 shortly
[20:52:42] <inflatador>	 No storms there, do you want me to try omega or chi cloudelastic?
[20:53:01] <ebernhardson>	 inflatador: not yet i haven't applied it, sec i'll do those too
[20:55:35] <ebernhardson>	 inflatador: ok should be applied everywhere in cloudelastic
[20:55:49] <ebernhardson>	 lets try a master in the big cluster, if no storm i think we can declare we found it
[20:58:42] <ebernhardson>	 (big cluster = cloudelastic:9243)
[20:58:54] <ebernhardson>	 i should be precise when talking about changing prod i guess :)
[21:00:00] <inflatador>	 ebernhardson ACK, restarting now
[21:01:23] <inflatador>	 confirmed, I restarted the active master on cloudelastic psi and chi with no storms
[21:02:23] <ebernhardson>	 nice! I'll apply to all the eqiad clusters then, the actual application seems very no-op (it's supposed to be)
[21:02:52] <inflatador>	 cool, LMK when ready and I'll depool/restart master on one of the small clusters
[21:06:23] <ebernhardson>	 inflatador: fully applied in eqiad
[21:07:10] <inflatador>	 ebernhardson ACK, just depooled psi
[21:08:08] <inflatador>	 restarting psi on active master (cirrussearch1118)
[21:09:04] <inflatador>	 We are storm free!
[21:09:22] <ebernhardson>	 wowzers!!
[21:09:32] <ebernhardson>	 this has been bugging me for weeks
[21:09:37] <ebernhardson>	 applying in codfw
[21:10:02] <inflatador>	 So everything is looking great, but is there any reason to wait to apply everywhere?
[21:10:20] <inflatador>	 NM, I guess we're past that now ;) . Should be easy enough to revert methinks
[21:10:55] <ebernhardson>	 inflatador: i don't think there is any risk, it seems to work on all of these.  I suppose the main risk going forward is not merging the cirrus fix and a new reindex changes it back.
[21:11:06] <ebernhardson>	 I've got the cirrus patch most of the way there, can finish and put it up for review today
[21:11:22] <inflatador>	 ebernhardson sounds good. I depooled eqiad chi and I'm gonna restart the active master there just for grins
[21:11:28] <ebernhardson>	 +1
[21:12:21] <inflatador>	 Same results: clear weather and smooth sailing ;)
[21:12:59] <inflatador>	 Repooled both clusters in EQIAD
[21:14:12] <ebernhardson>	 all done, 0 indices across 9 clusters have auto_expand_replicas
[21:14:36] <inflatador>	 {◕ ◡ ◕}
[21:14:50] <ebernhardson>	 do we need to ponder this on not-cirrus clusters?  I suppose if they have 10 indexes in the whole cluster it doesn't matter.  Maybe a mention to o11y
[21:14:54] <ebernhardson>	 not sure if they use it or not
[21:15:14] <ebernhardson>	 i also have claude in the background testing against opensearch 3.8.0 to see if the bug still exists
[21:16:35] <inflatador>	 Yeah, it's worth checking....guess it boils down to whether or not it's a default behavior or set by CirrusSearch
[21:17:12] <inflatador>	 But regardless, nice job. You made my week for sure!
[21:22:36] <ebernhardson>	 mine too, glad it's finally resolved. Been worrying me
[21:23:33] <inflatador>	 I'm especially glad it's software as opposed to hardware ;)
[21:23:49] <ebernhardson>	 looks like so far the 3.8.0 does trigger the reproduction, but at 50-100x smaller magnitude
[21:24:11] <ebernhardson>	 which would be like 600s -> 120s? still meh
[21:24:21] <ebernhardson>	 no, thats bad math. 12s
[21:24:24] <ebernhardson>	 wouldn't have noticed
[21:24:34] <ebernhardson>	 well...maybe more like wouldn't have cared as much?
[21:27:04] <ebernhardson>	 it says the 2.19.5 test storm'd for 282s in a smaller repro, 3.8.0 stormed for 3.3s on the same
[21:27:54] <inflatador>	 interesting, I wonder if that was ID'd as a deliberate problem or just happened to be fixed
[21:28:56] <ebernhardson>	 according to this its two parts,  basically the old one was because it had to re-enter the security manager on every index to get some system property.   On the new one that specific property is now cached, and also 3.x no longer has a security manager
[21:29:12] <ebernhardson>	 hard to say
[21:31:40] <inflatador>	 I guess 3.x is next on the slate! But it's pinball time for me, see ya tomorrow!
[21:32:07] <ebernhardson>	 see ya!
[21:32:11] <ebernhardson>	 Looks like it was indirectly fixed in https://github.com/opensearch-project/OpenSearch/issues/16519