[10:20:44] <dcausse>	 lunch
[13:21:23] <inflatador>	 <o/
[13:23:20] <dcausse>	 o/
[13:23:43] <ebernhardson>	 \o
[13:24:24] <inflatador>	 Looks like the OpenSearch ECS logs are working, can y'all see https://logstash.wikimedia.org/goto/096e065718f7f43cf6468d920fe7558d
[13:25:35] <ebernhardson>	 awesome! yea seeing some there
[13:27:31] <dcausse>	 o/
[13:27:58] <dcausse>	 nice thanks!
[13:28:46] <inflatador>	 ottomata and b-tullis deserve most of the credit, but I'll take it ;)
[13:31:18] <dcausse>	 :)
[14:13:44] <ebernhardson>	 should we just remove sonar scanner from SUP?  It's giving 403's which we could look into, but i don't know i've really use the reports it generates
[14:14:28] <dcausse>	 +1
[14:14:33] <ebernhardson>	 kk, will do
[15:12:03] <ebernhardson>	 inflatador: have we been doing any maint on cloudelastic?  I'm reviewing the commonswiki_file reindexing failures and pretty much all of them are because a node wasn't available
[15:12:17] <ebernhardson>	 i'm wondering if thats expected because we were doing things, or unexpected instability
[15:15:38] <dcausse>	 I see some operations in https://wikitech.wikimedia.org/wiki/Server_Admin_Log for yesterday
[15:16:12] <dcausse>	 looks like a rolling-restart for the logging settings
[15:16:32] <ebernhardson>	 i wonder what we need to think about for cloudelastic...more shards?
[15:17:08] <ebernhardson>	 the summary seems to be if we need a host to be up for 5-10 days uninterrupted, that's not a guarantee
[15:17:13] <dcausse>	 to make a reindex resilients to restarts? or make the reindex faster?
[15:17:29] <ebernhardson>	 to make it finish in a short enough time period it doesn't fail 3+ times in a row 
[15:17:44] <ebernhardson>	 it runs for 3 days, fails, runs for 7 days, fails, runs for 5 days, fails
[15:17:44] <dcausse>	 yes...
[15:18:33] <dcausse>	 if it's shard related, I barely remember a slice option to parallelize the re-index even more
[15:18:56] <dcausse>	 but if it's bound to overall node perf I'm unclear if that's going to help
[15:19:50] <ebernhardson>	 yea i'm not actually sure, i guess i'm thinking cloudelastic doesn't look all that busy, load stays around 10 with 48 cores
[15:20:25] <ebernhardson>	 but it's always hard to guess what happens when you push harder
[15:20:39] <dcausse>	 yes...
[15:21:28] <dcausse>	 7days is a hard limit anyways for backfills give kafka retention
[15:21:31] <ebernhardson>	 hmm, actually i should verify the way we slice the reindexing. Maybe something there doesn't align with current best practices, or maybe we can just slice it more to have 2 processes per shard of something
[15:21:43] <dcausse>	 sure
[15:33:42] <ebernhardson>	 sigh, ok maybe it is our fault.  we do min(numNodes, numShards, maxAllowed)
[15:33:44] <ebernhardson>	 so, 6
[15:34:10] <ebernhardson>	 so in prod it happily does the full 32 shards.  In cloudelastic it does 6.  I should have looked at that like two weeks ago :S
[15:34:24] <ebernhardson>	 actually we set the maxAllowed to 20, so prod doesn't do 32
[15:35:19] <dcausse>	 oh ok
[15:35:40] <dcausse>	 yes, that's very conservative
[15:38:20] <ebernhardson>	 we can also provide them directly on the commandline...almost tempted to do the detecting in the reindex orchestrator instead (at least until we have a solid answer, to void mediawiki deploys)
[15:39:47] <dcausse>	 +1
[15:49:55] <inflatador>	 ebernhardson sorry was AFK, yeah I did restart the nodes yesterday for the ECS changes. I wanna say Ryan was adding something to `rolling-operation` that would warn if a reindex was in place. FWiW we don't have any restarts planned at the moment
[15:52:34] <inflatador>	 * for cloudelastic, that is*. We'll still need to restart the prod clusters, probably next week
[15:53:16] <ebernhardson>	 inflatador: ok that does make sense then, it's not instability it's just the pace of regular maintenance.  Confirms i need to look closer into how we parallelize it and make sure it gets done in a narrower timeline
[15:54:23] <inflatador>	 ebernhardson for sure, and we can usually delay our operations if we're aware of the reindexes. Here's the change I was talking about: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1309320
[15:56:38] <ebernhardson>	 turns out opensearch has one more option around reindexing we don't use, but not sure it's worthwhile.  Instead of issuing 1 reindex request that fans out and has 32 sub-requests, we could issue those 32 sub-requests ourselves.  So they would fail separately instead of all as one.  But then i'm not sure how valuable it is to have 26 of 32 shards complete, probably not very.
[16:00:13] <dcausse>	 if you can identify the particular slices that failed restart only them this could be interesting, but that sounds a lot of babysitting code to write
[16:03:49] <ebernhardson>	 yea it seems plausible...but also tedious. I'm thinking the juice isn't worth the squeeze
[16:07:37] <ebernhardson>	 i kinda want a 'safe delete' tool for doing index maintenance...like i tripple checked these are 0 replica failed reindexes, but i still feel awkward pushing the button
[16:09:21] <dcausse>	 yes... always has to ship a couple curl queries to verify but that's very error-prone, eps because I tend to copy/paste the full index name after the -XDELETE, kind of dangerous
[16:10:49] <dcausse>	 a "safe_delete.sh foo_content" sounds much safer 
[16:11:36] <dcausse>	 it has to be easily available tho... not sure I'd pull the cirrus-toolbox just for that
[16:12:23] <dcausse>	 or we make the cirrus-toolbox easily available on the deploy hosts
[16:12:25] <ebernhardson>	 I wonder how hard it would be to containerize cirrus-toolbox, such that it becomes a shell script like `mwscript-k8s` but massively simpler to make a helmfile release
[16:12:54] <ebernhardson>	 the helmfile seems massive overkill though, maybe we can just install it and auto-git-pull to deploy hosts?
[16:13:45] <ebernhardson>	 i guess i also don't really want to run cirrus-toolbox inside k8s, that's just the default deploy method
[16:13:46] <dcausse>	 sure, many tools are doing this in /srv/deployement, so would not be entirely new
[16:13:59] <ebernhardson>	 i'll make a ticket, seems like something we should get to
[16:14:04] <dcausse>	 thanks!
[16:14:16] <inflatador>	 FWiW I have a bunch of crappy ansible playbooks that do some of that stuff https://gitlab.wikimedia.org/repos/search-platform/sre/ansible-playbooks/cirrussearch
[16:14:42] <dcausse>	 heading out, have a nice week-end
[16:14:56] <inflatador>	 .o/
[16:15:27] <inflatador>	 as long as you have shell access to the deploy servers, you can use the playbooks
[16:15:53] <inflatador>	 But I'm fine with making toolbox into a container or deb pkg or something instead
[16:16:03] <ebernhardson>	 hmm, that is an improvement over spicerack.  I saw the task for non-ops to run cookbooks has yet again been deprioritized
[16:19:04] * inflatador stifles a rant
[16:20:49] <inflatador>	 Those playbooks are pretty awful examples of ansible, https://docs.debops.org/en/stable-3.3/ and https://github.com/geerlingguy would be better places to start if there is interest