[00:20:09] <inflatador>	 Incident report for CODFW omega https://wikitech.wikimedia.org/wiki/Incidents/2024-03-27_Elasticsearch_Omega_Cluster_Failure
[01:43:27] <inflatador>	 https://etherpad.wikimedia.org/p/resurrect_omega has some rough notes on next steps for Omega, feel free to add/change/edit as needed
[01:52:10] <ryankemper>	 thanks! i'll do another pass on the incident report after dinner
[10:29:25] <dcausse>	 errand+lunch
[12:34:57] <pfischer>	 dcausse: I ran into a bug shortly after rolling out the new ES bulk sink. The fix is ready: https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/111. Could you have a look, please?
[12:43:33] <dcausse>	 pfischer: lgtm
[13:05:01] <pfischer>	 thx
[14:04:39] <pfischer>	 I can’t make it to our retrospective today.
[14:40:55] <gehel>	 I probably won't be able to make it to the retro either. Augustin still sick and more than likely to wake up around then (if not, I might actually wake him up myself to make sure he sleeps this night).
[14:41:02] <gehel>	 I'll cancel retro for today :/
[14:41:43] <gehel>	 @team: could you make sure to send your status updates to https://etherpad.wikimedia.org/p/search-standup early? Public holiday tomorrow, so I'd like to send the update tonight if possible.
[14:49:12] <dcausse>	 errand
[14:55:28] <inflatador>	 codfw-omega back to yellow
[15:01:26] <inflatador>	 codfw-omega back to green
[15:02:12] <inflatador>	 next step is to requeue the writes
[15:28:23] <ebernhardson>	 \o
[15:28:34] <inflatador>	 <o/
[15:28:52] <ebernhardson>	 inflatador: awesome1
[15:28:54] <ebernhardson>	 !
[15:29:36] <inflatador>	 ebernhardson yeah, i found out about elasticsearch-node last night, that was the key. Updating docs but this is basically what I did: https://etherpad.wikimedia.org/p/resurrect_omega
[15:30:11] <ebernhardson>	 oh interesting,  ann out-of-cluster bootstrap routine
[15:50:05] <inflatador>	 Added procedure to search docs: https://wikitech.wikimedia.org/wiki/Search#Cluster_Quorum_Loss_Recovery_Procedure
[15:54:09] <dcausse>	 inflatador: well done! glad it worked
[16:02:37] <inflatador>	 Me too, sorry it was necessary in the first place ;( . Working out, back in ~40
[16:02:38] <ryankemper>	 \o
[16:03:30] <ebernhardson>	 things break, it's no worry :)  If anything i guess our time to depool the cluster and move traffic to eqiad is the only real problem. Things go wrong, but fixing them should perhaps find a way to move outside the mw deploy process
[16:44:08] <inflatador>	 back
[16:46:43] <inflatador>	 ryankemper would you mind doing a quick pass of cat/indices between eqiad and codfw? I don't anticipate data loss since we banned all the nodes ahead of time, but just wanna make sure things look reasonably similar
[16:53:40] <ryankemper>	 inflatador: sure
[17:00:59] <ryankemper>	 inflatador: yeah I see an identical number of indices, and only (unsurprising) small differences in doc counts since I ran the commands a few secs apart
[17:03:43] <ebernhardson>	 another tool you could plausibly use is the check_indices.py script from CirrusSearch. This one figures out from config what all should exist, and then checks the configured clusters to see if those things do exist
[17:03:46] <inflatador>	 cool, I still need to requeue the writes, will get started on that shortly
[17:04:05] <ebernhardson>	 things == indics
[17:05:12] * ebernhardson is running it currently, partially of curiosity.  Report will show up in mwmaint1002:~ebernhardson/check_indices.20240328T170300.json
[17:07:34] <inflatador>	 Cool
[17:09:53] <ebernhardson>	 complains about glent indices that should have been deleted but were not, toolhub_lists since it hasn't been whitelisted yet, and a failed testwiki reindex that needs to be cleaned up on cloudelastic
[17:10:02] <ebernhardson>	 but otherwise, it looks like everything it expects is there
[17:10:37] <inflatador>	 toolhub_lists needs to be whitelisted?
[17:14:25] <inflatador>	 for the requeue command https://wikitech.wikimedia.org/wiki/Search#Recovering_from_an_Elasticsearch_outage/interruption_in_updates , "wiki" is the index without the part past the underscore? Like for "zuwikibooks_archive" I'd just put "zuwikibooks", right?
[17:14:48] <ebernhardson>	 inflatador: yea
[17:15:10] <inflatador>	 cool, will get it started shortly
[17:24:36] <ryankemper>	 inflatador: (less important) I see notifications are disabled for `elastic110[3-7]` in icinga. These hosts aren't in the search cluster yet, but I don't see a service implementation ticket up
[17:25:44] <inflatador>	 ryankemper should be covered by T353878
[17:25:45] <stashbot>	 T353878: Service implementation for elastic2087-2109 - https://phabricator.wikimedia.org/T353878
[17:25:59] <inflatador>	 ryankemper actually no
[17:26:04] <inflatador>	 I guess that's eqiad
[17:26:29] <inflatador>	 You might have to create a ticket then, sorry about that
[17:28:08] <inflatador>	 ryankemper FWiW I never killed that decom cookbook, it's still running as my user on cumin2002
[17:30:04] <ryankemper>	 oh didn't realize
[17:30:08] <ryankemper>	 yeah we should have finished that off already
[17:30:09] <ryankemper>	 okay exiting it
[17:30:44] <ryankemper>	 Alright hopefully spamming ctrl+c didn't mess anything up :P think it should be okay tho
[17:33:59] <ryankemper>	 other random wrangling (sorry for the noise, just lots of doozies here): I see `cloudelastic1003` marked as decomissioning in netbox whereas the other cloudelastic hosts are marked as `offline`. Perhaps was an incomplete decom, will try to scrounge for phab context
[17:34:39] <inflatador>	 yeah, cloudelastic1003 failed some part of decom. I handed ticket off to dc ops but may have not tagged properly
[17:35:54] <ryankemper>	 AFAICT the decom cookbook sets them to decomissioning but not offline, so I guess it doesn't get marked as offline until dc-ops handles it maybe
[17:40:15] <ryankemper>	 Okay, reopened the associated dc-ops ticket (https://phabricator.wikimedia.org/T358046#9670481). Suspect there was just a step or two missed by dc-ops
[17:54:05] <inflatador>	 Heading to my son's birthday lunch, back in ~1h
[17:57:31] <ryankemper>	 Okay, dc-ops got cloudelastic1103 taken care of
[17:57:50] <ryankemper>	 Separately, I made a ticket for `elastic110[3-7]`: https://phabricator.wikimedia.org/T361268
[18:12:06] <dcausse>	 dinner
[18:58:27] <inflatador>	 back
[19:20:59] <inflatador>	 reminder to self: wdqs1013 still needs data transfer
[19:22:01] <gehel>	 inflatador, ryankemper, dcausse (and probably others): big thanks and congratulations on the recovery of omega! We've learned a few things and good to see them documented!
[19:27:07] <ryankemper>	 will be few mins late to pairing
[19:35:55] <inflatador>	 gehel np, sorry it was necessary
[19:38:06] <inflatador>	 just requeued writes, hopefully done soon
[19:42:29] <pfischer>	 ebernhardson: I rolled out the new ES sink today and it currently fails due to a large page (~5.6mb)
[19:48:16] <pfischer>	 I’ll temporarily roll back to the last working version.
[19:48:38] <ebernhardson>	 pfischer: hmm, i had hoped those would get fixed on the cirrus side :(  ok
[19:54:00] <pfischer>	 Caused by: java.lang.IllegalArgumentException: The request entry sent to the buffer was of size [5606355], when the maxRecordSizeInBytes was set to [4194304]
[21:04:32] <inflatador>	 ebernhardson looks like the codfw omega backfill is done if you wanna re-enable it in mwconfig
[21:04:45] <ebernhardson>	 inflatador: alrighty, can do