[06:18:17] <ryankemper>	 inflatador: codfw restarts kept failing on read timeouts (probably just unlucky with the time of day). to resume tomorrow: `sudo -E cookbook sre.elasticsearch.rolling-operation search_codfw "codfw cluster restart to pickup swift-s3 plugin" --restart --nodes-per-run 3 --start-datetime 2022-06-29T04:25:11 --task-id T309648`
[06:18:17] <stashbot>	 T309648: Restore lost index in cloudelastic - https://phabricator.wikimedia.org/T309648
[07:18:14] <tanny411>	 Will be out couple of hours, going home to do some clean up. Will be back 7pm my time.
[07:42:56] <gehel>	 tanny411: good luck!
[10:21:40] <dcausse>	 lunch
[13:10:50] <inflatador>	 ryankemper thanks
[13:45:49] <inflatador>	 dropping off kids, back in ~15
[14:05:30] <inflatador>	 back
[16:13:31] <inflatador>	 ebernhardson : I couldn't raise ryankemper , so if you don't mind taking a look. Would also like to try a snapshot once I get back, maybe in deployment-prep or relforge
[17:28:30] <inflatador>	 back
[17:29:52] <inflatador>	 looks like the codfw restart finished w/out errors, will take a closer look after lunch
[17:53:59] <ebernhardson>	 \o/
[18:05:54] <inflatador>	 {◕ ◡ ◕}
[18:31:25] <inflatador>	 eqiad rolling operation is happening now
[20:03:42] <ebernhardson>	 always see the most curious things when looking at query logs. A random clickthrough to Articles_for_creation was the query 'is pineapple for pregnancy'
[20:39:01] <inflatador>	 LOL
[21:08:55] <inflatador>	 picking up Rx, back in ~20
[21:42:01] <inflatador>	 back
[22:21:00] <ebernhardson>	 down to ~1k deprecation logs from saneitizer instead of the normal ~3k, expect that once the trains rolls far enough forward those will finally stop
[23:33:59] <inflatador>	 eqiad restarts are done, cluster back to green. See ya tomorrow