[09:09:02] <dcausse>	 errand
[10:19:02] <dcausse>	 lunch
[13:10:41] <ebernhardson>	 \o
[13:14:52] <inflatador>	 <o/
[13:15:45] <Trey314159>	 I will be late to the Wednesday meeting. I have a dental thing I have to take care of and the next appointment is the same time as the meeting. Should be quick, though.
[13:31:21] <ebernhardson>	 cloudelastic IO is looking great this morning, i think we can be relatively certain we found a fix
[13:32:25] <inflatador>	 {◕ ◡ ◕}
[13:32:35] <dcausse>	 o/
[13:32:57] <inflatador>	 I'm gonna kick off a rolling restart just to force some shards to move around
[13:33:03] <ebernhardson>	 +1
[13:33:13] <inflatador>	 after that I'll probably reimage one or two to make sure Puppet's doing all the things
[14:06:50] <inflatador>	 dcausse got anything for pairing? NP if not, I'm just working on the above
[14:15:58] <dcausse>	 inflatador: no, sorry for missing the call, was distracted
[14:21:20] <inflatador>	 dcausse no worries, I don't have anything urgent
[14:23:32] <inflatador>	 I/O looks good ( https://w.wiki/Mmcm ) but we did get a flink alert for cloudelastic
[14:36:10] <dcausse>	 kind of expected if it recovers
[17:20:26] <dcausse>	 dinner
[17:45:53] <inflatador>	 cloudelastic cookbook is hung up, it appears that cumin can't access cloudelastic.wikimedia.org:9243 anymore. I had that happen to me last week, I think there must be some rate-limiting happening or something
[17:47:47] <inflatador>	 test test
[17:51:23] <inflatador>	 oh, that's not it. David/Cathal already figured it out, ref T425300
[17:51:24] <stashbot>	 T425300: lvs on cloudelastic1012 is misconfigured - https://phabricator.wikimedia.org/T425300
[18:48:24] <inflatador>	 Cloudelastic cluster restart is done, gonna try reimaging 1012 (since it has a broken LVS config anyway)
[18:53:52] <inflatador>	 gonna try my hand at writing a puppet role for relforge/beta cluster using https://www.thelinuxvault.net/blog/how-to-run-podman-containers-under-systemd-with-quadlet/ . That should allow us to swap OpenSearch versions quickly without reimaging everything
[19:25:23] <inflatador>	 cloudelastic1012 is back up, readahead looks reasonable
[19:56:37] <inflatador>	 Forgot to fix the hieradata before reimaging and the LVS iface is still goofy. One more time!
[21:19:46] <inflatador>	 OK, looks like 2012 is fixed
[21:19:52] <inflatador>	 err...1012 that is