[13:51:31] <Krinkle>	 vgutierrez: nudge re https://gerrit.wikimedia.org/r/c/operations/puppet/+/1182652 - I'd like to avoid a merge conflict and to bring back beta cherry-picks to "only" 4 historic hacks, nothing recent :)
[13:51:38] <Krinkle>	 Beta currently depends on this patch to function.
[14:01:14] <vgutierrez>	 that looks good 
[15:46:50] <swfrench-wmf>	 FYI, in a couple of minutes, I'll be restarting etcd on a single non-PyBal conf node in codfw for [0].
[15:46:50] <swfrench-wmf>	 * this will result in a brief disruption for clients connecting to that node (note: codfw is the read replica cluster, so conftool etc. writes will not be affected).
[15:46:50] <swfrench-wmf>	 * as usual, once things are stable, I'll be following this with a rolling restart of all codfw-associated confds.
[15:46:50] <swfrench-wmf>	 [0] https://phabricator.wikimedia.org/T352245
[15:46:50] <swfrench-wmf>	 cc: elukey arnoldokoth
[15:47:19] <elukey>	 \o/ ack
[15:47:56] <swfrench-wmf>	 vgutierrez: I'll ping you here post-restart to check Liberica
[15:48:45] <arnoldokoth>	 Ack. Thank you.
[15:53:57] <swfrench-wmf>	 moritzm: I'm going to do this now :)
[15:55:51] <moritzm>	 ack!
[15:55:55] <moritzm>	 I'm around
[15:57:12] <swfrench-wmf>	 restarted
[15:57:18] <swfrench-wmf>	 kicking various tires :)
[16:01:05] <swfrench-wmf>	 alright, I think we're good
[16:02:27] <vgutierrez>	 lol
[16:02:45] <swfrench-wmf>	 vgutierrez: if you could take a quick look at Liberica in codfw-associated sites (eqsin, ulsfo, codfw) at your convenience, that would be awesome
[16:03:27] <vgutierrez>	 Nov 10 15:56:20 lvs5004 libericad[518250]: time=2025-11-10T15:56:20.188Z level=WARN msg="ErrorCodeEventIndexCleared detected" key=/conftool/v1/pools/eqsin/ncredir/nginx
[16:03:27] <vgutierrez>	 Nov 10 15:56:20 lvs5004 libericad[518250]: time=2025-11-10T15:56:20.188Z level=WARN msg="etcd watcher finished unexpectedly" service=ncredirlb_80 error="etcd error code 401. Retryable: false"
[16:03:31] <vgutierrez>	 the regular 401s but all good
[16:04:01] <swfrench-wmf>	 awesome, so exactly what we expend for some long-running idle watches that got interrupted
[16:04:04] <swfrench-wmf>	 thank you!
[16:04:11] <swfrench-wmf>	 *expect
[16:05:49] <swfrench-wmf>	 alright, I'll get started on those confd restarts
[16:05:52] <swfrench-wmf>	 thanks, all!
[17:02:12] <vgutierrez>	 swfrench-wmf: BTW.. quick way of checking impacted instances: https://grafana.wikimedia.org/goto/OY5yZlzDg?orgId=1
[17:06:38] <swfrench-wmf>	 vgutierrez: ah, thank you! this is quite handy
[19:43:32] <cdanis>	 anyone care to stamp https://gerrit.wikimedia.org/r/c/operations/puppet/+/1203462 ?
[19:46:36] <sukhe>	 TIL. I think can be helpful
[19:48:14] <cdanis>	 it's nice, I've also started using fzf recently which we already have in that list
[19:49:11] <sukhe>	 (for those who try to run it, it is actually batcat and not bad, the executable)
[19:49:18] <sukhe>	 er s/bad/bat
[20:01:03] <cdanis>	 moritzm: sigh, you are right
[20:01:05] <cdanis>	 fixing
[20:02:06] <sukhe>	 yeah sorry, I for one forgot about the buster hosts completely. I guess a conditional for bullseye and above is all we need there I guess
[20:02:33] <cdanis>	 yep
[20:03:59] <cdanis>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1203512
[20:04:10] <cdanis>	 thx!
[20:08:26] <moritzm>	 hopefully these are history in a few weeks...
[20:08:48] <cdanis>	 I can't even ssh to them anymore 😅 had to use cumin to see if puppet had indeed failed