[06:37:26] <NovemLinguae>	 This seems like a major bug but hasn't gotten any WMF response yet. Any tips for routing this to the correct team? https://phabricator.wikimedia.org/T423206
[07:24:31] <_joe_>	 NovemLinguae: I guess no one updated the task, but I think we took corrective actions
[07:24:56] <_joe_>	 specifically, that was due to the session datastore having to work cross-datacenter for a while due to hardware failures
[07:25:18] <_joe_>	 we have repooled the sessions datastore last night and you can see the effect on session loss
[07:25:20] <_joe_>	 https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&from=now-7d&to=now&timezone=utc&viewPanel=panel-13
[07:26:09] <NovemLinguae>	 thanks. want me to copy paste that into the ticket then mark it resolved? although the tail on that grafana graph still shows error levels higher than baseline, hmm...
[07:30:21] <_joe_>	 NovemLinguae: I just commented on the task :)
[07:30:53] <_joe_>	 not really since midnight, which is ~ the time when the session store was repooled
[07:31:19] <NovemLinguae>	 ah i see now, the right floating panel was covering up the rest of the tail
[07:31:46] <_joe_>	 yes this is a new grafana thing that I *really* hate
[07:32:18] <NovemLinguae>	 i like to write custom ublock origin adblock rules for annoying floating stuff. this might have to be one of my targets. hehe
[07:33:03] <_joe_>	 I might need it from time to time sadly, but I didn't find a way to tell grafana "keep it compressed by default"
[07:35:33] <NovemLinguae>	 did today's hardware incident have a phab ticket that we should cross-link to this one?
[07:36:44] <_joe_>	 I'm sure there is one but adding it would mean I need to use phab's search
[07:36:53] <_joe_>	 and my religion forbids me to do so before 1 pm
[07:37:07] <_joe_>	 jokes aside, yes we should :D
[07:40:05] <NovemLinguae>	 it is very important to respect religious requirements :)
[07:42:44] <_joe_>	 {{done}}
[07:46:43] <NovemLinguae>	 i don't see the cross-link. is it acl*sre?
[07:59:30] <NovemLinguae>	 oh i found it, you edited your post. all good
[07:59:34] <NovemLinguae>	 thanks for your help!
[08:13:38] <p858snake|cloud>	 <_joe_> I might need it from time to time sadly, but I didn't find a way to tell grafana "keep it compressed by default" <something something filing a upstream task (if there isn't already) to benefit everyone that uses the software
[08:14:34] <_joe_>	 p858snake|cloud: as most FLOSS developers, I am a terrible FLOSS user. I just complain and wait for the fixes to flow down the river
[08:17:14] <volans>	 it seems that could be disabled by config via viewPanelPane https://grafana.com/docs/grafana/latest/visualizations/panels-visualizations/panel-overview/#view-mode-panel-controls
[08:18:22] <_joe_>	 volans: patches welcome!
[08:34:12] <XioNoX>	 hey, my bad for bast3007 to be unreachable, I forgot to move the VMs out of rack BW in esams before rebooting the switch... Servers and site are depooled so no user impact
[08:36:22] <XioNoX>	 switch is back up
[09:53:25] <taavi>	 godog: ok to puppet-merge yours?
[09:53:32] <godog>	 taavi: yes please
[09:53:36] <godog>	 and thank you
[09:53:43] <taavi>	 done
[09:53:47] <godog>	 \o/
[10:20:09] <btullis>	 Hello. Heads-up, I plan to disable puppet for a few minutes on `C:k8s::proxy` (664 hosts) so that I can roll out https://gerrit.wikimedia.org/r/c/operations/puppet/+/1343023 and verify that it is a no-op on all clusters. I'll do so in 5 mins, unless anyone objects. 
[10:20:52] <marostegui>	 federico3 cezmunsta ^ are we fully done with replication changes for DC switchover?
[10:21:05] <federico3>	 we are
[10:23:35] <elukey>	 as FYI I just completed the reimages of all registry VMs to trixie, nothing to report but in case you see issues it is my fault :D
[10:27:04] <btullis>	 Proceeding to disable puppet on those nodes...
[10:37:20] <btullis>	 Testing complete. Puppet runs cleanly and confirms a no-op on a worker node from each cluster. Re-enabling puppet on those workers again. 
[10:37:26] <btullis>	 Thanks all.
[13:36:10] <tappof>	 Hi there, I've merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1327496 and need to run a scap deploy --k8s-only. The diff looks okay (https://phabricator.wikimedia.org/P96494), but I think it may trigger a massive rolling restart since it touches a lot of deployments.
[13:36:14] <tappof>	 Would it be okay to proceed with the deployment?
[13:39:17] <moritzm>	 tappof: probably rather not since some part of the DC switchover is starting in 20 minutes, but Simon can you more definitevely
[13:39:19] <moritzm>	 ^ slyngs
[13:40:27] <slyngs>	 tappof: We might want to hold of for a bit if possible. The plan is to start: https://phabricator.wikimedia.org/T435443 in 20 - 30 minutes
[13:42:22] <tappof>	 As long as I leave the brokers with the `insetup` role, it shouldn't be a problem to hold off. Can I leave the deployment pending in the meantime, or do I need to revert the patch? slyngs moritzm 
[13:43:03] <slyngs>	 I honestly. don't know, but aren't we risking that something else rolls it out?
[13:43:47] <elukey>	 yeah if we stall it, we'd need to revert it, otherwise somebody else will roll it out
[13:44:07] <elukey>	 but in theory the patch shouldn't interfere with the CDN/services switchover
[13:44:10] <tappof>	 ack elukey slyngs 
[13:44:25] <elukey>	 so we could proceed anyway, I think some pods will restart etc..
[13:44:30] <elukey>	 but you choose slyngs!
[13:45:30] <slyngs>	 I'd feel better about waiting if possible
[13:57:16] <tappof>	 ok slyngs reverted 
[13:58:14] <slyngs>	 thank you :-)
[14:01:33] <slyngs>	 We'll just be a little late on depooling eqiad while verifying the final bits
[14:51:29] <slyngs>	 We are finally ready. Sorry about the wait
[14:54:10] <vgutierrez>	 🍿 and 🍺
[15:24:13] <slyngs>	 Depooling eqiad is completed
[20:39:01] <lizimedia>	 o/
[20:39:34] <cdanis>	 o/
[20:47:06] <rzl>	 o/
[20:51:37] <lizimedia>	 okay so if prod and wmcs need different ssh keys, and I have two different yubikeys, is that four keys total? or is it supposed to be one for each
[20:51:54] <lizimedia>	 (and apologies if this is the wrong venue)
[20:52:54] <rzl>	 fine venue! only two physical yubikeys (one primary and one backup) and only the prod ones need to be on there. I think most of us just use a regular file on disk for the wmcs key
[20:55:07] <cdanis>	 you'll want an on-disk key for gerrit/gitlab as well, likely (I think I just use the wmcs key for this one which isn't great but shrug)
[20:55:34] <rzl>	 I *think* you can put multiple ssh keys on a yubikey, in which case you could just put a prod and a wmcs key on each?? but I don't know if anyone does that, you might be a test pilot
[20:55:55] <cdanis>	 you totally can
[20:56:53] <lizimedia>	 I found an old nano yubikey I was planning to use for the day-to-day stuff
[20:57:25] <cdanis>	 I think to do what rzl said you want ssh-keygen -O application=ssh:foobar
[20:57:26] <lizimedia>	 but unfortunately it's too old for ed25519 
[21:26:00] <jhathaway>	 o/ lizimedia!