[08:52:17] <jayme>	 I've created a wikitech page describing the upgrade procedure to k8s 1.31 at https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/Upgrade/1.31 - while this is obviously based on our experience with upgrading wikikube clusters it should be applicable to the other clusters as well
[09:02:58] <elukey>	 nice!
[09:03:17] <elukey>	 how was the experience so far? I know that we already spent a huge amount of time on PSP migrations etc..
[09:09:15] <jayme>	 upgrade experience was okay-ish...there where a couple of rough edges but those should have been smoothed out now after wikikube-codfw is done. It was mainly due to packaging issues with the kubernetes-client package and workloads beeing undeployable because of pending, broken changes in deployment-charts git
[09:09:49] <jayme>	 that's why the documentation now suggests to deploy all services before wiping the cluster - making sure they can be deployed (best effort)
[09:10:38] <jayme>	 wikikube-codfw took us a good 4 hours because of those issues and low default puppet run batch sized, so I suspect other clusters to be done quicker
[09:57:51] <elukey>	 okok nice
[13:03:26] <inflatador>	 jayme I added instructions on how to deal with rdf-streaming-updater here: https://phabricator.wikimedia.org/T341984#10967612 TLDR is just depool and let us know, we'll handle the rest
[13:05:02] <jayme>	 inflatador: just saw, thanks. Would it be good enough if we properly announce the upgrade (and it's completion) via the ops mailinglist? I'd like to avoid a situation where we have to leave notes in N channels and platforms
[13:16:20] <inflatador>	 jayme if you can give us 2-3 days advance notice, that works
[13:25:48] <jayme>	 inflatador: cool. I've added that, and an additional question, to the task