[00:46:50] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11576930 (10Scott_French) [00:58:47] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11577077 (10Scott_French) [14:51:00] sorry Iā€Œ can't find the documentation. I'm sure after merging this, Iā€Œ need to deploy a service in k8s but Iā€Œ can't remember which: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1225119 [15:55:14] Amir1: it should be the same as https://wikitech.wikimedia.org/wiki/Mw-cron_jobs#Deploying_periodic_jobs [15:56:18] (with s/eqiad/codfw/ in the example helmfile command) [16:09:55] thanks! [16:31:24] swfrench-wmf: šŸ‘‹ I have this patch for node 18-vs-22 memory issue for mobileapps https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1236332. Let me know if you are available to try it on prod [16:32:57] nemo-yiannis: awesome! taking a look right now [16:34:50] +1 [16:44:05] thanks! [16:46:22] i am deploying now [16:59:46] nemo-yiannis: great, so it looks like nothing has immediately gone sideways, but as we've seen before, this might take a little while. I'll check in periodically on the "usual" set of metrics throughout the day (pod availability, downstream latency, resource usage, etc.) [17:00:24] sounds good, yeah it took ~1,2 hours to show the latency issue [19:04:28] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11580318 (10RLazarus) > Before the next upgrade, we may want to give charlie the ability to optionally exclude mediawiki se...