[08:00:06] Cteam: welcome to today 🦄! Don’t forget to post your update in thread. [08:00:06] Feel free to include: [08:00:06] 1. 🕫 Anything you'd like to share about your work [08:00:06] 2. ☏ Anything you'd like to get help with [08:00:06] 3. ⚠ Anything you're currently blocked on [08:00:06] (this message is from a toolforge job under the admin project) [14:31:09] today: [14:31:09] * https://phabricator.wikimedia.org/T316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25 [14:31:09] * https://phabricator.wikimedia.org/T369163 toolforge: prepare deb packages for k8s 1.25 [14:31:09] * https://phabricator.wikimedia.org/T366061 [infra,k8s] package k9s for use in kubernetes [14:31:42] * https://phabricator.wikimedia.org/T365681 toolforge: kubernetes can't revoke certificates [15:09:28] k8s 1.25 upgrade status: [15:09:28] * all components upgraded and deployed: https://phabricator.wikimedia.org/T329671 [15:09:28] * kubernetes 1.25 successfully tested in lima-kilo: https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/161 [15:09:28] * api deprecations: some PSPs still lurking? https://grafana.wmcloud.org/d/dVVFcEAVz/deprecated-kubernetes-api-calls?orgId=1&var-cluster=prometheus-tools&var-versions=1.25&from=now-12h&to=now [15:23:14] Done: [15:23:14] * [cookbooks,ceph] Merged all the pending patches for ceph cookbooks, this migrates to spicerack alerting code, and adds batching to add_and_bootstrap and depool_and_destroy cookbooks [15:23:14] * [1st of the month upgrades] they are done :), fixed also the code to ignore archived projects [15:23:14] * [lima-kilo] Improved the toolforge_get_version script to highlight "development (yellow)" running versions and "deprecated (red)" running versions of components [15:23:14] * [toolforge,functional] added the direct-api functional tests suite to check the API direct access [15:23:14] * [toolforge,api-gateway] Deployed the authorization-checking api-gateway [15:23:14] Doing: [15:23:15] * [ceph,upgrade to bullseye] Reimaging cloudcephosd1011, upgrading the network firmware to 21.X pre-reimage seems to work \o/ https://phabricator.wikimedia.org/T369026 [15:23:15] * [puppet] Will re-think the patch that enables cloud.yaml check for missing hiera vars, might change the way hiera looks up in cloud https://gerrit.wikimedia.org/r/c/operations/puppet/+/1051332 [15:23:16] * [toolforge,auth] Continue with oidc diagram/tests (tomorrow have a dedicated slot for it) [15:23:16] * [ceph,disks] Will keep trying to load a single drive adding+removing from the cluster, has to be done in-between reimaging hosts though [15:23:17] * [toolforge,fourohfour] investigating all the alerts of it being down during the weekend