[07:56:28] greetings [09:36:39] morning! I'm guessing this should have been taken of as part of clinic duty too or is it separate from other pywikibot processes? https://phabricator.wikimedia.org/T431056#12244172 [09:38:29] aputhin: my impression was that all of the pywikibot regular update work was now handled by platform internally [09:45:29] I don't recall this being part of the discussion. was it explicitly called out when dividing ownership / making adjustments to clinic duty? (I might just not have been at the sync and missed notes :)) [10:09:54] I think the k8s upgrade might have broken toolforge-deploy, helmfile fails because it expects an older helm version with different flags [10:10:30] or maybe that was separate from the k8s upgrade, but anyway I think the helm version currently installed in tools-k8s-control-7 is not compatible with the version of helmfile [10:10:36] I will open a task [10:11:47] dhinus: Unsure if it's the same issue, but there's a chance something in there could help https://phabricator.wikimedia.org/T388390 [10:12:07] claime: thanks, looking [10:12:08] dhinus: i suspect you need to just merge https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1370 [10:12:21] taavi: nice, thanks [10:13:19] aputhin: not sure :/ [10:18:19] taavi: merged, but same error :/ [10:18:55] which error is that? [10:19:03] T436228 [10:19:03] T436228: toolforge-deploy fails with Helmfile error - https://phabricator.wikimedia.org/T436228 [10:20:02] does it work in toolsbeta? [10:20:05] testing [10:20:33] it does work in toolsbeta [10:21:21] ok, so now we get to play a game of 'spot the difference' [10:21:56] helm is at 3.17 on toolsbeta control-10 [10:22:13] but on 4.23 on control-11 [10:22:37] and 4.23 also in tools-k8s-control-7 which is where the cookbook is failing [10:22:53] taavi@toolsbeta-test-k8s-control-10:~$ which helm [10:22:59] /usr/local/bin/helm [10:23:06] why is that from /usr/local??? [10:23:30] no idea :/ [10:23:57] file is dated Aug 13 [10:24:30] anyway, seems like we either need to upgrade Helmfile or downgrade Helm [10:25:01] yes, I suspect that /usr/local might be a ugly hotfix, judging from this line in .bash_history "install -m 0755 /tmp/linux-amd64/helm /usr/local/bin/helm" [10:25:11] bookworm-wikimedia comes with helm 3.11, will that work? [10:25:29] seems likely, I hope that's v4 that changed the args [10:26:42] or we use the helm 3.17 build that c.laime was referring to earlier, that seems even better [10:28:06] one sec [10:30:31] dhinus: got to run but https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330345/ [10:31:07] thanks +1d, I will merge and deploy [10:38:50] hmm puppet in tools-k8s-control-7 failed with "Error: '/usr/bin/update-alternatives --force --set helm /usr/bin/helm3.17' returned 2 instead of one of [0]" [10:40:47] ok just a race condition, I ran update-alternatives manually and it worked [10:51:53] ok in tools the deploy cookbook is now working fine [10:53:27] in toolsbeta, the /usr/bin/helm is missing for some reason [10:57:07] the /usr/bin/helm symlink is actually missing everywhere except tools-k8s-control-7, where I ran "update-alternatives" manually [11:20:52] I think the patch above with alternatives::select is not enough to create the symlink... I will look at this after lunch [13:07:04] dhinus: very strange https://phabricator.wikimedia.org/P96269 [13:07:14] note the 2025 date [13:08:15] I suspect these boxes used to use the wikimedia-managed helm3 binary (which created the alternatives entry), then migrated to the mirrored one (which managed /usr/bin/helm binary directly), and now uninstalling the mirrored one got rid of /usr/bin/helm but didn't manage to re-create it as alternatives managed [13:10:54] wait no ignore the date thing, it's a symlink and the date is when the helm317 package was built. otherwise that theory still works, helm would've been uninstalled only after helm317 was installed so it blocked the creation of alternatives-managed /usr/bin/helm [13:12:11] and `sudo apt reinstall helm317` fixes the issue, which matches that theory [13:56:03] taavi: thanks for the sleuthing ^ have you already run "apt reinstall" on all the affected hosts? [13:56:24] i ran it on all k8s controls and bastions on both projects [13:56:32] so yes unless there are more surprise helm installations [13:57:36] oh yeah just read https://phabricator.wikimedia.org/T436228#12261374 [16:27:28] What is the lowest frequency of metric that prometheus tolerate? Can we ingest a daily metric without getting a lot of 'stale metric' complaints? [17:57:10] aputhin: we didn't discuss pywikibot - I'll add an entry in our table for that - grey area - we can discuss next thursday, but perhaps an easy thing I can have Riccardo handle when he's back next week (don't know how easy/hard it will be to upgrade and not break stuff) [17:58:13] For the current version, thilp will handle it after all since he was on clinic duty anyway and wants to get familiar with the process [18:00:01] oh cool! thanks thilp !