[02:07:07] Quota reduction https://phabricator.wikimedia.org/T434023 . Inclined to just do that right away, but I should wait for a +1 [08:05:12] is mirrors.wikimedia.org not a thing anymore? [08:06:15] yep, xd https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/LJYRIS4WB66HIRCAO4GIDTXCMDVZRBMA/ [08:10:55] I had locally cached an old image of docker-registry.wikimedia.org/trixie:latest, that had that in the sources.list and was failing to run apt commands [10:25:15] toolforge haproxy got alerts [13:36:31] I just had a look at a VM (statanalyser.statanalyser) that has a seriously scrambled network after an in-place upgrade from Bullseye. I'm not going to spend a lot of time on it and don't know exactly how it got this way, but... I guess this is one more vote against in-place upgrade. [13:44:22] dcaro: I am admiring the graphic on https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy and trying to figure out what happens next with https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1355 -- it seems to say that I deploy first, and merge later, is that right? [13:45:28] andrewbogott: yep [13:45:51] does that mr look ready to deploy on toolsbeta? [13:45:58] as you created that branch by hand, you'll have to pass `--git-branch ` to the deploy cookbook [13:46:07] ok [13:46:30] have you tried in lima-kilo already? [13:46:48] yes, although I've only tried the automated tests, haven't done any by-hand testing [13:47:14] are there things you think should be tested that aren't covered by the test suite? [13:48:13] I think that for cert-manager should be enough [13:48:39] not sure if there's an easy way to force the timeout to happen (to double check that it's applying it) [13:49:11] you mean, cert renewal? [13:50:14] nono, the helm timeout that we changed [13:50:41] oh that [13:50:52] well that won't break runtime things anyway [13:51:06] yep [13:52:01] ok, so I'll deploy on toolsbeta now [13:52:44] 🎉 [13:52:53] oh, I should be doing this on the tools cumin host, or cloudcumin? [13:53:27] cloudcumin [13:53:42] (or at least I've always used cloudcumin) [13:53:43] andrewbogott: yeah, I finally got around to updating https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_instance_operating_system_migration to discourage that more clearly [13:53:58] dcaro: indeed, project cumin boxes don't support running cookbooks [13:54:31] looks good taavi, thanks! [13:54:32] (they're primarily around for puppetdb access, since cumin can only talk to one puppetdb instance and we have multiple projects with separate instances) [13:56:36] sudo cookbook wmcs.toolforge.component.deploy --cluster-name toolsbeta --git-branch certmanager --run-all-tests --component cert-manager --task-id T408786 [13:56:37] T408786: Upgrade cert-manager past 1.15 - https://phabricator.wikimedia.org/T408786 [13:56:44] dcaro: look right? [14:01:32] :+1 [14:30:58] what is the equivalent to toolforge_get_versions.sh that I can run on toolsbeta to make sure I got what I expected to get? [14:31:14] toolforge_get_versions.sh [14:31:57] can't remember if it supports the `kubectl sudo` thing available on the bastions but should work as root on a k8s-control at least [14:33:52] https://www.irccloud.com/pastebin/PLiFyZZw/ [14:34:40] ah, you'll need a clone of toolforge-deploy for it, it's a script in the utils/ folder [14:36:10] I suspect that it should be around cloned already, though I don't remember exactly where xd [14:36:25] ok, will look [14:37:26] buh, i'm still doing something silly [14:38:26] https://www.irccloud.com/pastebin/yCRRJRlx/ [15:02:10] taavi: any idea about ^ ? And, how long shall I wait after deploying to toolsbeta before deploying to tools? [15:04:07] sorry meeting [15:04:15] np [15:04:58] oh, it should run on the bastions, as it tries to fetch the packages intstalled (clis) [15:05:03] I think it might be silently failing [15:05:44] https://www.irccloud.com/pastebin/z8gU4auu/ [15:06:06] the table is meant to be copy-pasted in a gitlab comment that would format it nicely xd [15:06:20] | cert-manager | chart | cert-manager | cert-manager-v1.20.3 | | [15:06:30] is that the version that you expected? [15:06:51] btw. it also pasted the output in the merge request: [15:06:52] https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1355 [15:06:58] (when you deployed) [15:07:48] yep, it's the version you changed [15:09:53] yes, that's the version I expected. thank you! [15:10:22] (and now I'm in a meeting, but keeping half an eye on this) [15:11:01] ack [15:12:14] to test you can try creating a new tool, and see if the certificate is created [15:12:36] (might take a minute for maintain-kubeusers to create it though) [15:12:38] how is cert-manager involved in tool certificates? [15:13:03] doesn't it do a certificate signing request? [15:14:23] from kubernetes directly, yes, but that's not what cert-manager is involved in [15:15:19] hmpf... confusing, then we might have to force a renewal of one of the APIs certs? Those go through cert-manager right? [15:15:31] most of them do, yes [15:16:41] is there any reason we don't use cert-manager to renew the tool certificates? [15:18:40] tool k8s api certificates are used in a completely different way than what cert-manager is designed for, so you would just get more moving parts and custom glue code for the same result [15:18:40] (besides time/not gotten there yep of course xd) [15:18:59] if cert-manager can even issue certs signed by the cluster internal ca, which it might not [15:22:25] that rings a bell yep [15:22:58] still in a meeting but... is there a straightforward way for me to force cert-manager to do a thing? [15:23:31] basically, what cert-manager is good is maintaining certificates that exist as kubernetes Secrets [15:23:54] and tool certificates exist as files on NFS, and NFS files are definitely not represented as Secret objects [15:25:52] ack, so we would still need to implement the extraction of that cert whenever it's renewed somehow (the glue code you mentioned) [15:27:15] yeah, which imo is not worth it since maintain-kubeusers would still have to handle all other parts of the user config and so sticking certificates in there in addition is pretty easy [15:50:19] andrewbogott: let me have a look then to see if I find a way to trigger it without having to remove/redeploy a component (an option, though might break something) [15:50:37] ty [15:56:13] you can run something similar to this [15:56:16] https://www.irccloud.com/pastebin/YGjmZqBj/ [15:56:24] that will force that certificate to get renewed [15:56:50] you should see a certificaterequest going through (`kubectl get certificaterequest -n ingress-admission -w`) [15:57:53] and see the certificate being renewed in the describe [15:57:57] https://www.irccloud.com/pastebin/mpFLiqxk/ [16:02:50] andrewbogott: https://os-deprecation.toolforge.org/ still has fastcci-nfs-1 and wikidumpparse-nfs-1 shut down, is there a special reason why those can't be deleted like the rest? [16:03:20] no, I thought I did already but I'll look now [16:05:16] ty! [16:38:24] quick review https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/333 (I missed it a few times in the last week) [16:40:33] lgtm although I do not know what "stern" is... [16:40:54] it tails logs from pods, picking up new ones if the deployment creates new and such [16:41:07] (among other things) [16:41:27] cool [16:41:32] oh, also, one for you: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1321058 [16:41:54] backups, it's been a while :) [16:43:08] sure has [16:43:19] fortunately you don't need to remember anything to understand that patch [16:56:02] +1d [16:59:17] thx [17:32:03] * dcaro off [17:32:05] cya tomorrow!