[04:55:35] there seems to be a problem in toolsbeta. Someone else can help verify but it seems like the root cause is that the `CN` is set to `tool-fourohfour` (note the tool- prefix) in toolsbeta bastion client.crt file, while in tools bastion client.crt `CN` is `maintain-harbor` (no tool- prefix). As a result when you run `toolforge jobs list` on the toolsbeta bastion you get `No such file or directory: [04:55:35] '/data/project/tool-fourohfour/.kube/config'`, which makes sense because the config is located here `/data/project/fourohfour/.kube/config` [04:57:22] `No such file or directory: '/data/project/tool-fourohfour/.kube/config'`, which makes sense because the config is located here `/data/project/fourohfour/.kube/config` [07:18:15] Roo.k: I'd try to avoid it as there's a slight chance that it might break the python installation itself (not only the package management), but if you test it and it works should be ok, creating a small venv might be simple enough too, but can be done later for the long run [07:18:19] Raymond_Ndibe: looking [07:20:28] Raymond_Ndibe: this sounds like maitain-kubeusers, artur.o deployed the newer version there [07:52:03] Raymond_Ndibe: I've deployed the old maintain-kubeusers in toolsbeta and forced a rotation of the certs (deleting the cm `maitnain-kubeusers` from all the tool namespaces), that seemed to restore a working state [07:52:11] let me know if you see anything else weird going on [07:59:52] :-( [08:00:20] I'm fighting systemd in my laptop, will look at the maintain-kubeusers problem later [08:01:59] looks easy enough to fix (we are prepending `tools-` to the CN, where we should not) [08:13:05] I see [08:13:11] I fixed the systemd problem in my laptop [08:27:29] Raymond_Ndibe: I did a bit of a mess with the jobs-api MR on toolforge-deploy, the new one is https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/291 [08:28:53] Oh, I got a heisenbug in toolsbeta! Xd when I add a debugging statement the bug does not happen 🐛 [08:29:10] (/me playing with functional tests there) [08:43:03] * arturo back later [11:39:20] * dcaro lunch [11:52:34] I can't seem to create or delete magnum clusters in codfw, is anything going on there? [11:52:43] not that I know of [12:06:43] rook: thanks for quarto! <3 [12:06:53] np [13:11:05] @dcaro is there a reason we have different openapi schema versions across components? [13:11:05] envvars-api, base-yaml (the one used in api-gateway to merge specs): 3.0.1 [13:11:05] builds-api, jobs-api: 3.1.0 [13:11:05] [13:11:51] not that I know of, probably they were developed separatedly and then copy-pasted around [13:13:08] might even be a typo 1<->0 [13:13:11] xd [13:13:18] (3.1.0 is the latest though) [13:16:39] I'm also not quite sure how we are dealing with conflicting security schemes when merging the specs. Testing locally with a script that checks for this I get [13:16:52] https://www.irccloud.com/pastebin/9NuNZ2DT/ [13:30:32] We don't, and probably shouldn't, the auth at the sub-api level is done with a header, and at the api-gateway with TLS, but that will change [13:31:39] Once we decide the auth to use for the public API we should set it there, unless we want to generate the clis before that [13:54:25] 👍 [15:59:30] * arturo offline