[10:09:12] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) [10:10:50] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) Thank you @brennen! GitLab is now OK. Do you know who can help with the things on Diffusion and Gerrit? [10:11:31] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) [10:35:51] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10Peachey88) >>! In T318342#8289818, @valerio.bozzolan wrote: > Thank you @brennen! GitLab is now OK. > > Do you know who can help with the things on Diffusion and Ge... [13:28:58] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) 05Open→03Resolved p:05Triage→03Medium a:03brennen [13:29:29] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) [13:29:42] 10GitLab (Project Migration), 10WMCH-Infrastructure: Create new GitLab project group: wikimedia-ch - https://phabricator.wikimedia.org/T318342 (10valerio.bozzolan) [13:41:24] 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token - https://phabricator.wikimedia.org/T308501 (10Jelto) The findings of yesterdays troubleshooting s... [13:44:39] 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab, 10Patch-For-Review, and 2 others: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10Jelto) Cross-post from T308501#8290690: it seems we have our first successful [buildkitd build job](https://gitlab.wik... [13:56:35] dduval, dancy: buildkit builds seem to work now after fixing the issues discovered yesterday. Pulling the new image works too 🥳 [15:38:09] jelto: yes! thanks again for pairing on that yesterday [15:38:52] nice [15:39:11] i did some more debugging yesterday and i believe that buildkitd is caching the token passed to it by the auth provider on the client side. it does so according to the `issuedAt` and `expires` properties of the token [15:39:50] Interesting. [15:40:07] my patch to upstream sets both to their go "zero" values for the statically configured token since it can't otherwise determine an exact value for either [15:40:48] the server side in turn sets some defaults for tokens that have such "zero" values on those properties, but it's unclear to me what the resulting values become [15:41:25] i patched the client code to instead set `time.Now()` for the `issuedAt` and `10` for `expires`, and I got much better behavior [15:41:45] however, i'm still a little concerned about how the server caches the tokens in general [15:41:56] nod. Seems dangerous. [15:42:21] i know it caches them according to their "scope" [15:42:40] but e.g. do multiple clients shared that cache? [15:42:49] that would not be good [15:42:49] What is the scope? [15:43:22] the token scope, so like `respos/releng/gitlab-runner,push` [15:43:24] something like that [15:43:30] er gitlab-runner-test [15:43:33] I see. [15:43:36] So I could make a .gitlab-ci.yml that repeatedly tries to push an image that I shouldn't be able to push... eventually another job that does have permission to push will run and I can piggyback on that cached credential. [15:43:42] exactly [15:43:46] it's unclear [15:43:52] i want to test it [15:44:14] but! the server jwt auth part seems to be working [15:44:21] so... incremental progress! [15:44:21] yesssss [16:16:23] the buildkitd code in question is https://github.com/moby/buildkit/blob/master/util/resolver/authorizer.go#L339 and https://github.com/moby/buildkit/blob/master/util/resolver/authorizer.go#L303 [16:17:03] in this case, the issuedAt is the unix epoch, and expires is 0 [16:18:00] so it will never be the case that `time.Now()` is before the token expiry, so `expires` remains 0 [16:19:46] during the cache check, `expires.IsZero()` is checked (meaning "is this a time.Time zero value?"), and if true, the entry is used unconditionally (meaning it never expires from the cache) [16:21:23] the auth cache is per image resolver (interface responsible for pushing and pulling images from registries) which is in turn cached per scope :) [16:21:35] scope being image ref and either push or pull [16:21:40] wee! [18:28:55] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) 05Open→03In progress p:05Triage→03High a:03dduvall [18:30:04] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) [20:07:54] 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab, 10Patch-For-Review, and 2 others: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dduvall) [20:08:51] 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token - https://phabricator.wikimedia.org/T308501 (10dduvall) 05In progress→03Resolved >>! In T30850... [21:38:04] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) [23:35:57] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) [23:51:11] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/1 [23:51:23] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) [23:52:19] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall)