[11:09:22] 10GitLab (CI & Job Runners), 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, and 3 others: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10jbond) >>! In T308350#7928087, @thcipriani wrote: > Sounds good from from my... [16:23:42] dduvall: interesting notion [16:35:25] yeah, i looked into it a bit more this weekend. it seems there's an upstream blocker with that approach, namely that while docker does support storing a bearer token in its client config (see https://pkg.go.dev/github.com/docker/cli@v20.10.16+incompatible/cli/config/types#AuthConfig), it doesn't seem that buildkit's auth looks for it at all. it only tries to use username/password credentials [16:36:03] I was thinking of hacking buildkitd to work in the way that we originally hoped (i.e., that it can read credentials from a JSON file). [16:37:03] right, server side (buildkitd)? [16:37:29] yeah [16:37:40] that could be a good model as well. i like the bearer token too, though, and that's what the v2 docker registry actually users [16:37:43] uses [16:38:11] and i do see a place to hack in support for the `registrytoken` field from the client config [16:38:12] i think [16:39:08] dancy: brennen and i have a 11am (PDT) standing meeting. do you want to crash it and talk options? [16:39:21] or brennen could crash our 10am :) [16:39:47] Either of times times works for me. [16:49:05] dduvall, dancy: i'm in a waiting room and _might_ not be back to my Real Computer by 11 PDT, but if you all want to get started without me around then i'll try to jump in. [16:49:22] sounds good [16:51:30] 10GitLab (CI & Job Runners), 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, and 3 others: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10thcipriani) >>! In T308350#7930458, @jbond wrote: >>>! In T308350#7928087, @... [17:10:23] 10GitLab (CI & Job Runners), 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, and 3 others: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10jbond) >>! In T308350#7931549, @thcipriani wrote: >>> @lmata/@MoritzMuehlenh... [18:36:32] dduvall, dancy - back at my desk, did i miss any excitement? [18:37:10] brennen: we discussed some options for registry auth but it's not clear which are feasible yet [18:37:16] need to experiment a bit more [18:37:21] right on [18:38:06] i'm going to try to keep a lightweight decision log at https://etherpad.wikimedia.org/p/gitlab-sync [18:41:02] i'm looking at a token based auth scheme where we would: 1) use the existing `CI_JOB_JWT` token that is provided to each gitlab job as the registry credential (unclear how exactly to provide this to buildctl or `docker push` but the latter in theory looks for a `registrytoken` field in the auth config); 2) configure the registry to allow this kind of JWT auth from trusted runners only; 3) validate the token against GitLab [18:41:25] and 4) provide push/pull access to a registry namespace based on the payload of the token [18:42:26] "registry namespace" here meaning in our prod registry? [18:43:07] yeah [18:43:23] but they would correlate to GitLab project namespaces [18:43:35] ::nod:: [18:44:06] currently the `CI_JOB_JWT` is used for things like Vault, but in theory it's a general auth feature of GitLab [19:09:27] dancy, brennen: yay, i've verified that at least `docker push` does use the `registrytoken` from the `auths` section of the docker client config and sends it as `Authorization: Bearer {registrytoken}` [19:10:08] Excellent.. so next is to see if buildkitd passes it along properly? [19:10:30] i'm 100% sure it won't :( [19:10:31] but! [19:10:43] we can export from buildctl and do `docker import` followed by `docker push` [19:11:25] while we work with upstream on getting support for that config field into buildctl [19:11:31] or make a contribution ourselves [19:12:14] Running the real docker commands will require access to a dockerd won't it? [19:12:28] * dduvall facepalms [19:12:46] bloody hell [19:13:00] Sorry. :-) [19:13:01] why is docker so stupidly architected [19:13:40] ok, so... work on an upstream contribution to buildctl :) [19:13:51] and suss out the registry auth changes with SRE? [19:14:15] Sounds reasonable. [19:14:20] meanwhile set up the mirror-er in jenkins? [19:14:46] alrighty. i'll write a task with the proposal for the registry changes [19:14:54] Ah right you had found a nice Jenkins plugin that seemed to do what we want, right? [19:15:18] polls docker registries, yeah [19:15:28] 👍🏾 [19:15:33] but honestly i think a script + curl would work just as well [19:16:03] OK. I'd prefer a script that we can stuff in puppet [19:16:33] i'd hope the buildctl contribution would be easily accepted. "the docker cli does {this}, buildctl should also do {this}" [19:16:40] dancy: sounds good [19:18:01] btw, i found a nice little tool while debugging: `docker run --net container:registry byfcz/tcpflow -c` [19:18:28] (does a tcpflow command for the interface of a running container) [19:19:06] a nice reminder that encryption on container networks is important :) [19:20:37] Nice tip. tcpflow is new to me [20:34:54] TIL JWT is pronounced "jot" [20:44:03] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, and 2 others: Re-implement semgrep ci includes - https://phabricator.wikimedia.org/T307962 (10sbassett) New semgrep-rules-merge tool created here: https://toolsadmin.wikimedia.org/tools/id/semgrep-rule-merge [21:12:04] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Figure out authentication scheme for WMF production registry from trusted GitLab runners - https://phabricator.wikimedia.org/T308501 (10dduvall) [21:20:06] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Figure out authentication scheme for WMF production registry from trusted GitLab runners - https://phabricator.wikimedia.org/T308501 (10dduvall) [21:20:42] re: "jot" huh :| [21:21:09] that seems like a stretch but i'll say it [21:21:14] haha [21:21:40] wrote up https://phabricator.wikimedia.org/T308501 hopefully it's coherent [21:22:48] Looks good to me. [21:23:40] cool cool