[03:23:55] i took a shot at it and verified that it works https://github.com/moby/buildkit/pull/2868 [03:24:03] we'll see how review goes [03:27:23] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Figure out authentication scheme for WMF production registry from trusted GitLab runners - https://phabricator.wikimedia.org/T308501 (10dduvall) I went ahead and hacked up an upstream contribution to support statically... [03:33:53] nice dduvall [10:15:28] 10GitLab (Misc), 10Release-Engineering-Team, 10wikimedia.biterg.io, 10User-AKlapper: How to identify affiliation of indexed GitLab accounts - https://phabricator.wikimedia.org/T306770 (10Aklapper) Thanks. That means that I could at least run manual checks on https://ldap.toolforge.org/user/username (replac... [12:05:00] 10GitLab (CI & Job Runners), 10Security-Team, 10serviceops, 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) I added a first version on how to get access to Trusted Runners. [12:14:40] 10GitLab (CI & Job Runners), 10Security-Team, 10serviceops, 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) ## Trusted Runner automation and access request I added a first version on how to get and manage access to Trusted... [12:16:33] 10GitLab (CI & Job Runners), 10Security-Team, 10serviceops, 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [12:17:39] Hey folks. I created a first prototype for access request and management for Trusted Runners. Described here: https://phabricator.wikimedia.org/T295481#7934229 Let me know what you think :) [12:18:30] jelto: is that file supposed to be yaml or json? [12:19:57] taavi: good catch! currently it's json. I'll fix that. But we can also move to yaml if that makes more sense [14:26:51] 10GitLab, 10Gerrit, 10serviceops-radar, 10Release-Engineering-Team (GitLab-a-thon 🦊): Request for a gitlab repo for the kubernetes workshop - https://phabricator.wikimedia.org/T308563 (10akosiaris) [14:29:28] 10GitLab, 10Gerrit, 10serviceops-radar, 10Release-Engineering-Team (GitLab-a-thon 🦊): Request for a gitlab repo for the kubernetes workshop - https://phabricator.wikimedia.org/T308563 (10akosiaris) Gitlab wise, I can create a personal repo easily, but my guess says that's not ideal either. I guess I should... [14:34:27] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen, 10User-dduvall: Write a GitLab "Migrating a Project" runbook / manual based on Blubber migration - https://phabricator.wikimedia.org/T307538 (10Volans) >>! In T307538#7926644, @hashar wrote: > SRE foundations pointe... [15:35:54] 10GitLab, 10Gerrit, 10serviceops-radar, 10Release-Engineering-Team (GitLab-a-thon 🦊): Request for a gitlab repo for the kubernetes workshop - https://phabricator.wikimedia.org/T308563 (10brennen) If [[https://gitlab.wikimedia.org/repos/sre|/repos/sre]] seems like the right group, that's already present and... [16:26:32] dduvall: am i correct in thinking that CI_JOB_JWT isn't dependent on the GitLab Container Registry features in any way? [16:26:57] 10GitLab, 10Gerrit, 10serviceops-radar, 10Release-Engineering-Team (GitLab-a-thon 🦊): Request for a gitlab repo for the kubernetes workshop - https://phabricator.wikimedia.org/T308563 (10Dzahn) @akosiaris I can take this and import it into something under /repos/sre. [16:29:41] nice, jelto! [16:29:48] brennen: that's right [16:29:54] AFAICT [16:29:58] cool cool [16:30:01] i believe it was implemented for Valut [16:30:04] er Vault [16:30:23] but it's a general feature/solution [17:18:09] dduvall: Are you working on the puppet/nginx side of T308501? [17:18:09] T308501: Figure out authentication scheme for WMF production registry from trusted GitLab runners - https://phabricator.wikimedia.org/T308501 [17:18:29] dancy: not currently. feel free! [17:19:35] ok [17:43:37] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Figure out authentication scheme for WMF production registry from trusted GitLab runners - https://phabricator.wikimedia.org/T308501 (10dancy) @dduvall https://docs.nginx.com/nginx/admin-guide/security-controls/configu... [17:46:58] dduvall: We might be boned. ^^ [17:48:02] gah [17:48:32] i'm not sure which release we run [17:48:50] I would assume we use the open source version. [17:48:57] fair assumption [17:49:01] Foiled at every turn! [17:49:07] seriously :D [17:52:32] fcgi program? [17:52:57] I did see some open source nginx jwt extensions. Written in Lua? [17:53:09] interesting [18:04:15] god i hate open core. [18:37:18] same [18:37:44] "Plus" is an especially irritating commercial software name as well [19:41:56] brennen: 💯 can we get t-shirts that say that? [19:42:39] :D [21:30:54] dduvall: `curl -D -` Thanks for that too. Much better than -v for https hosts. [21:31:32] :) [21:31:42] yeah, a little less chatty [21:33:37] A notable difference between Gitlab registry and production is that gitlab registry requires auth for all operations as far as I can tell. Can you confirm? [21:34:20] oh interesting [21:34:29] i'm not sure that i tried a pull without auth [21:34:31] sec [21:34:35] Put differently.. you can pull public containers from docer reg... not from gitlab (at least I haven't successfully so far) [21:34:55] s/docer/prod/ :-P [21:36:55] hmm, i _was_ able to do `docker pull gitlab.devtools.wmcloud.org:5050/dduvall/registry-push-test:foo` after deleting the auth entry in `~/.docker/config.json` [21:37:45] and did `docker rmi gitlab.devtools.wmcloud.org:5050/dduvall/registry-push-test:foo` just to make sure i didn't have the layers [21:38:26] hm.. I'll retry. I'm accessing using curl so there's plenty of room for getting things wrong [21:38:40] didn't occur to me to use the docker cli. hehe [21:38:51] This is what this project has done to me. [21:38:55] haha [21:39:53] suddenly you find yourself writing a lua implementation of regularly available tooling because of some random snafu [21:40:11] btw have you found any magic environment variables or command line switches to make docker and/or buildctl show the HTTP requests it is making? [21:40:24] i haven't [21:40:40] that's why i went with tcpflow on the registry side [21:41:08] but that's only if the registry isn't using tls of course [21:41:23] nod [21:41:45] alright. I can docker pull too. [21:41:45] you _could_ build buildctl locally (it's fairly easy) and fire up gdb... [21:41:56] gdb... not so easy [21:42:12] Does gdb support go binaries properly? I've never tried that. [21:42:17] it does [21:42:35] i haven't used it in a while so i don't remember if there are limitations [21:42:43] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Next), 10User-brennen: Add DigitalOcean resource monitoring for cloud runner nodes - https://phabricator.wikimedia.org/T308615 (10brennen) [21:43:05] it might require some extra `-ldflags` [21:43:32] I'd be more inclined to add prints() to the source [21:43:35] i used `go build -mod=vendor -trimpath -ldflags "-s -w" -o bin/buildctl ./cmd/buildctl` to build buildctl [21:44:53] https://go.dev/doc/gdb [21:46:53] OK, I see what's going on. A JWT is required for all accesses.. The auth server will give them out w/o authenticating the user if the access is for a public repo. [21:48:56] A JWT without a "sub" field is returned. [21:50:36] oh, right. i think i remember seeing something about that in the docker registry auth documentation [21:50:41] there's an anonymous token request or something [21:52:04] fwiw https://docs.docker.com/registry/spec/auth/jwt/#getting-a-bearer-token [21:52:40] so no sub == no subject == anonymous i'm guessing [21:52:51] makes sense. [22:06:49] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Next), 10User-brennen: Provision untrusted instance-wide GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10jeena) [22:11:15] 10GitLab (CI & Job Runners), 10Security-Team, 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10brennen) Tested against a shared runner in WMCS. Works as expect... [22:11:42] 10GitLab (CI & Job Runners), 10Security-Team, 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10brennen) 05In progress→03Stalled [22:44:20] Looks like the gitlab folks have been working on online registry gc: https://gitlab.com/gitlab-org/container-registry/-/issues/300 [22:44:43] and https://gitlab.com/groups/gitlab-org/-/epics/2313 [22:44:52] oooh "the third and last stage of online GC" that's good [22:45:46] Looks like they finally decided to use a database. [22:46:29] gitdb i hope :D [22:46:35] haha [22:47:08] maybe by the time we migrate google will buy gitlab and "merge" it with gerrit [22:47:57] 10GitLab (CI & Job Runners), 10Security-Team, 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10Jdforrester-WMF) Nice! Don't suppose there's a way to get it to s... [22:53:05] nah, i'm just kidding. we're going to figure this registry auth situation out and really get rolling [23:00:37] 10GitLab (CI & Job Runners), 10Security-Team, 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10brennen) > Don't suppose there's a way to get it to spit out a cu... [23:15:11] 10GitLab, 10Gerrit, 10serviceops-radar, 10Release-Engineering-Team (GitLab-a-thon 🦊): Request for a gitlab repo for the kubernetes workshop - https://phabricator.wikimedia.org/T308563 (10Dzahn) a:03Dzahn [23:41:45] 10GitLab, 10Gerrit, 10serviceops-radar, 10Release-Engineering-Team (GitLab-a-thon 🦊): Request for a gitlab repo for the kubernetes workshop - https://phabricator.wikimedia.org/T308563 (10Dzahn) > If Gitlab, what's the process for requesting a repo like that one? I made a first quick version of docs for th... [23:49:07] 10GitLab, 10Gerrit, 10serviceops-radar, 10Release-Engineering-Team (GitLab-a-thon 🦊): Request for a gitlab repo for the kubernetes workshop - https://phabricator.wikimedia.org/T308563 (10Dzahn) repo imported (with the "repo by URL" not the "github" feature because that requires creating a personal access t...