[14:43:23] 10GitLab (CI & Job Runners), 10Security-Team, 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10brennen) 05Open→03In progress p:05Triage→03High a:03bren... [14:43:56] ^ now i need to remember how to test this. [15:35:48] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Next), 10User-brennen: Provision untrusted instance-wide GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10Jelto) [15:37:46] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Next), 10User-brennen: Provision untrusted instance-wide GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10Jelto) @thcipriani I added some more open topics to the description.... [16:41:11] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Next), 10User-brennen: Provision untrusted instance-wide GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10thcipriani) >>! In T297426#7917974, @Jelto wrote: > @thcipriani I add... [17:27:17] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10brennen) [17:27:19] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845 (10brennen) [17:27:27] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): Assess GitLab-provided docker container registry as a default for docker-in-docker build processes - https://phabricator.wikimedia.org/T307537 (10brennen) 05Open→03In progress p:05... [17:28:33] stuff we discussed just now: buildkit security concerns, buildkit vs. kaniko, practicalities of using the gitlab container registry, defining a set of wrapper images / scripts / CI includes to handle build boilerplate [17:28:49] i'm probably missing some things. [18:26:22] this is kind of interesting: https://docs.gitlab.com/ee/administration/packages/container_registry.html#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint [18:32:28] * dancy reads [18:34:44] it says it loses some features, but i'm not sure which ones. [18:34:54] quite probably just a whole can of worms. [18:36:59] Looks like "nested image names" functionality is affected. [18:52:10] 10GitLab (Infrastructure), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): Assess GitLab-provided docker container registry as a default for docker-in-docker build processes - https://phabricator.wikimedia.org/T307537 (10brennen) [18:53:58] 10GitLab (Infrastructure), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): Assess GitLab-provided docker container registry as a default for docker-in-docker build processes - https://phabricator.wikimedia.org/T307537 (10brennen) @Jelto, @Dzahn - we'd appreciate your thoughts, p... [18:54:09] ^ tried to capture questions from discussion earlier, feel free to correct / expand on anything in that description. [18:55:14] 10GitLab (Infrastructure), 10serviceops, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): Assess GitLab-provided docker container registry as a default for docker-in-docker build processes - https://phabricator.wikimedia.org/T307537 (10Dzahn) [18:58:28] 10GitLab (Infrastructure), 10serviceops, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): Assess GitLab-provided docker container registry as a default for docker-in-docker build processes - https://phabricator.wikimedia.org/T307537 (10dancy) [19:06:25] (off for a bit) [19:06:35] 10GitLab, 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate buildkitd instances as image builders for GitLab - https://phabricator.wikimedia.org/T307810 (10dancy) [19:07:33] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Establish image trust system for GitLab/Blubber - https://phabricator.wikimedia.org/T307541 (10dduvall) We (@brennen and others but I don't want to subscribe every to the task, Ahmon, Jeena, Jaime) talked a bit more about signing this... [19:09:02] dancy: re "strace: attach: ptrace(PTRACE_SEIZE, 27): Operation not permitted" well that's good [19:09:34] killing buildkitd isn't great but at least it's still in the realm of DoS [19:35:36] I wish I could delete the _subversion_ respos hosted on Phabricator. (lol) [19:35:42] maybe one day: https://docs.gitlab.com/ee/user/project/import/svn.html :p [19:35:53] 'Migrating from SVN to GitLab' hah [19:43:12] 10GitLab, 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10Dzahn) [19:43:33] 10GitLab, 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10Dzahn) [19:48:45] Taking a break [19:52:42] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Implement linting and unit tests for Blubber on GitLab CI - https://phabricator.wikimedia.org/T307534 (10jeena) a:03jeena [20:02:17] brennen: I am trying to use "transfer project" in gitlab admin (to move a repo from personal namespace to the cloudvps-repos namespace. (The alternative would be to just import it a second time). But in both cases I don't see that new namespace yet. Only everything that is under /repos. I remember you once gave me privs for /repos. Do I not have them on /cloudvps-repos? I am also admin though [20:02:23] meanwhile and before I was only like "repo admins". [20:08:51] 10GitLab (Infrastructure), 10serviceops, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): Assess GitLab-provided docker container registry as a default for docker-in-docker build processes - https://phabricator.wikimedia.org/T307537 (10Dzahn) For "docker-registry.wikimedia.org... [20:17:25] mutante: /cloudvps-repos is a separate hierarchy from /repos. I invited you as a "maintainer" to /cloudvps-repos which hopefully will let you move things there. [20:17:49] mutante: but poke me if that's not enough rights and I'll level you up to "owner" [20:21:01] bd808: I could now see the namespace and 'transfer' seems to have worked. thank you! [20:21:07] yw [21:34:07] seems like we're a bit dead in the water (re: blubber image publishing) until we figure out a registry solution [21:34:14] does anyone want to pair/brainstorm? [21:34:39] I think we should turn on the gitlab registry so we can exercise it. [21:35:11] seems like one approach would be to figure out the puppet changes necessary to auth per-group or per-project. another is to get the gitlab registry up and running [21:35:28] but the latter won't necessarily get us a deployable image [21:35:33] Puppet changes for the existing registry? [21:36:12] tweaking the auth in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/docker_registry_ha/manifests/web.pp [21:37:05] to provision htpasswd entries for a list of gitlab projects and also restrict access to registry prefixes for each project [21:37:23] Gotcha. [21:37:27] then the credentials for a given project can be bound to gitlab variables [21:37:40] for just the project or group of projects [21:38:26] just a thought [21:38:42] it doesn't seem like the best long term approach but it would get us something deployable [21:38:50] Are you proposing some manual edits for now? [21:39:07] no, we should at least write a patch to operations/puppet i think [21:39:24] sorry, that's what I meant.. manual edits to web.pp (versus some automated processes that groks gitlab) [21:39:27] er, what do you mean by manual edits? [21:39:34] ah, yes [21:39:36] more or less [21:39:40] hiera entries at least [21:39:42] Gotcha. [21:40:09] hmm, what would an automated solution look like [21:40:12] ? [21:40:13] So we could do it for the releng group now.. and use the ci-build token (which is already configured). [21:40:52] we could just grab the ci-build token, that's true [21:41:00] i'm not sure if sre would be cool with that [21:41:39] automation... there' [21:42:10] there's probably some way to get a list of projects.... so for each group/project... if a token has not already been generated, generate one and add the variable to the group/project.... [21:42:28] hmm, interesting [21:42:31] it would have to generate an output file to be moved to the puppet/private. [21:42:42] right [21:43:10] i'm not sure how that last part (move to puppet/private) would happen [21:44:17] That's SRE-only territory I presume [21:44:45] yeah [21:44:53] we do have a meeting with them tomorrow morning :) [21:47:30] i wonder if instead of using puppet/private at all, it could just rely on the gitlab variable config as the store [21:48:59] so, 1) get list of projects, and for each: 2) check gitlab project variables for an existing registry token; 3) if not found, generate a new one; 4) if found, grab it; 5) generate htpasswd entry using the token and configure acl to project prefix [21:49:29] that's puppet doing that :D (wee!) [21:49:48] what could go wrong? [21:53:01] * brennen returneth [21:53:48] re: turning on the gitlab registry, it's a simple switch to flip, but i think it's then available instance-wide [21:54:13] and i _don't_ think there's a way to cap its disk usage [21:54:20] * dduvall stares wide-eyed at the switch [21:54:45] perfect. who would want a quota? [21:54:49] in practice: might be fine, there aren't a lot of projects, but you know somebody'll start using it [21:55:10] there's still the question of: how do we get a deployable image out of this? [21:55:48] since we ostensibly wouldn't be configuring our helm values to point to said registry [21:56:06] we'd still need the replication part [21:58:37] nod. [21:58:51] I think the little replicator is the easiest hack we can employ [22:01:32] how do you see it working? [22:02:49] 10GitLab, 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate buildkitd instances as image builders for GitLab - https://phabricator.wikimedia.org/T307810 (10dancy) [22:02:56] btw: https://gitlab.wikimedia.org/dancy/builtkitd-abuse [22:02:59] hmm.. good question. [22:03:00] * dancy thinks [22:03:55] ah right. it subscribes to gitlab registry event notifications.. when it seems that an image of interest has been pushed to gitlab registry, it pulls the image from gitlab and uses docker-pusher to push to the official registry. [22:04:30] It could also poll the gitlab registry for images of interest, to allow for missed notifications [22:06:00] nice abuser! [22:06:44] so, to use docker-pusher it would probably have to run on contint then [22:07:24] which is a bit ironic but it would work for now [22:08:18] nod..yeah. [22:08:31] i'm down to try that [22:11:08] You can use operations/puppet/modules/profile/files/kubernetes/deployment_server/deploy-mwdebug.py for a registry polling example. [22:14:05] right on [22:14:34] On that note I need to step out to retrieve a child. [22:15:36] the little docker replicator approach really does seem like the most practical way forward, assuming we have somewhere for it to replicate _from_. [22:26:48] we could enable the registry but require people to turn it on: https://docs.gitlab.com/ee/administration/packages/container_registry.html#disable-container-registry-for-new-projects-site-wide [22:28:55] brennen: ooh. would turning it on require our approval? [22:29:47] or does it just require project owners to know where to look? [22:30:01] (i'll be retrieving children soon as well) [22:33:28] i think it just requires project owners to know where to look. [22:34:03] we could add it to the list of project settings that get auto-disabled by https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/blob/main/configure-projects and actually get that running in a scheduled job. [22:36:14] i say do it [22:36:20] :) [22:36:57] i filed https://phabricator.wikimedia.org/T308080 as a subtask to track the idea of setting up a replicator [22:37:29] if you flip the switch on the container registry, i can hack on the replicator tomorrow [22:38:18] oops, i got the parent/child relationship wrong [22:38:22] * dduvall fixes [22:39:04] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845 (10dduvall) [22:39:13] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845 (10dduvall) [22:41:10] Excellent [22:45:56] k, i'll put together a patch for the registry and the little configurator script. [22:46:53] (this might be where i find out the registry requires dealing with things like dns.)