[05:32:23] I might be misunderstanding things, but I'm not sure why CI here is failing https://gitlab.wikimedia.org/repos/commtech/toolforge-docs/-/merge_requests/23 it seems to be trying to run on the source repo (where no job runners are available), rather than the target (where there are). [05:48:47] oh, actually maybe it's this? https://phabricator.wikimedia.org/T297426 [08:12:20] samwilson: the project hokwelum/toolforge-docs has no Runners configured. You either have to add your own self-managed runner or migrate to /repos group. Instance wide Runners are not available yet [08:13:44] jelto: that makes sense. But it's a MR to a repo under `/repos`, so I thought it'd use the runner there? Like on Github, where a repo owner has to approve the running of a PR from an untrusted fork. [08:27:55] samwilson: to do MR from forks we need instance wide Runners (T297426) afaik. So you have to do the MR from a feature branch in the same project to also have CI [08:27:56] T297426: Provision untrusted instance-wide GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 [09:18:49] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen, 10User-dduvall: Write a GitLab "Migrating a Project" runbook / manual based on Blubber migration - https://phabricator.wikimedia.org/T307538 (10hashar) For the archival of Gerrit repository it is done manually via #... [16:02:07] 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dduvall) The image built from https://gerrit.wikimedia.org/r/791427 works and I'm working on the operation... [16:58:39] > So you have to do the MR from a feature branch in the same project to also have CI -- that is going to confuse literally everyone [16:59:46] mutante: thanks for the review! [17:03:46] we do intend to have instance-wide runners [17:04:12] current status is the confusing in-between state. [17:07:19] thcipriani: ack. thanks for the clarification :) [17:08:00] sure thing, thank you for caring <3 [17:08:21] I suppose one of the challenges is the sadly needed "how to prevent crypto miners from abusing ci" mess [17:08:37] heh, that is exactly the current mess [17:08:57] jel.to has been doing some awesome work on that [17:09:35] https://phabricator.wikimedia.org/T297426 is the place to follow along [17:11:35] bd808: just keep tanking the price of crypto? :) [17:12:37] "destroy cryptocurrency" is an option i'm 100% ok with. [17:13:09] it needs to go down a lot more to make it unattractive for run script for free, get pennies in return to stop. [17:13:58] it's not "worth it" if you are comparing to other US grifts, but there are a lot of places where pennies could keep you alive [17:15:48] good to have that plan in our pocket, though, as a backup. [17:37:23] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate Kaniko as an option to build CI images - https://phabricator.wikimedia.org/T308213 (10jnuche) 05Open→03Resolved Even though RelEng already decided in T307810 on using BuildKit as the way to build images in GitLab , I'm w... [17:37:26] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10jnuche) [17:40:25] 10GitLab (CI & Job Runners), 10SRE-Access-Requests, 10User-brennen: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10brennen) [18:17:40] 10GitLab (CI & Job Runners), 10SRE-Access-Requests, 10serviceops, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10brennen) [18:23:01] 10GitLab (CI & Job Runners), 10SRE-Access-Requests, 10serviceops, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10thcipriani) [18:24:24] 10GitLab (CI & Job Runners), 10SRE-Access-Requests, 10serviceops, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10thcipriani) Sounds good from from my side: seems... [18:28:45] 10GitLab (CI & Job Runners), 10SRE-Access-Requests, 10serviceops, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10RLazarus) We're past the European work day, so I... [18:28:58] 10GitLab (CI & Job Runners), 10SRE-Access-Requests, 10serviceops, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10RLazarus) p:05Triage→03Medium [18:36:09] 10GitLab (CI & Job Runners), 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, and 3 others: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10RLazarus) Hmm, also: As a group access change, this should be reviewed and a... [18:39:34] dancy: o [18:39:55] whoops - was just gonna say i'm gonna grab some lunch, then happy to pair on that for a bit if extra eyeballs are useful [18:40:10] ok [18:49:45] Taking a food break myself. [19:13:27] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) DNS change deployed. ` host gitlab-replica-new.wikimedia.org gitlab-replica-new.wikimedia.org has address 208.80.154.15 gitlab-replica-new.wi... [19:26:50] re [19:28:52] dancy: have returned from burger truck [19:31:54] I'm pondering ways to inject the registry credentials into the container safely [19:33:20] 10GitLab (Project Migration), 10Release-Engineering-Team (Doing), 10User-brennen, 10Voice & Tone: Rename mainline Git branch from "master" to "main" on all WMF-hosted repositories during GitLab migration - https://phabricator.wikimedia.org/T281593 (10Aklapper) [19:34:20] dancy: if we make the config owned by buildkitd and ensure the buildkitd user on the host and buildkit user in the container have the same UID, would that work? [19:34:50] I believe so. [19:35:56] So we'll have to reserve an ID for buildkitd. That was recently done for the new `scap` user [19:36:38] https://wikitech.wikimedia.org/wiki/UID [19:36:53] ah, ok [19:37:02] not the most terrible thing [19:37:25] did you figure out a good way to abstract the "trusted" runner state in puppet [19:37:26] ? [19:37:34] Not yet. [19:37:40] k [19:37:59] i think what you mentioned about starting with the host hiera makes sense [19:38:41] sure would be nice if puppet had inheritance [19:39:02] `class profile:::gitlab::trusted_runner extends profile::gitlab::runner` ... [19:39:30] hmm.. looks like s/extends/inherits/ [19:39:43] oh [19:39:49] :) [19:40:09] great! [19:40:57] That said, I don't see any uses of it in operations/puppet [19:41:26] and there's still the question of where to put the extra hieradata for trusted_runner [19:41:44] so maybe just figuring that part out is the key [19:49:25] or we just use `inherits` and see what happens in review [20:03:26] Update: I misunderstood how buildctl/buildkitd use the registry credentials. It's buildctl that uses them, not buildkitd. [20:05:34] i think it's both? [20:05:56] At the very least, it's buildctl that reads config.json. I haven't figured out if buildctl passes the info onto buildkitd or if it pushes by itself. [20:06:06] otherwise this gets a little more complicated re: credential management [20:06:22] buildctl will pass credentials [20:06:37] but i thought buildkitd would also look for credentials in its own `.docker/config.json` [20:06:46] i have never verified eitheer [20:07:02] Nothing I've tried has been able to get buildkitd to read any config.json (According to strace) [20:08:39] oh bother [20:09:57] so are we back to "how do we extend the registry auth to support project specific credentials"? [20:11:48] * dancy thinks. [20:12:45] maybe what i had seen where .docker/config.json was used were the daemonless invocations [20:12:48] e.g. https://github.com/moby/buildkit/blob/master/examples/kubernetes/job.rootless.yaml [20:13:05] so that's also just `buildctl` reading it and passing to buildkitd [20:14:03] seems a little like re-working the registry auth setup is the elephant in the room [20:14:18] yeah. :-/ [20:14:25] or go back to gitlab registry + mirroring [20:14:39] or that [20:14:46] but that's also a stop gap imo [20:15:04] since it relies on something (jenkins) that has access to the shared cred [20:15:07] True [20:15:12] the shared cred is the problem [20:15:32] how tangled is it to address that at the level of the existing registry? [20:15:44] my assumption is: probably pretty tangled [20:15:45] it's "just" puppet [20:15:47] :) [20:16:17] i don't think it would be that hard to do really, but it would require more collaboration and fussing over how to manage the credentials [20:16:49] i.e. whether to automate credential generation, semi-automate, or manage them all manually [20:18:53] it'd be nice if users didn't really have to think about that management. [20:19:14] i.e. integrate the wmf registry with gitlab auth [20:19:20] probably never going to happen [20:19:33] it seems like the kind of thing people would balk at pretty hard. [20:20:03] but what if we had credentials per gitlab group. they could be manually added to private ops/puppet but it would have to happen a lot less frequently [20:20:50] that sounds less painful than per-project, at least [20:22:16] something like that would be a lot more likely to get through review [20:23:48] they wouldn't really have to be 1:1 to groups either, they're just service accounts but the main thing is that the registry config would support n of them and separate authn from authz [20:25:15] tangent: this all makes me wonder where operations/puppet will end up living in the end. if it's in gitlab, this is all a bit absurd [20:25:50] although i guess the private repo is always somewhere secure, so maybe not [20:26:05] nod [20:26:57] anyway, i think i have to call it a week (for real this time) to try to peal my kiddo from the tv and get some air. happy hacking everyone and cheers to wrapping up a great sprint! [20:27:15] Good luck Dan and have a good weekend! [20:27:16] *peel [20:27:23] thanks! you too [20:29:26] see ya dan! [23:02:20] 10GitLab (CI & Job Runners), 10Security-Team, 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10brennen) See also: [[https://gitlab.wikimedia.org/repos/releng/gi... [23:04:40] i'm not sure if i understand more or less about how this works: https://gerrit.wikimedia.org/r/c/operations/puppet/+/724472/18//COMMIT_MSG#17 [23:05:11] anyway, realized we'll probably want that regardless of whether the wmcs tier of runners goes away [23:06:22] any rate, i'm off for the week - thanks everybody for all the help on gitlab stuff these last few days. [23:36:30] Have a good weekend.