[07:16:21] first successful experiment pushing an image using buildkit to a registry that's authenticating using the gitlab JWT https://gitlab.wikimedia.org/repos/releng/blubber/-/jobs/18503 \o/ [07:16:30] now time to sleep [07:18:56] nginx config is here https://gitlab.wikimedia.org/dduvall/gitlab-buildkitd-eval/-/blob/main/registry.yaml#L80 [07:19:27] jwt-authorizer service is here https://gitlab.wikimedia.org/dduvall/gitlab-buildkitd-eval/-/blob/main/cmd/jwt-authorizer/main.go [13:01:04] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Jelto) I solved the installation/puppet issues with `gitlab1003`. The `gitlab-ce` package was installed and login using CAS/IDP worked. Synced backups... [14:40:24] dduvall: Burnin the midnight oil I see. [15:03:31] Yeah it was a mistake. I already need a nap :) [16:04:07] The code looks good. [16:04:29] The only bit that needs adjustment is the hard-coded "foo" at https://gitlab.wikimedia.org/dduvall/gitlab-buildkitd-eval/-/blob/main/cmd/jwt-authorizer/main.go#L70 [16:09:24] and we'll need a mapping of the gitlab project names to the corresponding registry image names. [16:13:57] Nice work! [16:14:29] I spent most of the evening wondering if using the job token is safe enough. [16:14:56] It would be worth running by security team i think [16:15:23] Nod. I think since this approach doesn't reveal any secrets, it has parity with the Jenkins setup. [16:15:27] Funny enough the www-authentication challenge doesn't really come into play here [16:15:39] Nod. [16:16:03] hmm. [16:16:27] It's just to satisfy the parser on the client end. The change i made to buildctl has it short circuit before making the follow up token request [16:17:13] If it has the token already that is, in the `registrytoken` auth config field [16:17:20] nod. [16:23:53] do you want to talk it over at all? i hacked it up in a stupor right before bed so i'm sure there are other problems :) [16:25:58] Sure I can do that. You have it all nicely packaged up. [16:26:13] s/I/we/ [16:26:38] It'll have to be later though. I have an appointment to get to. [16:26:42] cool cool [16:26:45] standing meeting? [16:27:31] yeah, we can use that. I'll hit you up when I'm availab.e [16:29:27] sounds good [17:13:06] 10GitLab (CI & Job Runners), 10Airflow, 10Data-Engineering-Kanban: Allow a shared, protected runner for the data-engineering group in GitLab - https://phabricator.wikimedia.org/T295045 (10BTullis) 05Open→03Declined Declining this task as we have no time to work on it at the moment. [17:15:29] 10GitLab (CI & Job Runners), 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, and 3 others: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10thcipriani) 05Open→03Resolved a:03jbond Confirmed working: ` thcipri... [17:36:20] brennen: can gitlab have any number of nested groups or is there a limit? [17:37:22] oh. https://docs.gitlab.com/ee/user/group/subgroups/ says 20!? [17:37:25] i'm not _aware_ of a limit. i intuitively suspect that there might be one but that... [17:37:29] yeah, so you could go pretty deep i guess. [17:37:34] that is a high limit [17:37:49] i tend to feel that if you hit that limit, you are probably doing something quite wrong. [17:37:58] haha yeah sounds right [17:38:14] now i have to ask what the root of this question is. :) [17:38:17] and i'm guessing the registry mirrors that structure? [17:38:28] to the best of my understanding. [17:39:04] fair enough. :) i'm wondering how to properly compare registry request paths to project paths [17:39:28] i hacked on a little JWT authorizer last night but the nginx config is pretty terrible as is [17:40:09] "nginx config" is a string that brings on some unpleasant flashbacks for me, so i can believe it [17:40:10] i want it to pass only the part of the request path that pertains to the repo/image ref to the backend authorizer [17:40:53] but the tl;dr is that the jwt auth seems to work. we "just" need to poke at it and see if there are security issuers with the model [17:41:37] this seems like good news [17:41:46] /repos/notwmf/affiliates/orgs/europe/germany/wikimedia/wikidata/base/deploy but only half way there [17:41:56] hahah [17:42:02] oh my :) [17:42:41] repos/milky-way/sol-system/earth/primates/homo-sapiens/... [17:42:52] ;) hihi [17:43:11] and you started at milky way [17:43:35] maybe there should be an alpha-quadrant in there [17:43:43] :D [17:43:47] and specify a branch of the multiverse [17:44:04] and where is that multiverse hosted? [17:44:08] probably on github [17:44:11] ^ [17:44:18] lulz [17:44:54] it's corporations all the way down [18:10:59] We might need a label for which multiverse as well. It is currently unknown if the Marvel multiverse is the same as the Rick and Morty multiverse. Basically the open question is if there is a single multiverse containing infinite universes, or if there are infinite multiverses as well. [18:11:42] metamultiverse [18:12:11] dduvall: Cramming food in my face. I'll be ready soon [18:12:17] I'm sure https://en.wikipedia.org/wiki/Aleph_number is connect somehow too (cardinality of infinite sets) [18:13:37] https://en.wikipedia.org/wiki/Elias_Levy [18:15:22] dancy: no prob [18:16:25] dancy: that link reminds me that life would probably be different if i had understood any of what was in phrack in the 90s. [18:16:32] i went to mcdonald's this morning after dropping my kiddo off and ate a sausage mcmuffin on the beach. it was lovely, and the brick in my stomach should keep me for a bit [18:16:57] haha totally [18:17:13] speaking of which... Are you looking to buy a motorcyle? I have one for sell [18:17:15] *sale [18:18:06] me or brennen? [18:18:07] :) [18:18:21] i'm not looking, no [18:18:29] * dduvall wonders if he should be [18:18:40] i need a nicer bandsaw first [18:18:42] motorcycles are on my short but firmly established "don't, you'll die" list. [18:18:51] holding out for a laguna to pop up on craigslist [18:18:53] along with downhill snow sports and needle drugs. [18:19:42] also probably climbing that requires ropes. [18:20:49] i was in the middle of taking the MSF (Motorcycle Safety Foundation) course years ago when just about all of my friends had bad crashes, so it sort of stalled my ambitions to ride [18:21:23] There is soooome risk of getting killed. [18:21:53] (to be clear, i have no opinions on whether *other* people should do (most of) those things, i just know my own tendencies well enough to make a decision.) [18:24:16] it looks _really_ fun and that's what worries me the most [18:24:29] although i've gotten a lot tamer for taking risks in my old age [18:25:04] * dduvall looks over at workshop where there are so many tools that kill [18:25:19] I'm in https://meet.google.com/eve-esmx-wea now [18:30:39] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Jelto) [19:51:27] quality hacking session, even if i was mostly a spectator [21:10:17] Agreed. That was fun/informative. [21:19:05] dduvall: https://github.com/stakater/Reloader [21:19:21] `A Kubernetes controller to watch changes in ConfigMap and Secrets and do rolling upgrades on Pods with their associated Deployment, StatefulSet, DaemonSet and DeploymentConfig` [21:20:52] dancy: w00t [21:20:54] that's nice [21:21:16] i'm hacking on nginx again :D [21:21:31] just about to test another approach [21:21:50] haha.. ok good luck. I'm working on the python version of what you implemented [21:22:43] right on [21:24:03] geez, variables in nginx are a nightmare [21:25:00] some directives support them, others don't [21:32:13] i reiterate my dislike of nginx configuration. [21:33:02] i wrote up a long rant about this at one point when i actually had a lot of the model in my head. it looks like a language with a reasonable syntax, but in reality it's like 6 or 8 mutually contradictory state machines in a trenchcoat. [21:33:17] yeah it's totally deceptive! [21:33:40] people get tricked into thinking it's reasonable because you start out with a 20 line config and it looks totally sensible. [21:45:47] lol at "mutually contradictory state machines in a trenchcoat" [21:46:12] i'm imagining a noir scene now [21:47:49] so close but so far, now i've lost the request path in the request to jwt-authorizer [21:48:15] i've gotta run an errand pre rush-hour, but happy to stare at it when i'm back [21:48:55] basically `auth_request` doesn't support variables so i'm doing a `rewrite` within an `auth_request` `location = /auth { ... }`... [21:49:06] i don't know who i am anymore [22:33:45] 10GitLab (Initialization), 10Release-Engineering-Team (Doing), 10User-brennen: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10thcipriani) [22:33:48] 10GitLab (Auth & Access), 10SRE, 10Release-Engineering-Team (Doing), 10User-brennen: Define auth strategy for GitLab - https://phabricator.wikimedia.org/T274461 (10thcipriani) 05Open→03Resolved a:03brennen [22:48:14] how's it looking dduvall? [22:52:04] blerg. [22:54:44] heh [22:54:46] that good eh. [22:56:17] any non-trivial nginx config expands until it is rewritten in lua [22:57:07] i've never used the lua stuff. is it an end-run around the absurdities? [22:57:13] i'm trying to do an `auth_basic` within a location referenced by `auth_request` and it is acting strange [22:57:26] i got it to route to the right auth_request based on ip though [22:58:20] https://gitlab.wikimedia.org/dduvall/gitlab-buildkitd-eval/-/blob/main/registry.yaml#L89 [22:58:31] maybe bd808 has ideas :) [22:59:34] the jwt auth works [22:59:38] the basic auth is not working [22:59:58] any associated errors? [23:00:32] just that nginx tries to serve a file within the /auth/basic request and throws a 404 [23:00:46] which it then errors on saying an `auth_request` shouldn't give that [23:00:49] which makes sense [23:01:04] but if i'm not giving an `Authorization` why would it even get that far [23:01:54] so it seems to me like the `auth_basic` does nothing the way i have it [23:02:13] brennen: some of them, yes. Basically the pile of broken state machines have knobs exposed to lua and you can kind of use understandable logic to turn the knobs. [23:02:42] Yuvi wrote things with nginx+lua that have poked and sometimes think I understand like https://github.com/wikimedia/puppet/blob/production/modules/dynamicproxy/files/domainproxy.lua [23:03:40] * bd808 squints at dduvall's nginx config snippet [23:07:20] maybe the rewrite within the auth_request location is mucking everything up [23:07:41] can i just use an if please? [23:08:45] maybe, unless you can't. [23:09:39] you can wrap rewrites and returns in if blocks, but pretty much anything else will go boom [23:12:16] :( [23:15:22] dduvall: have you tried hoisting the things in the `location = /auth` block up to the `location ~ ^/v2/(.*)$` block directly? Or is that working fine and it's just that things don't do what you want when hitting the `location = /auth/basic` block? [23:16:00] the latter i think [23:23:12] "maybe the rewrite within the auth_request location is mucking everything up" is what made me wonder about hoisting the `/auth/$auth_type` logic up. But now I see "basically `auth_request` doesn't support variables" which makes my idea invalid :( [23:23:45] yeah, that's how i got to the weird /auth location routing [23:24:46] so here's what _does_ work: if i make any request from the host specified in the `geo` section, i am challenged with `WWW-Authentication: Bearer ...` [23:25:16] if i make a `GET` request from another host i get a 200 from the backend registry [23:25:39] what doesn't work: if i make a `POST` request from another host i get a 500 [23:26:37] https://www.irccloud.com/pastebin/jYFCMorV/ [23:27:01] a `POST` request with no credentials [23:28:17] well time to pickup my kiddo so i'll give this a rest [23:28:30] have a good one [23:28:37] that open() fail is confusing... [23:29:24] it's just falling back to looking for a file there, right? [23:29:48] yeah, within the auth_request for some reason [23:30:17] ok, going for real [23:30:29] Have a good evening [23:30:46] have a good one too! good hacking and incremental progress today [23:34:43] the lua hack for this would be using https://github.com/openresty/lua-nginx-module#access_by_lua_block or https://github.com/openresty/lua-nginx-module#access_by_lua_file to to replace `auth_request`