[02:04:51] 10GitLab (Project Migration), 10Striker, 10Tools, 10Developer-Advocacy (Jul-Sep 2022), and 3 others: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 (10bd808) [02:04:57] 10GitLab (Project Migration), 10Phabricator, 10Epic, 10Release-Engineering-Team (Bonus Level 🕹ī¸), and 2 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10bd808) [02:16:36] 347 Diffusion repos ended up migrating to GitLab. T191182 is no longer stalled on Striker! [02:16:36] T191182: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 [02:16:48] 10GitLab (Project Migration), 10Phabricator, 10Epic, 10Release-Engineering-Team (Bonus Level 🕹ī¸), and 2 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10bd808) 05Stalled→03Open Removing the stalled status. #striker no longer allows... [02:22:54] 10GitLab, 10Striker, 10User-bd808: Inactive Diffusion repos for missingpages and wiki2email were migrated to GitLab - https://phabricator.wikimedia.org/T317260 (10bd808) Diffusion repos that were mirrored: * https://phabricator.wikimedia.org/source/tool-wiki2email/ * https://phabricator.wikimedia.org/source/... [02:23:39] 10GitLab, 10Striker, 10User-bd808: Inactive Diffusion repos for missingpages and wiki2email were migrated to GitLab - https://phabricator.wikimedia.org/T317260 (10bd808) [05:14:56] looks like the research/ team has repos with issues active: https://gitlab.wikimedia.org/groups/repos/-/issues [05:15:09] bd808: yay! [07:05:38] bd808: congratulations! [14:54:32] I have a strong feeling that if issues are not disabled at a deployment level folks are going to use them if for not other reason than they are low friction/self-service vs creating Phabricator projects. [14:56:53] bd808: isssues are disabled by default for all new projects. But it is possible for project maintainers to enable them. [14:57:25] I share some concerns for running GitLab issues and phab in parallel is a good idea [15:10:12] the research thing is apparently deliberate, but has no feedback yet: T304614 [15:10:12] T304614: GitLab Issues experiment with Research - https://phabricator.wikimedia.org/T304614 [15:34:29] 10GitLab (Project Migration), 10Quarry: Move quarry to gitlab or github - https://phabricator.wikimedia.org/T308978 (10rook) As mentioned above github appears to serve the project requirements where gitlab is not quite ready. I don't believe I mentioned it here but one of the main problems is the lack of being... [15:39:50] 10GitLab, 10Striker, 10User-bd808: Inactive Diffusion repos for missingpages and wiki2email were migrated to GitLab - https://phabricator.wikimedia.org/T317260 (10bd808) 05Open→03Resolved [15:56:03] brennen, thcipriani: I am now wondering if Striker should setup a Diffusion mirror for each gitlab repo it creates. This happened as a side effect for the repos I just migrated so there will be a feature gap growing for each new repo going forward. Github gives a much better (IMO) code browser than gitiles, but It can be really nice to be able to link to a git hash in Phabricator. Thoughts? [15:56:36] Deciding on this would change my work for T317272 [15:56:36] T317272: Remove legacy Diffusion related code from Striker - https://phabricator.wikimedia.org/T317272 [16:14:54] this was always mukundas argument for why differential should mirror everything. It is a really nice feature. I think having a mirror would make sense. [16:26:01] 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab, 10Security: Findings in Security Readiness Reviews of Trusted GitLab Runners - https://phabricator.wikimedia.org/T317341 (10Jelto) [16:29:17] 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab, 10Security: Findings in Security Readiness Reviews of Trusted GitLab Runners - https://phabricator.wikimedia.org/T317341 (10Jelto) [16:29:47] thcipriani, bd808: concur with tyler. [16:31:22] cool. that gives me a direction at least. It also makes me feel that I may have purged some data prematurely... but I can recreate it I think. [16:38:14] 10GitLab (Project Migration), 10Striker, 10Tools, 10Developer-Advocacy (Jul-Sep 2022), and 3 others: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 (10bd808) [18:39:34] 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab, 10Security: Findings in Security Readiness Reviews of Trusted GitLab Runners - https://phabricator.wikimedia.org/T317341 (10Mstyles) [19:26:01] bd808: thanks a log [19:26:04] bd808: thanks a lot [19:26:20] hashar: I saw the shout-out in the meeting when I watched the video, thanks for that too [20:04:51] mutante: well deserved given all the work you did to migrate those legacy applications [20:05:07] and I totally missing thanking you directly for doc.wikimedia.org or for gerrit while I was on vacations :D [20:05:35] without you I would probably still run on Precise *grin* [20:05:50] <3 [20:15:28] :)) [21:20:11] TIL the "your projects" view in gitlab becomes useless when you are the owner of a group with a lot of projects in it (toolforge-repos in my case) [21:21:27] yeah, there are some general problems with getting an overview of repos. [21:21:58] i can't say it's _worse_ than having to find-in-page on gitiles every time i forget where something lives on gerrit, but it's not as much better as i'd like it to be. [21:24:58] I guess I will try using stars as a way to find things for now [21:25:36] I accepted my first MR in gitlab today :) [21:30:19] heh, this made me realize I have 120 projects because of /repos/ [21:30:53] but there is a big different between "All" and "Personal" [21:31:22] "personal" repos I have only one [21:31:42] they are defined as all repos under the user namespace [21:32:46] bd808: adding ?personal=true to the URL does not help for your case, does it? [21:32:58] that would be repos under /bd808/ I suppose [21:33:25] yeah, I guess if I forked all of the /toolforge-repos/ ones to my personal space that would work [21:33:49] I did not mean to say it's a good idea to move everything under personal namespaces :p [21:33:51] which is how things are for me on gihub too [21:34:00] actually I recommended the opposite before, heh [21:34:41] stars / starred seems better [21:34:54] personal forks is the easiest way to follow a striker merge request workflow, but for most toolforge things I don't care about that much [21:35:10] *strict merge request workflow [21:47:17] anyone have a clue about this error? https://gitlab.wikimedia.org/repos/releng/gitlab-runner-test/-/jobs/24114#L41 [21:48:01] my take is that a DNS request is being attempted over ipv6 during the buildkit LLB solve [21:48:06] why that would be, i have no idea [21:50:34] https://www.irccloud.com/pastebin/1aWgsNyW/ [21:50:45] no ipv6 nameservers listed there... [21:56:39] dduvall: on production host we also don't have IPv6 nameservers in /etc/resolv.conf but that does not keep us from looking up docker-registry.wikimedia.org and getting both IPv4 and IPv6 records back [21:57:50] 'connect: cannot assign requested address' seems more like it can't use an IPv6 adress itself [22:00:33] as in "IPv6 binding error: Cannot assign requested address" to make [22:00:43] ah ok [22:00:44] some outgoing connection that comes after the DNS lookup [22:01:58] no sure how to debug further, but i'm thinking i could: 1) write a `FROM scratch` Dockerfile and attempt to build via buildctl, then see what the resolv.conf is for that image [22:02:17] where does it get the name "docker-registry.wikimedia.org" from [22:02:24] can that place take an IP for debugging? [22:02:49] i believe it's during the LLB graph solve in the blubber-buildkit container [22:03:00] can you edit the resolv.conf in the running container [22:03:30] and hardcode just 208.80.154.224 as the IP for docker-registry [22:03:41] so, i.e. it's reading in the `.pipeline/blubber.yaml` in that project and sending it to the container spawned for the buildkitd frontend using this image https://gitlab.wikimedia.org/repos/releng/gitlab-runner-test/-/jobs/24114#L27 [22:03:49] hmm, i could try that [22:04:17] maybe i can try spawning that frontend container manually and seeing what its resolv.conf looks like [22:04:48] so many layers here... [22:05:39] one other thing maybe, to compare the output of iptables -L with ip6tables -L [22:05:50] indeed [22:07:11] or docker does not use any IPv6 by default https://github.com/MatthewVance/unbound-docker-rpi/issues/12 [22:11:49] well, it's trickier than that i believe. buildkitd is running inside of a docker container. however, it spawns its own containers via OCI [22:12:14] and grepping the buildkit source, it looks like this may be a result of the default resolv.conf it uses [22:13:36] ACK, so "the inner docker" or something in the sentence above [22:13:49] er, well more exactly, it doesn't seem to pass on its own resolver config to the underlying OCI container used during the solve [22:14:01] yeah, basically [22:14:21] https://github.com/moby/buildkit/issues/1004 has clues [22:14:32] aha [22:14:33] and https://github.com/moby/buildkit/blob/master/executor/oci/resolvconf_test.go [22:15:44] and https://github.com/moby/buildkit/blob/master/vendor/github.com/docker/docker/libnetwork/resolvconf/resolvconf.go#L75 [22:15:54] we don't get "bad address" though or NXDOMAIN like a failed DNS lookup [22:16:47] nevertheless this is very suspicous indeed "Buildkit IMO definitely has a bug where it handles resolv.conf different than legacy Docker (with the latter working fine). [22:18:02] yeah, i don't know why it would be trying to use ipv6 to perform the lookup but then fail at the basic ipv6 udp "socket" [22:18:46] looking into this change to see if we can configure it somehow https://github.com/moby/moby/pull/39295 [22:19:01] the resolv.conf within the oci buildkit executor that is [22:57:20] a little more info from the buildkitd logs: [22:57:29] https://www.irccloud.com/pastebin/tpSuF7Io/ [22:58:47] it seems that because the `/etc/resolv.conf` in the buildkitd container only contains local nameserver entries (it's `nameserver 127.0.0.11` because of the custom Docker network) buildkit (via libnetwork) is setting up a default `/etc/resolv.conf` with those entries [23:01:00] i really hate docker [23:01:23] * dduvall sighs and carries on