[01:09:09] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): Assess GitLab Container Registry as a default for container build processes - https://phabricator.wikimedia.org/T307537 (10brennen) [[https://gitlab.wikimedia.org/rep... [08:27:20] * jnuche finishes getting up to speed with yesterday's chat [08:27:40] this is where not being in the same timezone gets a bit painful [08:27:56] looks like we're leaning heavily towards buildkitd [08:33:44] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate Kaniko as an option to build CI images - https://phabricator.wikimedia.org/T308213 (10jnuche) [09:05:53] 10GitLab: gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10Jelto) 05Resolved→03Open Reopening, puma still fails to stop: ` May 12 01:33:19 gitlab2001 gitlab-restore.sh[26219]: timeout: run: puma: (pid 6268) 44600s, want down ` I'll try to find out why thi... [14:11:33] jnuche: yeah, i think so. [14:21:47] concur re: timezones. [14:25:35] 10GitLab (Infrastructure): gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10brennen) [14:25:49] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10brennen) [14:26:20] 10GitLab (CI & Job Runners), 10serviceops: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10brennen) [14:27:24] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate buildkitd instances as image builders for GitLab - https://phabricator.wikimedia.org/T307810 (10brennen) [14:27:55] 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Replicate select published images from GitLab container registry to WMF prod registry - https://phabricator.wikimedia.org/T308080 (10brennen) [14:56:44] jelto: any chance you could take a look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/790778/ ? [14:58:37] i think that's sort of our main blocker at the moment. [14:59:34] I'm getting "No route to host" when trying to access gitlab.devtools.wmcloud.org [14:59:55] yeah, same. [14:59:56] * brennen looks [15:00:55] no problem sshing in... [15:01:37] nginx is down [15:03:03] That'll do it. [15:03:39] bind() to 172.16.7.146:80 failed (99: Cannot assign requested address) [15:08:37] Which nginx is that? [15:08:44] (what host does it run on?) [15:09:07] gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud runs a production-like test instance in WMCS/VPS serving https://gitlab.devtools.wmcloud.org/ [15:09:12] ^ https://wikitech.wikimedia.org/wiki/GitLab [15:09:49] at a guess, i suspect i broke something here messing around with the standalone puppetmaster yesterday [15:09:52] Looks like I don't have permission to ssh there [15:10:05] one sec [15:10:30] eth0 is missing the second ip address "up ip addr add 172.16.7.146/32 dev eth0" [15:11:04] dancy: you should be good now [15:11:18] thx. I'm in now. [15:16:51] brennen: are you looking into the test instance? I can fix the instance and check why the address is not applied automatically [15:17:36] jelto: if you don't mind. fwiw i think i likely broke it by following instructions on https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster and deleting certs. [15:17:50] (which didn't need to be done; i didn't realize that it was already using the standalone puppetmaster.) [15:58:37] gitlab.devtools.wmcloud.org should work again. I fixed the configuration for the second ip address which was clashing with DHCP config [15:59:51] thx! [16:03:16] thanks! [16:28:25] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10hashar) [17:16:41] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate buildkitd instances as image builders for GitLab - https://phabricator.wikimedia.org/T307810 (10dduvall) 05In progress→03Resolved [17:16:45] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10dduvall) [17:17:45] 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Replicate select published images from GitLab container registry to WMF prod registry - https://phabricator.wikimedia.org/T308080 (10dancy) 05Open→03Declined Not needed if buildkitd's run on t... [17:20:31] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10dduvall) 05In progress→03Resolved The consensus is that buildkitd will provide us more features and c... [17:20:35] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10dduvall) [17:31:25] brennen: I am guessing updating the migration runbook is a bit too early of a task given all the unknown isn't it [17:31:54] hashar: yeah, it might be. [17:31:59] it felt like a low hanging fruit :( [17:32:17] looks like each step described raises a whole lot more questions :-\ [17:32:25] could write some of the parts about archiving old projects [17:32:56] k9s! k9s! k9s! [17:33:35] don't mind me. just chanting [17:33:43] it does seem pretty great. [17:34:16] it's great for watching gitlab runners kick off [17:34:48] filed https://phabricator.wikimedia.org/T308271 (!) [17:35:53] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10brennen) [17:36:06] 👍🏾 [17:36:57] brennen: yeah archiving sounds like a good first step [17:37:36] there's sort of an outline here: https://www.mediawiki.org/wiki/GitLab/Migrating_a_Project [17:38:49] brennen: yeah excellent, I will expand that after dinner [17:39:01] thx! [17:57:41] i'm writing up some notes summarizing today's discussion here: https://etherpad.wikimedia.org/p/gitlab-sync#L7 [18:11:06] ^ i think i got the highlights. [18:17:48] brennen: looks good. i made one small change from my recollection [18:18:51] yeah, that's more accurate [18:37:14] off for a bit to walk in the sun and acquire a sandwich. [19:48:29] i have to help with my sick kiddo for a bit but i've pushed a change for review for buildkitd https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/791427 [19:48:52] this is for packaging the image. we'll still need puppet config [19:49:12] Cool! I'll check it out after watercooler. [19:49:24] thanks! [20:27:28] 10GitLab (Infrastructure), 10Patch-For-Review: gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10Dzahn) after deploying the change above, now this is what happens when manually starting the unit: ` May 12 19:55:23 gitlab2001 gitlab-restore.sh[17479]: timeout... [20:28:58] 10GitLab (Infrastructure), 10Patch-For-Review: gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10Dzahn) 05Open→03In progress [22:54:04] dancy: hmm, i can try using rootlesskit from the binary releases [23:26:44] negative. tried with a newer rootlesskit and a newer newuidmap/newgidmap and same thing