[08:37:53] 10GitLab: Set new owner in wmit-wikimedia GitLab group - https://phabricator.wikimedia.org/T323196 (10valerio.bozzolan) [09:18:04] 10GitLab: Set new owner in wmit-wikimedia GitLab group - https://phabricator.wikimedia.org/T323196 (10valerio.bozzolan) [09:50:17] 10GitLab (Infrastructure), 10serviceops-collab: Evaluate a high available GitLab architecture - https://phabricator.wikimedia.org/T323201 (10Jelto) [09:52:32] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Jelto) 05In progress→03Resolved All GitLab machines in codfw and eqiad are migrated to physical hosts. I created a follow-up task T323201 to... [09:53:41] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Jelto) [16:28:36] mutante: i forgot to say thanks for unblocking us on the terraform package. thank you! [16:29:26] dduvall: :) you're welcome [16:36:25] I'm going to pause runner-1030 for some tests :) [16:55:53] ack. ty :) [16:56:59] jelto: oh, i was thinking about the role/profile issue. can you just modify the instance puppet config and swap out `profile::gitlab::runner` in the puppet classes for the role class? or is there something more tricky about it? [17:04:23] dduvall: We are testing exactly that :) thanks [17:41:01] 10GitLab, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Provision Horizontal Pod Autoscaler (HPA) for GitLab cloud runners - https://phabricator.wikimedia.org/T323164 (10thcipriani) [17:42:32] 10GitLab, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try local directory export + GitLab cache for buildkit caching - https://phabricator.wikimedia.org/T323150 (10thcipriani) [17:42:49] 10GitLab, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try DigitalOcean object storage for buildkit caching - https://phabricator.wikimedia.org/T323147 (10thcipriani) [17:43:25] 10GitLab, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try Reggie for buildkit caching - https://phabricator.wikimedia.org/T323149 (10thcipriani) [17:43:30] 10GitLab, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try DigitalOcean registry for buildkit caching - https://phabricator.wikimedia.org/T323148 (10thcipriani) [17:48:15] 10GitLab (Infrastructure), 10serviceops-collab: ensure Gitlab logs end up in logstash - https://phabricator.wikimedia.org/T322261 (10brennen) [17:48:19] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try DigitalOcean registry for buildkit caching - https://phabricator.wikimedia.org/T323148 (10brennen) [17:48:22] 10GitLab (Auth & Access): Set new owner in wmit-wikimedia GitLab group - https://phabricator.wikimedia.org/T323196 (10brennen) [17:48:28] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Provision Horizontal Pod Autoscaler (HPA) for GitLab cloud runners - https://phabricator.wikimedia.org/T323164 (10brennen) [17:48:45] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try DigitalOcean object storage for buildkit caching - https://phabricator.wikimedia.org/T323147 (10brennen) [17:48:50] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try Reggie for buildkit caching - https://phabricator.wikimedia.org/T323149 (10brennen) [17:49:17] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try local directory export + GitLab cache for buildkit caching - https://phabricator.wikimedia.org/T323150 (10brennen) [17:49:30] 10GitLab (Project Migration), 10MediaWiki Train Development Environment, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Migrate train-dev to GitLab - https://phabricator.wikimedia.org/T323225 (10thcipriani) [17:49:45] 10GitLab (Project Migration), 10MediaWiki Train Development Environment, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Migrate train-dev to GitLab - https://phabricator.wikimedia.org/T323225 (10brennen) [17:49:51] 10GitLab (CI & Job Runners), 10serviceops-collab, 10Release-Engineering-Team (Priority Backlog 📥): Build and import new release of jwt-authorizer (1.1.0) - https://phabricator.wikimedia.org/T322691 (10brennen) [17:50:05] 10GitLab (CI & Job Runners), 10Data Pipelines, 10Data-Engineering-Planning, 10Release-Engineering-Team, 10serviceops-collab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10brennen) [17:50:42] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10serviceops-collab: Migrate GitLab Shared Runners from profile::gitlab::runner to role::gitlab_runner - https://phabricator.wikimedia.org/T322409 (10Jelto) p:05Triage→03Medium a:03Jelto I converted one of the Shared Runners `runner-1030.gitlab-r... [17:52:37] runner-1030 was converted from profile to role successfully (some test jobs were successful). I'll leave it running as a "canary". I'll convert all other Shared Runners in a few days if nothing comes up. Feel free to pause the runner-1030 if anything happens :) [17:53:02] the whole process to change the role/profile takes around 5 minutes, so quite fast [17:57:36] Nice work! [18:01:46] \o/ [19:25:07] 10GitLab (CI & Job Runners), 10serviceops-collab, 10Release-Engineering-Team (Priority Backlog 📥): Build and import new release of jwt-authorizer (1.1.0) - https://phabricator.wikimedia.org/T322691 (10Jelto) 05Open→03Resolved a:03Jelto I built and published the new version `jwt-authorizer | 1.1.0-1` fo... [21:00:22] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try local directory export + GitLab cache for buildkit caching - https://phabricator.wikimedia.org/T323150 (10jeena) This was done, but disabled due to poor performance. A summary of results is in this commit message: https:... [21:03:34] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Provision Horizontal Pod Autoscaler (HPA) for GitLab cloud runners - https://phabricator.wikimedia.org/T323164 (10jeena) a:03jeena