[16:00:54] skipping the gitlab ic sync meeting since it looks like most folks are doing the wmf tech dept one instead. [17:04:33] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Automate GitLab version upgrade process - https://phabricator.wikimedia.org/T323569 (10Jelto) [17:04:47] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Automate GitLab version upgrade process - https://phabricator.wikimedia.org/T323569 (10Jelto) The `sre.gitlab.upgrade` cookbook was merged and is available on cumin hosts now. Some minor errors were fixed with laser fast reviews from Riccard... [17:47:53] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Holiday Leftovers 🥡), 10User-brennen: Build a widget to display GitLab changes on related Phabricator tasks - https://phabricator.wikimedia.org/T324149 (10brennen) [17:48:42] 10GitLab (Project Migration), 10MediaWiki Train Development Environment, 10Patch-For-Review, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Migrate train-dev to GitLab - https://phabricator.wikimedia.org/T323225 (10thcipriani) 05Open→03Resolved [17:49:06] 10GitLab (Project Migration), 10Release-Engineering-Team (Holiday Leftovers 🥡), 10User-brennen: Migrate mediawiki/tools/release/ to GitLab - https://phabricator.wikimedia.org/T290260 (10thcipriani) [17:50:26] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10serviceops-radar, and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10thcipriani) [17:50:33] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Holiday Leftovers 🥡), 10User-brennen: Comment on Phabricator tasks for new, merged, and abandoned changes on GitLab - https://phabricator.wikimedia.org/T324150 (10thcipriani) [17:51:16] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try Reggie for buildkit caching - https://phabricator.wikimedia.org/T323149 (10thcipriani) 05Open→03Resolved a:03dduvall [17:53:36] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Holiday Leftovers 🥡), 10User-brennen: Add DigitalOcean resource monitoring for cloud runner nodes - https://phabricator.wikimedia.org/T308615 (10thcipriani) [17:53:49] 10GitLab (CI & Job Runners), 10serviceops-collab, 10Release-Engineering-Team (Holiday Leftovers 🥡), 10User-brennen: Provision untrusted instance-wide GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10thcipriani) [23:21:46] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) I noticed we were nowadays only at "9/10" instances quota-wise.. so I jumped on that and created gitlab-prod-1002 just now. ..which filled up the quota but we got away... [23:29:25] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) instance created, puppetmaster changed via Hiera: ` -server = puppetmaster.cloudinfra.wmflabs.org -ca_server = puppetmaster.cloudinfra.wmflabs.org +server = puppetmas... [23:30:57] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) and FAIL.. ` Warning: Unable to fetch my node definition, but the agent run will continue: Warning: SSL_connect returned=1 errno=0 state=error: certificate verify fai... [23:33:43] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) this is fixed by `rm -rf /var/lib/puppet/ssl` and running puppet again to create new requests. keeps coming up every time we switch masters, more or less, since years:... [23:55:58] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) next issue: ` Setting up gitlab-ce (15.5.7-ce.0) ... It looks like there was a problem with public attributes; run gitlab-ctl reconfigure manually to fix. dpkg: error... [23:57:00] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) running `gitlab-ctl reconfigure` manually results in: ` Running handlers: [2023-01-11T23:56:28+00:00] ERROR: Running exception handlers There was an error running git... [23:58:15] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) This does seem like a general issue with Letsencrypt / acme_chief in cloud as in the past.