[12:14:41] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Jelto) Checklist for todays gitlab-replica migation from `gitlab2001` to `gitlab1003`:  **Preparations before downtime:**  [x] register second service...
[16:32:28] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Jelto) Checklist for gitlab migation from `gitlab1001` to `gitlab1004`:  **Preparations before downtime:**  [x] register second service IPs for `gitlab...
[17:23:58] <wikibugs>	 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token - https://phabricator.wikimedia.org/T308501 (10dduvall) 05In progress→03Stalled Blocked on review...
[17:24:02] <wikibugs>	 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dduvall)
[17:52:30] <wikibugs>	 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token - https://phabricator.wikimedia.org/T308501 (10dduvall) @akosiaris and/or @Jelto do you have time to...
[17:53:49] <wikibugs>	 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token - https://phabricator.wikimedia.org/T308501 (10dduvall)
[23:32:12] <wikibugs>	 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) Also, after manually starting the "backup-restore" service on gitlab2001, which was still alerting in Icinga, we now have:  23:2...
[23:36:57] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10Dzahn) After T274463#7966543 ff we now have a working backup-restore service again:  <+icinga-wm> RECOVERY - Check systemd state on gitlab2001 is O...
[23:37:20] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10Dzahn) 05In progress→03Resolved