[08:05:36] GitLab will be down at 10:00 UTC for around 2 hours. The active host will be switched to the other datacenter. See https://phabricator.wikimedia.org/T329931 for more information. [08:24:11] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [09:35:40] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [09:59:50] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) a:03eoghan [10:06:31] GitLab maintenance starting now [10:07:30] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [10:18:17] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [10:19:53] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [10:28:59] 10GitLab, 10serviceops-collab: Clarify the process of logging into gitlab-replica using 2FA - https://phabricator.wikimedia.org/T330639 (10LSobanski) [10:30:40] 10GitLab, 10serviceops-collab: Clarify the process of logging into gitlab-replica using 2FA - https://phabricator.wikimedia.org/T330639 (10LSobanski) p:05Triage→03Low [11:03:23] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [11:25:39] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [11:36:55] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [11:51:23] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [11:58:10] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [12:00:37] Maintenance finished, GitLab is back again. If you face any issues feel free to comment in T329931. [12:00:37] T329931: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 [12:02:26] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [12:03:46] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) Maintenance finished, GitLab is back again. If you face any issues feel free to post it here. I'm leaving the task... [12:59:15] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) A "Backup freshness" alert with summary:"No backups: 1 (gitlab2002), Fresh: 116 jobs"" triggered. This should resol... [15:59:32] 10GitLab (Infrastructure), 10serviceops-collab: Investigate gitlab maintenance/read-only mode - https://phabricator.wikimedia.org/T330277 (10eoghan) 05Open→03Resolved [16:00:04] 10GitLab (Infrastructure), 10serviceops-collab: Investigate gitlab maintenance/read-only mode - https://phabricator.wikimedia.org/T330277 (10eoghan) Nothing further to do here for the time being, we have options that we can use and notes on where to look in the future if we want to improve the user experience... [16:59:10] 10GitLab (Infrastructure), 10serviceops-collab: gitlab_default_can_create_group setting deprecation - https://phabricator.wikimedia.org/T330282 (10Jelto) p:05Triage→03Medium a:03eoghan [21:12:12] 10GitLab (CI & Job Runners), 10serviceops-collab, 10serviceops-radar, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Set up mirror of the docker hub registry for gitlab-runners - https://phabricator.wikimedia.org/T329679 (10dancy) p:05Triage→03Medium a:03dancy [21:17:09] 10GitLab (Project Migration), 10Gerrit, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Script closing/archiving of migrated repositories on Gerrit - https://phabricator.wikimedia.org/T330345 (10hashar) This task is similar to what is done when archiving Gerrit project as part of th... [21:23:15] 10GitLab (CI & Job Runners), 10serviceops-collab, 10serviceops-radar, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Set up mirror of the docker hub registry for gitlab-runners - https://phabricator.wikimedia.org/T329679 (10dancy) [21:46:50] 10GitLab (Project Migration), 10Gerrit, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-Kizule, 10User-brennen: Script closing/archiving of migrated repositories on Gerrit - https://phabricator.wikimedia.org/T330345 (10Kizule) [22:16:16] 10GitLab (Auth & Access), 10serviceops-collab: Clarify the process of logging into gitlab-replica using 2FA - https://phabricator.wikimedia.org/T330639 (10brennen)