[08:01:34] <jelto>	 GitLab maintenance window starts in one hour: 09:00-13:00 UTC. We will switch GitLab from eqiad to codfw. See T345531
[08:01:35] <stashbot>	 T345531: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531
[08:44:25] <wikibugs>	 10GitLab (Infrastructure), 10collaboration-services, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 (10CodeReviewBot) jelto merged https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/46  [gitlab/switchover] Up...
[08:55:55] <wikibugs>	 10GitLab (Infrastructure), 10collaboration-services, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 (10Jelto)
[08:59:23] <jelto>	 GitLab maintenance starting now - T345531
[08:59:24] <stashbot>	 T345531: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531
[09:00:03] <jelto>	 eoghan: I'm starting the cookbook now
[09:00:23] <eoghan>	 Cool cool! Good luck
[09:01:22] <wikibugs>	 10GitLab (Infrastructure), 10collaboration-services, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.failover (Failover of gitlab from gitlab1004.wikimedia.org to gitlab2002.wikime...
[09:02:18] <jelto>	 gitlab is down and backup in progress
[10:26:55] <jelto>	 backup done, sync to the other host happening
[10:43:32] <jelto>	 restore happening on the codfw instance
[11:46:35] <wikibugs>	 10GitLab (Infrastructure), 10collaboration-services, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.failover (Failover of gitlab from gitlab1004.wikimedia.org to gitlab2002.wikime...
[11:49:08] <jelto>	 Gitlab is back, maintenance done
[12:27:03] <wikibugs>	 10GitLab (Infrastructure), 10collaboration-services: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 (10LSobanski)
[12:33:02] <urbanecm>	 welcome, gitlab! :)
[13:33:02] <wikibugs>	 10GitLab (Infrastructure), 10collaboration-services: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 (10Jelto)
[15:04:12] <thcipriani>	 kudos on the failover \o/
[15:06:35] <jelto>	 thanks :)
[16:11:18] <brennen>	 nicely done.
[18:11:00] <fab>	 When navigating to https://gitlab.wikimedia.org/, I see a read banner "THIS IS A REPLICA". But url in the bar and the network traffic in the developer tool show only gitlab.wikimedia.org. Since the redeploy is done, I presume this is an issue for me only? Clicking on the link in the banner drops me on the same page again.
[18:12:52] <fab>	 I have had a similar problem before, which was resolved by clearing the cache (https://phabricator.wikimedia.org/T344156). But I wonder how I can prevent this in the future?