[05:43:53] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11045733 (10Jelto) [05:45:30] Hey, GitLab maintenance starting soon 06:00–08:00 UTC, see T399306 for more details. I'll let you know when the maintenance is finished. [05:45:31] T399306: upgrade gitlab hosts to bookworm - https://phabricator.wikimedia.org/T399306 [05:45:42] and T400252 [05:45:42] T400252: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252 [05:51:14] arnaudb: the daily restore is still running on gitlab1004, so we will start a bit later, probably around 06:30 UTC [05:52:02] ack [05:53:22] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11045748 (10Jelto) [06:36:04] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11045780 (10Jelto) [06:37:20] GitLab maintenance is starting now [06:37:24] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11045781 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.failover (Failover of gitlab from gitlab2002.wikimedia.org to gitlab1004.wikimedia.org... [06:37:54] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11045782 (10Jelto) [07:11:00] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11045817 (10Jelto) [07:11:09] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11045818 (10Jelto) [08:01:28] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11045870 (10Jelto) [08:03:32] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11045873 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.failover (Failover of gitlab from gitlab2002.wikimedia.org to gitlab1004.wikimedia.org... [08:05:17] GitLab maintenance finished. If you have any problem please report here or in T400252 [08:05:18] T400252: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252 [08:16:56] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11045935 (10Jelto) [08:17:59] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11045943 (10Jelto) The switchover to `gitlab1004` was successful. There are two missing packages: ` /var/opt/gitlab/gitlab-rails/shared/packages/... [08:57:27] 10GitLab (Infrastructure), 06collaboration-services: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11046027 (10Jelto) >>! In T400252#11045935, @Jelto wrote: > The switchover to `gitlab1004` was successful. > > There are two missing packages: > > ` > /var/opt/gitlab/... [08:59:25] 10GitLab (Infrastructure), 06collaboration-services: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11046033 (10Jelto) [09:19:12] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11046056 (10Jelto) [11:27:43] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11046466 (10Jelto) [12:59:45] hmm.... gitlab stopped working for me, is anything else going on? (/me starts checking from other servers) [13:00:19] still working from toolforge [13:00:27] so on my laptop side :/ [13:02:03] hmm.... it works with ip6, but not ip4 [13:03:24] https://www.irccloud.com/pastebin/jBIvUdP0/ [13:03:48] interesting... for ip6 it passes through cern xd [13:04:16] it seems to get to equinix though, maybe some routing issue somewhere? [13:04:31] let me ask in -sre [13:13:30] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11046794 (10dcaro) Just for the record, I lost ipv4 connectivity for a few minutes twice today to `wikimedia.gitlab.com`, where the packets were bei... [14:04:59] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11046950 (10Jelto) Thanks for reporting the issue @dcaro . It looks like the rate-limiting is more active on the eqiad host than on codfw. I'll disa... [14:37:26] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11047112 (10Jelto) [15:04:13] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for NIGHTFURY609 - https://phabricator.wikimedia.org/T400453#11047253 (10A_smart_kitten) 05Open→03Resolved a:03Jayprakash12345 Looks like your GitLab account got automatically approved ([[https://wikitech.wi... [15:09:38] 10GitLab (Account Approval): Requesting GitLab account activation for Aadhi13 - https://phabricator.wikimedia.org/T400498#11047270 (10A_smart_kitten) 05Open→03Resolved a:03Jayprakash12345 Looks like your GitLab account got automatically approved ([[https://wikitech.wikimedia.org/w/index.php?title=Tool:... [15:12:43] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation - jyothikat22 - https://phabricator.wikimedia.org/T400455#11047283 (10A_smart_kitten) 05Open→03Resolved a:03Jayprakash12345 Looks like your GitLab account got automatically approved ([[https://wikitech.wikim... [15:15:52] 10GitLab (Account Approval): Requesting GitLab account activation - flykrth - https://phabricator.wikimedia.org/T400451#11047296 (10A_smart_kitten) 05Open→03Resolved a:03Jayprakash12345 Looks like your GitLab account got automatically approved ([[https://wikitech.wikimedia.org/w/index.php?title=Tool:Gi... [15:16:15] 10GitLab (Support), 07Essential-Work, 10Release-Engineering-Team (Doing 😎): GitLab Private Repository Request for: research/npov-workstream-research - https://phabricator.wikimedia.org/T399696#11047300 (10Pablo) That works. Thank you, @thcipriani! [15:19:09] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for MalavikaM05 - https://phabricator.wikimedia.org/T400450#11047322 (10A_smart_kitten) 05Open→03Resolved a:03Jayprakash12345 Looks like your GitLab account got automatically approved ([[https://wikitech.wik... [15:22:20] 10GitLab (CI & Job Runners), 06collaboration-services, 10mwbot-rs, 06Release-Engineering-Team: Random GitLab CI failures with ContainersNotReady - https://phabricator.wikimedia.org/T380680#11047331 (10dancy) [20:16:52] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Swayam_Agrahari - https://phabricator.wikimedia.org/T400449#11048227 (10A_smart_kitten) 05Open→03Resolved Looks like your GitLab account got automatically approved ([[https://wikitech.wikimedia.org/w/index... [20:57:46] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Chimnayyyy - https://phabricator.wikimedia.org/T400448#11048354 (10A_smart_kitten) 05Open→03Resolved a:03Jayprakash12345 Looks like your GitLab account got automatically approved ([[https://wikitech.wiki... [21:06:04] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation - HridyaKK - https://phabricator.wikimedia.org/T400447#11048383 (10A_smart_kitten) 05Open→03Resolved a:03Jayprakash12345 Looks like your GitLab account got automatically approved ([[https://wikitech.wikimedi...