[08:27:31] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: ProbeDown - https://phabricator.wikimedia.org/T330717 (10Jelto) [08:33:21] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) As mentioned in T330717, the new production host `gitlab2002` still had the restore enabled and executed a restore... [08:35:13] Hi all, GitLab was switched over to the other datacenter yesterday. Unfortunately due to a misconfiguration we lost some data. All actions (like pushes, merges, builds and comments) between Feb 28th 0:30 UTC and [08:35:13] Feb 28th 2:00 UTC (so one and a half hours) are lost. If you used GitLab [08:35:13] during that time make sure to re-apply your changes. [08:35:13] See also: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/MK34K3LQUSYE6FHMWGV5W7B4MYRG3TAT/ [09:01:42] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) I started a incident report in [2023-02-28_GitLab_data_loss](https://wikitech.wikimedia.org/wiki/Incidents/2023-02-... [10:38:14] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: ProbeDown - https://phabricator.wikimedia.org/T330717 (10Jelto) 05Open→03Resolved After merging the above change the restore timer is gone form `gitlab2002`. So we should not see a ProbeDown alert again due to restores on the production... [11:19:23] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) The rsync jobs between production host and replica are only created but not removed, when the list of replicas chan... [11:57:33] 10GitLab: gitlab backup timer failing - https://phabricator.wikimedia.org/T330744 (10jbond) 05Open→03In progress p:05Triage→03Medium [12:01:36] 10GitLab, 10serviceops-collab: gitlab backup timer failing - https://phabricator.wikimedia.org/T330744 (10Jelto) thanks for keeping an eye on that! I also posted this issue to the gitlab switchover task: T329931#8652138 I would be happy to review a patch! Thanks a lot. [12:12:09] 10GitLab, 10serviceops-collab: gitlab backup timer failing - https://phabricator.wikimedia.org/T330744 (10jbond) [12:13:29] 10GitLab, 10serviceops-collab, 10Patch-For-Review: gitlab backup timer failing - https://phabricator.wikimedia.org/T330744 (10jbond) [12:13:34] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10jbond) [12:14:25] 10GitLab, 10serviceops-collab, 10Patch-For-Review: gitlab backup timer failing - https://phabricator.wikimedia.org/T330744 (10jbond) >>! In T330744#8652227, @Jelto wrote: > thanks for keeping an eye on that! > Oncall so just cleaning up active alerts :) > I also posted this issue to the gitlab switchover t... [13:13:52] 10GitLab, 10serviceops-collab: gitlab backup timer failing - https://phabricator.wikimedia.org/T330744 (10Jelto) Time/jobs for `gitlab2002` were removed on `gitlab1004`: ` Notice: /Stage[main]/Gitlab::Rsync/Systemd::Timer::Job[rsync-data-backup-gitlab2002.wikimedia.org]/Systemd::Timer[rsync-data-backup-gitlab... [17:11:09] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10jbond) [17:11:13] 10GitLab, 10serviceops-collab: gitlab backup timer failing - https://phabricator.wikimedia.org/T330744 (10jbond) 05In progress→03Resolved thanks, i have ran `sudo systemctl reset-failed` to clean up the systemd status. closing [19:58:30] 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Isito interferes with HTTP traffic from buildkitd build containers - https://phabricator.wikimedia.org/T330433 (10dduvall) p:05Triage→03Medium a:03dduvall [20:01:03] 10GitLab (Misc), 10Upstream: GitLab truncates commit messages over 1k of text - https://phabricator.wikimedia.org/T330790 (10brennen) [20:31:12] 10GitLab (Administration, Settings & Policy), 10Privacy Engineering, 10Product-Analytics, 10Privacy, and 2 others: Request for Private repos to be enabled - https://phabricator.wikimedia.org/T305082 (10JFishback_WMF) [21:03:42] 10GitLab (Auth & Access), 10Infrastructure-Foundations, 10SRE, 10serviceops-collab, and 3 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10demon) From the looks of it, we can add OIDC as a second [omniauth provider](https://docs.gitlab.com/ee/integration/omni... [21:47:45] 10GitLab (Project Migration), 10Phabricator, 10serviceops-collab, 10Epic, and 3 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10Aklapper) [22:58:24] 10GitLab (Project Migration), 10Gerrit, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-Kizule, 10User-brennen: Script closing/archiving of migrated repositories on Gerrit - https://phabricator.wikimedia.org/T330345 (10brennen) > In my imagination the perfect migration and least error pron...