[05:17:20] 10GitLab: Ocassionally pushing to GitLab takes very long - https://phabricator.wikimedia.org/T332967 (10Legoktm) [05:18:48] I'm not sure what to do about https://phabricator.wikimedia.org/T332528 [05:19:18] should I leave it in a broken state so it can be debugged? or should I amend it or something to retrigger CI and the pipeline? [13:44:04] 10GitLab: Disable GitLab snippers - https://phabricator.wikimedia.org/T332993 (10taavi) [13:44:16] 10GitLab: Disable GitLab snippets - https://phabricator.wikimedia.org/T332993 (10taavi) [13:55:59] 10GitLab: Disable GitLab snippets - https://phabricator.wikimedia.org/T332993 (10Aklapper) Currently in GitLab there is only the `Default snippet visibility` setting under `admin/application_settings/general`. https://gitlab.com/gitlab-org/gitlab-foss/-/issues/65355 got rejected in favor of https://gitlab.com/g... [14:50:17] legoktm: hrm - sure would be helpful if it said _what_ error occurred while merging, huh. [15:03:02] i'm not entirely sure where to look in logs here, but digging around a bit [15:13:30] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10User-brennen: Ocassionally pushing to GitLab takes very long - https://phabricator.wikimedia.org/T332967 (10brennen) Confirming I've also experienced this a couple of times recently. At the time I thought it was a local network hiccup. [15:15:09] 10GitLab, 10Release-Engineering-Team, 10mwbot-rs, 10User-brennen: GitLab "An error occurred while merging. Try again." but unable to try again - https://phabricator.wikimedia.org/T332528 (10brennen) Trying to figure out where to even look in logs for this... [15:24:29] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team, 10Patch-For-Review, 10User-brennen: GitLab MR widget sometimes errors out on a missing index - https://phabricator.wikimedia.org/T332607 (10brennen) [16:12:59] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team, 10Upstream: Disable GitLab snippets - https://phabricator.wikimedia.org/T332993 (10brennen) See also: {T264231}. I have two thoughts: 1. This isn't quite the same as Phabricator's pastebin functionality. It's a GitHub gists clone: It suppor... [16:28:30] 10GitLab, 10Release-Engineering-Team, 10mwbot-rs, 10User-brennen: GitLab "An error occurred while merging. Try again." but unable to try again - https://phabricator.wikimedia.org/T332528 (10dancy) Narrowing down the the request that is generating that error response: ` name=/tmp/payload [{"operationName":... [18:17:08] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Addshore) [18:18:03] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Addshore) Looks like it is set up in the gitlab settings for the repo... {F36926040} [18:19:12] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Addshore) ` 13:get remote references: create git ls-remote: exit status 128, stderr: "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @\r\n@@@@@@@@@@@@@@@@@@@... [18:20:46] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10thcipriani) hrm, github did change their rsa key: https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/ [18:21:05] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10brennen) GitHub [[https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/|changed their host key]]... [18:22:46] ^ wonder if deleting and recreating would fix. [18:29:30] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10brennen) Tried deleting and recreating. I'm unclear on what pubkey this relies or where it's set... [18:51:09] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Dzahn) Curious, how are people using these mirrors? They are just readonly on github, right? Is there any difference in that case to just using our own Gitlab when there is no submittint anyways? [19:13:37] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10brennen) Recreating the mirror and clicking "detect host keys" resolves that part of the issue. Now get a permission denied. [22:15:46] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Dzahn) > It seems this has been broken for some time. The RSA host key change only happened yesterday. The same thing was affecting Gerrit at T332972 which is fixed. So I think brennen also fixed everything related to the... [22:40:57] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10hashar) Note: https://github.com/wikimedia/mediawiki-tools-cli is a mirror of the Gerrit repository https://gerrit.wikimedia.org/r/admin/repos/mediawiki/tools/cli And it is indeed replicated: ` name=gerrit1001:/var/log/gerr... [23:19:31] 10GitLab, 10mwcli: Gitlab-Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Dzahn) [23:20:03] 10GitLab, 10mwcli: Gitlab-Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Dzahn) renaming ticket to make clear this is about "gitlab-github" and the other is about "gerrit-github".