[02:52:54] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog ๐Ÿ“ฅ), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) [05:13:47] (03PS11) 10Hashar: Display Zuul status of jobs for a change in Gerrit [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/859127 (https://phabricator.wikimedia.org/T214068) [05:15:38] (03CR) 10Hashar: [C: 03+2] wm-checks-api: support the Early Warning bot [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/894099 (https://phabricator.wikimedia.org/T330850) (owner: 10Hashar) [05:16:09] (03Merged) 10jenkins-bot: wm-checks-api: support the Early Warning bot [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/894099 (https://phabricator.wikimedia.org/T330850) (owner: 10Hashar) [05:16:41] (03CR) 10Hashar: [C: 03+2] Display Zuul status of jobs for a change in Gerrit [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/859127 (https://phabricator.wikimedia.org/T214068) (owner: 10Hashar) [05:17:12] (03Merged) 10jenkins-bot: Display Zuul status of jobs for a change in Gerrit [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/859127 (https://phabricator.wikimedia.org/T214068) (owner: 10Hashar) [05:17:20] 10GitLab: Ocassionally pushing to GitLab takes very long - https://phabricator.wikimedia.org/T332967 (10Legoktm) [05:40:09] (03CR) 10Hashar: "I have to further amend the permissions:" [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/902371 (https://phabricator.wikimedia.org/T281593) (owner: 10Hashar) [05:52:51] (03PS1) 10Hashar: Revert "Properly block pushes to 'main' branch" [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/902586 [05:52:59] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Seen), 10Zuul: Display Zuul status of jobs for a change on Gerrit UI - https://phabricator.wikimedia.org/T214068 (10Krinkle) Very nice! {F36925186 height=300} [05:53:01] (03CR) 10Hashar: [V: 03+2 C: 03+2] Revert "Properly block pushes to 'main' branch" [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/902586 (owner: 10Hashar) [05:55:02] (03PS1) 10Hashar: Revert "Block access to the main branch" [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/902587 [05:55:11] (03CR) 10Hashar: [V: 03+2 C: 03+2] Revert "Block access to the main branch" [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/902587 (owner: 10Hashar) [05:55:41] (03PS1) 10Hashar: Revert "Block access to the main branch" [skins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/902588 [05:55:52] (03CR) 10Hashar: [V: 03+2 C: 03+2] Revert "Block access to the main branch" [skins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/902588 (owner: 10Hashar) [06:25:34] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Seen), 10Zuul: Display Zuul status of jobs for a change on Gerrit UI - https://phabricator.wikimedia.org/T214068 (10hashar) >>! In T214068#8723722, @Krinkle wrote: > Very nice! Thanks Timo for the TypeScript linting and your QU... [06:28:17] (03PS1) 10Hashar: Review access change [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/902589 [06:31:54] (03PS2) 10Hashar: Block access to the main branch (take 2) [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/902589 (https://phabricator.wikimedia.org/T281593) [06:36:49] (03CR) 10Hashar: "This time I have applied the rules to the test repository test/gerrit-ping https://gerrit.wikimedia.org/r/admin/repos/test/gerrit-ping,acc" [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/902589 (https://phabricator.wikimedia.org/T281593) (owner: 10Hashar) [07:15:01] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Seen), 10Zuul: Display Zuul status of jobs for a change on Gerrit UI - https://phabricator.wikimedia.org/T214068 (10Michael) This is great! Thank you! ๐Ÿ™‡ [08:33:59] 10Gerrit, 10SRE, 10Wikimedia-GitHub: Github RSA ssh host key updated 2023-03-23 - https://phabricator.wikimedia.org/T332972 (10hashar) [08:58:00] 10Gerrit, 10SRE, 10Wikimedia-GitHub: Github RSA ssh host key updated 2023-03-23 - https://phabricator.wikimedia.org/T332972 (10hashar) 05Openโ†’03Resolved a:03hashar At least for #Gerrit that is fixed. There might be other use cases though (TranslateWiki comes to mind) [09:37:53] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Seen), 10Zuul: Display Zuul status of jobs for a change on Gerrit UI - https://phabricator.wikimedia.org/T214068 (10Lucas_Werkmeister_WMDE) Very cool, thank you! One thing I noticed was this `null` on a change that I just pushe... [09:51:45] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Seen), 10Zuul: Display Zuul status of jobs for a change on Gerrit UI - https://phabricator.wikimedia.org/T214068 (10hashar) >>! In T214068#8723990, @Lucas_Werkmeister_WMDE wrote: > Very cool, thank you! > > One thing I noticed... [10:36:20] hashar: im getting the following error in gerrit [10:36:22] Error while fetching results for wm-zuul-status: TypeError: status.jobs[0] is undefined [10:36:33] and it just dissapeared [10:36:50] was on this change looked like a js issue https://gerrit.wikimedia.org/r/c/operations/puppet/+/902682/1 [10:37:22] ahh it went away after pcc uploaded a report [11:07:44] jbond: yeah it is part of a frontend plugin I have deployed very early this morning [11:08:00] ack [11:08:13] I need to add some guards, some data are not entirely set when the change enters CI [11:08:37] ack not cuasing an issue on my end just wanted to make sure you had seen it :) [11:08:44] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Seen), 10Zuul: Display Zuul status of jobs for a change on Gerrit UI - https://phabricator.wikimedia.org/T214068 (10hashar) A similar issue has been reported: > Error while fetching results for wm-zuul-status: TypeError: status... [11:09:04] jbond: I have added your report on the task ( https://phabricator.wikimedia.org/T214068#8724008 ) [11:09:28] cheers [11:09:49] that is despite a bunch of unit tests but I haven't written integration tests against the CI output :/ [11:13:07] jbond: for that specific case the change was not being tested in Zuul, but a child change was being processed [11:13:24] and Zuul represents the dependency even though the parent change has nothing running [11:13:28] easy to filter out ;) [11:17:17] ahh ok thanks [12:03:38] (03PS1) 10Kosta Harlan: jjb: ci-fullrun-extensions should not trigger npm install [integration/config] - 10https://gerrit.wikimedia.org/r/902695 (https://phabricator.wikimedia.org/T332386) [12:52:15] (03PS1) 10Hashar: wm-zuul-status: filter out non-live item [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/902705 (https://phabricator.wikimedia.org/T214068) [13:04:59] Lucas_WMDE: I have reproduced the issue with `null` being shown in the Gerrit check chipsets below the commit message :] [13:05:15] it is for a change that has just entered Zuul but for which no jobs have been running yet \o/ [13:12:48] ah, when theyโ€™re all qeueued? [13:13:10] (that word didnโ€™t have enough vowels yet so I added one more ๐Ÿ˜Œ) [13:44:04] 10GitLab: Disable GitLab snippers - https://phabricator.wikimedia.org/T332993 (10taavi) [13:44:16] 10GitLab: Disable GitLab snippets - https://phabricator.wikimedia.org/T332993 (10taavi) [13:51:19] (03CR) 10Hashar: "INFO:jenkins_jobs.builder:Reconfiguring jenkins job integration-quibble-fullrun-extensions" [integration/config] - 10https://gerrit.wikimedia.org/r/902695 (https://phabricator.wikimedia.org/T332386) (owner: 10Kosta Harlan) [13:51:28] (03CR) 10Hashar: [C: 03+2] jjb: ci-fullrun-extensions should not trigger npm install [integration/config] - 10https://gerrit.wikimedia.org/r/902695 (https://phabricator.wikimedia.org/T332386) (owner: 10Kosta Harlan) [13:52:42] (03Merged) 10jenkins-bot: jjb: ci-fullrun-extensions should not trigger npm install [integration/config] - 10https://gerrit.wikimedia.org/r/902695 (https://phabricator.wikimedia.org/T332386) (owner: 10Kosta Harlan) [13:55:59] 10GitLab: Disable GitLab snippets - https://phabricator.wikimedia.org/T332993 (10Aklapper) Currently in GitLab there is only the `Default snippet visibility` setting under `admin/application_settings/general`. https://gitlab.com/gitlab-org/gitlab-foss/-/issues/65355 got rejected in favor of https://gitlab.com/g... [14:44:35] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Math: Jenkins should run test for extension math with PHP8 - https://phabricator.wikimedia.org/T333002 (10Physikerwelt) [15:13:30] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10User-brennen: Ocassionally pushing to GitLab takes very long - https://phabricator.wikimedia.org/T332967 (10brennen) Confirming I've also experienced this a couple of times recently. At the time I thought it was a local network hiccup. [15:15:09] 10GitLab, 10Release-Engineering-Team, 10mwbot-rs, 10User-brennen: GitLab "An error occurred while merging. Try again." but unable to try again - https://phabricator.wikimedia.org/T332528 (10brennen) Trying to figure out where to even look in logs for this... [15:24:29] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team, 10Patch-For-Review, 10User-brennen: GitLab MR widget sometimes errors out on a missing index - https://phabricator.wikimedia.org/T332607 (10brennen) [15:50:05] 10Gerrit: [wm-checks-api] support EarlyWarningBot - https://phabricator.wikimedia.org/T330850 (10hashar) 05Openโ†’03Resolved a:03hashar I went to pick the latest change "reviewed" by EarlyWarningBot via https://gerrit.wikimedia.org/r/q/reviewer:EarlyWarningBot https://gerrit.wikimedia.org/r/c/mediawiki/exte... [16:12:59] 10GitLab (Upstream pit of despair ๐Ÿ•ณ๏ธ), 10Release-Engineering-Team, 10Upstream: Disable GitLab snippets - https://phabricator.wikimedia.org/T332993 (10brennen) See also: {T264231}. I have two thoughts: 1. This isn't quite the same as Phabricator's pastebin functionality. It's a GitHub gists clone: It suppor... [16:28:30] 10GitLab, 10Release-Engineering-Team, 10mwbot-rs, 10User-brennen: GitLab "An error occurred while merging. Try again." but unable to try again - https://phabricator.wikimedia.org/T332528 (10dancy) Narrowing down the the request that is generating that error response: ` name=/tmp/payload [{"operationName":... [18:17:08] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Addshore) [18:18:03] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Addshore) Looks like it is set up in the gitlab settings for the repo... {F36926040} [18:19:12] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Addshore) ` 13:get remote references: create git ls-remote: exit status 128, stderr: "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @\r\n@@@@@@@@@@@@@@@@@@@... [18:20:46] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10thcipriani) hrm, github did change their rsa key: https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/ [18:21:05] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10brennen) GitHub [[https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/|changed their host key]]... [18:29:30] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10brennen) Tried deleting and recreating. I'm unclear on what pubkey this relies or where it's set... [18:51:09] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Dzahn) Curious, how are people using these mirrors? They are just readonly on github, right? Is there any difference in that case to just using our own Gitlab when there is no submittint anyways? [19:13:37] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10brennen) Recreating the mirror and clicking "detect host keys" resolves that part of the issue. Now get a permission denied. [21:02:12] (03PS1) 10Bartosz Dziewoล„ski: Add my other email address to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/902804 [21:45:13] 10Release-Engineering-Team (GitLab V: Event Horizon ๐ŸŒ„), 10Release, 10Train Deployments: 1.41.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T330208 (10thcipriani) p:05Triageโ†’03Medium [21:45:55] 10Release-Engineering-Team (Priority Backlog ๐Ÿ“ฅ), 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10thcipriani) p:05Triageโ†’03Medium a:03demon [21:46:21] 10Release-Engineering-Team (Priority Backlog ๐Ÿ“ฅ), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T330207 (10dancy) 05Openโ†’03Resolved [21:47:35] 10Release-Engineering-Team (Priority Backlog ๐Ÿ“ฅ), 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10thcipriani) a:05demonโ†’03hashar [21:48:12] 10Release-Engineering-Team (Priority Backlog ๐Ÿ“ฅ), 10Release, 10Train Deployments: 1.41.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T330210 (10thcipriani) p:05Triageโ†’03Medium a:03demon [22:15:46] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Dzahn) > It seems this has been broken for some time. The RSA host key change only happened yesterday. The same thing was affecting Gerrit at T332972 which is fixed. So I think brennen also fixed everything related to the... [22:40:57] 10GitLab, 10mwcli: Github mirror broken - https://phabricator.wikimedia.org/T333029 (10hashar) Note: https://github.com/wikimedia/mediawiki-tools-cli is a mirror of the Gerrit repository https://gerrit.wikimedia.org/r/admin/repos/mediawiki/tools/cli And it is indeed replicated: ` name=gerrit1001:/var/log/gerr... [23:19:31] 10GitLab, 10mwcli: Gitlab-Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Dzahn) [23:20:03] 10GitLab, 10mwcli: Gitlab-Github mirror broken - https://phabricator.wikimedia.org/T333029 (10Dzahn) renaming ticket to make clear this is about "gitlab-github" and the other is about "gerrit-github".