[08:28:02] (03CR) 10Hashar: [C: 03+2] Gerrit v3.5.4 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/824200 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [08:28:33] (03Merged) 10jenkins-bot: Gerrit v3.5.4 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/824200 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [08:46:20] (03CR) 10Hashar: [C: 03+2] Zuul: Archive the IndexFunction extension [integration/config] - 10https://gerrit.wikimedia.org/r/855856 (https://phabricator.wikimedia.org/T296158) (owner: 10Zoranzoki21) [08:47:14] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ): Automate integration Jenkins deployment and config changes - https://phabricator.wikimedia.org/T319406 (10jnuche) 05Open→03In progress a:03jnuche [08:48:02] (03Merged) 10jenkins-bot: Zuul: Archive the IndexFunction extension [integration/config] - 10https://gerrit.wikimedia.org/r/855856 (https://phabricator.wikimedia.org/T296158) (owner: 10Zoranzoki21) [09:10:10] 10Continuous-Integration-Config, 10WikiLambda, 10Abstract Wikipedia team (Phase Îș – Clean-up), 10MW-1.39-notes (1.39.0-wmf.22; 2022-07-25), and 2 others: Run some full end-to-end browser integration tests on Beta Cluster Wikifunctions now it exists - https://phabricator.wikimedia.org/T294388 (10zeljkofilipi... [09:27:02] 10Gerrit, 10Release-Engineering-Team: Fix up Gerrit usernames having upper case character(s) - https://phabricator.wikimedia.org/T323097 (10hashar) [09:28:12] 10Gerrit, 10Release-Engineering-Team: Fix up Gerrit usernames having upper case character(s) - https://phabricator.wikimedia.org/T323097 (10hashar) [09:30:02] (03PS2) 10Zfilipin: jjb: Fix link to Jenkins credential store [integration/config] - 10https://gerrit.wikimedia.org/r/856613 (https://phabricator.wikimedia.org/T323047) [09:34:15] (03PS2) 10Zfilipin: jjb: Remove unused site variable in wdio-selenium-daily project [integration/config] - 10https://gerrit.wikimedia.org/r/856622 (https://phabricator.wikimedia.org/T323047) [09:43:25] (03PS3) 10Zfilipin: jjb: Fix link to Jenkins credential store [integration/config] - 10https://gerrit.wikimedia.org/r/856613 (https://phabricator.wikimedia.org/T303739) [09:43:44] (03PS3) 10Zfilipin: jjb: Remove unused site variable in wdio-selenium-daily project [integration/config] - 10https://gerrit.wikimedia.org/r/856622 (https://phabricator.wikimedia.org/T303739) [09:44:00] (03PS2) 10Zfilipin: WIP Run Selenium tests targeting production [integration/config] - 10https://gerrit.wikimedia.org/r/856618 (https://phabricator.wikimedia.org/T303739) [09:53:25] (03PS1) 10Dom Walden: WIP make the credentials and username variables for selenium daily job [integration/config] - 10https://gerrit.wikimedia.org/r/856925 (https://phabricator.wikimedia.org/T303739) [09:57:11] (03CR) 10Zfilipin: WikiLambda: run e2e tests daily on betacluster (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/856646 (https://phabricator.wikimedia.org/T294388) (owner: 10Stef Dunlap) [10:06:15] (03CR) 10Zfilipin: WIP make the credentials and username variables for selenium daily job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/856925 (https://phabricator.wikimedia.org/T303739) (owner: 10Dom Walden) [10:33:35] (03PS2) 10Dom Walden: jjb: make the credentials and username variables for selenium daily job [integration/config] - 10https://gerrit.wikimedia.org/r/856925 (https://phabricator.wikimedia.org/T303739) [10:34:35] (03CR) 10Dom Walden: jjb: make the credentials and username variables for selenium daily job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/856925 (https://phabricator.wikimedia.org/T303739) (owner: 10Dom Walden) [10:35:11] (03CR) 10CI reject: [V: 04-1] jjb: make the credentials and username variables for selenium daily job [integration/config] - 10https://gerrit.wikimedia.org/r/856925 (https://phabricator.wikimedia.org/T303739) (owner: 10Dom Walden) [10:36:47] (03PS3) 10Dom Walden: jjb: make the credentials and username variables for selenium daily job [integration/config] - 10https://gerrit.wikimedia.org/r/856925 (https://phabricator.wikimedia.org/T303739) [11:23:41] 10GitLab, 10Release-Engineering-Team (Priority Backlog đŸ“„), 10serviceops-collab: Build and import new release of jwt-authorizer (1.1.0) - https://phabricator.wikimedia.org/T322691 (10Jelto) @Dzahn you described the initial build process of `jwt-authorizer` package in T309646. I created docs in wikitech for j... [11:32:54] (03CR) 10Hashar: [C: 04-1] "This job act on" [integration/config] - 10https://gerrit.wikimedia.org/r/856033 (owner: 10Ladsgroup) [11:53:12] (03CR) 10Zfilipin: [C: 03+2] jjb: make the credentials and username variables for selenium daily job [integration/config] - 10https://gerrit.wikimedia.org/r/856925 (https://phabricator.wikimedia.org/T303739) (owner: 10Dom Walden) [11:55:14] (03Merged) 10jenkins-bot: jjb: make the credentials and username variables for selenium daily job [integration/config] - 10https://gerrit.wikimedia.org/r/856925 (https://phabricator.wikimedia.org/T303739) (owner: 10Dom Walden) [13:08:34] (03CR) 10Dom Walden: WIP Run Selenium tests targeting production (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/856618 (https://phabricator.wikimedia.org/T303739) (owner: 10Zfilipin) [13:12:25] 10Phabricator: WikiLearn project - https://phabricator.wikimedia.org/T323112 (10Asaf) [13:13:42] 10Project-Admins: WikiLearn project - https://phabricator.wikimedia.org/T323112 (10taavi) [13:23:42] 10Project-Admins: WikiLearn project - https://phabricator.wikimedia.org/T323112 (10Asaf) Thank you, @taavi . I have updated the request to include a tag as well, to tag issues outside the project as affecting it. [13:24:34] 10Project-Admins: WikiLearn project and affects-WikiLearn tag - https://phabricator.wikimedia.org/T323112 (10Asaf) [13:31:01] (03PS3) 10Zfilipin: WIP Run Selenium tests targeting production [integration/config] - 10https://gerrit.wikimedia.org/r/856618 (https://phabricator.wikimedia.org/T303739) [13:31:37] (03CR) 10Zfilipin: "PS3 resolves the merge conflict." [integration/config] - 10https://gerrit.wikimedia.org/r/856618 (https://phabricator.wikimedia.org/T303739) (owner: 10Zfilipin) [13:35:32] (03PS4) 10Zfilipin: WIP Run Selenium tests targeting production [integration/config] - 10https://gerrit.wikimedia.org/r/856618 (https://phabricator.wikimedia.org/T303739) [13:35:55] (03CR) 10Zfilipin: WIP Run Selenium tests targeting production (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/856618 (https://phabricator.wikimedia.org/T303739) (owner: 10Zfilipin) [13:38:37] 10Project-Admins: WikiLearn project and affects-WikiLearn tag - https://phabricator.wikimedia.org/T323112 (10Peachey88) 05Open→03Resolved a:03Peachey88 Requested public projects #WikiLearn (https://phabricator.wikimedia.org/project/view/6264/) and #affects-wikilearn (https://phabricator.wikimedia.org/proje... [13:49:14] (03PS1) 10Daimona Eaytoy: zuul: [mediawiki/extensions/CampaignEvents] Add Echo as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/856982 (https://phabricator.wikimedia.org/T317231) [13:52:04] (03CR) 10Dom Walden: [C: 03+1] "Looks good to me. We just need to add the appropriate credentials to Jenkins with ID "selenium-user-production"." [integration/config] - 10https://gerrit.wikimedia.org/r/856618 (https://phabricator.wikimedia.org/T303739) (owner: 10Zfilipin) [14:46:46] (03PS3) 10Stef Dunlap: WikiLambda: run e2e tests daily on betacluster [integration/config] - 10https://gerrit.wikimedia.org/r/856646 (https://phabricator.wikimedia.org/T294388) [14:48:34] (03CR) 10CI reject: [V: 04-1] WikiLambda: run e2e tests daily on betacluster [integration/config] - 10https://gerrit.wikimedia.org/r/856646 (https://phabricator.wikimedia.org/T294388) (owner: 10Stef Dunlap) [14:50:16] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Fix up Gerrit usernames having upper case character(s) - https://phabricator.wikimedia.org/T323097 (10hashar) I have manually deleted `username:Kaldari` in `All-Users.git`: ` commit 495ec6989a771ca345d89d6df8055c66dd91e888 (HEAD -> meta/external-ids,... [14:50:45] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Fix up Gerrit usernames having upper case character(s) - https://phabricator.wikimedia.org/T323097 (10hashar) [15:10:43] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Fix up Gerrit usernames having upper case character(s) - https://phabricator.wikimedia.org/T323097 (10hashar) [15:11:29] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Fix up Gerrit usernames having upper case character(s) - https://phabricator.wikimedia.org/T323097 (10hashar) I have converted `username:Fran McCrory` to be all lower case: ` commit 6c09bae902bfe56bfce222439468d79148d5dc38 (HEAD -> meta/external-ids,... [15:16:41] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Fix up Gerrit usernames having upper case character(s) - https://phabricator.wikimedia.org/T323097 (10hashar) I have converted SamanthaNguyen to all lower case: ` commit f8b7cc2dc1844518cdd92495d8df1cf5b94bc3d5 (HEAD -> meta/external-ids, origin/meta/... [15:20:03] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Fix up Gerrit usernames having upper case character(s) - https://phabricator.wikimedia.org/T323097 (10hashar) I have reindexed all accounts with `ssh -p 29418 gerrit.wikimedia.org -- gerrit index start accounts --force` then queried again the three ac... [15:21:53] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Fix up Gerrit usernames having upper case character(s) - https://phabricator.wikimedia.org/T323097 (10hashar) 05Open→03Resolved a:03hashar There is no more any upper case letter in Gerrit accounts (scheme `username:`): ` $ git grep -P 'username:... [15:24:15] !log gerrit: converted, to all lower case, the Gerrit accounts `username:Kaldari`, `username:Fran McCrory` and `username:SamanthaNguyen` # T323097 [15:24:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:24:18] T323097: Fix up Gerrit usernames having upper case character(s) - https://phabricator.wikimedia.org/T323097 [15:42:44] 10Release-Engineering-Team (Priority Backlog đŸ“„), 10Gerrit (Gerrit 3.5): Upgrade to Gerrit 3.5 - https://phabricator.wikimedia.org/T307334 (10hashar) [15:42:46] 10Gerrit, 10Release-Engineering-Team: Fix up Gerrit usernames having upper case character(s) - https://phabricator.wikimedia.org/T323097 (10hashar) [15:51:29] 10GitLab, 10Release-Engineering-Team, 10Data Pipelines, 10Data-Engineering-Planning, 10serviceops-collab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10LSobanski) p:05Triage→03Low Likely needs a design discussion between RelEng and ServiceOps. [15:53:53] (03CR) 10Jforrester: [C: 03+2] zuul: [mediawiki/extensions/CampaignEvents] Add Echo as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/856982 (https://phabricator.wikimedia.org/T317231) (owner: 10Daimona Eaytoy) [15:55:40] (03Merged) 10jenkins-bot: zuul: [mediawiki/extensions/CampaignEvents] Add Echo as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/856982 (https://phabricator.wikimedia.org/T317231) (owner: 10Daimona Eaytoy) [15:57:34] !log Zuul: [mediawiki/extensions/CampaignEvents] Add Echo as phan dependency for T317231 [15:57:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:57:36] T317231: Implement automatic email after event registration - https://phabricator.wikimedia.org/T317231 [15:57:51] 10Release-Engineering-Team (Radar), 10serviceops-collab, 10serviceops-radar: give releng access to logs to debug buildkit-to-wmf-registry publishing - https://phabricator.wikimedia.org/T322579 (10LSobanski) p:05Triage→03High [16:01:46] 10Release-Engineering-Team, 10serviceops-collab: reconsider https://git.wikimedia.org link - https://phabricator.wikimedia.org/T323073 (10LSobanski) 05Open→03Stalled p:05Triage→03Low Pending a decision from RelEng. [16:03:11] 10GitLab, 10Release-Engineering-Team (Priority Backlog đŸ“„), 10serviceops-collab: Build and import new release of jwt-authorizer (1.1.0) - https://phabricator.wikimedia.org/T322691 (10LSobanski) p:05Triage→03Medium [16:06:41] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10LSobanski) p:05High→03Medium [16:20:10] (03PS1) 10Jforrester: Zuul: [mediawiki/services/parsoid] Make MW jobs voting in test [integration/config] - 10https://gerrit.wikimedia.org/r/857002 [16:20:12] (03PS1) 10Jforrester: Zuul: [mediawiki/services/parsoid/deploy] Move to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/857003 [16:23:25] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/services/parsoid] Make MW jobs voting in test [integration/config] - 10https://gerrit.wikimedia.org/r/857002 (owner: 10Jforrester) [16:24:37] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/services/parsoid/deploy] Move to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/857003 (owner: 10Jforrester) [16:25:14] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/parsoid] Make MW jobs voting in test [integration/config] - 10https://gerrit.wikimedia.org/r/857002 (owner: 10Jforrester) [16:25:50] !log Zuul: [mediawiki/services/parsoid] Make MW jobs voting in test [16:25:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:26:21] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/parsoid/deploy] Move to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/857003 (owner: 10Jforrester) [16:34:58] 10Gerrit: Fix up Gerrit duplicate LDAP external ids - https://phabricator.wikimedia.org/T323135 (10hashar) [16:35:17] 10Release-Engineering-Team (Priority Backlog đŸ“„), 10Gerrit (Gerrit 3.5): Upgrade to Gerrit 3.5 - https://phabricator.wikimedia.org/T307334 (10hashar) [16:35:19] 10Gerrit: Fix up Gerrit duplicate LDAP external ids - https://phabricator.wikimedia.org/T323135 (10hashar) [16:53:00] 10Gerrit: Fix up Gerrit duplicate LDAP external ids - https://phabricator.wikimedia.org/T323135 (10hashar) The reason is we have `ldap.localUsernameToLowerCase=true`. The upstream script to do the conversion had issue and instead @thcipriani crafted a shell script for that: {P8606} The conversion commit is from... [16:57:11] 10Gerrit: Fix up Gerrit duplicate LDAP external ids - https://phabricator.wikimedia.org/T323135 (10hashar) I have verified all of the Gerrit external ids for LDAP (scheme `gerrit:`) having an upper case letter have an equivalent all lower case. The only differences I have spotted is some email addresses have bee... [17:07:05] 10Continuous-Integration-Config, 10phpunit-patch-coverage: Most (all?) phpunit-patch-coverage jobs failing with "String could not be parsed as XML" instead of actually running - https://phabricator.wikimedia.org/T323139 (10Jdforrester-WMF) [17:09:56] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Scap: Ensure efficient Gitlab CI operations for scap - https://phabricator.wikimedia.org/T323140 (10dancy) [17:10:42] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Scap: Ensure efficient Gitlab CI operations for scap - https://phabricator.wikimedia.org/T323140 (10dancy) [17:12:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:17:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:19:25] 10Continuous-Integration-Config, 10MediaWiki-extensions-Linter, 10MediaWiki-extensions-Translate, 10MW-1.40-notes (1.40.0-wmf.4; 2022-10-03), and 2 others: ApiPrefixUniquenessTest::testPrefixes fails because linterstats and languagestats are both using "ls" - https://phabricator.wikimedia.org/T319202 (10Jdf... [17:27:05] (03CR) 10Jforrester: [C: 03+2] jjb: Remove unused site variable in wdio-selenium-daily project [integration/config] - 10https://gerrit.wikimedia.org/r/856622 (https://phabricator.wikimedia.org/T303739) (owner: 10Zfilipin) [17:27:44] (03CR) 10Jforrester: [C: 03+2] jjb: Fix link to Jenkins credential store [integration/config] - 10https://gerrit.wikimedia.org/r/856613 (https://phabricator.wikimedia.org/T303739) (owner: 10Zfilipin) [17:29:00] (03Merged) 10jenkins-bot: jjb: Remove unused site variable in wdio-selenium-daily project [integration/config] - 10https://gerrit.wikimedia.org/r/856622 (https://phabricator.wikimedia.org/T303739) (owner: 10Zfilipin) [17:29:29] (03Merged) 10jenkins-bot: jjb: Fix link to Jenkins credential store [integration/config] - 10https://gerrit.wikimedia.org/r/856613 (https://phabricator.wikimedia.org/T303739) (owner: 10Zfilipin) [17:37:40] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:48:24] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Security: Jenkins plugins security advisory - 2022-11-15 - https://phabricator.wikimedia.org/T323054 (10Jdforrester-WMF) Doesn't look like we're affected. [17:54:43] 10Continuous-Integration-Config, 10MediaWiki-extensions-Linter, 10MediaWiki-extensions-Translate, 10MW-1.40-notes (1.40.0-wmf.4; 2022-10-03), and 2 others: ApiPrefixUniquenessTest::testPrefixes fails because linterstats and languagestats are both using "ls" - https://phabricator.wikimedia.org/T319202 (10mat... [18:15:25] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10User-brennen: Add DigitalOcean resource monitoring for cloud runner nodes - https://phabricator.wikimedia.org/T308615 (10brennen) [18:15:53] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10User-brennen: Provision untrusted instance-wide GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10brennen) [18:19:28] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10User-brennen: Migrate mediawiki/tools/release/ to GitLab - https://phabricator.wikimedia.org/T290260 (10brennen) [18:31:03] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Scap: Try DigitalOcean object storage for buildkit caching - https://phabricator.wikimedia.org/T323147 (10dancy) [18:31:28] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Scap: Try DigitalOcean object storage for buildkit caching - https://phabricator.wikimedia.org/T323147 (10dancy) [18:31:30] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Scap: Ensure efficient Gitlab CI operations for scap - https://phabricator.wikimedia.org/T323140 (10dancy) [18:32:23] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Scap: Try DigitalOcean registry for buildkit caching - https://phabricator.wikimedia.org/T323148 (10dancy) [18:32:44] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Scap: Try DigitalOcean registry for buildkit caching - https://phabricator.wikimedia.org/T323148 (10dancy) [18:32:46] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Scap: Ensure efficient Gitlab CI operations for scap - https://phabricator.wikimedia.org/T323140 (10dancy) [18:35:42] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ): Try Reggie for buildkit caching - https://phabricator.wikimedia.org/T323149 (10dancy) [18:36:14] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ): Try Reggie for buildkit caching - https://phabricator.wikimedia.org/T323149 (10dancy) [18:36:22] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Scap: Ensure efficient Gitlab CI operations for scap - https://phabricator.wikimedia.org/T323140 (10dancy) [18:36:56] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Scap: Try DigitalOcean object storage for buildkit caching - https://phabricator.wikimedia.org/T323147 (10dancy) [18:37:56] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Scap: Try DigitalOcean registry for buildkit caching - https://phabricator.wikimedia.org/T323148 (10dancy) [18:39:13] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ): Try local directory export + GitLab cache for buildkit caching - https://phabricator.wikimedia.org/T323150 (10dancy) [18:40:11] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ): Try local directory export + GitLab cache for buildkit caching - https://phabricator.wikimedia.org/T323150 (10dancy) [18:40:13] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Scap: Ensure efficient Gitlab CI operations for scap - https://phabricator.wikimedia.org/T323140 (10dancy) [19:20:41] heads up I'm getting "Remote side unexpectedly closed network connection" when trying to SSH into any of the swift hosts (tried some other hosts, they're ok) on the *beta cluster* — starting to diagnose now [19:24:00] ^ hosts were upgraded, I was trying to SSH into the old instance [19:25:15] 10Release-Engineering-Team (Priority Backlog đŸ“„), 10Release, 10Train Deployments: 1.40.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T320517 (10Jdlrobson) [19:26:06] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.40.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T320515 (10brennen) [19:59:29] 10Continuous-Integration-Config, 10MediaWiki-extensions-Linter, 10MediaWiki-extensions-Translate, 10MW-1.40-notes (1.40.0-wmf.4; 2022-10-03), and 2 others: ApiPrefixUniquenessTest::testPrefixes fails because linterstats and languagestats are both using "ls" - https://phabricator.wikimedia.org/T319202 (10dan... [19:59:37] 10Gerrit: Fix up Gerrit duplicate LDAP external ids - https://phabricator.wikimedia.org/T323135 (10hashar) Most of the 199 accounts having an upper case letter are the same as the lower case version (checked with `diff -i`). I though 7 accounts that had a mismatch: Two are a bit specific: `gerrit:Fomafix` lack... [20:19:06] 10Gerrit: Fix up Gerrit duplicate LDAP external ids - https://phabricator.wikimedia.org/T323135 (10hashar) I have pushed a commit to `All-Users.git` to adjust all those accounts (cecd04c3ccd7b40097b753d0c7226b6861c6d5a2). I took a snapshot of the user accounts details via the REST API using: ` lang=bash,name=ac... [20:20:07] !log gerrit: removed legacy mixed case accounts for `gerrit:Fomafix` and `gerrit:Ricordisamoa` T323135#8397539 [20:20:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:20:09] T323135: Fix up Gerrit duplicate LDAP external ids - https://phabricator.wikimedia.org/T323135 [20:21:15] !log gerrit: removed legacy mixed case accounts and moved the extra secondary email to a mailto id for `gerrit:krinkle`, `gerrit:revi`, `gerrit:daniel kinzler`, `gerrit:harej` and `gerrit:samanthanguyen` T323135#8397539 [20:21:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:24:06] $ colordiff -i compare/{lower,upper} [20:24:06] $ [20:24:10] * hashar flexes [20:25:40] I am 0.6294 sure I can delete the 192 remaining ones [20:25:57] but I think I will first try it out with a few canaries accounts tomorrow [20:48:19] 10Continuous-Integration-Config, 10MediaWiki-extensions-Linter, 10MediaWiki-extensions-Translate, 10MW-1.40-notes (1.40.0-wmf.4; 2022-10-03), and 2 others: ApiPrefixUniquenessTest::testPrefixes fails because linterstats and languagestats are both using "ls" - https://phabricator.wikimedia.org/T319202 (10mat... [21:10:22] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ), 10User-brennen: Add DigitalOcean resource monitoring for cloud runner nodes - https://phabricator.wikimedia.org/T308615 (10dduvall) I recently [[ https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/blob/main/git... [21:23:48] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸȘƒ): Provision Horizontal Pod Autoscaler (HPA) for GitLab cloud runners - https://phabricator.wikimedia.org/T323164 (10dduvall) [22:06:54] hashar: thx!