[00:02:50] 10Gerrit: reviewer comments missing on a specific change - https://phabricator.wikimedia.org/T322964 (10ori) ㅤ [01:22:02] teehee :0 [01:22:04] :) [08:41:13] 10Gerrit: reviewer comments missing on a specific change - https://phabricator.wikimedia.org/T322964 (10hashar) Even in Phabricator, vanish @ori comments! More seriously: when looking at the change in the Web UI with all comments expanded, the 6th and 7th comments by Ryan Kaldari are empty but show they increme... [09:14:33] Hello! I'm running (again) into a ssh issue while trying to push to gerrit. I get the error " Too many concurrent connections (4) - max. allowed: 4". I've tried using https for pushing but the authentication is also failing. Would it be possible to get my ssh connections flushed? Ty! [09:56:38] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10wdwb-tech, and 3 others: Wikibase test failures on REL1_39 - https://phabricator.wikimedia.org/T322467 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved a:03Michael Looks like the Math change successfully merged now. [10:21:53] sergi0: I wonder why Gerrit doesn't notice the connections are idling/got terminated [10:21:55] that is strange [10:26:57] and i have no idea how to terminate a ssh connection [10:29:16] hmmm what's your IP sergi0? [10:29:39] hashar: I don't know neither, I changed to use https for fetch after the first time it happened to me as suggested here by taavi. [10:29:43] vgutierrez: 93.176.132.139 [10:29:47] I just want to check what gerrit is reporting VS what the system is reporting [10:29:48] thx [10:30:01] ah `gerrit close-connection` [10:30:22] sergi0: hmm actually you got 4 ESTABLISHED connections according to gerrit1001 [10:30:50] closed them all [10:31:15] hashar: :_) [10:31:19] hard to debug right now then [10:31:25] getting the ssh sessions id from `gerrit show-connections -w` [10:31:35] sorry :-\ [10:31:51] I guess Gerrit missed some disconnect packet [10:31:54] nope [10:32:01] the kernel was reporting those as ESTABLISHED [10:32:24] root@gerrit1001:/var/log/gerrit# netstat -n |grep 93.176.132.139 [10:32:24] tcp6 0 420 208.80.154.137:29418 93.176.132.139:51790 ESTABLISHED [10:32:24] tcp6 0 420 208.80.154.137:29418 93.176.132.139:51788 ESTABLISHED [10:32:24] tcp6 0 420 208.80.154.137:29418 93.176.132.139:51787 ESTABLISHED [10:32:24] tcp6 0 420 208.80.154.137:29418 93.176.132.139:51789 ESTABLISHED [10:32:42] sergi0: how are you using gerrit? [10:32:53] IDE? git CLI? [10:34:34] cli, my suspicion is that it's the git shell prompt but I would expect the connection to close when I close my shell(s) [10:34:51] from logstash that has been going on for a while https://logstash.wikimedia.org/goto/2d5d39ac320e6083bfc6556d0ecd6887 [10:35:07] first occurrence on Sepmteber first [10:37:11] sergi0: maybe you are fetching multiple repositories in parallel over ssh? [10:37:14] sergi0: ouch, what are you rendering on the prompt? [10:37:25] you can check your remotes and verifies the url is set to http instead [10:37:47] just the branch name, eg: git:(review/sergio_gimeno/853378) [10:37:48] cause usually git only performs actions against a remote when you trigger a git fetch or push [10:37:58] sergi0: that should be a local operation [10:44:12] vgutierrez: then I really don't know. Do you know how can I check my open connections on mac os? [10:44:59] AFAIK netstat is available on mac os x [10:45:08] and of course you can use lsof as well [10:53:13] can't find a clue there. I'm uninstalling the only git plugin I have on an IDE (VSCode) and keep monitoring logstash. Thank you for the help! [12:04:49] hi folks! looking for a quick review on a one-line patch: https://gerrit.wikimedia.org/r/c/integration/config/+/855991. not urgent and I promise not to make a habit out of this but it's the last thing in the bullseye upgrade we are doing so! [13:08:42] is this expected? [13:09:00] 10GitLab, 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10serviceops-collab: Build and import new release of jwt-authorizer (1.1.0) - https://phabricator.wikimedia.org/T322691 (10jijiki) [13:09:09] failure running ops/pouppet utils/run_ci_locally.sh https://www.irccloud.com/pastebin/WYgsQJyB/ [13:27:28] 10Project-Admins: Create project tag for MediaWiki-libs-Bcp47Code - https://phabricator.wikimedia.org/T322381 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #bcp47code has been created: https://phabricator.wikimedia.org/project/view/6257/ (In case you need to edit the project or proj... [13:36:20] 10Continuous-Integration-Config, 10translatewiki.net: Run rake tests for translatewiki repo - https://phabricator.wikimedia.org/T323027 (10Nikerabbit) [13:55:02] (03PS1) 10Hashar: Merge tag 'v3.5.4' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) [13:55:57] (03CR) 10KBach: [C: 03+1] [pywikibot/core] increase tox-doc-docker timeout [integration/config] - 10https://gerrit.wikimedia.org/r/855868 (https://phabricator.wikimedia.org/T322979) (owner: 10Xqt) [14:14:50] (03PS1) 10Jforrester: Zuul: [mediawiki/services/function-schemata] Move from node 12 to 16 [integration/config] - 10https://gerrit.wikimedia.org/r/856559 [14:17:24] (03CR) 10Hashar: [C: 03+2] Merge tag 'v3.5.4' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [14:18:01] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/services/function-schemata] Move from node 12 to 16 [integration/config] - 10https://gerrit.wikimedia.org/r/856559 (owner: 10Jforrester) [14:19:46] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/function-schemata] Move from node 12 to 16 [integration/config] - 10https://gerrit.wikimedia.org/r/856559 (owner: 10Jforrester) [14:19:56] !log Zuul: [mediawiki/services/function-schemata] Move from node 12 to 16 [14:19:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:21:20] (03CR) 10CI reject: [V: 04-1] Merge tag 'v3.5.4' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [14:22:17] (03CR) 10Hashar: [C: 03+2] "CI failed due to:" [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [14:26:14] (03CR) 10CI reject: [V: 04-1] Merge tag 'v3.5.4' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [14:38:55] (03CR) 10Hashar: [C: 03+2] Merge tag 'v3.5.4' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [14:42:53] (03CR) 10CI reject: [V: 04-1] Merge tag 'v3.5.4' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [14:54:34] :( [15:04:02] (03CR) 10CI reject: [V: 04-1] Merge tag 'v3.5.4' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [15:04:49] guess will have to debug that [15:09:02] (03CR) 10CI reject: [V: 04-1] Merge tag 'v3.5.4' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [15:39:52] 10Gerrit, 10Release-Engineering-Team (Seen), 10Machine-Learning-Team: gerrit: scoring/ores/editquality takes a long time to git gc - https://phabricator.wikimedia.org/T237807 (10hashar) I ran Tyler's script from `/home/thcipriani/elapsed_gc_time.py` ` gerrit1001:~$ python /home/thcipriani/elapsed_gc_time.py|... [15:42:19] 10GitLab (Infrastructure), 10serviceops, 10serviceops-collab: Reduce usage of public IPv4 addresses on GitLab hosts - https://phabricator.wikimedia.org/T310265 (10Jelto) @Volans , @ayounsi , @cmooney , @BBlack and I had a chat about this topic during the SRE summit. We talked about multiple options which wo... [15:44:25] 10Gerrit, 10Release-Engineering-Team (Seen), 10Machine-Learning-Team: gerrit: scoring/ores/editquality takes a long time to git gc - https://phabricator.wikimedia.org/T237807 (10thcipriani) >>! In T237807#8392885, @hashar wrote: > I ran Tyler's script from `/home/thcipriani/elapsed_gc_time.py` > ` > gerrit10... [15:53:47] (03CR) 10Hashar: [C: 03+2] "I have manually populated the cache in Castor for wmf/stable-3.5" [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [15:57:52] (03CR) 10CI reject: [V: 04-1] Merge tag 'v3.5.4' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [15:58:23] * hashar whistles [16:04:24] 10GitLab (Infrastructure), 10serviceops, 10serviceops-collab: Reduce usage of public IPv4 addresses on GitLab hosts - https://phabricator.wikimedia.org/T310265 (10thcipriani) Thanks for the detailed write up as always @Jelto 🎉 >>! In T310265#8392890, @Jelto wrote: > For this and the previous option I have... [16:09:36] 10GitLab (Infrastructure), 10serviceops, 10serviceops-collab: Reduce usage of public IPv4 addresses on GitLab hosts - https://phabricator.wikimedia.org/T310265 (10dancy) [16:21:01] 10GitLab (Infrastructure), 10serviceops, 10serviceops-collab: Reduce usage of public IPv4 addresses on GitLab hosts - https://phabricator.wikimedia.org/T310265 (10dancy) Seconding what @thcipriani said, I'm strongly against disabling git over ssh. Using HTTP only requires plaintext passwords to be stored on... [16:28:10] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (GitLab III: GitLab in LA đŸĒƒ): Automate integration Jenkins deployment and config changes - https://phabricator.wikimedia.org/T319406 (10thcipriani) [16:54:18] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Release, 10Train Deployments, 10User-brennen: 1.40.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T320515 (10brennen) [16:55:29] (03PS1) 10Zfilipin: WIP Fix link to credential store [integration/config] - 10https://gerrit.wikimedia.org/r/856613 (https://phabricator.wikimedia.org/T323047) [17:07:25] (03PS1) 10Zfilipin: WIP Run Selenium tests targeting production [integration/config] - 10https://gerrit.wikimedia.org/r/856618 (https://phabricator.wikimedia.org/T323047) [17:23:25] (03PS1) 10Zfilipin: WIP cleanup selenium daily job [integration/config] - 10https://gerrit.wikimedia.org/r/856622 (https://phabricator.wikimedia.org/T323047) [17:25:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Castor cache is ineffective due to a split brain - https://phabricator.wikimedia.org/T323051 (10hashar) [17:28:38] grbmbm [17:30:06] (03PS1) 10Hashar: jjb: save Castor caches at /srv/cache [integration/config] - 10https://gerrit.wikimedia.org/r/856626 (https://phabricator.wikimedia.org/T323051) [17:30:27] hashar: Incremental progress! [17:30:38] also, hello. [17:33:09] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10serviceops-collab, 10Patch-For-Review: decom phab2001 - https://phabricator.wikimedia.org/T322250 (10Papaul) [17:34:45] !log Changing Castor cache saving from `/srv/jenkins-workspace/caches/` to `/srv/cache/caches/` which is the one served by rsync. [17:34:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:35:58] !log Changing Castor cache saving from `/srv/jenkins-workspace/caches/` to `/srv/cache/caches/` which is the one served by rsync T323051 [17:36:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:36:00] T323051: Castor cache is ineffective due to a split brain - https://phabricator.wikimedia.org/T323051 [17:36:17] !log Nuking unused Castor cached files in `/srv/jenkins-workspace/caches` # T323051 [17:36:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:37:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Cloud-VPS (Debian Stretch Deprecation), 10Patch-For-Review: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10hashar) Part of this task reloc... [17:38:07] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/856626 (https://phabricator.wikimedia.org/T323051) (owner: 10Hashar) [17:39:49] (03Merged) 10jenkins-bot: jjb: save Castor caches at /srv/cache [integration/config] - 10https://gerrit.wikimedia.org/r/856626 (https://phabricator.wikimedia.org/T323051) (owner: 10Hashar) [17:40:55] dancy: yeah looks like I have broke the CI infra back in March :-\\\\ [17:41:00] well partly [17:41:18] and nobody noticed! [17:44:48] 10Continuous-Integration-Infrastructure, 10Jenkins: Jenkins plugins security advisory - 2022-11-15 - https://phabricator.wikimedia.org/T323054 (10hashar) [17:45:01] 10Continuous-Integration-Infrastructure, 10Jenkins: Jenkins plugins security advisory - 2022-11-15 - https://phabricator.wikimedia.org/T323054 (10hashar) [17:47:30] so the short story is [17:47:51] as part of phasing out Stretch WMCS instances back in March, I have rebuild the integration-castor03 instance to integration-castor05 [17:48:05] and AT THE SAME TIME , I have dediced to move the cache storage to a new directory /srv/cache [17:48:19] the puppet patch, cinder volume magic etc got all deployed and I marked the task fixed [17:48:21] two issues: [17:48:45] 1) I should have created a dedicated task and split the Stretch > Buster migration from the file path relocation [17:49:05] 2) the rsync servers serves caches from `/srv/caches` which the Jenkins job saves them to /srv/jenkins-workspace/caches [17:49:17] end result: the caches have been stall since ~ March [17:49:30] expect CI to have some slow down this evening as caches are being repopulated [17:49:41] notably jobs will be waiting for https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ to complete [17:49:53] and that one is a bottleneck cause only one triggers at a time [17:51:12] I can see `/srv/castor/caches/operations-software-gerrit/wmf-stable-3.5/gerrit-docker/` now [18:00:10] of course I screwed it up [18:13:26] (03CR) 10Hashar: [C: 03+2] "Also found out Castor cache was broken and fixed it with T323051 . There should be a fully populated cache now." [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [18:13:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Castor cache is ineffective due to a split brain - https://phabricator.wikimedia.org/T323051 (10hashar) 05Open→03Resolved a:03hashar I think it is good now :-] [18:19:33] (03Merged) 10jenkins-bot: Merge tag 'v3.5.4' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/856555 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [18:23:59] finally merged ;) [18:24:06] dinner & [18:26:45] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Security: Jenkins plugins security advisory - 2022-11-15 - https://phabricator.wikimedia.org/T323054 (10Reedy) [19:10:43] (03PS1) 10Stef Dunlap: WikiLambda: run e2e tests daily on betacluster [integration/config] - 10https://gerrit.wikimedia.org/r/856646 (https://phabricator.wikimedia.org/T294388) [19:12:21] (03CR) 10CI reject: [V: 04-1] WikiLambda: run e2e tests daily on betacluster [integration/config] - 10https://gerrit.wikimedia.org/r/856646 (https://phabricator.wikimedia.org/T294388) (owner: 10Stef Dunlap) [19:44:35] 10Beta-Cluster-Infrastructure: Unable to confirm email address on beta cluster - https://phabricator.wikimedia.org/T285527 (10LucasWerkmeister) So if you have shell access, you can work around this / T291679 using: `lang=sh mwscript resetUserEmail.php metawiki 'Lucas Werkmeister CI' 'wikimediaci@lucaswerkmeiste... [20:03:48] (03PS2) 10Stef Dunlap: WikiLambda: run e2e tests daily on betacluster [integration/config] - 10https://gerrit.wikimedia.org/r/856646 (https://phabricator.wikimedia.org/T294388) [20:05:39] (03CR) 10CI reject: [V: 04-1] WikiLambda: run e2e tests daily on betacluster [integration/config] - 10https://gerrit.wikimedia.org/r/856646 (https://phabricator.wikimedia.org/T294388) (owner: 10Stef Dunlap) [21:00:03] (03PS4) 10Hashar: Gerrit v3.5.4 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/824200 (https://phabricator.wikimedia.org/T307334) [21:00:31] (03CR) 10CI reject: [V: 04-1] Gerrit v3.5.4 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/824200 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [21:12:32] 10Release-Engineering-Team, 10serviceops-collab: reconsider https://git.wikimedia.org link - https://phabricator.wikimedia.org/T323073 (10Dzahn) [21:16:49] (03CR) 10Hashar: "recheck after having uploaded artifacts to Archiva" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/824200 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [21:43:05] 10Phabricator, 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) after https://gerrit.wikimedia.org/r/c/operations/puppet/+/856013 set the mysql_port to 3323 we still c... [21:56:31] (03CR) 10Jforrester: "We don't allow production code to disable selenium tests on merge as part of our development policy, but we could waive it here. Will need" [integration/config] - 10https://gerrit.wikimedia.org/r/856646 (https://phabricator.wikimedia.org/T294388) (owner: 10Stef Dunlap) [22:05:31] maintenance-disconnect-full-disks build 439045 integration-agent-docker-1023 (/: 29%, /srv: 95%, /var/lib/docker: 45%): OFFLINE due to disk space [22:10:29] maintenance-disconnect-full-disks build 439046 integration-agent-docker-1023 (/: 29%, /srv: 31%, /var/lib/docker: 45%): RECOVERY disk space OK [22:17:04] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸĒƒ), 10Release, 10Train Deployments, 10User-brennen: 1.40.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T320515 (10Jdlrobson) ##### Risky Patch! 🚂đŸ”Ĩ * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/854129/ * **Su... [22:37:51] thcipriani: tldr our All-Users.git is borked ;) [22:38:08] I was testing against a local copy of All-Users.git which only has a few references [22:38:18] against the real one, the script explodes quickly :] [22:40:49] neat [22:41:01] well. glad you were able to test it anyway. [23:04:37] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Gerrit (Gerrit 3.5), 10Patch-For-Review: Upgrade to Gerrit 3.5 - https://phabricator.wikimedia.org/T307334 (10hashar) The major change which affects us is normalizing username to be all lower case. I initially gave it a try with a local checkout of the `All-... [23:10:36] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Gerrit (Gerrit 3.5), 10Patch-For-Review: Upgrade to Gerrit 3.5 - https://phabricator.wikimedia.org/T307334 (10hashar) The reason for those accounts originates from when we turned on `ldap.localUsernameToLowerCase` T152640, T197083, T197257 etc. The upstre... [23:13:52] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Gerrit (Gerrit 3.5), 10Patch-For-Review: Upgrade to Gerrit 3.5 - https://phabricator.wikimedia.org/T307334 (10hashar) My plan: * Ensure all 199 `gerrit:` users having an upper case character have a matching all lower case `gerrit:` entry. * `git rm` the fil... [23:14:24] thcipriani: yeah and thanks for all the details in the task! [23:14:29] I found the script you wrote originally https://phabricator.wikimedia.org/T307334#8394491 [23:14:43] note sure why it did not delete the old files, apparently it skips when a target file exists [23:14:59] I will create a dry run copy of that script and see what happens :-] [23:15:24] I am happy to have thoroughly read the release notes and verified them \o/ [23:16:30] yeah, looks like if the new file exists, it bails. Otherwise it renames the user. [23:17:04] TIL about bash syntax `${variable,,}` (convert to lower case) [23:17:13] so there are likely old gerrit:MixedCaseAccounts if they have a cooresponding gerrit:mixedcaseaccouts [23:17:57] I definitely did not remember ${variable,,} :) [23:19:07] 42 if [ -f "$new_file" ]; then [23:19:07] 43 println "The new file '${new_file}' exists!!!! for '${username}'. Aborting!" [23:19:23] gerrit would be a lot happier if we were using uid instead of cn as the LDAP foreign key [23:19:25] your script skipped when gerrit:Foo already had a gerrit:foo [23:19:45] uid == unix shell name [23:19:45] in which case instead of aborting it should have deleted the erroneous `gerrit:Foo` [23:20:11] yeah that would be worth pursuing [23:20:44] the sn/cn/unix mess is a bit intimidating [23:21:07] as a bonus, gitlab uses uid so it would get folks used to logging into git/code review using their uid instead of their cn. [23:21:13] I think I came to the same conclusion about uid. My alternate conclusion at the time was I didn't want to try it since the benefits were slight and the risks seemed high. [23:21:50] but maybe there are more benefits piling up all the time [23:22:07] * bd808 remembers that the uid login is for gitlab wired to LDAP directly which we aren't using [23:22:34] it's for CAS which gitlab is...deprecating. :\ [23:22:40] and maybe drop the direct LDAP in favor of oauth via wikitech [23:23:10] we are the largest open instance of gerrit NOT using oauth as far as I'm aware [23:24:00] hashar: don't count on wikitech remaining an LDAP powered wiki please. [23:24:02] which is why we were the only instance with a giant memory leak between versions 2.18->3.2, IIRC :) [23:24:35] hmm slightly [23:24:58] idp.wikimedia.org is the "safe" thing to integrate with for Developer account authn [23:25:01] the leak was due to anonymous git http causing the creation of jetty sessions/monitoring tracker for each git fetches [23:25:23] which pilled up quickly due to so many bots / anonymous fetches hitting us [23:25:47] yeah idp, then it doesn't have the wikitech/LDAP accounts does it? [23:27:29] idp is just a wrapper service over the LDAP directory [23:28:27] but it is a wrapper service intended to provide SSO for Developer accounts. Wikitech is working towards being able to be converted into a "normal" SUL wiki [23:29:08] that's not happening quickly, but I would really hate to see a new thing become dependent on wikitech as a Developer account authn system [23:29:18] yeah [23:29:39] I had the idea of moving Gerrit to OAuth and use Wikitech for that [23:29:44] but that sounds counterproductive [23:30:07] then idp.wikimedia.org iirc I asked about it for Gerrit and that got rejected cause Gerrit has too many users [23:30:17] maybe things have changed now and I could ask again [23:30:28] gitlab uses it so... [23:30:38] +1 :) [23:31:30] (03CR) 10Hashar: "recheck" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/824200 (https://phabricator.wikimedia.org/T307334) (owner: 10Hashar) [23:39:48] ah yeah https://gerrit.googlesource.com/plugins/oauth/+/refs/heads/master/src/main/resources/Documentation/config.md#cas-oauth [23:39:51] guess I will ask [23:43:55] 10Continuous-Integration-Config, 10MediaWiki-extensions-Linter, 10MediaWiki-extensions-Translate, 10MW-1.40-notes (1.40.0-wmf.4; 2022-10-03), and 2 others: ApiPrefixUniquenessTest::testPrefixes fails because linterstats and languagestats are both using "ls" - https://phabricator.wikimedia.org/T319202 (10mat...