[08:43:19] <wikibugs>	 10Release-Engineering-Team, 10castor, 10ci-test-error: Multiple extensions' CI blocked again by cache corruption error in mwgate-node16-docker - https://phabricator.wikimedia.org/T352305 (10hashar)
[08:43:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 10castor: npm cache saved by castor get corrupted for unknown reason - https://phabricator.wikimedia.org/T295351 (10hashar)
[08:43:42] <wikibugs>	 10Continuous-Integration-Infrastructure, 10castor: Castor rsync causes: rsync: failed to set times on "/cache/.": Operation not permitted (1) - https://phabricator.wikimedia.org/T188488 (10hashar) >>! In T188488#9475158, @Muhammad_Yasser_Jazirahly_WMDE wrote: > Hello, this issue seems to be blocking Wikibase C...
[08:46:58] <hashar>	 !log integration-castor05: `sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwgate-node18-docker` # T295351
[08:47:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:47:02] <stashbot>	 T295351: npm cache saved by castor get corrupted for unknown reason - https://phabricator.wikimedia.org/T295351
[08:51:30] <wikibugs>	 10Continuous-Integration-Infrastructure: Spurious job failures - https://phabricator.wikimedia.org/T355000 (10hashar)
[08:51:33] <wikibugs>	 10Continuous-Integration-Infrastructure: Spurious job failures - https://phabricator.wikimedia.org/T355000 (10hashar) The very first lines of output are: ` + ls -laF --color=always drwxr-xr-x  2 root           root    4096 Jan 13 01:55 cache/ `  The `cache` directory is owned by `root` (while the containers run...
[08:51:53] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Priority Backlog 📥), 10Upstream, 10ci-test-error: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T282893 (10hashar)
[09:11:06] <wikibugs>	 10Release-Engineering-Team, 10Gerrit (Gerrit 3.7): Upgrade to Gerrit 3.7 - https://phabricator.wikimedia.org/T354885 (10hashar) ` $ java -jar bin/gerrit.war init --batch --skip-plugins Migrating data to schema 185 ... Migrating label configurations ... using 48 threads ... ... (0.609 s) migrated label configur...
[09:13:01] <wmf-insecte>	 Project beta-code-update-eqiad build #480036: 04FAILURE in 0.71 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/480036/
[09:17:35] <wikibugs>	 10Release-Engineering-Team, 10Gerrit (Gerrit 3.7): Upgrade to Gerrit 3.7 - https://phabricator.wikimedia.org/T354885 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e51413d8-1282-4a8d-9ae6-23fe479263fe) set by cgoubert@cumin1002 for 4:00:00 on 2 host(s) and their services with reason: Gerri...
[09:23:01] <wmf-insecte>	 Project beta-code-update-eqiad build #480037: 04STILL FAILING in 0.69 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/480037/
[09:23:39] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Gerrit commit not found - https://phabricator.wikimedia.org/T355504 (10hashar)
[09:25:05] <wikibugs>	 10Release-Engineering-Team, 10Gerrit (Gerrit 3.7): Upgrade to Gerrit 3.7 - https://phabricator.wikimedia.org/T354885 (10hashar) ` Reindexing changes: changes: 99% (979933/980136), failed: 203, project-slices: 100% (3240/3240), Slicing projects: 100% (2862/2862), done     [2024-01-22 09:22:44,510] [main] WARN...
[09:32:37] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10awight)
[09:33:01] <wmf-insecte>	 Project beta-code-update-eqiad build #480038: 04STILL FAILING in 0.68 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/480038/
[09:39:26] <TheresNoTime>	 `fatal: unable to access 'https://gerrit.wikimedia.org/r/operations/mediawiki-config/': The requested URL returned error: 503`
[09:39:29] <TheresNoTime>	 that's new
[09:40:40] <ihurbain>	 hashar mentioned he was updating gerrit about now, i suppose it's related/still in progress
[09:41:19] <apergos>	 yeah I got one of those, went away on reload
[09:42:06] <wmf-insecte>	 Yippee, build fixed!
[09:42:07] <wmf-insecte>	 Project beta-code-update-eqiad build #480039: 09FIXED in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/480039/
[09:42:29] <TheresNoTime>	 looks like trying again might have worked :D
[09:42:58] <ihurbain>	 :magic:
[09:43:15] <wikibugs>	 10Release-Engineering-Team, 10Gerrit (Gerrit 3.7): Upgrade to Gerrit 3.7 - https://phabricator.wikimedia.org/T354885 (10hashar) 05Open→03Resolved
[09:44:53] <hashar>	 TheresNoTime: ihurbain: apergos yeah that was definitely me upgrading Gerrit
[09:44:59] <hashar>	 it is back now
[09:45:04] <apergos>	 \o/
[09:45:11] <hashar>	 with a new brok^H^H^H^Hrelease
[09:45:15] <apergos>	 heh
[09:45:23] <apergos>	 gonna check out the new features!
[09:45:28] <hashar>	 more seriously, it is well tested nowadays
[09:45:37] <hashar>	 they have a nice acceptance test suite
[09:45:40] <TheresNoTime>	 cool cool :D
[09:45:52] * TheresNoTime only notices when https://www.isbetabroken.com complains
[09:46:12] <ihurbain>	 i'm happy i remembered seeing your notifications from last week, that made me Not Panic and Not Get Grumpy when I got "ssh: connect to host gerrit.wikimedia.org port 29418: Connection refused" first thing in the morning :D
[09:46:14] <hashar>	 and since we upgrade  from a later patchset version, it has been deployed by a wide range of setup already
[09:46:41] <ihurbain>	 (it's indeed working now)
[09:47:06] * Lucas_WMDE will have to get used to the apparently new dark scheme
[09:47:47] <TheresNoTime>	 night mode gerrit??
[09:48:29] <TheresNoTime>	 oooh, nope no thank you :P
[09:49:12] <apergos>	 huh I am still logged in, that's a nice surprise
[09:52:01] <hashar>	 yeah that is automatic now
[09:52:06] <hashar>	 based on system preferences
[09:52:22] <hashar>	 https://www.gerritcodereview.com/3.7.html#auto-theme-mode
[09:52:26] <hashar>	 "The theme is automatically changed to light/dark based on your OS preference. You can change the theme in users preferences.
[09:52:27] <hashar>	 "
[09:52:46] <hashar>	 I am guessing it is a gift from paladox :)
[09:58:00] <wikibugs>	 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for Michaelcochez - https://phabricator.wikimedia.org/T355494 (10Michaelcochez) @thcipriani  can you approve my account for gitlab, so I can complete https://phabricator.wikimedia.org/T344746
[10:02:43] <wikibugs>	 10Release-Engineering-Team, 10Data-Engineering, 10Gerrit (Gerrit 3.7), 10Upstream: git clone and git pull commands fail for refinery repo - https://phabricator.wikimedia.org/T355173 (10hashar) 05Stalled→03Resolved a:03hashar I went to leave it as is over the week-end knowing upgrading Gerrit was on t...
[10:03:17] <apergos>	 mine is light themed which works fine for me
[10:12:30] <MichaelG_WMDE>	 mh, I get the following modal when trying to open my Gerrit dashboard: `Error 400 (Bad Request): Unrecognized value: ignored Endpoint: /changes/`Is that known?
[10:15:56] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10Bugreporter)
[10:18:09] <Lucas_WMDE>	 feels like Gerrit emails might be a bit slow at the moment… the “successfully merged” email I got for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/991396 (happened 10:05) was apparently only received around 10:15?
[10:18:42] <Lucas_WMDE>	 it says `Date: Mon, 22 Jan 2024 10:05:34 +0000`, but `Received: from [127.0.0.1] … by gerrit1003 … Mon, 22 Jan 2024 10:15:55 +0000`
[10:18:59] <Lucas_WMDE>	 it might just be nothing, but I thought I’d mention it… let’s see if it keeps happening I guess
[10:19:39] <MichaelG_WMDE>	 strange, unchecking all the boxes in Settings/"Change Table Columns", saving, and re-checking them caused the error to go away it seems.
[10:20:22] <MichaelG_WMDE>	 Maybe for me just some outdated cache lingering somewhere?
[10:20:25] * MichaelG_WMDE shrugs
[10:25:44] <wikibugs>	 10Gerrit, 10Release-Engineering-Team, 10collaboration-services: Refine alert manager silences for Gerrit upgrades - https://phabricator.wikimedia.org/T355516 (10hashar)
[10:26:48] <Lucas_WMDE>	 post-merge build email also had a 10-minute delay between localhost and gerrit1003
[10:31:17] <ihurbain>	 hashar: the "Depends on" field stays empty even when there's a depends-on line. (see for instance https://gerrit.wikimedia.org/r/c/translatewiki/+/991281 or https://gerrit.wikimedia.org/r/c/mediawiki/core/+/992094). I THINK this is is a display issue and not a proper issue because the dependent patch gets a "Needed by" field (... also empty), so there's something happening, but.
[10:31:25] <hashar>	 oh joy
[10:33:57] <hashar>	 MichaelG_WMDE: Lucas_WMDE: for the email lags, I don't know but they show up at https://grafana.wikimedia.org/d/Zh_ncGsWk/queues-upstream?orgId=1&refresh=1m&viewPanel=14 
[10:34:03] <hashar>	 and it looks like the queue is processing 
[10:34:18] <hashar>	 at leaast I don't see email tasks pending
[10:34:23] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10Bawolff) An interesting point is that its not the same zordzord but different zordzord captchas. The fact it starts with z suggests it might be near the...
[10:35:31] <hashar>	 ihurbain: je vais regarder
[10:35:34] <hashar>	 ça sent le rollback :-\
[10:35:40] <ihurbain>	 :(
[10:36:08] <hashar>	 et après je shutdown toute la stack
[10:36:12] <wikibugs>	 10Project-Admins: Create a workboard/project for #Tool-link-dispenser - https://phabricator.wikimedia.org/T355518 (10Soda)
[10:37:13] <ihurbain>	 also, in the "vague annoyance" category, I'm fairly sure that when I used to click on the pencil in the "Reviewers" column, the cursor would go in the popup, like, in the Reviewer field, and I could start typing people's names there. now it goes to the "Comment" field so I end up commenting people's names :P 
[10:37:50] <hashar>	 ihurbain: yeah that was is intentional unfortunately
[10:37:57] <ihurbain>	 >_<
[10:38:07] <hashar>	 :-]
[10:38:33] <hashar>	 I am going to file the task about hte depends-on issue
[10:38:58] <hashar>	 which would not affect Zuul/CI itself (which instead does a search query)
[10:39:07] <hashar>	 but might have affect scap backport which relies on that API
[10:40:15] <ihurbain>	 want me to test that? i'm supposed to self-+2 our core patch so that it gets merged at the same time as the vendor patch during our vendor patch process; i haven't done that because i saw the discrepancy before i clicked, but i can tentatively click and add a -2 if it actually tries to start the merge
[10:40:31] <wikibugs>	 10Phabricator, 10Striker: Phabricator workboard creation is broken - https://phabricator.wikimedia.org/T355519 (10Soda)
[10:40:51] <wikibugs>	 10Phabricator, 10Striker: Phabricator workboard creation is broken - https://phabricator.wikimedia.org/T355519 (10taavi) If I had to guess this is caused by the change to require a description on all newly created workboards.
[10:41:45] <wikibugs>	 10Phabricator, 10Striker: Phabricator workboard creation is broken - https://phabricator.wikimedia.org/T355519 (10taavi)
[10:43:30] <hashar>	 ihurbain: for sure:
[10:43:31] <hashar>	 !
[10:43:40] <wikibugs>	 10Project-Admins: Create a workboard/project for #Tool-link-dispenser - https://phabricator.wikimedia.org/T355518 (10taavi) a:03taavi
[10:43:42] <hashar>	 or let met at least capture some trace
[10:44:05] <hashar>	 ah yeah that does not show up 
[10:44:06] <hashar>	 fun
[10:44:36] <hashar>	 TypeError: Gerrit.Nav is undefined
[10:44:36] <ihurbain>	 waiting for CI to go to +2 before I slap it
[10:44:51] <ihurbain>	 so yeah, display error?
[10:46:58] <wikibugs>	 10Project-Admins: Create a workboard/project for #Tool-link-dispenser - https://phabricator.wikimedia.org/T355518 (10taavi) 05Open→03Resolved Done as #tool-link-dispenser.
[10:47:35] <hashar>	 ihurbain: yeah I think
[10:47:41] <hashar>	 CI / Zuul is not affected anyway
[10:47:44] <hashar>	 it does not rely on that API
[10:47:55] <hashar>	 I am filing the task
[10:47:55] <ihurbain>	 k
[10:51:02] <ihurbain>	 hashar: in case that's relevant: I submitted my depended-on patch after the depending patch (I expect this is not the case for the translatewiki patch I linked) and, before I uploaded the depending patch, it was showing "can't find the depends-on patch in gerrit", or something like that
[10:51:21] <ihurbain>	 so it was able to display something at that time :P
[10:53:25] <hashar>	 ihurbain: that is a javascript API that got removed in Gerrit 3.7 but the Gerrit plugin hasn't been updated to match it
[10:53:38] <hashar>	 well as I get it
[10:53:44] <wikibugs>	 10Gerrit: Gerrit Zuul plugin does not show Depends-On/Needed-By since Gerrit 3.7 - https://phabricator.wikimedia.org/T355521 (10hashar)
[10:53:50] <hashar>	 here is the task
[10:54:20] <ihurbain>	 wunderbar
[10:54:45] <wikibugs>	 10Phabricator, 10Striker: Phabricator workboard creation is broken - https://phabricator.wikimedia.org/T355519 (10Peachey88) I was able to do it on #Tool-link-dispenser, I don't have anything other than #trusted-contributors similar to @Soda.  @Soda what project did you attempt this one?
[10:55:41] <hashar>	     Replace `GerritNav.navigateToRelativeUrl()` by new `setUrl()` service
[10:55:41] <hashar>	     
[10:55:41] <hashar>	     This was the last method. Let's remove `GerritNav` completely. :-)
[10:55:46] <hashar>	 Release-Notes: skip
[10:55:48] <hashar>	 \o/
[10:57:43] <ihurbain>	 i can confirm that my +2'd patch with a depends-on patch that's not merged yet is staying put and not triggering anything.
[10:58:32] <wikibugs>	 10Gerrit, 10Upstream: Gerrit Zuul plugin does not show Depends-On/Needed-By since Gerrit 3.7 - https://phabricator.wikimedia.org/T355521 (10hashar) Caused by 54934debee842065a9716c88db4a1ff441dfd188 / https://gerrit-review.googlesource.com/c/gerrit/+/346479 //Replace `GerritNav.navigateToRelativeUrl()` by new...
[10:59:33] <hashar>	 ihurbain: yeah as expected more or less
[11:01:25] <wikibugs>	 10Phabricator, 10Striker: Phabricator workboard creation is broken - https://phabricator.wikimedia.org/T355519 (10Peachey88) Wait, just confirming if you were having trouble creating a Phabricator Project or the workboard on the project?
[11:03:10] <ihurbain>	 "trust, but verify" :P :D
[11:05:06] <hashar>	 !log Branched stable-3.7 on upstream https://gerrit.googlesource.com/plugins/zuul # T355521
[11:05:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[11:05:09] <stashbot>	 T355521: Gerrit Zuul plugin does not show Depends-On/Needed-By since Gerrit 3.7 - https://phabricator.wikimedia.org/T355521
[11:12:24] <paladox>	 hashar: you’ll need to use window.location.href
[11:12:39] <paladox>	 At least that’s what delete-project was converted to use
[11:12:58] <hashar>	 to forge an url?
[11:13:10] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10Lucas_Werkmeister_WMDE)
[11:13:18] <hashar>	 looks like all the logics to create urls got moved behind typescript and is not exposed in the Gerrit Javascript API unfortunately
[11:13:26] <paladox>	 Yeh
[11:13:36] <paladox>	 https://github.com/GerritCodeReview/plugins_delete-project/commit/7dce6f70611cd8dbf1d38628698155258ee8ef82
[11:13:40] <hashar>	 the zuul plugin has  return Gerrit.Nav.getUrlForSearchQuery(changeInfo.change_id);
[11:13:40] <hashar>	 ;)
[11:19:51] <paladox>	 That’s just /q/<query>
[11:20:17] <hashar>	 yeah it is quite simple
[11:20:32] <paladox>	 https://github.com/GerritCodeReview/plugins_delete-project/blob/master/web/gr-delete-repo.ts#L169
[11:20:33] <hashar>	 I am browsing the code to find whether there is an API to craft an URL though
[11:20:43] <paladox>	 Don’t forget the base url function :)
[11:20:50] <hashar>	 without having to convert the plugin to typescript
[11:21:07] <paladox>	 Oh
[11:44:02] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10TheresNoTime) If it helps as a jumping off point, I did a [[ https://beta-logs.wmcloud.org/app/discover#/context/5f0c9be0-0b6f-11ec-9cde-3f4490e09a26/ntj...
[11:46:37] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10Ladsgroup) I think there is something really broken about swift handling in beta cluster. I ran the script again. Does it work now?
[11:51:06] <Lucas_WMDE>	 email delay seems to be a lot better now fwiw
[11:51:22] <Lucas_WMDE>	 (https://grafana.wikimedia.org/d/Zh_ncGsWk/queues-upstream?orgId=1&refresh=1m&viewPanel=14 also looks healthier again)
[11:52:15] <hashar>	 Lucas_WMDE: yeah I am not sure what has happened to be fair :\
[11:52:51] <Lucas_WMDE>	 :S
[11:53:07] <Lucas_WMDE>	 I guess a mysterious problem that resolves itself is at least better than a mysterious problem that stays around
[11:57:36] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10Ladsgroup) >>! In T355505#9476303, @Ladsgroup wrote: > I think there is something really broken about swift handling in beta cluster. I ran the script ag...
[11:58:54] <hashar>	 Lucas_WMDE: if you have the source of an email that lagged, I guess I can look in the exim queue and see how it processed it
[11:59:06] <hashar>	 but my guess is that the lags comes from the Gerrit side
[12:00:41] <wikibugs>	 10Phabricator, 10Striker, 10cloud-services-team: Striker can't create Phabricator projects - https://phabricator.wikimedia.org/T355519 (10taavi)
[12:00:45] <wikibugs>	 10Phabricator, 10Striker, 10cloud-services-team: Striker can't create Phabricator projects - https://phabricator.wikimedia.org/T355519 (10taavi) a:03taavi
[12:01:06] <Lucas_WMDE>	 hashar: https://phabricator.wikimedia.org/P55194 (private because I couldn’t be bothered to check if there’s anything sensitive in there)
[12:01:12] <Lucas_WMDE>	 (I can have another look later, afk now)
[12:01:19] <hashar>	 danke
[12:02:11] <hashar>	 those email sources remember me my very first job at an ISP
[12:02:14] <Lucas_WMDE>	 hm, wait, I’m not sure I did that for the right email though
[12:02:19] <hashar>	 which involved parsing emails headers for a living :D
[12:02:52] <hashar>	 if you have to lunch, that can wait
[12:02:54] <hashar>	 don't worry :)
[12:03:11] <Lucas_WMDE>	 well, I have a potential meeting before lunch, and so far nobody else is there ^^
[12:03:20] <Lucas_WMDE>	 so I can check for a few minutes while I wait if anyone else shows up
[12:04:50] <Lucas_WMDE>	 okay, https://phabricator.wikimedia.org/P55196 is the email with Date: … 10:05
[12:05:01] <Lucas_WMDE>	 that’s the “gate pipeline build succeeded” email
[12:05:24] <Lucas_WMDE>	 whereas “jenkins-bot submitted this change” was the earlier paste, and that one somehow has Date: … 10:15?
[12:06:00] <Lucas_WMDE>	 maybe gerrit delayed sending out the “submitted” email while “build succeeded” was… waiting somehow?
[12:06:06] <Lucas_WMDE>	 very strange
[12:06:40] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10LSobanski) Right now I see different captchas when reloading the page or hitting "refresh" in the captcha section.
[12:08:34] <hashar>	 Lucas_WMDE: yeah I don't haev much info from that email which looks all fine as far as Exim is concerned
[12:08:41] <hashar>	 so that is definitely Gerrit that lagged out for some reason
[12:08:45] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10LSobanski) However, account creation fails with "Incorrect or missing CAPTCHA".
[12:08:45] <Lucas_WMDE>	 ok
[12:09:09] <hashar>	 maybe it does not process as fast as it should o
[12:09:14] <hashar>	 or there is an initial delay
[12:09:18] <hashar>	 or it had too many at the same time
[12:09:41] <hashar>	 I am filing a task to investigate that later on
[12:09:52] <Lucas_WMDE>	 ok, thanks!
[12:10:21] <Lucas_WMDE>	 as far as I understand the email headers, it should be fine to publish the emails as far as WMDE’s gmail is concerned, I think
[12:10:41] <Lucas_WMDE>	 surely the crypto is resilient to the occasional signature being published
[12:11:07] * Lucas_WMDE handwaves in the direction of known-plaintext attacks
[12:14:43] <wikibugs>	 10Phabricator, 10Striker, 10cloud-services-team, 10Patch-For-Review: Striker can't create Phabricator projects - https://phabricator.wikimedia.org/T355519 (10Soda) >>! In T355519#9476166, @Peachey88 wrote: > I was able to do it on #Tool-link-dispenser, I don't have anything other than #trusted-contributors...
[12:16:23] <wikibugs>	 10Gerrit: Investigate Gerrit email delay following 3.7.6  upgrade - https://phabricator.wikimedia.org/T355529 (10hashar)
[12:17:03] <hashar>	 hmm
[12:17:16] <hashar>	 I am not sure how to read the various graphs I have pasted :D
[12:17:36] <hashar>	 maybe it needs more threads
[12:19:47] <wikibugs>	 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10serviceops: Allow release engineering to delete images - https://phabricator.wikimedia.org/T354786 (10Clement_Goubert) p:05Triage→03Low
[12:41:36] <wikibugs>	 10Gerrit: Investigate Gerrit email delay following 3.7.6  upgrade - https://phabricator.wikimedia.org/T355529 (10hashar) 05Open→03Resolved a:03hashar Gerrit went up on at 09:36:05: ` [2024-01-22T09:36:05.415Z] [main] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.7.6 ready `  Gerrit email config...
[12:41:43] <hashar>	 Lucas_WMDE: I think the delay was due to Gerrit 3.7 starting with no caches at all
[12:41:54] <hashar>	 which slowed down everything until they got populated
[12:41:58] <hashar>	 well that is my assumption
[12:47:21] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, 10serviceops: Move 40% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T355532 (10Clement_Goubert)
[12:48:38] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, 10serviceops: Move 40% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T355532 (10Clement_Goubert) p:05Triage→03High
[13:16:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10Ladsgroup) When the main changes got merged, I deployed them and they were okay but then we did some cleanups and now it seems broken. I think something...
[13:21:43] <wikibugs>	 10Continuous-Integration-Infrastructure, 10EntitySchema, 10Wikidata, 10wmde-wikidata-tech, and 2 others: EntitySchema CI broken on master with OutOfMemory error in browser test - https://phabricator.wikimedia.org/T354179 (10ItamarWMDE) Did you create a task for that investigation? If not, can you please cr...
[13:28:39] <wikibugs>	 10Release-Engineering-Team, 10Data-Engineering, 10Gerrit (Gerrit 3.7), 10Upstream: git clone and git pull commands fail for refinery repo - https://phabricator.wikimedia.org/T355173 (10gmodena) >>! In T355173#9475961, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=ht...
[13:33:13] <wikibugs>	 10Gerrit: Clicking "Add reviewer" button on Gerrit focuses the comment field, not the reviewers field - https://phabricator.wikimedia.org/T355535 (10matmarex)
[13:35:30] <Lucas_WMDE>	 hashar: alright, thanks for investigating!
[13:37:09] <wikibugs>	 10Gerrit: Links to commit hashes in commit messages on Gerrit no longer work - https://phabricator.wikimedia.org/T355537 (10matmarex)
[13:40:14] <wikibugs>	 10Gerrit: Links to commit hashes in commit messages on Gerrit no longer work - https://phabricator.wikimedia.org/T355537 (10Lucas_Werkmeister_WMDE) > Actual: 404 error (in this case at https://gerrit.wikimedia.org/q/ee91e0f2ebb8efb97145d95d34a90a2b33f0b467).  A working URL would be https://gerrit.wikimedia.org/r...
[13:44:11] <wikibugs>	 10GitLab (Project Migration), 10Data-Platform-SRE: Migrate Elasticsearch plugins repo to gitlab - https://phabricator.wikimedia.org/T353275 (10Gehel)
[13:44:29] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10Ladsgroup) Actually I have an idea why. Let me try something.
[13:44:50] <wikibugs>	 10Phabricator, 10Release-Engineering-Team, 10Trust-and-Safety: Account recovery help needed for Phabricator account Ifeatu_Nnaobi_WMDE - https://phabricator.wikimedia.org/T355414 (10Ifeatu_Nnaobi_WMDE) Thanks for the ticket @WMDE-leszek. And thanks for the action @taavi I'm now in :)
[13:52:29] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10Ladsgroup) Yeah, it's fixed now. [[https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ConfirmEdit/+/991780|I7b71c2f6b1bf05a4f00f2]] (with some modific...
[14:12:09] <Dreamy_Jazz>	 hashar: The Needed by and Depends On links are not working after the Gerrit upgrade.
[14:12:17] <Dreamy_Jazz>	 For example https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/989627
[14:12:46] <Dreamy_Jazz>	 I just see "Depends On" and "Needed by" with no links to patches underneath them.
[14:14:14] <Dreamy_Jazz>	 Ah I see this is https://phabricator.wikimedia.org/T355521
[14:16:39] <hashar>	 yeah
[14:17:14] <Dreamy_Jazz>	 Also I can get the set topic buttons to overflow
[14:17:34] <Dreamy_Jazz>	 Causing a vertical scroll in the small menu
[14:17:57] <Dreamy_Jazz>	 https://usercontent.irccloud-cdn.com/file/J7VKjp5R/image.png
[14:18:59] <Dreamy_Jazz>	 I get that when I paste into the box.
[14:20:03] <wmf-insecte>	 Project beta-update-databases-eqiad build #73354: 04FAILURE in 2.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/73354/
[14:25:01] <wikibugs>	 10Gerrit, 10Upstream: Gerrit Zuul plugin does not show Depends-On/Needed-By since Gerrit 3.7 - https://phabricator.wikimedia.org/T355521 (10hashar) I have send a patch #upstream https://gerrit-review.googlesource.com/c/plugins/zuul/+/404117 Inline removal of Gerrit.Nav in Gerrit 3.7
[14:25:04] <hashar>	 paladox: https://gerrit-review.googlesource.com/c/plugins/zuul/+/404117 Inline removal of Gerrit.Nav in Gerrit 3.7 :)
[14:25:47] <hashar>	 Dreamy_Jazz: maybe that got solved upstrema, I usually check against the Google instance at https://gerrit-review.googlesource.com/ 
[14:26:19] <paladox>	 hashar: for this https://phabricator.wikimedia.org/T355537 locally I changed the config which I got it working. Not sure if it'll work for 3.7 tho. https://gerrit.wikimedia.org/r/c/operations/puppet/+/992109
[14:26:33] <paladox>	 also fyi https://gerrit-review.googlesource.com/c/gerrit/+/351515
[14:26:46] <paladox>	 (that's for 3.8+)
[14:27:02] <hashar>	 ahh I am not looking at 3.8 yet!
[14:27:40] <hashar>	 though I did flag the removal of comment link in our 3.8 upgrade task https://phabricator.wikimedia.org/T354886
[14:29:29] <paladox>	 https://usercontent.irccloud-cdn.com/file/sB6oxKe9/Screenshot%202024-01-22%20at%2014.28.14.png
[14:32:43] <paladox>	 the use of link fixed it it seems hashar
[14:33:07] <hashar>	 is that my patch?
[14:34:20] <paladox>	 no, it's mine https://gerrit.wikimedia.org/r/c/operations/puppet/+/992109 
[14:35:08] <paladox>	 I couldn't use html as I run master but using link fixed it (which also explains why change-id was doubled linked when I added /r/)
[14:36:48] <hashar>	 I should do my branch on master
[14:36:53] <hashar>	 and create a legacy stable-3.6 branch
[14:36:54] <hashar>	 bah
[14:37:15] <wmf-insecte>	 Yippee, build fixed!
[14:37:16] <wmf-insecte>	 Project beta-update-databases-eqiad build #73355: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/73355/
[14:37:23] <hashar>	 or maybe not
[14:37:24] <hashar>	 well
[14:37:29] <hashar>	 paladox: thanks for the +2 :)
[14:37:36] <paladox>	 yw :)
[14:39:06] <wikibugs>	 (03PS1) 10Hashar: Update Zuul plugin for Gerrit 3.7 [software/gerrit] (wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/992191 (https://phabricator.wikimedia.org/T355521)
[14:40:30] <paladox>	 https://github.com/GerritCodeReview/gerrit/blob/stable-3.7/polygerrit-ui/app/utils/link-util.ts#L122 
[14:46:16] <paladox>	 https://github.com/GerritCodeReview/gerrit/commit/2971ae3941bf02f07b2f951aae7e1ebb76a96639 may be what you want hashar in the future to convert the rest from html
[14:46:35] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Update Zuul plugin for Gerrit 3.7 [software/gerrit] (wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/992191 (https://phabricator.wikimedia.org/T355521) (owner: 10Hashar)
[14:48:39] <wikibugs>	 (03PS1) 10Hashar: Update Zuul plugin for Gerrit 3.7 [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/992193 (https://phabricator.wikimedia.org/T355521)
[14:50:06] <hashar>	 paladox: yeah that is for Gerrit 3.8 though and I haven't quite looked at it yet :]
[14:50:19] <paladox>	 yup
[14:50:21] <hashar>	 paladox: but if you polish up that puppet config patch I am happy to reuse it!
[14:50:52] <paladox>	 polish?
[14:51:21] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Update Zuul plugin for Gerrit 3.7 [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/992193 (https://phabricator.wikimedia.org/T355521) (owner: 10Hashar)
[14:51:47] <hashar>	 paladox: s/polish/enhance/  https://gerrit.wikimedia.org/r/c/operations/puppet/+/992109 :)
[14:52:26] <paladox>	 oh yeh. But I mean what exactly?
[14:53:28] <hashar>	 oh you have amended it already
[14:53:29] <wikibugs>	 (03Merged) 10jenkins-bot: Update Zuul plugin for Gerrit 3.7 [software/gerrit] (wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/992191 (https://phabricator.wikimedia.org/T355521) (owner: 10Hashar)
[14:53:32] <wikibugs>	 (03Merged) 10jenkins-bot: Update Zuul plugin for Gerrit 3.7 [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/992193 (https://phabricator.wikimedia.org/T355521) (owner: 10Hashar)
[14:54:01] <hashar>	 paladox: can it be applied to 3.7 already?
[14:54:12] <paladox>	 yeh.
[14:56:23] <paladox>	 cleared it up as I found that the other stuff was needed for the patch hashar 
[14:58:55] <hashar>	 oh joy
[14:59:25] <hashar>	 the Zuul plugin got reloaded but Gerrit still serves the old static file bah
[15:06:46] <hashar>	 hm 
[15:06:48] <hashar>	 I blame some cache
[15:07:05] <Lucas_WMDE>	 always a sensible route
[15:08:12] <hashar>	 I feel like I am getting better at drawing a line to STOP investigating further
[15:08:12] <wikibugs>	 10Gerrit, 10Patch-For-Review, 10Upstream: Gerrit Zuul plugin does not show Depends-On/Needed-By since Gerrit 3.7 - https://phabricator.wikimedia.org/T355521 (10hashar) 05Open→03Resolved a:03hashar That is fixed!  Thank you @ihurbain for the report.
[15:31:34] <wikibugs>	 10Phabricator, 10Striker, 10cloud-services-team: Striker can't create Phabricator projects - https://phabricator.wikimedia.org/T355519 (10taavi) 05Open→03Resolved
[15:31:57] <wikibugs>	 10Phabricator (phabricator-next), 10Patch-For-Review: Disallow creating projects with an empty description - https://phabricator.wikimedia.org/T344610 (10taavi)
[15:32:12] <wikibugs>	 10Phabricator, 10Striker, 10cloud-services-team: Striker can't create Phabricator projects - https://phabricator.wikimedia.org/T355519 (10taavi)
[15:32:22] <wikibugs>	 10Phabricator (phabricator-next), 10Patch-For-Review: Disallow creating projects with an empty description - https://phabricator.wikimedia.org/T344610 (10taavi) 05Open→03Resolved
[15:44:48] <paladox>	 hashar: https://gerrit-review.googlesource.com/c/plugins/zuul/+/404017
[15:58:31] <wikibugs>	 10Gerrit: Gerrit team dashboards no longer load - https://phabricator.wikimedia.org/T355561 (10matmarex)
[16:00:58] <wikibugs>	 10Gerrit: Gerrit team dashboards no longer load - https://phabricator.wikimedia.org/T355561 (10Paladox) "ignored" needs to be removed. This was removed in gerrit 3.7.
[16:05:49] <wikibugs>	 10GitLab (CI & Job Runners), 10collaboration-services, 10serviceops: Create a staging apt repository for CI-based builds of Debian packages - https://phabricator.wikimedia.org/T347004 (10MoritzMuehlenhoff)
[16:06:30] <wikibugs>	 10Gerrit: Gerrit team dashboards no longer load - https://phabricator.wikimedia.org/T355561 (10matmarex) a:03matmarex I guess we just need to remove `-is:ignored` from the queries here: https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/+/refs/meta/dashboards/teams/platform (and in a few other places)  I...
[16:10:25] <wikibugs>	 10Gerrit: Gerrit team dashboards no longer load - https://phabricator.wikimedia.org/T355561 (10Paladox) >>! In T355561#9477465, @matmarex wrote: > I guess we just need to remove `-is:ignored` from the queries here: https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/+/refs/meta/dashboards/teams/platform (an...
[16:12:53] <wikibugs>	 10Gerrit: Investigate Gerrit email delay following 3.7.6  upgrade - https://phabricator.wikimedia.org/T355529 (10Lucas_Werkmeister_WMDE) I’m still seeing some delayed emails (but some are also speedy), are the caches still warming up?  `lines=4 Received: from gerrit1003.wikimedia.org ([2620:0:861:2:208:80:154:13...
[16:14:29] <wikibugs>	 10Gerrit: Gerrit team dashboards no longer load - https://phabricator.wikimedia.org/T355561 (10DAlangi_WMF)
[16:18:42] <hashar>	 paladox: ohh
[16:18:57] <hashar>	 paladox: note I have absolutely no idea how upstream test the plugin or against which branch of gerrit
[16:19:18] <paladox>	 plugins typically use Zuul 
[16:24:11] <hashar>	 paladox: oh also I got a fix in the newly created stable-3.7 branch
[16:24:20] <hashar>	 which should be merged up in master
[16:24:26] <hashar>	 should I send a merge commit for review?
[16:24:30] <hashar>	 or can I just push it?
[16:24:52] <wikibugs>	 10Phabricator: How to make me the top Phabricator autocompletion result when searching for my given name "Bartosz"? - https://phabricator.wikimedia.org/T353937 (10matmarex) Thanks for advice. This happened again, so I created myself a Herald rule: H431.
[16:27:03] <wikibugs>	 (03PS4) 10WQuarshie: stdin and stdout [tools/code-utils] - 10https://gerrit.wikimedia.org/r/991758 (https://phabricator.wikimedia.org/T354654)
[16:28:29] <wikibugs>	 (03CR) 10WQuarshie: stdin and stdout (033 comments) [tools/code-utils] - 10https://gerrit.wikimedia.org/r/991758 (https://phabricator.wikimedia.org/T354654) (owner: 10WQuarshie)
[16:29:28] <paladox>	 hashar: yeh you can send a commit for review
[16:32:55] <hashar>	 paladox: https://gerrit-review.googlesource.com/c/plugins/zuul/+/404157 :)
[16:32:59] <hashar>	 hopefully I have done it right
[16:33:31] <paladox>	 yup, looks correct. +2'd
[16:33:46] * hashar waits for Zuul to gate and submit it
[16:33:47] <hashar>	 oh no
[16:33:58] <hashar>	 :-]
[16:38:56] <wikibugs>	 10GitLab (Infrastructure), 10collaboration-services: Create a custom GitLab Prometheus exporter - https://phabricator.wikimedia.org/T354656 (10LSobanski) p:05Triage→03High
[16:53:56] <paladox>	 hashar: you can submit it
[16:54:02] <hashar>	 paladox: merci!
[16:54:09] <paladox>	 :)
[16:54:19] <hashar>	 done!
[16:56:32] <paladox>	 nice!~
[16:58:26] <wikibugs>	 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for Michaelcochez - https://phabricator.wikimedia.org/T355494 (10brennen) 05Open→03Resolved a:03brennen
[17:58:41] <hashar>	 thcipriani: gerrit has some system to recommend reviewers https://gerrit.googlesource.com/gerrit/+/refs/heads/master/java/com/google/gerrit/server/restapi/change/ReviewersUtil.java#209
[17:59:13] <hashar>	 with some weighting happening at https://gerrit.googlesource.com/gerrit/+/refs/heads/master/java/com/google/gerrit/server/restapi/change/ReviewerRecommender.java#94
[17:59:29] <hashar>	 so I'd guess we'd want something opposite, given a reviewer, suggest changes
[17:59:57] <hashar>	 I have no idea how they weight reviewers though
[18:03:37] <hashar>	 ah there https://gerrit.googlesource.com/gerrit/+/refs/heads/master/java/com/google/gerrit/server/restapi/change/ReviewerRecommender.java#190  and bunch of settings at https://gerrit.wikimedia.org/r/Documentation/config-gerrit.html#suggest
[18:04:25] <thcipriani>	 yeah, looks like it's storing suggestions *somewhere* indexed by patchset
[18:05:01] <thcipriani>	 taking that storage, inverting the index, and storing that would be the kind of thing to implement a "recommended patchset"
[18:07:09] <hashar>	 well that is the Lucene `change` index I believe
[18:09:14] <hashar>	 I am off
[18:09:32] <thcipriani>	 o/
[18:10:14] <hashar>	 `llm_embed_cosine()`  🤷
[18:11:43] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574 (10brennen)
[18:12:05] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574 (10brennen)
[18:12:57] <hashar>	 https://github.com/simonw/llm/blob/17b91e0c9cf37a954d0f58777885fc50216d5893/llm/__init__.py#L252-L256
[18:12:58] <hashar>	 ah
[18:13:32] <hashar>	 which leads me to https://en.wikipedia.org/wiki/Cosine_similarity
[18:13:43] <hashar>	 and I think I should have dinner else next thing is "it is midnight"
[18:14:03] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574 (10brennen)
[18:14:04] <hashar>	 🔫 🐰 🕳️
[18:52:37] <thcipriani>	 :D
[19:18:34] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574 (10Dzahn) https://gitlab.wikimedia.org/repos/phabricator/tools/-/merge_requests/3
[19:42:20] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Python3-Porting, 10User-brennen: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574 (10Peachey88)
[20:07:17] <wikibugs>	 (03CR) 10Daniel Kinzler: [C: 03+2] "nice!" [tools/code-utils] - 10https://gerrit.wikimedia.org/r/991758 (https://phabricator.wikimedia.org/T354654) (owner: 10WQuarshie)
[20:21:58] <wikibugs>	 (03Merged) 10jenkins-bot: stdin and stdout [tools/code-utils] - 10https://gerrit.wikimedia.org/r/991758 (https://phabricator.wikimedia.org/T354654) (owner: 10WQuarshie)
[20:35:17] <wikibugs>	 10Gerrit: Investigate Gerrit email delay following 3.7.6  upgrade - https://phabricator.wikimedia.org/T355529 (10Jdforrester-WMF) I'm still getting e-mails delayed much more from gerrit than previously (e.g. seeing notifications in IRC of merges before I see even the "starting gate and submit" on a patch).
[20:48:16] <wikibugs>	 10Gerrit: Investigate Gerrit email delay following 3.7.6  upgrade - https://phabricator.wikimedia.org/T355529 (10RhinosF1) 05Resolved→03Open
[20:48:59] <kostajh>	 seems like gate and submit is getting stuck on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MediaModeration/+/990739, not sure why
[20:58:57] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Release-Engineering-Team (Radar), 10Security Team AppSec, and 3 others: Gitlab unexpectedly creates multiple pipelines when certain includes/templates are defined within a repo's gitlab-ci.yml - https://phabricator.wikimedia.org/T354723 (...
[21:05:40] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security: Design and Build Application Security Pipeline Components for Gitlab - https://phabricator.wikimedia.org/T289290 (10sbassett)
[21:06:05] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Release-Engineering-Team (Radar), 10Security Team AppSec, and 3 others: Gitlab unexpectedly creates multiple pipelines when certain includes/templates are defined within a repo's gitlab-ci.yml - https://phabricator.wikimedia.org/T354723 (...
[23:58:23] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Python3-Porting, 10User-brennen: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574 (10CodeReviewBot) brennen updated https://gitlab.wikimedia.org/repos/phabricator/tools/-/merge_requests...