[04:52:27] 10Gerrit: Gerrit "Depends on" / "Needed by" display broken by 3.7 upgrade - https://phabricator.wikimedia.org/T355635 (10Tgr) [04:55:04] 10Gerrit: Gerrit "Depends on" / "Needed by" display broken by 3.7 upgrade - https://phabricator.wikimedia.org/T355635 (10Tgr) Sometimes it works but other sections are missing. I see JS errors like `SyntaxError: Unexpected end of JSON input` and `TypeError: Cannot read properties of undefined (reading 'getUrlFor... [06:49:11] @hashar my favorite gerrit query no longer works: https://gerrit.wikimedia.org/r/q/is:open+(owner:self+OR+reviewer:self+OR+assignee:self+OR+cc:self)+(-age:2w+OR+is:starred) [06:49:31] Error 400 (Bad Request): too many terms in query: 1217 terms (max = 1024)Endpoint: /changes/ [06:49:44] Is that a known issue? Should I file a phab task? [06:57:30] hashar: similar to last week's train, https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/992123 needs a backport for this week. The main patch has now been +2'ed so it shouldn't be an issue in the future. [06:58:24] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.42.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T354433 (10kostajh) @hashar https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/992123 (and possibly also https://gerrit.wikimedia.org/r/c/med... [08:01:11] 10GitLab (CI & Job Runners), 10collaboration-services, 10serviceops: Create a staging apt repository for CI-based builds of Debian packages - https://phabricator.wikimedia.org/T347004 (10Joe) Please let's make `apt-merge` less cumbersome to use. `apt-merge bookworm component/ci jenkins-glue` should work wit... [08:02:49] 10Diffusion: Enable git repo creation in Diffusion again - https://phabricator.wikimedia.org/T355644 (10QChris) [08:07:22] 10Diffusion, 10Phabricator: Enable git repo creation in Diffusion again - https://phabricator.wikimedia.org/T355644 (10taavi) [08:40:48] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.7): Gerrit notifications settings default/reset post upgrade - https://phabricator.wikimedia.org/T355646 (10fgiunchedi) [08:47:11] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.7): Gerrit notifications settings default/reset post upgrade - https://phabricator.wikimedia.org/T355646 (10fgiunchedi) I've set my notification preferences as to what they were before (I think?) i.e. all enabled, opening the task for followup/awareness in case it... [09:09:56] 10Gerrit, 10Regression, 10Upstream: Starred changes were not display as starred in change lists in Gerrit 3.7.6 - https://phabricator.wikimedia.org/T355650 (10Winston_Sung) [09:14:12] 10Gerrit, 10Regression, 10Upstream: Starred changes were not display as starred in change lists in Gerrit 3.7.6 - https://phabricator.wikimedia.org/T355650 (10Winston_Sung) 05Open→03Stalled https://issues.gerritcodereview.com/issues/40015194 [09:32:59] duesen: yes please file it as a task :) [09:37:17] duesen: funilly it works for me and works if I replace `self` with your email ( https://gerrit.wikimedia.org/r/q/is:open+(owner:dkinzler%2540wikimedia.org+OR+reviewer:dkinzler%2540wikimedia.org+OR+assignee:dkinzler%2540wikimedia.org+OR+cc:dkinzler%2540wikimedia.org)+(-age:2w+OR+is:starred) ) [09:38:48] 10Gerrit: Gerrit "Depends on" / "Needed by" display broken by 3.7 upgrade - https://phabricator.wikimedia.org/T355635 (10Lucas_Werkmeister_WMDE) This was supposed to have been fixed yesterday (T355521), and the examples in the task description are all working for me at the moment. [09:38:50] kostajh: looks like a method is missing a parameter :/ [09:39:14] hashar: which one? [09:40:00] I was looking at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/992123 [09:40:05] hmm. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/992277 was needed for the patch in master to go through [09:40:05] there are tests failures such as `ArgumentCountError: Too few arguments to function MediaWiki\User\CentralId\CentralIdLookupFactory::__construct(), 3 passed in /workspace/src/extensions/CentralAuth/tests/phpunit/CentralAuthIdLookupTest.php on line 100 and exactly 4 expected` [09:41:00] so I think we need a backport of https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/992277. I mentioned that in T354433#9479755 but not here, sorry [09:41:01] T354433: 1.42.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T354433 [09:41:20] ah I see [09:41:35] hashar: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/992367 [09:41:56] well I guess you can send a chain of backports [09:41:58] btw, seems like the "cherry pick" button fails on first attempt in Gerrit UI. [09:45:57] so many breakages :( [09:49:18] 10Gerrit: Gerrit "Depends on" / "Needed by" display broken by 3.7 upgrade - https://phabricator.wikimedia.org/T355635 (10hashar) The API to which belong `getUrlForSearchQuery` has been completely removed but was still used by our Zuul plugin and I fixed it yesterday. My guess is the file served ( https://gerrit.... [10:05:48] 10Gerrit: Gerrit "Depends on" / "Needed by" display broken by 3.7 upgrade - https://phabricator.wikimedia.org/T355635 (10Winston_Sung) Was broken for me but fixed after refreshed browser cache. [10:12:56] 10Gerrit: Investigate Gerrit email delay following 3.7.6 upgrade - https://phabricator.wikimedia.org/T355529 (10taavi) I'm currently getting Gerrit emails with a delay of over two hours. One example: ` Received: from [127.0.0.1] (port=33134 helo=localhost) by gerrit1003.wikimedia.org with esmtp (Exim 4.94.2) (e... [10:27:14] 10Gerrit: Investigate Gerrit email delay following 3.7.6 upgrade - https://phabricator.wikimedia.org/T355529 (10Lucas_Werkmeister_WMDE) FWIW, the big spike in the email queue starting 6:50 UTC ([Grafana](https://grafana.wikimedia.org/d/Zh_ncGsWk/queues-upstream?orgId=1&refresh=1m&viewPanel=14&from=1705983963397... [10:52:46] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Python3-Porting, 10User-brennen: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574 (10LSobanski) [11:11:03] 10Gerrit: Investigate Gerrit email delay following 3.7.6 upgrade - https://phabricator.wikimedia.org/T355529 (10hashar) @Lucas_Werkmeister_WMDE nice, that is definitely the email generation taking WAY longer than it did previously with Gerrit 3.6. I have looked at the thread dump via https://gerrit.wikimedia.o... [11:15:07] 10Gerrit: Gerrit "Depends on" / "Needed by" display broken by 3.7 upgrade - https://phabricator.wikimedia.org/T355635 (10hashar) [11:15:15] 10Gerrit, 10Upstream: Gerrit Zuul plugin does not show Depends-On/Needed-By since Gerrit 3.7 - https://phabricator.wikimedia.org/T355521 (10hashar) [11:16:05] 10Gerrit: Investigate Gerrit email delay following 3.7.6 upgrade - https://phabricator.wikimedia.org/T355529 (10Lucas_Werkmeister_WMDE) FWIW, I tried looking through the Gerrit log between 3.6 and 3.7 for email-related changes, but didn’t find much. There were [two](https://gerrit-review.googlesource.com/c/gerr... [11:16:38] * Lucas_WMDE sure is using the phrase “FWIW” a lot [11:17:00] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10awight) 05Open→03Resolved a:03awight Resolved from my perspective: I can create an account with spaces in the name eg. "Adamw test 3" on the beta c... [11:17:15] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation - https://phabricator.wikimedia.org/T355505 (10awight) a:05awight→03None [11:22:03] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster captcha prevents new account creation when username includes whitespace - https://phabricator.wikimedia.org/T355505 (10awight) [11:37:39] 10Gerrit: Investigate Gerrit email delay following 3.7.6 upgrade - https://phabricator.wikimedia.org/T355529 (10hashar) Beside the traces being fairly large, I found bunch of them having: ` java.base@11.0.21/java.nio.file.NoSuchFileException.(NoSuchFileException.java:61) com.google.gerrit.server.Starred... [11:46:59] 10Gerrit, 10Release-Engineering-Team, 10collaboration-services: Refine alert manager silences for Gerrit upgrades - https://phabricator.wikimedia.org/T355516 (10LSobanski) This will be ultimately addressed by {T284302}. For the time being my suggestion would be to schedule maintenance like this during the #c... [12:45:56] 10Gerrit: Investigate Gerrit email delay following 3.7.6 upgrade - https://phabricator.wikimedia.org/T355529 (10hashar) I have made a small configuration change which might have caused the Gerrit to enter some long/infinite loop due to `IsWatchedByPredicate`. [12:46:09] 10Gerrit: Investigate Gerrit email delay following 3.7.6 upgrade - https://phabricator.wikimedia.org/T355529 (10hashar) [13:54:09] 10Gerrit, 10Upstream: Investigate Gerrit email delay following 3.7.6 upgrade - https://phabricator.wikimedia.org/T355529 (10hashar) I am pretty sure I have found the trigger for the issue and I have reported it #upstream https://issues.gerritcodereview.com/issues/321784734 I will keep monitoring [[ https://g... [13:54:16] 10Gerrit, 10Upstream: Investigate Gerrit email delay following 3.7.6 upgrade - https://phabricator.wikimedia.org/T355529 (10hashar) [13:59:56] are gerrit issues not public? I get “access denied” at https://issues.gerritcodereview.com/issues/321784734 :< [14:08:02] yeah I have made that one private [14:42:14] 10Gerrit, 10Upstream: Investigate Gerrit email delay following 3.7.6 upgrade - https://phabricator.wikimedia.org/T355529 (10hashar) p:05Triage→03High [14:45:22] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.42.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T354433 (10hashar) [14:45:53] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.7): Upgrade to Gerrit 3.7 - https://phabricator.wikimedia.org/T354885 (10fgiunchedi) [14:45:59] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.7): Gerrit notifications settings default/reset post upgrade - https://phabricator.wikimedia.org/T355646 (10fgiunchedi) 05Open→03Invalid The issue I'm seeing is {T355529} [15:05:15] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Python3-Porting, 10User-brennen: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574 (10Dzahn) The good part here is that most (all except the dump script?) of... [15:06:20] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.7): Gerrit notifications settings default/reset post upgrade - https://phabricator.wikimedia.org/T355646 (10hashar) Indeed that is an issue caused by Gerrit 3.7 which causes emails processing to take a lot more longer than expected (minutes instead of hundred of m... [15:49:20] thcipriani: brennen btw the clock is unchanged (forgot to update :-P ) [15:53:21] 10Release-Engineering-Team (Deployment Training Requests), 10User-ArielGlenn: UTC morning backport and config training: who attended/were scheduled to attend - https://phabricator.wikimedia.org/T355687 (10thcipriani) [15:54:07] 10Release-Engineering-Team (Deployment Training Requests), 10User-ArielGlenn: UTC late backport and config training: who attended/were scheduled to attend - https://phabricator.wikimedia.org/T355688 (10thcipriani) [16:30:46] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Infrastructure-Foundations, 10User-brennen: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691 (10brennen) [16:35:23] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Infrastructure-Foundations, 10User-brennen: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691 (10brennen) Without having dug into the code at all yet, I expect that a `List-Unsubscr... [16:37:06] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Infrastructure-Foundations, 10User-brennen: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691 (10brennen) Relevant RFCs linked by Google: - https://datatracker.ietf.org/doc/html/rf... [16:47:57] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for USER[S] - https://phabricator.wikimedia.org/T355692 (10Cyberwolf) [17:10:25] 10Beta-Cluster-Infrastructure, 10RESTBase: RESTbase broken on beta cluster - https://phabricator.wikimedia.org/T351193 (10hnowlan) 05Open→03Resolved [17:37:07] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: [EPIC] Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10sbassett) [17:37:13] 10Release-Engineering-Team (They Live 🕶️🧟), 10Anti-Harassment, 10Projects-Cleanup, 10Security-Team, 10iPoid-Service: Migrate mediawiki/services/ipoid to GitLab - https://phabricator.wikimedia.org/T337714 (10sbassett) [17:38:55] 10Release-Engineering-Team (Radar), 10Anti-Harassment, 10Security-Team, 10iPoid-Service, 10Patch-For-Review: Use Gitlab Security Pipeline for ipoid - https://phabricator.wikimedia.org/T338238 (10sbassett) 05Stalled→03Resolved With the resolution of the above issue (T354723#9478839) and adding the osv... [18:09:14] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Infrastructure-Foundations, 10Upstream, 10User-brennen: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691 (10brennen) Filed upstream: https://we.phorge.it/T15719 [19:37:51] so there is this issue with Gerrit commit message links since the upgrade: https://phabricator.wikimedia.org/T355537 [19:38:12] Paladox uploaded https://gerrit.wikimedia.org/r/c/operations/puppet/+/992109 [19:38:17] and I believe we should deploy it [19:38:22] +1 [19:38:29] I confirmed that upstream gerrit removed that "html" thing in comments [19:38:39] that is in https://gerrit-review.googlesource.com/c/gerrit/+/351515 [19:38:52] so it makes sense that this is not used anymore and broke [19:39:35] yea, if anyone wants to +1, i'll do it [19:40:41] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Infrastructure-Foundations, 10Upstream, 10User-brennen: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691 (10Tgr) [19:46:21] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Infrastructure-Foundations, 10Upstream, 10User-brennen: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691 (10Dzahn) upstream ticket comment pointed out we can run `./mail/volume`... [20:16:21] kostajh: the issue you had yesterday with a CR+2 rebased patch not entering gate-and-submit, I think that will be fixed in the next Gerrit 3.7 version [20:16:50] good to know [20:17:02] hashar: did you end up not needing https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/992367? [20:17:46] the issue is that the CR+2 does get copied from the previous patchset but that is not associated as a vote in the event [20:18:01] I don't know whether you had a task filed for it [20:18:25] for CentralAuth, I don't know? [20:20:14] oh that is https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/992123 which was broken [20:22:23] I have done a rebase [20:22:43] and I guess we can deploy them tomorrow [20:23:44] I am off [20:23:45] :) [21:44:32] 10GitLab: Account pending approval banner shown on initial sign-in gives no context for how to proceed - https://phabricator.wikimedia.org/T353752 (10deni) Regarding my last comment, it was not ideal. I do believe that there is an issue here but I could've conveyed that in more considerate way. I am sorry for th... [22:47:22] I have a script which generates some container images based off our puppet repo. I would ideally like to rebuild the container images on a weekly basis and push them to our docker repository. Is there any documentation or advice on doing something like that? [23:40:00] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for Cyberwolf - https://phabricator.wikimedia.org/T355692 (10Pppery)