[00:44:04] 10Beta-Cluster-Infrastructure: deployment-restbase05.deployment-prep.eqiad1.wikimedia.cloud configured to talk to parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T389252#10681332 (10bd808) 05In progress→03Resolved [00:47:45] 10Beta-Cluster-Infrastructure: deployment-acme-chief0{5,6} puppet failures for 'profile::opensearch::cirrus::enable_remote_search' - https://phabricator.wikimedia.org/T390128 (10bd808) 03NEW [00:50:01] 10Beta-Cluster-Infrastructure: deployment-acme-chief0{5,6} puppet failures for 'profile::opensearch::cirrus::enable_remote_search' - https://phabricator.wikimedia.org/T390128#10681345 (10bd808) This looks like something random happened to the Horizon managed puppet config. It doesn't make any sense why `role::ci... [00:54:59] 10Beta-Cluster-Infrastructure: deployment-acme-chief0{5,6} puppet failures for 'profile::opensearch::cirrus::enable_remote_search' - https://phabricator.wikimedia.org/T390128#10681359 (10bd808) Looks like an accident by @bking while working on {T389971}: https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/insta... [00:55:34] !log Removed prefix puppet classes for deployment-acme-chief (T390128) [00:55:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:55:36] T390128: deployment-acme-chief0{5,6} puppet failures for 'profile::opensearch::cirrus::enable_remote_search' - https://phabricator.wikimedia.org/T390128 [03:17:29] (03PS1) 10Chlod Alejandro: zuul: Add Robertsky to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1131493 [08:45:05] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-1.44-notes (1.44.0-wmf.22; 2025-03-25), 05Release, 05Train Deployments: 1.44.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T386217#10681705 (10Aklapper) [09:12:26] 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.23; 2025-04-01), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10681765 (10Peter) P... [10:02:03] 10Scap: SecurityPatchBot: Failing due to malformed patch - https://phabricator.wikimedia.org/T390144 (10jnuche) 03NEW [10:02:52] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-1.44-notes (1.44.0-wmf.22; 2025-03-25), 05Release, 05Train Deployments: 1.44.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T386217#10681939 (10Aklapper) After I backported the fix for T390032 and deployed two other backports and deplo... [10:05:20] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T386218#10681951 (10SecurityPatchBot) [10:13:56] 10Scap, 13Patch-For-Review: SecurityPatchBot: Failing due to malformed patch - https://phabricator.wikimedia.org/T390144#10682004 (10jnuche) [10:59:10] 10Release-Engineering-Team (Doing 😎), 10MW-1.44-notes (1.44.0-wmf.22; 2025-03-25), 05Release, 05Train Deployments: 1.44.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T386217#10682199 (10Aklapper) [11:21:14] 10Release-Engineering-Team (Doing 😎), 10MW-1.44-notes (1.44.0-wmf.22; 2025-03-25), 05Release, 05Train Deployments: 1.44.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T386217#10682265 (10Aklapper) 05Open→03Resolved T390142 will get a SWAP deploy later today. Resolving. [13:17:04] bd808 thanks for cleaning up my mess in T390128 . I realized I made a mistake there and I thought I fixed it, but I guess not ;( [13:27:11] Project mediawiki-core-doxygen build #9241: 04FAILURE in 5 min 10 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/9241/ [13:45:31] 10Gerrit, 07Upstream: Gerrit editor replaces tabs with spaces - https://phabricator.wikimedia.org/T355816#10683096 (10Paladox) >>! In T355816#10676426, @hashar wrote: > @Paladox you are really a jewel to the wikis movements. Thank you! > > The fix is backported to 3.11 by https://gerrit-review.googlesource.co... [13:47:35] (03PS1) 10Kosta Harlan: zuul: Add dependency on ConfirmEdit to IPReputation [integration/config] - 10https://gerrit.wikimedia.org/r/1131712 (https://phabricator.wikimedia.org/T390197) [13:48:43] hashar: could you please deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1131712 as soon as possible? [13:48:58] (03CR) 10Dreamy Jazz: [C:03+1] zuul: Add dependency on ConfirmEdit to IPReputation [integration/config] - 10https://gerrit.wikimedia.org/r/1131712 (https://phabricator.wikimedia.org/T390197) (owner: 10Kosta Harlan) [13:49:06] needed to unblock some other work [13:49:36] (03CR) 10Majavah: [C:03+2] zuul: Add dependency on ConfirmEdit to IPReputation [integration/config] - 10https://gerrit.wikimedia.org/r/1131712 (https://phabricator.wikimedia.org/T390197) (owner: 10Kosta Harlan) [13:50:48] (03Merged) 10jenkins-bot: zuul: Add dependency on ConfirmEdit to IPReputation [integration/config] - 10https://gerrit.wikimedia.org/r/1131712 (https://phabricator.wikimedia.org/T390197) (owner: 10Kosta Harlan) [13:51:01] !log reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1131712 [13:51:12] Thanks taavi! [13:51:35] kostajh: looks like I wasn't fast enough :-]] [13:52:02] taavi: thank you [13:52:40] 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.23; 2025-04-01), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10683141 (10Peter) I... [13:52:53] (03CR) 10Hashar: [C:03+2] zuul: Add Robertsky to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1131493 (owner: 10Chlod Alejandro) [13:52:57] 10Continuous-Integration-Config, 10Math, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Allow control over which extra extensions are installed (Math REL1_43 jobs exceed 60min timeout) - https://phabricator.wikimedia.org/T389998#10683143 (10Physikerwelt) To give a very typical example of a very ann... [13:54:15] (03Merged) 10jenkins-bot: zuul: Add Robertsky to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1131493 (owner: 10Chlod Alejandro) [14:03:37] James_F: thanks for the wdio release patch. I wanted to try take a look at T390097, in case it has an easy fix. It would be great to include it in the release if so. [14:04:00] Hm no bot? https://phabricator.wikimedia.org/T390097 [14:10:04] Daimona: Yay for parallelism. [14:11:40] you parallelize stuff. That's what happens when [14:30:39] Daimona: I already filed a task for that [14:30:59] Oh really? [14:31:02] it is part of the series about running selenium tests in parallel T226869 [14:31:03] T226869: Run browser tests in parallel - https://phabricator.wikimedia.org/T226869 [14:31:20] the issue is the chromium browsers being run in parallel each share the same XVfb display [14:31:25] which we use in order to capture videos [14:31:27] Yippee, build fixed! [14:31:27] Project mediawiki-core-doxygen build #9242: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/9242/ [14:31:38] so the videos can show an entirely different session [14:31:55] the analysis is https://phabricator.wikimedia.org/T344754 [14:32:35] the idea I had was to spin up an Xvfb screen for each browser instance [14:32:54] andhave one of the parallel session to grab the screen number from a shared pool [14:32:56] Ah, that helps, thank you! [14:33:02] so at start you would have screens 0 1 2 3 4 5 [14:33:13] when the parallel sessions starts they would pop an entry frrom there [14:33:18] use that Xvfb screen [14:33:27] and we would have has many ffmpeg recording [14:33:34] but OF COURSE [14:33:36] That seems reasonable, I'm going to read the task. But do you think it would work if we specified the window to record? [14:34:01] the super happy doc I found which was so great relied on https://webdriver.io/docs/shared-store-service/ [14:34:08] which requires webdriver.io v8.7.0 [14:34:31] and my test bed implementation is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/951587 [14:34:37] Of course. Really looking forward to the update! [14:34:57] also .kill() does not do what you would expect :] [14:35:37] and somehow my task is blocked on upgrading to webdriver.IO to V8 but Zeljko removed the sub tasks for some reason [14:35:40] I will add it back [14:35:49] So yeah, that does look somewhat complext. That's why I was wondering if we could do it with window_id instead. If that works, it should be much simpler. There's that "if", though. [14:42:00] Daimona: if each of the parallel session has some of fixed identifier, then we could map that to some Xvfb screen [14:42:25] when running four in parallel one can see thing ssuch as Chrome [0-0] , Chrome [0-1] [14:42:43] so I imagine once a session starts there is a way to differentiate between the instances [14:42:59] and that numbre can be used to have each session on a standalone screen [14:43:49] I don't know if that or any other identifier is exposed. But if not, I imagine we could build our own. [14:44:00] My proposal though was to leave them in the same screen, but record different windows [14:44:08] my patch mentionned "cid of worker", so maybe I tried that approach [14:45:58] I think we can track the Xvfb display screen being used via the onWorkerStart hook but wdio v7 lacks the onWorkerComplete hook which could be used to clear them once the worker terminates. [14:45:59] I could not find how to share a global state between sessions to register assignments of screen. I guess I will revisit later :) [14:46:05] from my self review [14:46:06] anyway [14:46:28] the recording is done by ffmpeg which records the whole display regardless of the windows [14:46:42] it just reads the framebuffer [14:46:51] and ffmpeg has no knowledge about the windows [14:47:30] It does take a window_id according to the docs [14:47:36] I posted a link in the task [14:47:45] But I don't know if/how that works. [14:47:48] you are ahead of me in that area of expertise! :b [14:48:19] If 5 minutes of googling can be considered "area of expertise" then yeah, I'm an expert :P [14:49:14] Even if the option works, it looks like we'd run into other problems if someone resizes the window. But oh well. [14:50:43] then looking at my code again, there is a commented out section about onWorkerStart [14:50:51] it is passed a `cid` whichis is eg 0-0 [14:51:09] so from there you can determine an Xvfb screen to use and don't need a shared pool [14:51:28] but the blocker in the comment was there was no onWorkerEnd hook, so we could not dispose of the Xvfb [14:51:53] but that might not be needed if we drop the XvfbScreenAssignments map [14:51:57] hmm anyway [14:52:21] it is probably eachievalbe with wdio v7 by removing XvfbScreenAssignments [14:52:49] and instead have a deterministic way to map a cid to an xvfb screen number [14:58:22] inflatador: no worries. the UX in that particular screen sort of lays a trap for the unwary. #someday I'll get around to trying to fix it up... maybe. [15:05:11] !log Moved role::acme_chief::cloud from individual instance config to deployment-acme-chief Puppet prefix. [15:05:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:07:16] 10Beta-Cluster-Infrastructure: deployment-acme-chief0{5,6} puppet failures for 'profile::opensearch::cirrus::enable_remote_search' - https://phabricator.wikimedia.org/T390128#10683457 (10bd808) I moved `role::acme_chief::cloud` from instance level puppet config to the deployment-acme-chief prefix too. This may h... [15:08:22] a dumb workaround for that bug would be to create a prefix like `deployment-AAAAAAAAAAAAAAA` that just has roles and hiera set to soemthing saying "you are in the wrong place" [15:15:04] I'm digging into the component to see if I can figure out how to make the "These puppet settings will affect..." text more prominent. It's an upstream display widget so slightly more complicated than putting a tag in an existing template. [15:32:06] So, are the puppet failures mentioned above the reason beta is currently completely down (503 no server is available to handle this request), or should I be filing some new task about that? [15:34:17] Kemayo: not the same problem. I can reproduce, but haven't started looking for why yet. A task would be appreaciated [15:34:28] Okay, I will get that filed. [15:34:52] 10Beta-Cluster-Infrastructure: Beta cluster is completely down with 503 "No server is available to handle this request" - https://phabricator.wikimedia.org/T390209 (10DLynch) 03NEW [15:34:58] There you go. [15:35:50] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review, 07User-notice: Enable browser notifications system in Gerrit - https://phabricator.wikimedia.org/T389327#10683611 (10hashar) I have restarted Gerrit and the feature is now available. The following popups show up inviting... [15:36:03] 10Beta-Cluster-Infrastructure: Beta cluster is completely down with 503 "No server is available to handle this request" - https://phabricator.wikimedia.org/T390209#10683613 (10bd808) p:05Triage→03Unbreak! a:03bd808 https://beta.wmflabs.org/ is also down with the same message. [15:40:48] Having a latency problem on elasticsearch in codfw, anyone mind if i ship a mediawiki-config patch that moves all our traffic to eqiad? [15:41:59] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review, 07User-notice: Enable browser notifications system in Gerrit - https://phabricator.wikimedia.org/T389327#10683629 (10hashar) 05Open→03Resolved I have send an announcement on Slack and on wikitech-l. I'd like to... [15:42:18] ebernhardson: wrong channel? [15:42:30] bd808: hmm, don't i need to ask releng to deploy outside normal windows? [15:43:25] ah, I would expect that ask -operations and to mention that you were looking for an emergency deploy. [15:44:08] but really as long as there isn't an active deployment happening on a weekday then you should be good to go [15:44:41] sounds reasonable, i suppoes i worry that -operations is so full of automated messages. But this channel isn't that much less :) [15:44:51] Totally [15:45:43] the bot spam debate is like top posting or tabs v spaces :) [15:46:38] :) [15:52:22] Spaces, btw. [15:53:12] in python & yaml yes, in anything else no :) [15:54:30] Does anyone have good ideas/docs on how to debug the CDN in deployment-prep? [15:54:51] bd808: i was told there would be panicking [15:55:00] I'm about to hit the reboot switch on deployment-cache-text08 jsut to see if anything changes [15:55:25] :panic:+:disco: [15:55:43] 10Gerrit, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Remove explicit enablement of G1 garbage collector for Gerrit - https://phabricator.wikimedia.org/T387223#10683674 (10hashar) Gerrit heap is now 64G, we will see how the garbage collection behave on htt... [15:55:56] our not so helpful help link just goes to https://wikitech.wikimedia.org/wiki/Caching_overview [15:56:31] bd808: did you figure out which layer emits that 503 yet? [15:57:16] taavi: no, I'm not yet sure how to hit things below haproxy [15:57:27] my guess is that that's from haproxy which would be failing to talk to varnish [16:00:12] `curl -v --insecure -H 'wikidata.beta.wmflabs.org' https://localhost` returns the 503. [16:04:21] `sudo varnishncsa -n frontend` seems to return nothing. [16:05:24] !log `sudo systemctl restart varnish-frontend` on deployment-cache-text08 (T390209) [16:05:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:05:26] T390209: Beta cluster is completely down with 503 "No server is available to handle this request" - https://phabricator.wikimedia.org/T390209 [16:06:03] \o/ Restarting varnish seems to have done the trick [16:10:17] \o/ [16:16:19] 10Beta-Cluster-Infrastructure: Beta cluster is completely down with 503 "No server is available to handle this request" - https://phabricator.wikimedia.org/T390209#10683749 (10bd808) p:05Unbreak!→03Medium Boy oh boy to we need more runbooks for beta cluster. * I went to https://wikitech.wikimedia.org/wiki/N... [16:26:34] 10Scap, 13Patch-For-Review: Scap seemingly doesn't fully/properly clean backup deployment server - https://phabricator.wikimedia.org/T389830#10683770 (10dancy) 05Open→03Resolved The new `scap-master-sync` now deletes cache-only php-* dirs after rsyncing. This will prevent cruft from accumulating. [16:28:38] (03update) 10thcipriani: feat: Add Logstash link in the nav bar [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/719 [16:28:40] !log Moved Puppet configuration from deployment-cache-text08 to deployment-cache-text prefix Puppet [16:28:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:28:42] (03update) 10thcipriani: feat: Add Logstash link in the nav bar [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/719 [16:28:53] (03update) 10thcipriani: feat: Add Logstash link in the nav bar [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/719 [16:30:11] (03merge) 10dancy: feat: Add Logstash link in the nav bar [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/719 (owner: 10thcipriani) [16:33:38] 10Beta-Cluster-Infrastructure, 06Traffic, 07Documentation: Create a runbook for troubleshooting the CDN in deployment-prep - https://phabricator.wikimedia.org/T390213 (10bd808) 03NEW [16:39:52] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Investigate moving from Digital Ocean to WMF infrastructure - https://phabricator.wikimedia.org/T389936#10683861 (10aborrero) >>! In T389936#10677756, @Jelto wrote: > > OpenStack Magnum could be a potential platform for that. I... [16:43:14] (03open) 10dancy: scap.cfg: Update to latest spiderpig settings. [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/127 [16:43:17] (03update) 10dancy: scap.cfg: Update to latest spiderpig settings. [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/127 [16:44:15] (03merge) 10dancy: scap.cfg: Update to latest spiderpig settings. [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/127 [16:45:37] (03update) 10dancy: mods for deploy02 [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/93 [16:47:26] (03update) 10dancy: mods for deploy02 [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/93 [16:55:00] 10Beta-Cluster-Infrastructure: Beta cluster is completely down with 503 "No server is available to handle this request" - https://phabricator.wikimedia.org/T390209#10683934 (10bd808) 05Open→03Resolved [16:57:10] (03open) 10dancy: JobHistory.vue: Code cleanup [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/723 [16:57:12] (03update) 10dancy: JobHistory.vue: Code cleanup [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/723 [16:59:03] (03merge) 10dancy: JobHistory.vue: Code cleanup [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/723 [17:25:52] (03update) 10swfrench: kubernetes: support image "kind" and deprecate the debug boolean [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/702 (https://phabricator.wikimedia.org/T389499) [18:11:28] 10Scap, 06serviceops: Migrate scap's maintenance script invocations to PHP 8.1 - https://phabricator.wikimedia.org/T390225 (10Scott_French) 03NEW [18:11:43] 10Scap, 06serviceops: Migrate scap's maintenance script invocations to PHP 8.1 - https://phabricator.wikimedia.org/T390225#10684262 (10Scott_French) [18:14:33] 10Scap, 06serviceops: Migrate scap's maintenance script invocations to PHP 8.1 - https://phabricator.wikimedia.org/T390225#10684280 (10Scott_French) @dduvall - When you get a chance, could you confirm that my understanding here is correct? If so, then //making// the change is in and of itself fairly simple (i... [18:15:15] (03open) 10dancy: Release 4.146.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/724 [18:16:57] (03merge) 10dancy: Release 4.146.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/724 [18:40:48] (03approved) 10thcipriani: utils.get_real_username: Use SPIDERPIG_REAL_USER if available [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/722 (owner: 10dancy) [18:48:26] (03update) 10dancy: utils.get_real_username: Use SPIDERPIG_REAL_USER if available [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/722 [18:50:23] (03merge) 10dancy: utils.get_real_username: Use SPIDERPIG_REAL_USER if available [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/722 [18:59:14] (03open) 10dancy: Release 4.147.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/725 [19:00:17] (03merge) 10dancy: Release 4.147.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/725 [19:16:11] (03PS1) 10Abaris: layout.yaml: Remove CI for mediawiki/libs/StringUtils [integration/config] - 10https://gerrit.wikimedia.org/r/1131805 [19:17:30] (03CR) 10CI reject: [V:04-1] layout.yaml: Remove CI for mediawiki/libs/StringUtils [integration/config] - 10https://gerrit.wikimedia.org/r/1131805 (owner: 10Abaris) [19:23:05] (03PS2) 10Abaris: layout.yaml: Remove CI for mediawiki/libs/StringUtils [integration/config] - 10https://gerrit.wikimedia.org/r/1131805 [19:23:14] (03PS3) 10Abaris: layout.yaml: Remove CI for mediawiki/libs/StringUtils [integration/config] - 10https://gerrit.wikimedia.org/r/1131805 [20:10:20] (03CR) 10Jforrester: [C:04-1] "Sorry! I didn't mean "don't do that", quite the reverse. Let's get your library work landed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1131805 (owner: 10Abaris) [20:13:18] (03CR) 10Abaris: "Oh, okay then! 😂" [integration/config] - 10https://gerrit.wikimedia.org/r/1131805 (owner: 10Abaris) [20:13:21] (03Abandoned) 10Abaris: layout.yaml: Remove CI for mediawiki/libs/StringUtils [integration/config] - 10https://gerrit.wikimedia.org/r/1131805 (owner: 10Abaris) [21:39:52] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07User-notice: Enable browser notifications system in Gerrit - https://phabricator.wikimedia.org/T389327#10684908 (10Krinkle) The banner seemingly has no button to reject or opt-out, which is unfortunately a (likely inintended) dark patter... [21:40:39] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07User-notice: Enable browser notifications system in Gerrit - https://phabricator.wikimedia.org/T389327#10684909 (10Krinkle) [22:22:21] 10Deployments, 06Release-Engineering-Team: Consider using JSON content model for deployment calendar - https://phabricator.wikimedia.org/T366880#10685059 (10brennen) [23:06:24] (03update) 10swfrench: kubernetes: support image "kind" and deprecate the debug boolean [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/702 (https://phabricator.wikimedia.org/T389499) [23:14:24] 10Deployments, 06Release-Engineering-Team, 10Jouncebot, 10Tool-schedule-deployment: Consider using JSON content model for deployment calendar - https://phabricator.wikimedia.org/T366880#10685197 (10bd808) [23:25:30] 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.23; 2025-04-01), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10685243 (10Daimona)... [23:31:55] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" - https://phabricator.wikimedia.org/T329216#10685254 (10bd808) Per https://www.doc... [23:36:39] (03open) 10jforrester: releases: Bump qunit to 2.24.1 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/67 [23:36:45] (03merge) 10jforrester: releases: Bump qunit to 2.24.1 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/67