[00:24:36] 10Continuous-Integration-Infrastructure, 10Castor, 06cloud-services-team, 10Cloud-VPS (Quota-requests), 06Test Platform (Arusha 30): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12325721 (10Mhurd) I think retyping witho... [02:13:48] 10Continuous-Integration-Infrastructure, 10Castor, 07Spike, 06Test Platform (Arusha 30): Test out the https://github.com/git-pkgs/proxy/ for CI - https://phabricator.wikimedia.org/T437005#12325793 (10Mhurd) @Peter Yeah that's what Peter W saw too. And I came to same conclusion: that the web proxy request... [02:51:09] 10Continuous-Integration-Infrastructure, 10Castor, 07Spike, 06Test Platform (Arusha 30): Test out the https://github.com/git-pkgs/proxy/ for CI - https://phabricator.wikimedia.org/T437005#12325824 (10Mhurd) [03:02:00] holy cow: https://wikitech.wikimedia.org/wiki/Module:Scapsay [03:02:05] I mean pig [03:31:03] 10WikimediaDebug: Add Test Kitchen functionality to WikimediaDebug - https://phabricator.wikimedia.org/T437240#12325885 (10cjming) hi @phuedx - does my understanding of what needs to happen match yours? 1. Fetch active experiments from Test Kitchen in https://gerrit.wikimedia.org/g/performance/WikimediaDebug/ 2... [05:40:57] 10Gerrit: Users with a different name in the cn field compared to uid field cannot use http auth - https://phabricator.wikimedia.org/T225308#12326007 (10hashar) 05Declined→03Open I am reopening this since it might have been fixed when, last week, we upgraded Gerrit 3.10.6 to 3.10.9 (T400688). From the change... [05:41:29] 10Gerrit (Gerrit 3.10): Upgrade Gerrit from 3.10.6 to 3.10.9 - https://phabricator.wikimedia.org/T400688#12326018 (10hashar) [05:42:57] 10Gerrit, 07Upstream: Users with a different name in the cn field compared to uid field cannot use http auth - https://phabricator.wikimedia.org/T225308#12326021 (10hashar) [05:48:08] 10Gerrit, 07Upstream: Gerrit fudges multiple suggestions in comments - https://phabricator.wikimedia.org/T405625#12326023 (10hashar) Thank you @Paladox for the verification! [05:53:55] 10Gerrit (Gerrit 3.10): Upgrade Gerrit from 3.10.6 to 3.10.9 - https://phabricator.wikimedia.org/T400688#12326026 (10hashar) 05Open→03Resolved a:03thcipriani The upgrade has been completed by @thcipriani last week, thank you! I have looked at the tasks that were related to this upgrade in order to hav... [06:04:43] (03CR) 10Hashar: [C:03+2] Docker: [node26] Update Node from 26.4.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1342055 (owner: 10Jforrester) [06:06:16] (03Merged) 10jenkins-bot: Docker: [node26] Update Node from 26.4.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1342055 (owner: 10Jforrester) [06:15:07] (03CR) 10Hashar: [C:03+2] jjb: Update Node from 26.4.0 to 26.8.2 for generic Node jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342056 (owner: 10Jforrester) [06:15:07] !log Updating node jobs from NodeJS 26.4.0 to 27.8.2 | https://gerrit.wikimedia.org/r/c/integration/config/+/1342056 [06:15:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:16:58] (03Merged) 10jenkins-bot: jjb: Update Node from 26.4.0 to 26.8.2 for generic Node jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342056 (owner: 10Jforrester) [06:17:54] (03PS2) 10Hashar: Docker: [quibble-bookworm] Update Node from 24.18.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1342057 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [06:21:28] James_F: I am quite happy to see you have dropped jsduck (and as a result the ruby* stuff from the images) [06:22:36] (03CR) 10Hashar: [C:03+2] "`dockerfiles/quibble-bookworm/control` needed `node24` > `node26` as well. That is the file used by docker-pkg to track dependencies (rath" [integration/config] - 10https://gerrit.wikimedia.org/r/1342057 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [06:24:16] (03Merged) 10jenkins-bot: Docker: [quibble-bookworm] Update Node from 24.18.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1342057 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [06:45:30] (03PS2) 10Jforrester: jjb: Update Node from 24.18.0 to 26.8.2 for Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1342058 (https://phabricator.wikimedia.org/T438085) [06:49:38] (03CR) 10Hashar: "I have built the images, I am not sure what is going to break as part of upgrading Node 24 > Node 26, notably webdriver.io. I have commen" [integration/config] - 10https://gerrit.wikimedia.org/r/1342058 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [06:50:45] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/ThemeToggle] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1341995 (owner: 10SomeRandomDeveloper) [06:52:19] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ThemeToggle] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1341995 (owner: 10SomeRandomDeveloper) [06:53:13] (03CR) 10Hashar: "**Deployed!**" [integration/config] - 10https://gerrit.wikimedia.org/r/1341995 (owner: 10SomeRandomDeveloper) [07:06:19] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12326139 (10hashar) Confirmed, thx for the deploy! Note from https://gerrit.wikimedia.org/r/c/operations/puppet/+/1327569/comments/70c86841_bb... [07:23:00] (03CR) 10Phedenskog: [C:03+1] jjb: run extension browser tests without npm --prefix [integration/config] - 10https://gerrit.wikimedia.org/r/1342041 (https://phabricator.wikimedia.org/T437376) (owner: 10Mhurd) [07:25:59] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12326168 (10hashar) Now that we have Zookeeper logs on the zuul* hosts (T435503), I can see the connection failures: ` name=zuul1001.eqia... [08:12:10] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12326351 (10hashar) Found it, the client port is //wrong// https://gerrit.wikimedia.org/r/c/operations/puppet/+/1236730 //zookeeper/zuul... [08:13:18] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12326358 (10hashar) Looking at the firewall rules on `zuul1001` using `sudo iptables -L -v|grep dpt:2.8`: We get rules for all three ho... [08:15:49] (03CR) 10Phedenskog: "@hashar@free.fr we want your eyes/feedback on this change so we know how you feel about it!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1289330 (https://phabricator.wikimedia.org/T426684) (owner: 10Phedenskog) [08:31:24] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T430839#12326433 (10jnuche) [08:32:41] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, and 2 others: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12326440 (10fgiunchedi) I did a quick audit on the fleet (see below), my understanding is that we wan... [08:44:54] 06Project-Admins: Create project tag for DataMaps - https://phabricator.wikimedia.org/T438099#12326459 (10Peachey88) 05Open→03Resolved a:03Peachey88 Requested public project has been created: #MediaWiki-extensions-DataMaps [09:15:06] 06Release-Engineering-Team, 06Infrastructure-Foundations, 13Patch-For-Review: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297#12326580 (10elukey) ` elukey@deploy1003:~$ time sudo -i docker-registryctl -c /etc/docker-pusher/config.json delete-tags docker... [09:20:46] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Phab Scap template vs actual value out of sync for numerous values - https://phabricator.wikimedia.org/T398460#12326596 (10Aklapper) ` aklapper@phab1005:/srv/deployment/phabricator/deployment/phabricator$ sudo ./bin/config delete --database ui.hea... [09:30:07] (03CR) 10Hashar: [C:04-1] "My bad!!" [integration/config] - 10https://gerrit.wikimedia.org/r/1342041 (https://phabricator.wikimedia.org/T437376) (owner: 10Mhurd) [09:30:33] 06Release-Engineering-Team, 10Scap, 07Epic, 07OKR-Work, and 2 others: Proposal: scap deploy-service - https://phabricator.wikimedia.org/T412941#12326648 (10MLechvien-WMF) [10:12:02] 06Project-Admins: Create project tag for DataMaps - https://phabricator.wikimedia.org/T438099#12327021 (10SomeRandomDeveloper) Thanks! [10:13:01] (03CR) 10SomeRandomDeveloper: "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/1341995 (owner: 10SomeRandomDeveloper) [10:48:32] hashar: Thanks for nerd-sniping me. ;-) [10:50:56] 06Release-Engineering-Team, 10Scap, 07Essential-Work: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148 (10jnuche) 03NEW [11:04:14] (03open) 10arnaudb: Add repos/sre/aphlict [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/190 (https://phabricator.wikimedia.org/T436657) [11:07:01] (03update) 10arnaudb: Add repos/sre/aphlict [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/190 (https://phabricator.wikimedia.org/T436657) [11:07:40] (03update) 10arnaudb: Draft: Add repos/sre/aphlict [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/190 (https://phabricator.wikimedia.org/T436657) [11:11:14] (03update) 10arnaudb: Draft: Add repos/sre/aphlict [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/190 (https://phabricator.wikimedia.org/T436657) [11:22:31] 10Phabricator, 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Move aphlict to k8s - https://phabricator.wikimedia.org/T436657#12327381 (10ABran-WMF) [11:46:17] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T430839#12327552 (10jnuche) [12:00:49] James_F: you are welcome ;) [12:01:06] I have rolled some of the node 24 > 26 patches this morning [12:01:13] and the new Quibble images have been built [12:01:26] I am not sure what would happen if Quibble jobs get node upgraded [12:01:47] I can imagine Cypress / webdriver.io having issues with it [12:01:56] so I did not switch the jobs yet ( your patch is https://gerrit.wikimedia.org/r/c/integration/config/+/1342058 ) [12:02:20] (03PS2) 10Jforrester: Zuul: Migrate MediaWiki-land independent CI Node from 24 to 26 [integration/config] - 10https://gerrit.wikimedia.org/r/1342059 (https://phabricator.wikimedia.org/T438085) [12:05:18] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Rohitpal08] - https://phabricator.wikimedia.org/T438155 (10Rohitpal08) 03NEW [12:32:59] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 20 to Node 22 - https://phabricator.wikimedia.org/T395924#12327711 (10Krinkle) 05Open→03Resolved a:03Jdforrester-WMF ` [13:31 BST] krinkle at KrinkleMac in ~/Development/wikimedia/integration-config (master) $ git... [12:50:55] (03CR) 10Jforrester: "Hah, yes, the peril of writing these patches late in the day (which is why I didn't deploy). Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/1342057 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [12:51:44] (03PS2) 10Phedenskog: jjb: delete the workspace in the background [integration/config] - 10https://gerrit.wikimedia.org/r/1342006 (https://phabricator.wikimedia.org/T437022) [12:53:13] (03CR) 10CI reject: [V:04-1] jjb: delete the workspace in the background [integration/config] - 10https://gerrit.wikimedia.org/r/1342006 (https://phabricator.wikimedia.org/T437022) (owner: 10Phedenskog) [12:54:23] 10WikimediaDebug: Add Test Kitchen functionality to WikimediaDebug - https://phabricator.wikimedia.org/T437240#12327794 (10cjming) per discussion with @phuedx, we will allow for multiple experiments/groups selection in the WikimediaDebug UI and opt to send analytics events which means we need to ensure that `coo... [13:04:58] (03PS3) 10Phedenskog: jjb: delete the workspace in the background [integration/config] - 10https://gerrit.wikimedia.org/r/1342006 (https://phabricator.wikimedia.org/T437022) [13:06:19] (03CR) 10CI reject: [V:04-1] jjb: delete the workspace in the background [integration/config] - 10https://gerrit.wikimedia.org/r/1342006 (https://phabricator.wikimedia.org/T437022) (owner: 10Phedenskog) [13:42:44] (03open) 10phedenskog: Add summary tiles to the Castor dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/71 [13:42:49] (03merge) 10phedenskog: Add summary tiles to the Castor dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/71 [13:59:02] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Rohitpal08] - https://phabricator.wikimedia.org/T433793#12328224 (10Aklapper) →14Duplicate dup:03T438155 [13:59:03] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Rohitpal08] - https://phabricator.wikimedia.org/T438155#12328226 (10Aklapper) [13:59:53] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Rohitpal08] - https://phabricator.wikimedia.org/T438155#12328229 (10Aklapper) @Rohitpal08: Hi, please also [link your LDAP account to your Phabricator account](https://phabricator.wikimedia.org/settings/panel/ext... [14:12:35] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Rohitpal08] - https://phabricator.wikimedia.org/T438155#12328339 (10Rohitpal08) @Aklapper Done! I have linked my LDAP account to my Phabricator profile. Thank you! [14:15:03] hashar: Are you around to help with a full-disk issue in prod? See -operations [14:51:21] 06Release-Engineering-Team, 10Scap, 07Essential-Work: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12328727 (10jnuche) > 4:11 PM > James D. Forrester I’m getting No Space Left On Device from the MW docker build in Spiderpig. > 4:12 PM > <•logmsgbot> !... [14:53:04] 06Release-Engineering-Team, 10Scap, 07Essential-Work: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12328760 (10jnuche) I can't run `scap-clean-images` manually, I don't have permissions: ` jnuche@deploy1003:~$ systemctl start scap-clean-images Failed to execute... [14:54:16] 06Release-Engineering-Team, 10Scap, 07Essential-Work: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12328766 (10Jdforrester-WMF) Does this block the train at this point? [14:54:40] 06Release-Engineering-Team, 10Scap, 07Essential-Work: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12328779 (10jnuche) >>! In T438148#12328766, @Jdforrester-WMF wrote: > Does this block the train at this point? It does, was about to post saying that [14:54:52] 06Release-Engineering-Team, 10Scap, 07Essential-Work: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12328781 (10jnuche) p:05Triage→03Unbreak! [14:55:14] 06Release-Engineering-Team, 10Scap, 07Essential-Work: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12328782 (10jnuche) [14:55:19] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T430839#12328783 (10jnuche) [14:59:45] 06Release-Engineering-Team, 10Scap, 06ServiceOps, 07Essential-Work: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12328835 (10Jdforrester-WMF) [15:00:44] 06Release-Engineering-Team, 10Scap, 06ServiceOps, 07Essential-Work: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12328864 (10jnuche) Ok, so turns out scap has a command for it: ` jnuche@deploy1003:~$ scap clean-images 14:58:42 Re-executing as docker user: mwb... [15:04:41] 10GitLab, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Trusted-runner access for developer-activity-pipelines (publish a container image) - https://phabricator.wikimedia.org/T438042#12328905 (10Ahoelzl) p:05Triage→03High a:03Antoine_Quhen [15:23:20] 06Release-Engineering-Team, 10Scap, 06ServiceOps, 07Essential-Work: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12329031 (10jnuche) p:05Unbreak!→03Triage [15:23:33] 06Release-Engineering-Team, 10Scap, 06ServiceOps, 07Essential-Work: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12329032 (10jnuche) [15:23:38] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T430839#12329033 (10jnuche) [16:20:42] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12329557 (10Dzahn) Thanks for confirming! :) I would prefer if that is handled separately as this task was just about zuul machines and indee... [16:21:23] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12329560 (10Dzahn) >>! In T435503#12326139, @hashar wrote: > For the Zuul Zookeeper instances, we now have logs, I can see connection errors an... [17:55:46] (03CR) 10Hashar: "Ahmon and I briefly talked about this change and I am more than happy for him to step in and move it forward." [integration/config] - 10https://gerrit.wikimedia.org/r/1341139 (https://phabricator.wikimedia.org/T352319) (owner: 10Arendpieter) [18:05:44] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T438217#12330219 (10thcipriani) p:05Triage→03Medium a:03jeena [18:06:30] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218#12330225 (10thcipriani) p:05Triage→03Medium a:03brennen [18:07:16] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T438219#12330230 (10thcipriani) p:05Triage→03Medium a:03Aklapper [18:08:25] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T438220#12330237 (10thcipriani) p:05Triage→03Medium a:03hashar [18:11:09] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T438222#12330261 (10thcipriani) p:05Triage→03Medium a:03dancy [18:11:39] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T438223#12330266 (10thcipriani) p:05Triage→03Medium a:03dduvall [18:33:33] 06Release-Engineering-Team, 10Scap, 06ServiceOps, 07Essential-Work: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12330365 (10dancy) [18:59:01] 10Deployments, 06Release-Engineering-Team, 06ServiceOps: Request to increase the size of /srv on production deployment servers - https://phabricator.wikimedia.org/T438232 (10dancy) 03NEW [19:13:46] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12330543 (10dancy) Any news on rebuilt bullseye base images? [19:17:11] 06Release-Engineering-Team, 10Scap, 06ServiceOps, 07Essential-Work, 13Patch-For-Review: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12330546 (10Jdforrester-WMF) Note that we've bounded back from ~68% to 80% in the few hours since you last manually ran that. [19:22:08] (03CR) 10Ahmon Dancy: "That's promising news hashar!" [integration/config] - 10https://gerrit.wikimedia.org/r/1341139 (https://phabricator.wikimedia.org/T352319) (owner: 10Arendpieter) [19:38:03] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-mwmaint03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [19:39:29] 06Release-Engineering-Team, 10Scap: Recent "Branch cut test patches" job git concurrency failures - https://phabricator.wikimedia.org/T438236 (10dancy) 03NEW [20:27:25] 10Deployments, 06Release-Engineering-Team (Radar), 06ServiceOps: Request to increase the size of /srv on production deployment servers - https://phabricator.wikimedia.org/T438232#12330853 (10thcipriani) [20:28:11] 10Deployments, 06Release-Engineering-Team (Radar), 06ServiceOps: Request to increase the size of /srv on production deployment servers - https://phabricator.wikimedia.org/T438232#12330854 (10thcipriani) [20:28:12] 06Release-Engineering-Team, 10Scap, 06ServiceOps, 07Essential-Work, 13Patch-For-Review: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12330855 (10thcipriani) [20:31:07] 06Release-Engineering-Team (Priority Backlog 📥), 10Scap, 06ServiceOps, 07Essential-Work, 13Patch-For-Review: Image building is running out of space in `/srv` - https://phabricator.wikimedia.org/T438148#12330864 (10thcipriani) [21:09:18] (03open) 10southparkfan: Update Beta Cluster-related documentation [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1317 (https://phabricator.wikimedia.org/T435393 https://phabricator.wikimedia.org/T436470) [21:18:15] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12331033 (10Dzahn) 172.17.0.0/16 is allowed from this puppet code: ` # allow traffic between zuul services in docker and zookeeper... [21:19:53] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12331035 (10Dzahn) Before anything else - I suggest we talk in a group setting about how exactly the physical machines are replacing (or... [21:33:47] (03CR) 10Ahmon Dancy: "Thanks @arendpieter@proton.me. I like this idea. I will look at this more closely soon." [integration/config] - 10https://gerrit.wikimedia.org/r/1341139 (https://phabricator.wikimedia.org/T352319) (owner: 10Arendpieter) [21:40:43] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-rdb01 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436480#12331099 (10Southparkfan) Hi @Ladsgroup! I am currently working on getting the Beta Cluster off Bullseye. As the author https://gerrit.wikimedia.... [21:53:01] 10Beta-Cluster-Infrastructure, 06Growth-Team, 10GrowthExperiments-NewcomerTasks: [enwiki beta] Special:NewcomerTasksInfo gives Internal error - https://phabricator.wikimedia.org/T438254 (10Etonkovidova) 03NEW [21:55:45] 10Beta-Cluster-Infrastructure, 06Growth-Team, 10GrowthExperiments-NewcomerTasks: [enwiki beta] Special:NewcomerTasksInfo gives Internal error - https://phabricator.wikimedia.org/T438254#12331173 (10Etonkovidova) [22:17:35] 06Release-Engineering-Team, 10Scap: Recent "Branch cut test patches" job git concurrency failures - https://phabricator.wikimedia.org/T438236#12331253 (10dancy) ` dancy@releases1003:~$ time sudo -u jenkins-agent git -C "/srv/jenkins-agent/workspace/Branch cut test patches/work/mediawiki/php-branch_cut_pretest"... [23:23:52] 06Release-Engineering-Team, 10Scap: scap shouldn't attempt to !log messages with a newline - https://phabricator.wikimedia.org/T438270 (10RLazarus) 03NEW [23:26:51] ^ also though, I was able to recover from that error by just rerunning "scap backport" -- even though it had gotten partway through the first attempt, it picked up cleanly where it left off. that was *really* slick, nice job