[00:42:06] (03approved) 10swfrench: logstash: Match the fields of a query exactly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1313 (https://phabricator.wikimedia.org/T435419) (owner: 10dancy) [01:04:37] (03approved) 10swfrench: deploy-service: Name the group of the plan from the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1301 (https://phabricator.wikimedia.org/T412941) (owner: 10dancy) [05:05:15] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12288294 (10Peter) We have a [[ https://releng-data.wmcloud.org/... [05:11:22] 10Continuous-Integration-Infrastructure, 10Castor, 07Spike, 06Test Platform (Arusha 30): Test out the https://github.com/git-pkgs/proxy/ for CI - https://phabricator.wikimedia.org/T437005#12288299 (10hashar) [08:09:51] (03PS1) 10Phedenskog: Run the next npm install during the browser tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1335750 (https://phabricator.wikimedia.org/T436376) [08:49:37] (03CR) 10Phedenskog: [C:03+2] Rename cmd to cli [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293713 (https://phabricator.wikimedia.org/T300727) (owner: 10Hashar) [08:54:19] (03Merged) 10jenkins-bot: Rename cmd to cli [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293713 (https://phabricator.wikimedia.org/T300727) (owner: 10Hashar) [08:57:59] (03PS2) 10Phedenskog: Run the next npm install during the browser tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1335750 (https://phabricator.wikimedia.org/T436376) [09:23:31] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12288782 (10ABran-WMF) Given the volume of repositories we have, I asked claude with subagents to identify the most probable victims for the missing User-Agent issue that emer... [10:17:10] (03PS1) 10Hashar: dockerfiles: disable npm audit & fund in Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/1335785 (https://phabricator.wikimedia.org/T435975) [10:19:19] (03CR) 10Hashar: "I have to test it, I am not sure the env variables apply. We would need the same to be done for the `node*` images and the `sonar-scanner`" [integration/config] - 10https://gerrit.wikimedia.org/r/1335785 (https://phabricator.wikimedia.org/T435975) (owner: 10Hashar) [11:16:11] 10GitLab, 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: GitLab agents failing to connect - https://phabricator.wikimedia.org/T436902#12289128 (10jnuche) GitLab's CDN migration was temporarily reverted and @ABran-WMF has merged patches (thank you for that!) that should allo... [11:16:24] 10GitLab, 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: GitLab agents failing to connect - https://phabricator.wikimedia.org/T436902#12289129 (10jnuche) 05Open→03Resolved [11:37:28] (03PS8) 10Phedenskog: castor: load the cache without the delayed rename pass [integration/config] - 10https://gerrit.wikimedia.org/r/1330226 (https://phabricator.wikimedia.org/T436173) [11:37:54] (03CR) 10Phedenskog: "Acknowledged" [integration/config] - 10https://gerrit.wikimedia.org/r/1330226 (https://phabricator.wikimedia.org/T436173) (owner: 10Phedenskog) [11:38:43] (03CR) 10Phedenskog: "Think there's no hurry push this, I want to have some space as you said from other changes (the castor disk speedup) to make it easier to " [integration/config] - 10https://gerrit.wikimedia.org/r/1330226 (https://phabricator.wikimedia.org/T436173) (owner: 10Phedenskog) [11:44:23] (03PS1) 10Phedenskog: castor: Only add gate-and-submit to the save queue [integration/config] - 10https://gerrit.wikimedia.org/r/1335807 (https://phabricator.wikimedia.org/T436461) [12:13:53] is it just me or is the phabricator gitlab widget broken? [12:14:32] got link? [12:17:52] andre: T433690 for example has an attacked MR but no widget [12:17:53] T433690: [components-api] Create endpoint to get tools based on a set config - https://phabricator.wikimedia.org/T433690 [12:30:36] 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#12289396 (10Aklapper) 05Open→03Resolved I don't see anything left to do here? [12:36:57] (03open) 10jelto: digitalocean: create dedicated vpc and nat_gateway for each cluster [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/629 (https://phabricator.wikimedia.org/T425441) [12:37:13] 06Release-Engineering-Team (Priority Backlog 📥), 06Collaboration-Services: Upgrade phab (phorge) hosts to bookworm - https://phabricator.wikimedia.org/T372619#12289432 (10Aklapper) {T423727} and {T435087}/{T436635} are resolved. (Not sure what phab1006 is on though.) @Arnoldokoth, @LSobanski: Can this very tas... [12:39:33] (03update) 10jelto: digitalocean: create dedicated vpc and nat_gateway for each cluster [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/629 (https://phabricator.wikimedia.org/T425441) [12:44:39] (03open) 10jnuche: perform-release: add user agent to requests going out to GitLab [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1314 (https://phabricator.wikimedia.org/T425441) [13:38:37] taavi, I would assume it fails because the "Bug: Txxxxx" line is not on a line of its on [14:43:32] !log integration: set in Jenkins global config: NPM_CONFIG_AUDIT=false and NPM_CONFIG_FUND=false # T437008 [14:43:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:43:36] T437008: Wikibase and some other jobs become very slow 2026-09-03/04 - https://phabricator.wikimedia.org/T437008 [14:45:25] I think I am officially fed up with Docker [14:49:43] (03update) 10jnuche: perform-release: add user agent to requests going out to GitLab [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1314 (https://phabricator.wikimedia.org/T425441) [15:10:17] (03update) 10dancy: deploy-service: Name the group of the plan from the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1301 (https://phabricator.wikimedia.org/T412941) [15:12:07] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:15:32] (03PS2) 10Hashar: dockerfiles: disable npm audit/fund/update-notifier via npmrc [integration/config] - 10https://gerrit.wikimedia.org/r/1335785 (https://phabricator.wikimedia.org/T435975) [15:15:53] well hmm [15:16:01] I say [15:17:19] `⠘⣷⣶⢀⣶⣶⠇⣠⣶⣿⣿⣶⣦⡀⢰⣶⡶⠀⠀⢀⣴⣶⣿⣿⣶⣦⠀` [15:17:19] `⠀⠙⣿⣿⡿⠃⢸⣿⣟⠀⠀⢹⣿⡇⢸⣿⡇⠀⠀⣾⣿⡏⠀⠀⢻⣿⡇` [15:17:19] `⠀⠀⣿⣿⠃⠀⠘⢿⣿⣦⣴⣿⡿⠃⣸⣿⣧⣤⣤⠹⣿⣷⣤⣴⣿⡿⠁` [15:17:19] `⠀⠀⠉⠉⠀⠀⠀⠀⠉⠉⠉⠉⠀⠀⠈⠉⠉⠉⠉⠀⠀⠉⠉⠉⠉⠀⠀` [15:18:44] (03CR) 10Hashar: [C:03+2] dockerfiles: disable npm audit/fund/update-notifier via npmrc [integration/config] - 10https://gerrit.wikimedia.org/r/1335785 (https://phabricator.wikimedia.org/T435975) (owner: 10Hashar) [15:20:26] (03Merged) 10jenkins-bot: dockerfiles: disable npm audit/fund/update-notifier via npmrc [integration/config] - 10https://gerrit.wikimedia.org/r/1335785 (https://phabricator.wikimedia.org/T435975) (owner: 10Hashar) [15:20:50] 10Phabricator: Please un-archive the "New Private Task (WMF-FR)" Phabricator form - https://phabricator.wikimedia.org/T437067 (10Ejegg) 03NEW [15:24:42] (03PS1) 10Ottomata: Zuul: Run WikibaseClient job for EventBus [integration/config] - 10https://gerrit.wikimedia.org/r/1335904 (https://phabricator.wikimedia.org/T428176) [15:26:02] (03CR) 10CI reject: [V:04-1] Zuul: Run WikibaseClient job for EventBus [integration/config] - 10https://gerrit.wikimedia.org/r/1335904 (https://phabricator.wikimedia.org/T428176) (owner: 10Ottomata) [15:27:06] 06Release-Engineering-Team: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290291 (10dancy) p:05Triage→03High [15:30:58] (03PS2) 10Ottomata: Zuul: Run WikibaseClient job for EventBus [integration/config] - 10https://gerrit.wikimedia.org/r/1335904 (https://phabricator.wikimedia.org/T428176) [15:41:20] of course images fail to build [15:43:40] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI: Use Helm v4 - https://phabricator.wikimedia.org/T429768#12290375 (10jnuche) [15:46:20] (03PS1) 10Hashar: dockerfiles: add missing arg in COPY /usr/etc/npmrc [integration/config] - 10https://gerrit.wikimedia.org/r/1335924 (https://phabricator.wikimedia.org/T435975) [15:52:08] !log integration: remove from Jenkins global config: NPM_CONFIG_AUDIT=false and NPM_CONFIG_FUND=false # T437008 [15:52:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:52:12] T437008: Wikibase and some other jobs become very slow 2026-09-03/04 - https://phabricator.wikimedia.org/T437008 [16:03:35] (03PS1) 10Hashar: jjb: update node based jobs to disable audit/fund [integration/config] - 10https://gerrit.wikimedia.org/r/1335937 (https://phabricator.wikimedia.org/T435975) [16:09:57] (03CR) 10Hashar: [C:03+2] jjb: update node based jobs to disable audit/fund [integration/config] - 10https://gerrit.wikimedia.org/r/1335937 (https://phabricator.wikimedia.org/T435975) (owner: 10Hashar) [16:11:28] (03Merged) 10jenkins-bot: jjb: update node based jobs to disable audit/fund [integration/config] - 10https://gerrit.wikimedia.org/r/1335937 (https://phabricator.wikimedia.org/T435975) (owner: 10Hashar) [16:34:16] 06Release-Engineering-Team, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290602 (10BLiviero-WMF) [16:41:15] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290634 (10thcipriani) Bullseye is LTS EoL as of Monday, so they're starting to dismantle/rearrange the apt repo it seems. Ideally, w... [17:25:32] (03open) 10dancy: install-world: Add an --inventory mode [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1315 [17:40:43] (03update) 10dancy: install-world: Add an --inventory mode [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1315 [17:40:46] (03update) 10dancy: install-world: Add an --inventory mode [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1315 [17:42:52] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290843 (10dancy) Current production scap targets: ` ┌──────────┬─────────┬────────────────────────────────────────────� [17:44:12] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290845 (10dduvall) Noting that there are a number of images in operations/docker-images/production-images that depend on bullseye (d... [17:45:43] (03PS2) 10Hashar: Require Python 3.11 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334901 (https://phabricator.wikimedia.org/T429715) [17:45:53] (03PS2) 10Hashar: Use str.removesuffix() for clarity [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334924 (https://phabricator.wikimedia.org/T429715) [17:46:45] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290846 (10MoritzMuehlenhoff) Judging from what I saw on Debian IRC it looks like the removal of bullseye-security was accidental, th... [17:53:43] (03PS2) 10Hashar: Use Parenthesized Context Managers [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334902 (https://phabricator.wikimedia.org/T429715) [17:58:27] (03open) 10pwangai: Add p75 charts for MediawikiCore [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/60 (https://phabricator.wikimedia.org/T435977) [18:00:51] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290893 (10MoritzMuehlenhoff) Or specifically for scap maybe just disable CI for Bullseye for now? The remaining aqs/restbase/session... [18:03:48] 06Release-Engineering-Team, 06Content-Platform-Team, 10Quibble, 10Catalyst (PatchDemo), and 2 others: Not adding trailing / to autoload patches breaks Parsoid CI - https://phabricator.wikimedia.org/T400299#12290902 (10hashar) 05Open→03Resolved This got fixed in Quibble by https://gerrit.wikimedia.o... [18:04:01] 06Release-Engineering-Team, 06Content-Platform-Team, 10Quibble, 10Catalyst (PatchDemo), and 2 others: Not adding trailing / to autoload patches breaks Parsoid CI - https://phabricator.wikimedia.org/T400299#12290904 (10hashar) 05Resolved→03Open [18:04:15] (03update) 10dancy: install-world: Add an --inventory mode [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1315 [18:04:17] (03update) 10dancy: install-world: Add an --inventory mode [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1315 [18:06:25] 06Release-Engineering-Team, 06Content-Platform-Team, 10Quibble, 10Catalyst (PatchDemo), and 2 others: Not adding trailing / to autoload patches breaks Parsoid CI - https://phabricator.wikimedia.org/T400299#12290907 (10hashar) 05Open→03Resolved a:03cscott >>! In T400299#12290902, @hashar wrote: >... [18:06:26] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290910 (10dancy) Here's the full list of bullseye targets as of today: ` aqs1022.eqiad.wmnet aqs1023.eqiad.wmnet aqs1024.eqiad... [18:08:41] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-mwlog02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436469#12290915 (10Southparkfan) 05Open→03In progress [18:13:40] (03merge) 10phedenskog: Add p75 charts for MediawikiCore [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/60 (https://phabricator.wikimedia.org/T435977) (owner: 10pwangai) [18:17:47] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290930 (10MoritzMuehlenhoff) >>! In T437069#12290910, @dancy wrote: > Here's the full list of bullseye targets as of today: We're t... [18:23:36] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290932 (10dancy) Remaining beta cluster bullseye targets: ` bullseye (8 targets) deployment-deploy04.deployment-prep.eqiad1.wikime... [18:24:56] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290942 (10dancy) >>! In T437069#12290893, @MoritzMuehlenhoff wrote: > Or specifically for scap maybe just disable CI for Bullseye fo... [19:05:49] (03CR) 10Hashar: [C:03+2] zuul: add Thilio Liojo to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1335286 (owner: 10Apehitkey) [19:06:28] (03CR) 10Hashar: [C:03+2] zuul: Add Wikibase as phan dependency of GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1334082 (https://phabricator.wikimedia.org/T436682) (owner: 10Aude) [19:07:33] (03Merged) 10jenkins-bot: zuul: add Thilio Liojo to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1335286 (owner: 10Apehitkey) [19:08:05] (03Merged) 10jenkins-bot: zuul: Add Wikibase as phan dependency of GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1334082 (https://phabricator.wikimedia.org/T436682) (owner: 10Aude) [19:09:18] (03CR) 10Hashar: [C:03+2] "Deployed! @thilioliojo@gmail.com CI will now process patches you send and you can comment `recheck` on a change to retrigger CI." [integration/config] - 10https://gerrit.wikimedia.org/r/1335286 (owner: 10Apehitkey) [19:45:50] I have a patch to have Quibble to require Python 3.11 (from 3.9) given the CI image are no more using Bullseye :party: [19:46:18] and I sent a couple more that make use of parenthesis context manager ( with (x, y, z): ) and str.removesuffix() [19:46:19] https://gerrit.wikimedia.org/r/q/project:integration/quibble+status:open+bug:T429715 [19:46:20] T429715: Upgrade Quibble code base to Python3.11 - https://phabricator.wikimedia.org/T429715 [19:46:23] happy week-end [22:12:43] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-mwlog02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436469#12291658 (10Southparkfan) Rough outline: - Turn up `deployment-mwlog03` with Trixie (T417002) and a 20 GiB Cinder volume for `/srv` - Adjust `u... [22:56:39] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace deployment-mwlog02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436469#12291759 (10Southparkfan) I couldn't finish the Tofu run, because we have too many [[ https://wikitech.wikimedia.org/wiki... [22:57:12] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace deployment-mwlog02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436469#12291762 (10Southparkfan) 05In progress→03Stalled