[06:51:27] 06Release-Engineering-Team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): archiva1002 has stale jobs in /var/cache/archiva that uses all the disk space - https://phabricator.wikimedia.org/T425083#12200573 (10Gehel) [06:54:27] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep - https://phabricator.wikimedia.org/T424100#12200651 (10Gehel) [06:55:39] 10Phabricator-Bot-Requests, 06Discovery-Search, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Requesting Phab bot for Machine Learning/Data Engineering team - https://phabricator.wikimedia.org/T433543#12200668 (10Gehel) [06:57:17] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): No Puppet resources found on instance deployment-cirrussearch15 on project deployment-prep - https://phabricator.wikimedia.org/T430217#12200692 (10Gehel) [07:46:57] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for Mf212 - https://phabricator.wikimedia.org/T434455#12200901 (10Aklapper) 05Open→03Resolved a:03Aklapper Done! [08:12:55] 10GitLab, 06collaboration-services: SystemdUnitFailed - gitlab-backup-restore.service on gitlab2002:9100 - https://phabricator.wikimedia.org/T434464#12200992 (10ABran-WMF) p:05Triage→03Medium a:03ABran-WMF [08:15:01] 10GitLab, 06collaboration-services: SystemdUnitFailed - gitlab-backup-restore.service on gitlab2002:9100 - https://phabricator.wikimedia.org/T434464#12201000 (10ABran-WMF) 05Open→03In progress I restarted the service so it tries again to run to completion. The script disables the puppet agent so puppet wa... [08:30:56] 10GitLab, 06collaboration-services: SystemdUnitFailed - gitlab-backup-restore.service on gitlab2002:9100 - https://phabricator.wikimedia.org/T434464#12201058 (10ABran-WMF) 05In progress→03Resolved recovered upon retry [09:25:54] 10Phabricator Translations Extension, 06translatewiki.net, 10LPL sprints (S2 2026 August): Phabricator translations needs i18n check reviewed again - https://phabricator.wikimedia.org/T434026#12201434 (10abi_) I reviewed the translations today: https://gerrit.wikimedia.org/r/c/phabricator/translations/+/1323... [09:26:16] 10Phabricator Translations Extension, 06translatewiki.net, 10LPL sprints (S2 2026 August): Phabricator translations needs i18n check reviewed again - https://phabricator.wikimedia.org/T434026#12201441 (10abi_) a:03abi_ [09:50:03] maintenance-disconnect-full-disks build 832164 integration-agent-docker-1072 (/: 28%, /srv: 100%, /var/lib/docker: 33%): OFFLINE due to disk space [09:55:03] maintenance-disconnect-full-disks build 832165 integration-agent-docker-1072 (/: 28%, /srv: 27%, /var/lib/docker: 32%): RECOVERY disk space OK [10:31:53] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12201756 (10ABran-WMF) [10:46:40] 06Release-Engineering-Team, 06collaboration-services: Deploying Phab/Phorge in devtools to new Trixie instance does not restart PHP 8.4 - https://phabricator.wikimedia.org/T433988#12201815 (10LSobanski) p:05Triage→03High [10:47:45] o/ CI seems to be failing with a JDK spawn issue: https://integration.wikimedia.org/ci/job/inference-services-pipeline-pre-commit-check/887/console [11:40:50] (03open) 10jnuche: jenkins-rel: update plugins to address vulnerabilities [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/122 (https://phabricator.wikimedia.org/T434186) [11:41:19] (03merge) 10jnuche: jenkins-rel: update plugins to address vulnerabilities [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/122 (https://phabricator.wikimedia.org/T434186) [12:03:57] 10Phabricator: Phabricator TOTP token should be evaluated against the currently valid one instead of expiring - https://phabricator.wikimedia.org/T434520 (10Michael) 03NEW [12:30:50] 06Project-Admins, 06Data-Persistence (work done), 07Developer Productivity: Fold Phab tags for optimizer-bug and Slow-DB-Query tag under Wikimedia-database-issue - https://phabricator.wikimedia.org/T305639#12202248 (10Ladsgroup) >>! In T305639#9431660, @Aklapper wrote: > That leaves us with the proposal "Mov... [12:42:21] 10Gerrit, 07Upstream: Gerrit's syntax highlighting for PHP code breaks when encountering an apostrophe in a // comment in a function call - https://phabricator.wikimedia.org/T372404#12202331 (10Paladox) https://gerrit-review.googlesource.com/c/gerrit/+/617661 [13:01:39] 10Continuous-Integration-Infrastructure, 10Castor: Speedup mwext-codehealth-master-non-voting Castor job - https://phabricator.wikimedia.org/T427471#12202414 (10hashar) [13:06:32] (03Restored) 10Hashar: jjb: don't save castor cache after mwext-codehealth-master-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/1294992 (https://phabricator.wikimedia.org/T427471) (owner: 10Phedenskog) [13:06:44] (03PS3) 10Hashar: jjb: don't save castor cache after mwext-codehealth-master-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/1294992 (https://phabricator.wikimedia.org/T433331) (owner: 10Phedenskog) [13:07:09] (03Abandoned) 10Hashar: jjb: don't save castor cache after mwext-codehealth-master-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/1294992 (https://phabricator.wikimedia.org/T433331) (owner: 10Phedenskog) [13:12:41] 10Phabricator, 06Product Safety and Integrity, 06Security-Team, 07Security: Creating security tasks from email - https://phabricator.wikimedia.org/T434536 (10Reedy) 03NEW [13:14:53] 10Continuous-Integration-Infrastructure, 10Castor, 13Patch-For-Review: Speedup mwext-codehealth-master-non-voting Castor job - https://phabricator.wikimedia.org/T427471#12202576 (10hashar) [13:42:36] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform: Investigate multiple npm cache folders for mwext-codehealth-master-non-voting - https://phabricator.wikimedia.org/T427822#12202863 (10hashar) [13:43:04] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform: Investigate multiple npm cache folders for mwext-codehealth-master-non-voting - https://phabricator.wikimedia.org/T427822#12202869 (10hashar) [13:46:42] (03PS3) 10Mhurd: jjb: Skip npm install for mwext PHPUnit coverage jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1321626 (https://phabricator.wikimedia.org/T427922) [13:50:39] (03CR) 10Jforrester: [C:03+1] "Should be good to deploy now." [integration/config] - 10https://gerrit.wikimedia.org/r/1321626 (https://phabricator.wikimedia.org/T427922) (owner: 10Mhurd) [13:56:33] 10Phabricator (Upstream), 06Product Safety and Integrity, 06Security-Team, 07Security, 07Upstream: Creating security tasks from email - https://phabricator.wikimedia.org/T434536#12203076 (10Aklapper) p:05Triage→03Low This ticket seems to mix "Somehow set view policy for tasks created via email" and "... [14:12:03] (03PS1) 10Hashar: jjb: skip npm install for CodeHealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1324323 (https://phabricator.wikimedia.org/T427822) [14:13:34] (03PS2) 10Hashar: jjb: skip npm install for CodeHealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1324323 (https://phabricator.wikimedia.org/T427822) [14:14:03] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform, 13Patch-For-Review: Investigate multiple npm cache folders for mwext-codehealth-master-non-voting - https://phabricator.wikimedia.org/T427822#12203185 (10hashar) a:03hashar > And I think the problem is that mwext-codehealth-master-non-vo... [14:18:08] (03CR) 10Hashar: "Once deployed, we can nuke the useless `/cache/npm` entries from the Castor server using:" [integration/config] - 10https://gerrit.wikimedia.org/r/1324323 (https://phabricator.wikimedia.org/T427822) (owner: 10Hashar) [14:38:08] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/WandaVoice] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1324328 [14:38:08] (03CR) 10QChris: [V:03+2 C:03+2] Allow “Gerrit Managers” to import history [extensions/WandaVoice] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1324328 (owner: 10QChris) [14:38:12] (03PS1) 10QChris: Import done. Revoke import grants [extensions/WandaVoice] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1324329 [14:38:12] (03CR) 10QChris: [V:03+2 C:03+2] Import done. Revoke import grants [extensions/WandaVoice] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1324329 (owner: 10QChris) [14:43:05] (03approved) 10jnuche: Add dashboard that show agent load and how it affects performance [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/7 (https://phabricator.wikimedia.org/T428994) (owner: 10phedenskog) [14:43:10] (03approved) 10jnuche: Add last job that finish per patch [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/9 (owner: 10phedenskog) [14:43:11] (03approved) 10jnuche: Record per-stage pass/fail outcome for quibble sections [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/10 (https://phabricator.wikimedia.org/T432934) (owner: 10phedenskog) [15:07:56] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12203525 (10ABran-WMF) In anticipation of merging [[ https://gerrit.wikimedia.org/r/1324275 | the first DNS change ]], I added some [[ https://gerrit.wik... [15:12:25] (03merge) 10jnuche: Add dashboard that show agent load and how it affects performance [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/7 (https://phabricator.wikimedia.org/T428994) (owner: 10phedenskog) [15:12:27] (03merge) 10jnuche: Add last job that finish per patch [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/9 (owner: 10phedenskog) [15:12:30] (03merge) 10jnuche: Record per-stage pass/fail outcome for quibble sections [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/10 (https://phabricator.wikimedia.org/T432934) (owner: 10phedenskog) [15:19:51] (03merge) 10dancy: Record the staged checkouts in deployment-info.json [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1238 [15:20:58] (03open) 10dancy: Release 4.279.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1244 [15:21:09] 10Continuous-Integration-Infrastructure, 07Jenkins: inference-services-pipeline-pre-commit-check failing with JDK error after Java upgrade - https://phabricator.wikimedia.org/T434572 (10kevinbazira) 03NEW [15:24:06] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team: inference-services-pipeline-pre-commit-check failing with JDK error after Java upgrade - https://phabricator.wikimedia.org/T434572#12203645 (10hashar) ` java.io.IOException: Failed to exec spawn helper: pid: 251512, exit code... [15:24:12] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team: inference-services-pipeline-pre-commit-check failing with JDK error after Java upgrade - https://phabricator.wikimedia.org/T434572#12203651 (10hashar) 05Open→03Resolved a:03hashar [15:24:13] (03merge) 10dancy: Release 4.279.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1244 [15:37:46] (03CR) 10Hashar: "@jforrester@wikimedia.org why do you rebase the patch? It did not have any merge conflict." [integration/config] - 10https://gerrit.wikimedia.org/r/1321626 (https://phabricator.wikimedia.org/T427922) (owner: 10Mhurd) [15:41:46] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team: inference-services-pipeline-pre-commit-check failing with JDK error after Java upgrade - https://phabricator.wikimedia.org/T434572#12203775 (10kevinbazira) [15:43:23] hashar: I made it depend on the patch it said to not merge until it was landed. [15:45:24] James_F: ahh yeah "Parent: b001d907 (Update Quibble jobs to 1.19.0) [15:45:29] Bingo. [15:45:29] and I further rebased it [15:45:32] * hashar facepalms [15:45:36] :-) [15:45:45] anyway I'll sync with Monte to get those deployed and verified [15:45:51] <3 [15:45:58] thx! [15:46:04] If you need me for my timezone, just ask. [15:46:34] I am hoping to loop Vaughn in it, I trained him on using JJB last week [15:46:44] but I will mention you as a backup, thanks for offering your support [15:47:07] Always. [15:51:08] 10Continuous-Integration-Infrastructure, 06collaboration-services, 10doc.wikimedia.org, 10OOUI: OOUI PHP demos page on doc.wikimedia.org does not load due to old PHP version - https://phabricator.wikimedia.org/T434151#12203813 (10LSobanski) p:05Triage→03Low [15:53:06] (03CR) 10Hashar: "Once deployed, we can nuke the useless npm cache entries from the Castor server using:" [integration/config] - 10https://gerrit.wikimedia.org/r/1321626 (https://phabricator.wikimedia.org/T427922) (owner: 10Mhurd) [15:55:31] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Zuul, 06collaboration-services: Increase memory for Zuul VMs - https://phabricator.wikimedia.org/T425679#12203847 (10LSobanski) 05Open→03Resolved a:05dduvall→03Dzahn We now have large hosts available for Zuul in eqiad. [15:56:19] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 10observability: Enable Zuul OpenTelemetry tracing - https://phabricator.wikimedia.org/T430900#12203851 (10LSobanski) >>! In T430900#12090950, @LSobanski wrote: > Is the work here to configure Zuul, configure the receiving syst... [16:01:42] 10GitLab, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10Development-Metrics: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#12203884 (10LSobanski) @Aklapper could you take a look? [16:05:07] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Seen), 06collaboration-services, 06SRE: Upgrade CI Jenkins ssh key to ecdsa - https://phabricator.wikimedia.org/T177826#12203907 (10LSobanski) @hashar any updates on this one? [16:19:41] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Seen), 06collaboration-services, 06SRE: Upgrade CI Jenkins ssh key to ecdsa - https://phabricator.wikimedia.org/T177826#12203976 (10hashar) I am not working on it, last time I acted on that front was a couple years ago at T37... [16:20:18] (03CR) 10Pwangai: "Skipping npm install will make repos that run "npm run coverage" fail. The command is used to generate Jest coverage, which is then ingest" [integration/config] - 10https://gerrit.wikimedia.org/r/1324323 (https://phabricator.wikimedia.org/T427822) (owner: 10Hashar) [16:23:36] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12204014 (10hashar) I think we need a few builds to check the be... [16:34:55] (03PS3) 10Hashar: jjb: skip npm install for CodeHealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1324323 (https://phabricator.wikimedia.org/T427822) [16:35:46] (03CR) 10Hashar: "The answer is the `releng/node24-test` image has an entry point that does in short:" [integration/config] - 10https://gerrit.wikimedia.org/r/1324323 (https://phabricator.wikimedia.org/T427822) (owner: 10Hashar) [16:45:00] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12204174 (10Lars) Thanks @hashar. It's currently looking even be... [16:50:24] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 05FY252627 Reading Lists (Graduating from Beta), 07OKR-Work, and 2 others: ReadingLists tests not run in jenkins gate and submit job - https://phabricator.wikimedia.org/T403560#12204204 (10ldelench_wmf) 05Stalled→03Resolved [16:50:56] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 10observability: Enable Zuul OpenTelemetry tracing - https://phabricator.wikimedia.org/T430900#12204207 (10hnowlan) >>! In T430900#12203851, @LSobanski wrote: >>>! In T430900#12090950, @LSobanski wrote: >> Is the work here to c... [16:56:14] 10Phabricator: Add "thank you" to award token - https://phabricator.wikimedia.org/T434584 (10Jdlrobson-WMF) 03NEW [16:57:00] 10Phabricator: Add "thank you" to award token - https://phabricator.wikimedia.org/T434584#12204258 (10Jdlrobson-WMF) [17:45:54] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834#12204478 (10brennen) [17:53:34] (03approved) 10damilare: Add jupyter to fundraising-ml-trixie [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/115 (https://phabricator.wikimedia.org/T434031) (owner: 10ejegg) [18:18:19] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834#12204682 (10brennen) [18:26:09] 10Phabricator (Upstream), 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed, and 2 others: Creating security tasks from email - https://phabricator.wikimedia.org/T434536#12204724 (10Reedy) [18:39:21] 10Scap, 10Observability-Logging, 13Patch-For-Review: Scap support of http basic auth for Logstash queries - https://phabricator.wikimedia.org/T434114#12204793 (10dancy) 05Open→03Resolved a:03dancy [18:53:26] 10Phabricator (Upstream), 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed, and 2 others: Creating security tasks from email - https://phabricator.wikimedia.org/T434536#12204938 (10sbassett) Back-ordering for now. [18:53:35] 10Phabricator (Upstream), 06Security-Team, 07SecTeam-Processed, 07Security, 07Upstream: Creating security tasks from email - https://phabricator.wikimedia.org/T434536#12204939 (10sbassett) [18:54:26] 10Phabricator (Upstream), 06Security-Team, 07SecTeam-Processed, 07Security, 07Upstream: Creating security tasks from email - https://phabricator.wikimedia.org/T434536#12204950 (10Reedy) >>! In T434536#12203077, @Aklapper wrote: > This ticket seems to mix "Somehow set view policy for tasks created via ema... [19:01:41] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for WouterKo - https://phabricator.wikimedia.org/T434605 (10WouterKo) 03NEW [19:03:46] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834#12205066 (10brennen) Out of an abundance of caution, going to roll back for {T434597}. It's very far from ca... [20:02:30] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12205264 (10hashar) That table is amazing. That made me remembe... [20:22:41] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12205386 (10hashar) I pick 6893 and 6894 because they both start... [20:42:56] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12205454 (10hashar) I then looked at the underlying hosts for ea... [20:44:50] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12205460 (10Lars) Thanks @hashar, very interesting. Differences... [21:00:08] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12205518 (10hashar) Back in Spring 2019, I have discovered some... [21:43:41] Yippee, build fixed! [21:43:41] Project mwcore-phpunit-coverage-master build #5445: 09FIXED in 43 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/5445/