[01:39:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:39:10] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435153 (10wmcs-alerts) 03NEW [01:44:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:16:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:21:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:12:29] 06Release-Engineering-Team (Doing 😎), 10SyntaxHighlight, 13Patch-For-Review: New 2.21.0 upstream release for Pygments - https://phabricator.wikimedia.org/T421653#12224752 (10bd808) [04:27:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:47:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:07:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:17:34] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Spike, 06Test Platform (Nairobi 29): Create a plan on how to make Castor save jobs async - https://phabricator.wikimedia.org/T425411#12224784 (10Mhurd) Spike write-up. Short version: async is doable and I have a design for it, but I recommen... [05:27:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:29:13] (03merge) 10phedenskog: Stop inserting duplicate quibble section rows [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/12 [05:30:04] (03merge) 10phedenskog: Save triggered job data to make it easy to which Castor job is slow [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/15 (https://phabricator.wikimedia.org/T435075) [05:34:44] (03open) 10phedenskog: Store voting flag and change number for each build [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/16 [05:34:51] (03merge) 10phedenskog: Store voting flag and change number for each build [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/16 [05:54:03] (03open) 10phedenskog: Only store numeric change and patchset values [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/17 [05:54:21] (03merge) 10phedenskog: Only store numeric change and patchset values [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/17 [06:13:05] (03open) 10phedenskog: Show which jobs have slow castor saves [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/18 (https://phabricator.wikimedia.org/T435075) [06:13:50] (03merge) 10phedenskog: Show which jobs have slow castor saves [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/18 (https://phabricator.wikimedia.org/T435075) [06:15:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:20:42] (03open) 10phedenskog: Add castor saves sorted by number of runs [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/19 (https://phabricator.wikimedia.org/T435075) [06:20:48] (03merge) 10phedenskog: Add castor saves sorted by number of runs [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/19 (https://phabricator.wikimedia.org/T435075) [06:34:14] (03open) 10phedenskog: Move Castor data to its own dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/20 (https://phabricator.wikimedia.org/T435075) [06:37:04] (03update) 10phedenskog: Move Castor data to its own dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/20 (https://phabricator.wikimedia.org/T435075) [06:37:29] (03merge) 10phedenskog: Move Castor data to its own dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/20 (https://phabricator.wikimedia.org/T435075) [06:44:04] (03open) 10phedenskog: Add total castor save time per triggering job [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/21 (https://phabricator.wikimedia.org/T435075) [06:44:41] 10GitLab: Internal Server Error when publishing a maven package from GitLab CI - https://phabricator.wikimedia.org/T434834#12224872 (10ABran-WMF) upon investigation, it seems you were hitting a `QuotaExceeded` limit on Apus: `api error QuotaExceeded: UnknownError"` [06:44:58] 10GitLab: Internal Server Error when publishing a maven package from GitLab CI - https://phabricator.wikimedia.org/T434834#12224873 (10ABran-WMF) [06:45:00] 10GitLab (Infrastructure), 06Collaboration-Services: Make sure GitLab does not exceed apus object storage quotas - https://phabricator.wikimedia.org/T413871#12224874 (10ABran-WMF) [06:45:11] (03merge) 10phedenskog: Add total castor save time per triggering job [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/21 (https://phabricator.wikimedia.org/T435075) [06:45:11] 10GitLab, 06Collaboration-Services: Internal Server Error when publishing a maven package from GitLab CI - https://phabricator.wikimedia.org/T434834#12224876 (10ABran-WMF) [07:10:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:18:11] (03CR) 10Phedenskog: [V:03+1] castor: Add rsync stats and timing to the cache save [integration/config] - 10https://gerrit.wikimedia.org/r/1321049 (https://phabricator.wikimedia.org/T432685) (owner: 10Pwangai) [07:20:15] (03PS1) 10Fomafix: Zuul: [mediawiki/extensions/CategoryTree] Add Translate Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1326694 [07:32:22] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Coverage builds fail with HTTP 429 error - https://phabricator.wikimedia.org/T435112#12224973 (10hashar) 05Open→03Resolved I am assuming that one was related to yesterday GitHub outage. [08:05:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [08:10:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [08:28:08] 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T430835#12225111 (10Aklapper) [09:07:54] 10GitLab, 06Collaboration-Services: Internal Server Error when publishing a maven package from GitLab CI - https://phabricator.wikimedia.org/T434834#12225358 (10Don-vip) Thanks, I have added a package prune mechanism in my pipeline, it solved the issue: https://gitlab.wikimedia.org/toolforge-repos/spacemedia/... [09:24:17] !log zuul: restarted zuul-web [09:24:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:25:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:40:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:56:31] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186 (10hashar) 03NEW [11:10:00] (03open) 10phedenskog: Split quibble failure rate by voting and non-voting jobs [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/22 [11:10:04] (03merge) 10phedenskog: Split quibble failure rate by voting and non-voting jobs [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/22 [11:23:55] (03open) 10marostegui: deployments-calendar.yaml: Remove Amir, add Ceri [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/261 [11:24:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:44:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:49:14] 10GitLab (Infrastructure), 06Collaboration-Services: Make sure GitLab does not exceed apus object storage quotas - https://phabricator.wikimedia.org/T413871#12225962 (10ABran-WMF) current quota usage: ` arnaudb@gitlab1004:~ $ s3cmd du -H s3://gitlab-packages/ 286G 5231 objects s3://gitlab-packages/ arnaudb... [11:49:29] 10GitLab (Infrastructure), 06Collaboration-Services: Make sure GitLab does not exceed apus object storage quotas - https://phabricator.wikimedia.org/T413871#12225963 (10ABran-WMF) [11:51:10] 10GitLab, 06Collaboration-Services: Internal Server Error when publishing a maven package from GitLab CI - https://phabricator.wikimedia.org/T434834#12225965 (10LSobanski) 05Open→03Resolved a:03ABran-WMF Good to see the quotas working, this looks OK to resolve. [11:51:50] (03CR) 10Gkyziridis: [C:03+1] "LGTM!" [integration/config] - 10https://gerrit.wikimedia.org/r/1326329 (https://phabricator.wikimedia.org/T429675) (owner: 10AikoChou) [11:57:03] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services, 13Patch-For-Review: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12225978 (10LSobanski) p:05Triage→03Medium [11:57:09] 10GitLab, 06Collaboration-Services: Internal Server Error when publishing a maven package from GitLab CI - https://phabricator.wikimedia.org/T434834#12225983 (10ABran-WMF) Thanks @Don-vip, you might free up to 35GB in the package bucket: ` repos/machine-learning/ml-pipelines... [12:02:14] 10Gerrit, 06Collaboration-Services, 06Infrastructure-Foundations, 13Patch-For-Review, 07Puppet: Change puppet-merge git origin to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org - https://phabricator.wikimedia.org/T420184#12226002 (10ABran-WMF) [12:18:14] (03open) 10taavi: Add Toolforge deployment configuration [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/126 [12:20:43] (03merge) 10taavi: Add Toolforge deployment configuration [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/126 [12:25:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:25:15] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://phabricator.wikimedia.org/T435193 (10wmcs-alerts) 03NEW [12:25:59] (03merge) 10hashar: deployments-calendar.yaml: Remove Amir, add Ceri [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/261 (owner: 10marostegui) [12:31:27] (03open) 10kamila: make-container-image: add mediawiki publish-85 flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/262 (https://phabricator.wikimedia.org/T432988) [12:31:53] (03update) 10kamila: make-container-image: add mediawiki publish-85 flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/262 (https://phabricator.wikimedia.org/T432988) [12:32:03] (03open) 10phedenskog: Add a dashboard for gate retry cost and suspected flaky jobs [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/23 [12:32:19] 06Release-Engineering-Team (Priority Backlog 📥), 06Collaboration-Services, 10Projects-Cleanup: Migrate SRE repositories to GitLab - Archiving unused Gerrit repositories - https://phabricator.wikimedia.org/T343707#12226134 (10LSobanski) 05Stalled→03Resolved a:03LSobanski This work is no longer activ... [12:32:42] (03merge) 10phedenskog: Add a dashboard for gate retry cost and suspected flaky jobs [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/23 [12:32:59] 06Release-Engineering-Team (Priority Backlog 📥), 06Collaboration-Services, 10Projects-Cleanup: Migrate SRE repositories to GitLab - Archiving unused Gerrit repositories - https://phabricator.wikimedia.org/T343707#12226141 (10LSobanski) 05Resolved→03Declined [12:33:22] 10GitLab (Project Migration), 06Collaboration-Services, 10Observability-Alerting: Migrate SRE repositories to GitLab - operations/alerts - https://phabricator.wikimedia.org/T349626#12226143 (10LSobanski) 05Stalled→03Declined This work is no longer actively tracked. [12:33:29] 10GitLab (Project Migration), 06Collaboration-Services: Migrate SRE repositories to GitLab - operations/software - https://phabricator.wikimedia.org/T341504#12226149 (10LSobanski) 05Stalled→03Declined This work is no longer actively tracked. [12:34:10] 10GitLab (Project Migration), 06Collaboration-Services: Migrate SRE repositories to GitLab - https://phabricator.wikimedia.org/T341468#12226152 (10LSobanski) 05Stalled→03Declined This work is no longer actively tracked. [12:40:03] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:13:03] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services, 13Patch-For-Review: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12226262 (10hashar) a:03hashar [13:40:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:40:21] (03CR) 10Hashar: [C:03+2] inference-services: Fix stale and incorrect Zuul job file filters [integration/config] - 10https://gerrit.wikimedia.org/r/1326329 (https://phabricator.wikimedia.org/T429675) (owner: 10AikoChou) [13:40:46] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/CategoryTree] Add Translate Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1326694 (owner: 10Fomafix) [13:42:24] (03Merged) 10jenkins-bot: inference-services: Fix stale and incorrect Zuul job file filters [integration/config] - 10https://gerrit.wikimedia.org/r/1326329 (https://phabricator.wikimedia.org/T429675) (owner: 10AikoChou) [13:43:14] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CategoryTree] Add Translate Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1326694 (owner: 10Fomafix) [13:44:38] (03CR) 10Hashar: [C:03+2] "deployed and I have triggered a recheck on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CategoryTree/+/1326693" [integration/config] - 10https://gerrit.wikimedia.org/r/1326694 (owner: 10Fomafix) [14:03:44] (03CR) 10Phedenskog: [V:03+2 C:03+1] castor: Add rsync stats and timing to the cache save [integration/config] - 10https://gerrit.wikimedia.org/r/1321049 (https://phabricator.wikimedia.org/T432685) (owner: 10Pwangai) [14:12:51] (03update) 10jforrester: fundraising: Drop libc-client-dev from the bookworm PHP 8.2 image [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/119 [14:21:54] (03open) 10jforrester: fab: Set base_images so builds are more reproducible [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/120 [14:22:00] (03update) 10jforrester: fab: Set base_images so builds are more reproducible [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/120 [14:22:16] 10Phabricator, 07I18n: Date format shows extra quotes in Chinese (zh_CN / zh_TW) i18n - https://phabricator.wikimedia.org/T435211 (10SD_hehua) 03NEW [14:22:56] 10Phabricator, 07I18n: Date format shows extra quotes in Chinese (zh_CN / zh_TW) i18n - https://phabricator.wikimedia.org/T435211#12226665 (10SD_hehua) [14:25:19] 10Phabricator, 07I18n: Date format shows extra quotes in Chinese (zh_hans / zh_hant) i18n - https://phabricator.wikimedia.org/T435211#12226682 (10SD_hehua) [14:27:34] 10Phabricator, 07I18n: Date format shows extra quotes in Chinese (zh-hans / zh-hant) i18n - https://phabricator.wikimedia.org/T435211#12226691 (10SD_hehua) [14:30:16] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work: Deploy Phab/Phorge 2026-08-18 - https://phabricator.wikimedia.org/T435213 (10brennen) 03NEW [14:30:47] (03open) 10brennen: update submodules for 2026-08-18 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/118 (https://phabricator.wikimedia.org/T435213) [14:31:42] 10Phabricator, 07I18n: Date format shows extra quotes in Chinese (zh-hans / zh-hant) i18n - https://phabricator.wikimedia.org/T435211#12226746 (10Pppery) > Translations of Phabricator do not seem to be on translatewiki.net. Actually they are https://translatewiki.net/wiki/Translating:Phabricator [14:33:09] 10Phabricator, 07I18n: Date format shows extra quotes in Chinese (zh-hans / zh-hant) i18n - https://phabricator.wikimedia.org/T435211#12226766 (10Pppery) 05Open→03Invalid Already fixed by Shizhao in https://translatewiki.net/w/i.php?title=Phabricator:arcanist-core-e0ac3c4e0da81dab/zh-hans&diff=prev&old... [14:33:40] 10Phabricator Translations Extension, 07I18n: Date format shows extra quotes in Chinese (zh-hans / zh-hant) i18n - https://phabricator.wikimedia.org/T435211#12226774 (10Pppery) [14:35:23] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Deploy Phab/Phorge 2026-08-18 - https://phabricator.wikimedia.org/T435213#12226779 (10ABran-WMF) [14:40:39] 10Phabricator Translations Extension, 07I18n: Date format shows extra quotes in Chinese (zh-hans / zh-hant) i18n - https://phabricator.wikimedia.org/T435211#12226822 (10SD_hehua) OK... [14:42:15] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:43:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:48:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:49:49] (03merge) 10brennen: fab: Set base_images so builds are more reproducible [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/120 (owner: 10jforrester) [14:56:49] (03merge) 10brennen: update submodules for 2026-08-18 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/118 (https://phabricator.wikimedia.org/T435213) [14:57:11] 10Phabricator (2026-08-18), 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Deploy Phab/Phorge 2026-08-18 - https://phabricator.wikimedia.org/T435213#12226905 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=56fc3e41-adf6-4c78-ae57-1fa6aa7c82... [14:58:35] 10Phabricator (2026-08-18), 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Deploy Phab/Phorge 2026-08-18 - https://phabricator.wikimedia.org/T435213#12226917 (10Aklapper) Things to be included in this week's deployment, or fixed since the last deployment: * Tr... [15:10:06] update.php is blowing up in beta cluster with `Fatal error: Declaration of Flow\Hooks::onCategoryViewerGenerateLink(MediaWiki\Context\IContextSource $context, string $type, MediaWiki\Page\PageReference $page, string $html, ?string &$link) must be compatible with MediaWiki\Category\Hook\CategoryViewerGenerateLinkHook::onCategoryViewerGenerateLink(MediaWiki\Context\IContextSource $context, string $type, MediaWiki\Page\PageReference $page, [15:10:06] ?string $html, ?string &$link) in /srv/mediawiki-staging/php-master/extensions/Flow/includes/Hooks.php on line 1348` [15:10:44] 10Phabricator (2026-08-18), 06Release-Engineering-Team (Doing 😎), 06Collaboration-Services, 07Essential-Work: Deploy Phab/Phorge 2026-08-18 - https://phabricator.wikimedia.org/T435213#12226963 (10Aklapper) 05Open→03Resolved [15:12:04] 10GitLab (CI & Job Runners), 13Patch-For-Review: gitlab-runners: WMCS fleet became ~6.5x slower at BuildKit filesystem operations overnight on 8/9 July 2026 - https://phabricator.wikimedia.org/T433505#12226965 (10Jdforrester-WMF) 05Open→03Resolved a:03dancy Update: Yes, the WCMS runners are fixed, th... [15:14:03] 06Release-Engineering-Team, 06Collaboration-Services: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12226971 (10brennen) Noting here after discussion with @Aklapper that we're pretty sure the current https://phabricator.wmcloud.org is working correctly. [15:14:28] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12226972 (10brennen) [15:16:07] 06Release-Engineering-Team, 10dev-images, 10MediaWiki-Docker, 07Essential-Work, 13Patch-For-Review: Upgrade MediaWiki-Docker PHP images to PHP 8.5 - https://phabricator.wikimedia.org/T407430#12226979 (10brennen) [15:17:20] (03approved) 10dancy: make-container-image: add mediawiki publish-85 flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/262 (https://phabricator.wikimedia.org/T432988) (owner: 10kamila) [15:17:29] 10Phabricator (2026-08-18), 06Release-Engineering-Team (Doing 😎): Remove diffusion.ssh-user setting - https://phabricator.wikimedia.org/T434728#12226992 (10Aklapper) 05Open→03Resolved [15:29:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:42:28] 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T430835#12227143 (10brennen) [15:53:28] (03merge) 10dancy: Add editcheck-headless to projects.json [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/182 (https://phabricator.wikimedia.org/T433514) (owner: 10kemayo) [15:55:24] bd808: that's also https://phabricator.wikimedia.org/T435161 (probably you know already by now, but sharing here for anyone else reading the backlog) [15:56:10] going to deploy those backports in two hours (next train window) [16:03:58] tgr_: Hi, if you have a minute I'd welcome your judgment on https://phabricator.wikimedia.org/T435167 - I believe it is not a trainer blocker; if you think it is, please mark it as such. Thanks! [16:06:29] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12227324 (10Aklapper) See also https://docs.google.com/document/d/1b-K6JUluT3c5I08mhcxsuPDTaismPDh01qnYBRQmMGA/edit?usp=sharing [16:23:17] (03open) 10pppery: Use PhutilNumber for $weeks [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/127 [16:30:42] (03open) 10dancy: deployment-info.json: Report the submodules of every checkout [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1263 (https://phabricator.wikimedia.org/T434726) [16:33:13] (03merge) 10dancy: deployment-info.json: Report the submodules of every checkout [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1263 (https://phabricator.wikimedia.org/T434726) [16:34:12] (03update) 10dancy: Release 4.282.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1264 [16:34:13] (03open) 10dancy: Release 4.282.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1264 [16:36:52] (03merge) 10dancy: Release 4.282.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1264 [16:37:10] andre: I hadn't looked for a phab task, but I did find the upstream cause and left a note there. Thanks for running the train and tracking these things. [16:37:51] 10GitLab (Project Migration), 10CopyPatrol: Migrate CopyPatrol repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395401#12227476 (10MusikAnimal) [16:51:15] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#12227559 (10Aklapper) @MelWMDE, @mfossati: Added you both. //Standard disclaimer: Please follow the [guidelines](https://www.mediawiki.org/wiki/Phabricator/Cr... [17:06:22] (03update) 10dancy: kubernetes.py: Split the command execution into K8sRunner [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1257 (https://phabricator.wikimedia.org/T412941) [17:13:24] 06Release-Engineering-Team, 10dev-images, 10MediaWiki-Core-Profiler, 10MediaWiki-Docker, and 2 others: Replace php-tideways_xhprof with php-xhprof in MediaWiki-Docker - https://phabricator.wikimedia.org/T401115#12227660 (10Jdforrester-WMF) This is now live in MW-master (and so MW REL1_47+). Is that suffici... [17:13:30] 10Continuous-Integration-Infrastructure, 10MediaWiki-Vendor, 07Composer, 13Patch-For-Review: Upgrade composer to >= 2.10.1 - https://phabricator.wikimedia.org/T428570#12227663 (10Jdforrester-WMF) [17:13:42] 10Continuous-Integration-Infrastructure, 10MediaWiki-Vendor, 07Composer, 13Patch-For-Review: Upgrade composer to >= 2.10.1 - https://phabricator.wikimedia.org/T428570#12227665 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF I believe we can (finally!) close this. [17:13:56] 06Release-Engineering-Team, 10dev-images, 10MediaWiki-Docker, 07Essential-Work, 13Patch-For-Review: Upgrade MediaWiki-Docker PHP images to PHP 8.5 - https://phabricator.wikimedia.org/T407430#12227668 (10Jdforrester-WMF) 05In progress→03Resolved [17:14:01] 06Release-Engineering-Team, 10dev-images, 13Patch-For-Review, 07PHP 8.5 support: create bookworm-php85-fpm and bookworm-php85-jobrunner images - https://phabricator.wikimedia.org/T413817#12227669 (10Jdforrester-WMF) 05In progress→03Resolved [17:25:55] Hmm. https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page isn't working [17:26:37] hmm. [17:27:07] what it says? (I just get the boring IP block page here) [17:27:19] "Our servers are currently under maintenance or experiencing a technical issue [17:27:32] Request served via deployment-cache-text08 deployment-cache-text08, Varnish XID 447119391 [17:27:32] Error: 503, Backend fetch failed at Tue, 18 Aug 2026 17:27:03 GMT [17:27:58] ooh, I see some ENOSPC errors in the logs. [17:28:30] Root FS is full on deployment-mediawiki14.deployment-prep [17:29:00] (03PS1) 10Hslater: Zuul: [BlueSpiceWikiFarm] Add StandardDialogs dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1326889 [17:29:19] /var/log/apache2 12.7GB [17:29:53] "log file filled up the filesystem" remains undefeated [17:31:31] !log rm /var/log/apache2/*.gz on deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud to free up ~6GB. [17:31:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:32:28] !log Rebooting deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud for good measure [17:32:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:33:32] I think the root problem is that beta is being scraped, resulting in too much traffic, resulting in too much logging. [17:33:46] https://en.wikipedia.beta.wmcloud.org/wiki/Main_Page is back in the meantime. [17:33:54] php-fpm processes very busy [17:45:27] (03approved) 10swfrench: kubernetes.py: Split the command execution into K8sRunner [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1257 (https://phabricator.wikimedia.org/T412941) (owner: 10dancy) [18:26:32] 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T430835#12228117 (10Aklapper) [18:52:41] Still a bit worried about the unknown impact of T435167 but all-in-all train looks fine so far [18:52:41] T435167: TypeError: MediaWiki\Extension\OAuth\Frontend\SpecialPages\SpecialMWOAuthListConsumers::updateLink(): Argument #2 ($user) must be of type MediaWiki\Extension\OAuth\Entity\UserEntity, null given, called in SpecialMWOAuthListConsumers.php on line 341 - https://phabricator.wikimedia.org/T435167 [19:12:58] note to myself: ^^ only happens when logged out of a wiki, thus not a train blocker [20:01:23] (03update) 10addshore: Add `ip_range` to reserve DNS IP address [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/660 (owner: 10haak) [20:23:17] (03approved) 10addshore: Add `ip_range` to reserve DNS IP address [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/660 (owner: 10haak) [20:24:19] (03merge) 10dancy: kubernetes.py: Split the command execution into K8sRunner [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1257 (https://phabricator.wikimedia.org/T412941) [20:27:29] (03merge) 10addshore: Add `ip_range` to reserve DNS IP address [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/660 (owner: 10haak) [20:41:43] (03open) 10dancy: main.py: Clarify what review_diffs() is returning [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1265 [20:51:27] (03open) 10dancy: tests: Do not require an interrupt to reach every job [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1266 [20:54:21] (03merge) 10dancy: tests: Do not require an interrupt to reach every job [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1266 [20:54:40] (03update) 10dancy: main.py: Clarify what review_diffs() is returning [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1265 [21:22:29] (03open) 10addshore: Fix various update command failures and make improvements [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/661 (https://phabricator.wikimedia.org/T393087) [21:27:55] (03open) 10addshore: Keep existing linux/arm fallback behavior, but fix detection so it actually [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/662 (https://phabricator.wikimedia.org/T435037) [21:29:17] (03open) 10dancy: kubernetes.py: Rename rollout_is_progressing to rollout_was_abandoned [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1267 (https://phabricator.wikimedia.org/T375514) [21:30:46] (03update) 10addshore: Fix various update command failures and make improvements [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/661 [21:39:22] (03open) 10addshore: Improve MWDD dashboard service detection performance. [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/663 (https://phabricator.wikimedia.org/T413248) [21:55:25] (03update) 10addshore: Keep existing linux/arm fallback behavior, but fix detection so it actually [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/662 (https://phabricator.wikimedia.org/T435037) [22:02:34] (03update) 10addshore: Improve MWDD dashboard service detection performance. [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/663 (https://phabricator.wikimedia.org/T413248) [23:59:01] (03update) 10swfrench: main.py: Clarify what review_diffs() is returning [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1265 (owner: 10dancy) [23:59:18] (03approved) 10swfrench: main.py: Clarify what review_diffs() is returning [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1265 (owner: 10dancy)