[01:29:23] 06serviceops, 10envoy, 06SRE: Envoy config updates from v1.32 - https://phabricator.wikimedia.org/T409510#11474947 (10RLazarus) 05Open→03Resolved [01:29:33] 06serviceops, 10envoy, 06SRE: Upgrade Envoy to v1.32.12 - https://phabricator.wikimedia.org/T405808#11474949 (10RLazarus) 05Open→03Resolved [01:37:47] 06serviceops, 10envoy, 06SRE: Upgrade Envoy to >= 1.24 - https://phabricator.wikimedia.org/T380211#11474994 (10RLazarus) 05Open→03Resolved Still some hosts remaining to upgrade to 1.35 in T410975, but we don't need this umbrella task open to track the multi-stage upgrade anymore. [02:14:25] 06serviceops, 10MinT, 10Prod-Kubernetes, 06SRE, and 3 others: machinetranslation eqiad pods in state ContainerStatusUnknown - https://phabricator.wikimedia.org/T411058#11475082 (10RLazarus) Helm timed out again when I tried to deploy machinetranslation for the next round of envoy upgrades. I'll retitle thi... [02:15:20] 06serviceops, 10MinT, 10Prod-Kubernetes, 06SRE, and 3 others: Can't deploy machinetranslation due to exceeding resource quotas - https://phabricator.wikimedia.org/T411058#11475083 (10RLazarus) [08:13:12] Morning. Looking at the overdue alerts today I see a whole bunch of month old "Kubernetes worker wikikube-worker20xx.codfw.wmnet has been unschedulable for >= 24h k8s@codfw" [08:32:01] 06serviceops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, and 2 others: Cross-datacenter Docker Registry replication broken since 2025-04-27 - https://phabricator.wikimedia.org/T413008#11475596 (10MatthewVernon) This morning, there are 1519692 objects in `docker_registry_codfw` in eqiad, most... [09:05:50] 06serviceops, 06collaboration-services, 07Developer Productivity: deployment charts: automate testing on staging - https://phabricator.wikimedia.org/T413179 (10daniel) 03NEW [09:05:57] 06serviceops, 06collaboration-services, 07Developer Productivity: deployment charts: automate testing on staging - https://phabricator.wikimedia.org/T413179#11475719 (10daniel) [09:09:43] 06serviceops, 06collaboration-services, 07Developer Productivity, 13Patch-For-Review: deployment charts: automate testing on staging - https://phabricator.wikimedia.org/T413179#11475766 (10daniel) a:03daniel [09:16:39] 06serviceops, 06MediaWiki-Platform-Team, 06MW-Interfaces-Team, 05FY2025-26 KR 5.1, 07OKR-Work: rest gateway: add support for shadow policies (selective dry run) - https://phabricator.wikimedia.org/T413183 (10daniel) 03NEW [09:21:20] 06serviceops, 06MediaWiki-Platform-Team, 06MW-Interfaces-Team, 05FY2025-26 KR 5.1, 07OKR-Work: rest gateway: support multiple rate limits per route - https://phabricator.wikimedia.org/T413186 (10daniel) 03NEW [09:21:43] 06serviceops, 06MediaWiki-Platform-Team, 06MW-Interfaces-Team, 05FY2025-26 KR 5.1, 07OKR-Work: rest gateway: add support for shadow policies (selective dry run) - https://phabricator.wikimedia.org/T413183#11475852 (10daniel) [09:21:44] 06serviceops, 06MediaWiki-Platform-Team, 06MW-Interfaces-Team, 05FY2025-26 KR 5.1, 07OKR-Work: rest gateway: support multiple rate limits per route - https://phabricator.wikimedia.org/T413186#11475851 (10daniel) [09:21:45] 06serviceops, 06MW-Interfaces-Team, 06Traffic, 07Epic, and 3 others: Epic: API Rate Limiting Architecture - https://phabricator.wikimedia.org/T399291#11475853 (10daniel) [10:41:28] 06serviceops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, and 2 others: Cross-datacenter Docker Registry replication broken since 2025-04-27 - https://phabricator.wikimedia.org/T413008#11476200 (10MatthewVernon) ` Dec 19 10:25:35 ms-be2081 container-sync: Since Fri Dec 19 09:25:31 2025: 12 sy... [10:46:21] 06serviceops, 10Ceph, 06Data-Platform-SRE, 06Infrastructure-Foundations, and 3 others: Design and build the next generation of container-registry service for the WMF production realm - https://phabricator.wikimedia.org/T413080#11476221 (10BTullis) [11:19:20] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 4 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#11476299 (10MoritzMuehlenhoff) [11:19:59] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 4 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#11476300 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff All roles were migrated to Puppet 7 (except the remaining... [12:22:43] 06serviceops, 06collaboration-services, 07Developer Productivity, 13Patch-For-Review: deployment charts: automate testing on staging - https://phabricator.wikimedia.org/T413179#11476581 (10JMeybohm) FWIW there is the concept of helm test (https://helm.sh/docs/topics/chart_tests/) that is totally unused for... [14:50:57] 06serviceops, 10envoy, 06SRE: Upgrade Envoy to v1.32.12 - https://phabricator.wikimedia.org/T405808#11476968 (10MoritzMuehlenhoff) [15:02:10] 06serviceops, 07Epic, 06MediaWiki-Platform-Team (Kanban Board): Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11476998 (10Krinkle) >>! In T360995#11467234, @Novem_Linguae wrote: > Any thoughts yet on what version of PHP we should target next? 8.4 or 8.5?... [15:03:33] 06serviceops, 06MediaWiki-Engineering: Migrate Wikimedia production from PHP 8.4 to PHP 8.4 or PHP 8.5 - https://phabricator.wikimedia.org/T413223 (10Krinkle) 03NEW [15:04:10] 06serviceops, 06MediaWiki-Engineering: Migrate Wikimedia production from PHP 8.3 to PHP 8.4 or PHP 8.5 - https://phabricator.wikimedia.org/T413223#11477014 (10Krinkle) [15:04:15] 06serviceops, 07Epic, 06MediaWiki-Platform-Team (Kanban Board): Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11477015 (10Krinkle) [15:05:14] 06serviceops, 06MediaWiki-Engineering: Migrate Wikimedia production from PHP 8.3 to PHP 8.4 or PHP 8.5 - https://phabricator.wikimedia.org/T413223#11477028 (10Krinkle) >>! In T360995#11467234, @Novem_Linguae wrote: > Any thoughts yet on what version of PHP we should target next? 8.4 or 8.5? Picking this would... [16:39:34] 06serviceops, 07Epic, 06MediaWiki-Platform-Team (Kanban Board): Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11477407 (10Krinkle) //Final report for WE6.4.8 Support PHP 8.3 upgrade ([m:FY2025-2026#Q2](https://meta.wikimedia.org/wiki/Wikimedia_Foundation... [16:43:41] 06serviceops, 05WE4.2 Bot detection (WE4.2 hCaptcha editing trial): Increase in memcached get misses from mc1047 and mc1042 - https://phabricator.wikimedia.org/T412947#11477417 (10jijiki) By and large, identifying which memcached key is receiving the most misses is not straightforward, unless the application e... [17:36:39] 06serviceops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, and 2 others: Cross-datacenter Docker Registry replication broken since 2025-04-27 - https://phabricator.wikimedia.org/T413008#11477686 (10MatthewVernon) This has continued to proceed today: ` Dec 19 17:26:37 ms-be2081 container-sync:... [22:05:56] 06serviceops, 07Epic, 06MediaWiki-Platform-Team (Kanban Board): Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11478810 (10Scott_French) I've made a handful of (mostly minor) additional revisions to the SRE-driven items in the checklist: * ([[ https://wik...