[00:08:18] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: Backplane failure for wikikube-worker1243.eqiad.wmnet - https://phabricator.wikimedia.org/T397851#10965830 (10Jclark-ctr) @Clement_Goubert it has cleared for the time i am still working with dell since this seems to be reoccurring issues i... [00:09:02] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: Backplane error for wikikube-worker1069.eqiad.wmnet - https://phabricator.wikimedia.org/T397829#10965831 (10Jclark-ctr) @Clement_Goubert it has cleared for the time i am still working with dell since this seems to be reoccurring issues if... [00:40:42] 06serviceops, 06MediaWiki-Engineering, 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08): Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition - https://phabricator.wikimedia.org/T394556#10965853 (10Scott_French) [00:46:01] 06serviceops, 06MediaWiki-Engineering, 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08): Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition - https://phabricator.wikimedia.org/T394556#10965860 (10Scott_French) My sincere thanks for all of your help here @tstarling. With the error handling fix for `... [00:47:12] 06serviceops, 06MediaWiki-Engineering, 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08): Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition - https://phabricator.wikimedia.org/T394556#10965861 (10Scott_French) [01:11:26] 06serviceops, 06MediaWiki-Engineering, 05MW-1.43-notes, 05MW-1.44-notes, 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08): Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition - https://phabricator.wikimedia.org/T394556#10965874 (10Scott_French) [01:26:11] 06serviceops, 06MediaWiki-Engineering, 05MW-1.43-notes, 05MW-1.44-notes, 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08): Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition - https://phabricator.wikimedia.org/T394556#10965877 (10Scott_French) 05Open→03Resolved p:05Triage→03Medium [08:48:59] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Update wikikube codfw to kubernetes 1.31 - https://phabricator.wikimedia.org/T397148#10966464 (10JMeybohm) [08:50:30] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Update wikikube codfw to kubernetes 1.31 - https://phabricator.wikimedia.org/T397148#10966466 (10JMeybohm) 05Open→03Resolved I've addressed all action items (apart from the ones with dedicated tasks) and moved the upgrade documentat... [09:01:26] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Ensure the correct helm version is used for each cluster - https://phabricator.wikimedia.org/T388390#10966512 (10JMeybohm) >>! In T388390#10807496, @kamila wrote: > further progress soft-blocked by T388969 -- wh... [10:37:04] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: Backplane failure for wikikube-worker1243.eqiad.wmnet - https://phabricator.wikimedia.org/T397851#10966992 (10Clement_Goubert) We don't particularly need the node in production as we have spare capacity, if you need them depooled for testing w... [10:37:34] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: Backplane error for wikikube-worker1069.eqiad.wmnet - https://phabricator.wikimedia.org/T397829#10966999 (10Clement_Goubert) We don't particularly need the node in production as we have spare capacity, if you need them depooled for testing we... [11:56:56] 06serviceops: Provide nodejs22 base images for production - https://phabricator.wikimedia.org/T393437#10967335 (10Jdforrester-WMF) It'd be great if we could do Node 24 at the same time. [12:04:45] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: Backplane failure for wikikube-worker1243.eqiad.wmnet - https://phabricator.wikimedia.org/T397851#10967364 (10Jclark-ctr) if you can deploy them that would be great so there is some load @Clement_Goubert [12:59:13] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10967612 (10bking) Per IRC conversation with @JMeybohm and @Clement_Goubert , here are the instructions for failing over `rdf-streamin... [13:04:40] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: Backplane error for wikikube-worker1069.eqiad.wmnet - https://phabricator.wikimedia.org/T397829#10967629 (10Jclark-ctr) Dell just responded they are going to send a new backplane for this device it will probably not arrive till Thursday /Satur... [13:12:30] 06serviceops, 10Data-Engineering-Roadmap, 10Dumps-Generation, 10MW-on-K8s, and 3 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10967678 (10BTullis) 05Open→03Resolved a:03BTullis I have created... [13:24:38] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10967766 (10JMeybohm) >>! In T341984#10967612, @bking wrote: > Per IRC conversation with @JMeybohm and @Clement_Goubert , here are the... [13:42:46] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10967868 (10bking) Thanks @JMeybohm ! I'm confirming that the previous request no longer applies, and we can handle this through email... [14:10:18] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 3 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#10967996 (10MoritzMuehlenhoff) [14:10:28] 06serviceops, 10Data-Platform-SRE (2025.06.13 - 2025.07.04), 10SRE Observability (FY2025/2026-Q1): Ensure DPE SRE can receive alerts for applications hosted in wikikube - https://phabricator.wikimedia.org/T398073#10967998 (10lmata) [14:39:53] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 3 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#10968153 (10MoritzMuehlenhoff) [18:53:06] 06serviceops: Prepare WMF PHP 8.3 packages for bullseye - https://phabricator.wikimedia.org/T398245#10969415 (10Reedy) [19:06:08] 06serviceops, 06collaboration-services, 10GitLab, 07Kubernetes: Trusted gitlab runner containers need access to staging k8s cluster - https://phabricator.wikimedia.org/T325385#10969469 (10Dzahn) a:05Dzahn→03None [21:05:38] 06serviceops, 10Beta-Cluster-Infrastructure, 10Thumbor, 07Beta-Cluster-reproducible: deployment-prep needs a Thumbor instance - https://phabricator.wikimedia.org/T344605#10969961 (10bd808) 05Open→03Resolved a:03taavi Thumbor is running in a docker container on deployment-imagescaler04. There appe... [22:44:41] 06serviceops, 06collaboration-services, 10GitLab, 07Kubernetes: Trusted gitlab runner containers need access to staging k8s cluster - https://phabricator.wikimedia.org/T325385#10970341 (10dancy) 05Open→03Declined