[09:00:35] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118300 (10CWilliams-WMF) [09:05:27] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118310 (10CWilliams-WMF) [09:54:30] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12118444 (10Gehel) [10:00:02] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07sre-alert-triage: Alert in need of triage: SmartNotHealthy (instance ml-serve1001:9100) - https://phabricator.wikimedia.org/T414969#12118461 (10Gehel) [10:01:23] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Add the ML team to the POSIX group `docker` on the ML lab machines. - https://phabricator.wikimedia.org/T393566#12118467 (10Gehel) [10:03:02] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Build the debian packages for amd node labeller and gpu plugin to pick up the new changes - https://phabricator.wikimedia.org/T423459#12118477 (10Gehel) [10:57:19] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07sre-alert-triage: Alert in need of triage: SmartNotHealthy (instance ml-serve1001:9100) - https://phabricator.wikimedia.org/T414969#12118642 (10klausman) a:03klausman [11:02:00] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Implement TTS Section Generator Phase 1: API contract and core library (without transcoding, deployment, and the batch pilot) - https://phabricator.wikimedia.org/T432009#12118687 (10kevinbazira) In P94816, the TTS section generator implemented in th... [11:08:19] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118704 (10cmooney) [11:08:58] 06Machine-Learning-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118705 (10taavi) [11:16:38] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12118738 (10cmooney) [11:17:33] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops: hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105 (10klausman) 03NEW [11:18:12] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07sre-alert-triage: Alert in need of triage: SmartNotHealthy (instance ml-serve1001:9100) - https://phabricator.wikimedia.org/T414969#12118754 (10klausman) Disk is indeed bad, have filed T432105 with DCOps to get a replace... [11:18:43] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07sre-alert-triage: Alert in need of triage: SmartNotHealthy (instance ml-serve1001:9100) - https://phabricator.wikimedia.org/T414969#12118756 (10klausman) p:05Triage→03Medium [11:22:34] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Add the ML team to the POSIX group `docker` on the ML lab machines. - https://phabricator.wikimedia.org/T393566#12118760 (10klausman) The scope here has changed a bit: `docker` membership is not needed on `ml-lab1002`, but o... [11:24:06] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Add the ML team to the POSIX group `docker` on `ml-build1001` - https://phabricator.wikimedia.org/T393566#12118767 (10klausman) [11:45:36] 06Machine-Learning-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118824 (10cmooney) [11:48:59] 06Machine-Learning-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118846 (10BTullis) I have fully drained dse-k8s-worker2003 and depooled cirrussearch2110. Good to go from our side. [12:02:11] (03PS1) 10Kevin Bazira: tts_section_generator: Add TTS Section Generator Phase 1 (API contract and core library) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1310559 (https://phabricator.wikimedia.org/T432009) [12:05:16] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-eqiad: hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105#12118961 (10cmooney) @VRiley-WMF I set the Netbox status for this one back to 'active'. I'm not 100% if that was the correct thing t... [12:22:55] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work, 13Patch-For-Review: Implement TTS Section Generator Phase 1: API contract and core library (without transcoding, deployment, and the batch pilot) - https://phabricator.wikimedia.org/T432009#12119038 (10kevinbazira) Unit tests are passing for the p... [12:24:57] (03CR) 10Kevin Bazira: [C:03+2] tts_section_generator: Add TTS Section Generator Phase 1 (API contract and core library) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1310559 (https://phabricator.wikimedia.org/T432009) (owner: 10Kevin Bazira) [12:26:28] (03Merged) 10jenkins-bot: tts_section_generator: Add TTS Section Generator Phase 1 (API contract and core library) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1310559 (https://phabricator.wikimedia.org/T432009) (owner: 10Kevin Bazira) [12:39:42] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Implement TTS Section Generator Phase 1: API contract and core library (without transcoding, deployment, and the batch pilot) - https://phabricator.wikimedia.org/T432009#12119126 (10kevinbazira) Below is the error taxonomy (deterministic 4xx) from t... [12:50:31] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105#12119155 (10VRiley-WMF) Thanks @cmooney, currently looking into this right now. It's not under warrenty, so I'll need to sou... [12:55:38] 06Machine-Learning-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12119177 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8b130fc6-8cf0-455b-b0d3-cfde4861384c) set by cmooney@cumin100... [12:57:32] 06Machine-Learning-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12119202 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=bdeaaae5-a950-451d-a6e0-365a469fd764) set by cmooney@cumin100... [13:01:10] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105#12119208 (10VRiley-WMF) @cmooney I just checked our invintory of hdds, and it seems like we don't have this one covered. We... [13:14:33] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Implement TTS Section Generator Phase 1: API contract and core library (without transcoding, deployment, and the batch pilot) - https://phabricator.wikimedia.org/T432009#12119301 (10kevinbazira) Here are results from the `/generate-section` endpoint... [13:20:49] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105#12119338 (10klausman) >>! In T432105#12119208, @VRiley-WMF wrote: > @cmooney I just checked our invintory of hdds, and it se... [13:29:52] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105#12119383 (10VRiley-WMF) Thanks for the update @klausman (I was going by who last replied, sorry about that!) I'll check wi... [13:38:15] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Implement TTS Section Generator Phase 1: API contract and core library (without transcoding, deployment, and the batch pilot) - https://phabricator.wikimedia.org/T432009#12119422 (10kevinbazira) Below is the error taxonomy (deterministic 4xx) from t... [13:51:43] 06Machine-Learning-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12119568 (10cmooney) Upgrade is now complete, all looks healthy. Thanks everyone for their assistance. [14:13:50] 06Machine-Learning-Team (Q1 FY2026-27): Create dataset of article titles for TTS V1 - https://phabricator.wikimedia.org/T432128 (10Sucheta-Salgaonkar-WMF) 03NEW