[03:16:29] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: TTS Section Generator Phase 3: containerization with baked NeMo cache, staging deployment, and the timeout and blob-write de-risking spikes - https://phabricator.wikimedia.org/T432308#12126827 (10kevinbazira) We have containerized the TTS section ge... [04:45:46] (03PS1) 10Kevin Bazira: tts_section_generator: Add Phase 3 deployment readiness and de-risking spikes [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311170 (https://phabricator.wikimedia.org/T432308) [07:07:57] (03CR) 10Kevin Bazira: [C:03+2] tts_section_generator: Add Phase 3 deployment readiness and de-risking spikes [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311170 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [07:08:57] (03PS1) 10Kevin Bazira: tts-section-generator: Configure CI pipeline to run tests for this server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311315 (https://phabricator.wikimedia.org/T432308) [07:09:22] (03Merged) 10jenkins-bot: tts_section_generator: Add Phase 3 deployment readiness and de-risking spikes [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311170 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [07:11:36] (03PS2) 10Kevin Bazira: tts-section-generator: Configure CI pipeline to run tests and publish this server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311315 (https://phabricator.wikimedia.org/T432308) [07:17:16] (03CR) 10CI reject: [V:04-1] tts-section-generator: Configure CI pipeline to run tests and publish this server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311315 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [07:31:35] (03CR) 10Kevin Bazira: "The failing edit-check model-server CI pipeline will be fixed in:" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311315 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [08:16:04] (03CR) 10Hashar: "recheck after having deployed CI config https://gerrit.wikimedia.org/r/c/integration/config/+/1311317" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311315 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [08:28:11] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12127189 (10CWilliams-WMF) [08:28:25] (03CR) 10Kevin Bazira: [V:03+2 C:03+2] "thank you for deploying the CI config Antoine <3" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311315 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [08:31:27] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07Essential-Work: Move Lab machines into analytics net for DL access and switch to homedirs on Ceph - https://phabricator.wikimedia.org/T380279#12127199 (10klausman) [08:33:13] (03CR) 10CI reject: [V:04-1] tts-section-generator: Configure CI pipeline to run tests and publish this server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311315 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [08:34:04] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07Essential-Work: Move Lab machines into analytics net for DL access and switch to homedirs on Ceph - https://phabricator.wikimedia.org/T380279#12127210 (10klausman) 05In progress→03Resolved Most of this has been d... [08:34:19] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07Essential-Work: Move Lab machines into analytics net for DL access and switch to homedirs on Ceph - https://phabricator.wikimedia.org/T380279#12127213 (10klausman) [09:47:20] 06Machine-Learning-Team (Q1 FY2026-27): Publish V0 LiftWing LLM documentation on Wikitech - https://phabricator.wikimedia.org/T431847#12127578 (10achou) **Update: V0 documentation is live** The Lift Wing LLM documentation is now up to date on Wikitech: [[ https://wikitech.wikimedia.org/wiki/Machine_Learning/Lif... [09:48:12] 06Machine-Learning-Team (Q1 FY2026-27): Publish V0 LiftWing LLM documentation on Wikitech - https://phabricator.wikimedia.org/T431847#12127580 (10achou) [10:19:41] (03PS1) 10Kevin Bazira: tts-section-generator: Fix image name [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311417 (https://phabricator.wikimedia.org/T432308) [10:19:52] (03CR) 10CI reject: [V:04-1] tts-section-generator: Fix image name [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311417 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [10:24:13] (03PS2) 10Kevin Bazira: tts-section-generator: Fix image name [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311417 (https://phabricator.wikimedia.org/T432308) [11:31:06] (03CR) 10Kevin Bazira: [V:03+2 C:03+2] tts-section-generator: Fix image name [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311417 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [11:42:03] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12127936 (10cmooney) [12:25:20] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12128065 (10cmooney) [12:27:42] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12128073 (10cmooney) [14:12:11] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12128456 (10ops-monitoring-bot) Completed depooling of db2161 by cmooney@cumin1003: switch maintenance codfw rack b6 [14:12:46] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12128459 (10ops-monitoring-bot) Completed depooling of db2162 by cmooney@cumin1003: switch maintenance codfw rack b6 [14:12:59] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12128461 (10ops-monitoring-bot) Depooled db1152.eqiad.wmnet and db2251.codfw.wmnet switch maintenance codfw rack b6 -... [14:13:15] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12128462 (10ops-monitoring-bot) Depooled pc1022.eqiad.wmnet and pc2022.codfw.wmnet switch maintenance codfw rack b6 -... [14:19:22] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12128488 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=849e5c39-c3d5-4b31-9b03-154acdaf6b7f) se... [14:20:39] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12128491 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=017de456-7dae-4e7e-a386-dbeb39c04198) se... [14:25:42] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105#12128533 (10VRiley-WMF) CODFW is sending replacment disks. Thanks @Jhancock.wm [14:48:20] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12128631 (10cmooney) Switch is back online after reboot and all looks good at first glance. ` root@lsw1-b6-codfw> sho... [14:50:55] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12128645 (10ops-monitoring-bot) Starting pool of db2161 by cmooney@cumin1003: switch maintenance completed codfw rack b6 [14:52:17] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12128653 (10cmooney) @CWilliams-WMF I have triggered the re-pool for db2161 and db2162. Are you able to look at pc20... [15:29:19] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: Develop a persistence model for edit suggestions and their state - https://phabricator.wikimedia.org/T431606#12128820 (10dchan) See also T423465 [15:36:27] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12128848 (10ops-monitoring-bot) Completed pooling of db2161 by cmooney@cumin1003: switch maintenance completed codfw... [15:36:34] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12128849 (10ops-monitoring-bot) Starting pool of db2162 by cmooney@cumin1003: switch maintenance completed codfw rack b6 [15:46:39] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 06ServiceOps new, 10ServiceOps-SharedInfra, 06MediaWiki-Core-Platform-Team (Radar): Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12128911 (10jijiki) Tested https://gerrit.wikimedia.org/r/131010... [16:22:01] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12129166 (10ops-monitoring-bot) Completed pooling of db2162 by cmooney@cumin1003: switch maintenance completed codfw... [16:33:18] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12129249 (10cmooney) [18:35:47] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12129775 (10cmooney) 05Open→03Resolved