[07:42:42] (03CR) 10Bartosz Wójtowicz: "Thank you for the work here! Left a comment on how we could simplify the patch." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1308631 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [08:32:06] (03PS2) 10Kevin Bazira: tts: build Kokoro ONNX session with full graph optimization [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1308631 (https://phabricator.wikimedia.org/T430536) [08:35:14] (03CR) 10Kevin Bazira: tts: build Kokoro ONNX session with full graph optimization (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1308631 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [08:39:17] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: [Epic] Develop a framework for invalidating edit suggestions - https://phabricator.wikimedia.org/T429426#12103435 (10bmartinezcalvo) This task is related to {T420648}. [08:49:35] (03CR) 10Bartosz Wójtowicz: "Thank you! Left 2 more comments." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1308631 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [08:51:15] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12103481 (10ayounsi) 05Open→03Resolved Maintenance done, last repool in progress. [08:55:29] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12103507 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage... [08:57:46] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12103515 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by klausman@cum... [09:28:10] (03PS3) 10Kevin Bazira: tts: set explicit intra_op_num_threads on Kokoro's ONNX session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1308631 (https://phabricator.wikimedia.org/T430536) [09:38:47] (03CR) 10Bartosz Wójtowicz: [C:03+1] "The results seem amazing, LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1308631 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:39:51] (03CR) 10Kevin Bazira: [C:03+2] tts: set explicit intra_op_num_threads on Kokoro's ONNX session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1308631 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:40:44] (03Merged) 10jenkins-bot: tts: set explicit intra_op_num_threads on Kokoro's ONNX session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1308631 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:49:45] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12103823 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2003 for hosts: `testvm2007.codfw.... [09:50:12] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12103825 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by klausman@cumin10... [09:50:23] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12103826 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by klausman@cum... [10:00:47] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12103893 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2003 for hosts: `testvm2008.wikime... [10:21:31] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12104024 (10klausman) old: ml-serve1003.eqiad.wmnet has address 10.64.32.81 ml-serve1003.eqiad.w... [10:26:11] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12104039 (10ayounsi) 05Resolved→03Open [10:35:26] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12104080 (10ops-monitoring-bot) Draining ganeti2033.codfw.wmnet of running VMs [10:35:30] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12104082 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by klausman@cumin10... [10:47:59] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12104158 (10MoritzMuehlenhoff) [10:52:10] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: ORES flagging not working on simplewiki - https://phabricator.wikimedia.org/T431159#12104176 (10Samwalton9-WMF) >>! In T431159#12102643, @Ladsgroup wrote: >>>! In T431159#12087917, @Johannnes... [11:01:04] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12104219 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2003 for hosts: `testvm2005.codfw.wmne... [12:25:48] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): [draft] ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12104876 (10Gehel) [12:34:51] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): [draft] ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12104935 (10klausman) a:03klausman [12:53:01] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work, 13Patch-For-Review: Set ephemeral-storage requests/limits on ML predictor containers to prevent node disk eviction - https://phabricator.wikimedia.org/T431089#12105065 (10BWojtowicz-WMF) I've done first batch of updates targeting our LLM models, each... [13:19:38] 06Machine-Learning-Team (Q1 FY2026-27), 06Infrastructure-Foundations: Move all Machine Learning Docker images under the /ml prefix in the Docker Registry - https://phabricator.wikimedia.org/T428022#12105193 (10achou) Thanks for outlining the plan, Luca! We're aware of this task. Just let us know when you plan... [13:19:42] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12105196 (10klausman) 05In progress→03Resolved [13:29:49] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 06Traffic, 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12105235 (10BWojtowicz-WMF) I removed the `DTYPE=float16` and exchanged it with `DTYPE=auto` env var and service still responds happily,... [14:07:57] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: ORES flagging not working on simplewiki - https://phabricator.wikimedia.org/T431159#12105426 (10Ladsgroup) It works just fine for me o.O. It timed out before the clean up but not anymore. [14:11:02] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: ORES flagging not working on simplewiki - https://phabricator.wikimedia.org/T431159#12105438 (10Samwalton9-WMF) >>! In T431159#12105426, @Ladsgroup wrote: > It works just fine for me o.O. It... [15:43:44] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 10Community Feedback (Growth), 06Growth-Team: Investigate strategies to reduce prevalence of erroneous add-a-link suggestions of proper noun homonyms - https://phabricator.wikimedia.org/T409138#12106048 (10KStoller-WMF) [15:59:51] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: ORES flagging not working on simplewiki - https://phabricator.wikimedia.org/T431159#12106136 (10Ladsgroup) For me, it works logged in lol [16:07:29] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12106164 (10klausman) [16:17:06] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12106178 (10klausman) [16:22:14] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12106191 (10klausman) All ML worder nodes done. ML cache is already half-decommed and will e... [17:18:22] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12106350 (10cmooney)