[05:06:12] 06Machine-Learning-Team (Q4 FY2025-26): TTS batch generation pipeline for Featured Articles - https://phabricator.wikimedia.org/T428435#12068719 (10kevinbazira) Here are the measured file sizes from the batch-generation run so far: | Metric | Per section (.mp3) | Per section (.vtt) | Per article (total) | | Aver... [05:12:35] 06Machine-Learning-Team (Q4 FY2025-26): Explore options to run TTS models for evaluation - https://phabricator.wikimedia.org/T424378#12068722 (10kevinbazira) [05:22:57] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor Suggestion Mode, 07Epic: [Epic] Develop a framework for invalidating edit suggestions - https://phabricator.wikimedia.org/T429426#12068727 (10ppelberg) [05:23:05] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor Suggestion Mode, 07Epic: [Epic] Develop a framework for invalidating edit suggestions - https://phabricator.wikimedia.org/T429426#12068731 (10ppelberg) [05:52:07] 06Machine-Learning-Team (Q4 FY2025-26): Explore options to run TTS models for evaluation - https://phabricator.wikimedia.org/T424378#12068767 (10kevinbazira) We built a TTS prototype that provides the following API endpoints: | Endpoint | Method | What it does | | `/articles` | GET | List every pre-generated art... [07:32:19] 06Machine-Learning-Team (Q4 FY2025-26): Migrate TTS inference logic from v0 to v1 (without orchestration, storage, and serving layers) - https://phabricator.wikimedia.org/T430536#12068923 (10kevinbazira) [07:32:46] 06Machine-Learning-Team (Q4 FY2025-26): Explore options to run TTS models for evaluation - https://phabricator.wikimedia.org/T424378#12068939 (10kevinbazira) 05Open→03Resolved [08:41:14] 06Machine-Learning-Team (Q4 FY2025-26), 06Research, 05Goal, 13Patch-For-Review: Q4 FY2025-26 Goal: Text-to-Speech - https://phabricator.wikimedia.org/T419288#12069303 (10kevinbazira) **Satus Update:** Over time, we've gathered the Apps team's requirements and built the [TTS prototype](https://wiki-tts.too... [08:44:05] 06Machine-Learning-Team (Q4 FY2025-26), 06Research, 05Goal, 13Patch-For-Review: Q4 FY2025-26 Goal: Text-to-Speech - https://phabricator.wikimedia.org/T419288#12069306 (10kevinbazira) a:03kevinbazira [09:44:05] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 10ServiceOps-SharedInfra: Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12069654 (10Clement_Goubert) [09:49:05] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 10ServiceOps-SharedInfra: Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12069684 (10Clement_Goubert) [09:52:23] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 10ServiceOps-SharedInfra: Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12069694 (10Clement_Goubert) [09:54:55] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 10ServiceOps-SharedInfra: Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12069714 (10Clement_Goubert) I think the simplest way to go about this is to remove `X-Wikimedia-Debug` fr... [10:15:28] 06Machine-Learning-Team (Q4 FY2025-26), 10WE4.12 Content policy model evaluation, 05Goal: Q2 FY2025-26 Goal: Host a content policy evaluation model on LiftWing - https://phabricator.wikimedia.org/T418267#12069806 (10kevinbazira) a:03kevinbazira [10:22:10] 06Machine-Learning-Team (Q4 FY2025-26), 10WE4.12 Content policy model evaluation, 05Goal: Q2 FY2025-26 Goal: Host a content policy evaluation model on LiftWing - https://phabricator.wikimedia.org/T418267#12069850 (10kevinbazira) We deployed three content policy evaluation models on LiftWing: 1.In T418350 t... [10:22:21] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 10ServiceOps-SharedInfra, 13Patch-For-Review: Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12069858 (10hnowlan) I think given the API portal is deprecated and the /core/ APIs... [10:28:32] 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - Increase/refresh LLM generated suggestions. - https://phabricator.wikimedia.org/T428882#12069873 (10OKarakaya-WMF) We have received the final batch with 10381 suggestions equal or above to score 8.0 suggestion type specific scores are similar to prev... [10:30:33] 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - Increase/refresh LLM generated suggestions. - https://phabricator.wikimedia.org/T428882#12069875 (10OKarakaya-WMF) Combined the suggestions. We have 29350 suggestions in total. I share below the number of suggestions per suggestion type. {F91244475} [10:48:45] 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Editing Suggestions - Increase/refresh LLM generated suggestions. - https://phabricator.wikimedia.org/T428882#12069953 (10OKarakaya-WMF) [11:25:25] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06Traffic, 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12070060 (10gkyziridis) **Attempt: deploy qwen36-27b (FP8) on ml-serve1012 ** Changes (vs `ml-serve1013/1014`: - `affinity` → `ml-serve1... [12:01:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [12:01:49] Deployment revertrisk-multilingual-predictor-00002-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - ... [12:01:49] https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-multilingual-predictor-00002-deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [12:16:08] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 10ServiceOps-SharedInfra: Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12070270 (10Clement_Goubert) >>! In T428909#12069858, @hnowlan wrote: > I think given the API portal is de... [12:16:20] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 10ServiceOps-SharedInfra: Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12070272 (10Clement_Goubert) For now, I've removed any X-W-D handling on `api.wikimedia.org`. Other endp... [12:25:27] 06Machine-Learning-Team, 10decommission-hardware: decommission ml-cache* - https://phabricator.wikimedia.org/T430654 (10klausman) 03NEW [12:27:06] 06Machine-Learning-Team, 10decommission-hardware: decommission ml-cache* - https://phabricator.wikimedia.org/T430654#12070305 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8d4009dc-b938-46e5-bce2-0a024e9a27ad) set by klausman@cumin2002 for 365 days, 0:00:00 on 6 host(s) and their services... [12:27:14] 06Machine-Learning-Team, 10decommission-hardware: decommission ml-cache* - https://phabricator.wikimedia.org/T430654#12070306 (10klausman) [12:33:00] (03PS1) 10Gkyziridis: Qwen-model: add ENFORCE_EAGER env var to optionally disable CUDA graph capture [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1306680 (https://phabricator.wikimedia.org/T425680) [12:46:43] 06Machine-Learning-Team, 06Data-Platform-SRE: Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12070413 (10klausman) [12:47:13] 06Machine-Learning-Team, 06Data-Platform-SRE: Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12070415 (10klausman) ml-cache1002 has been decom'd (cookbook running atm) I'll take care of the ml-serve machines this week. [12:50:46] 06Machine-Learning-Team, 10decommission-hardware, 13Patch-For-Review: decommission ml-cache* - https://phabricator.wikimedia.org/T430654#12070439 (10klausman) [12:56:46] (03CR) 10Kevin Bazira: [C:03+1] Qwen-model: add ENFORCE_EAGER env var to optionally disable CUDA graph capture [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1306680 (https://phabricator.wikimedia.org/T425680) (owner: 10Gkyziridis) [12:57:14] 06Machine-Learning-Team, 10decommission-hardware, 13Patch-For-Review: decommission ml-cache* - https://phabricator.wikimedia.org/T430654#12070458 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by klausman@cumin2002 for hosts: `ml-cache[2001-2003].codfw.wmnet,ml-cache[1001-1003].eqiad.wmnet`... [13:02:29] 06Machine-Learning-Team: Investigate LW Consumer lag alerts - https://phabricator.wikimedia.org/T351735#12070489 (10isarantopoulos) 05Open→03Resolved a:03isarantopoulos Resolving as it is not relevant anymore [13:02:30] 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Editing Suggestions - Increase/refresh LLM generated suggestions. - https://phabricator.wikimedia.org/T428882#12070488 (10OKarakaya-WMF) [13:04:04] 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Editing Suggestions - Increase/refresh LLM generated suggestions. - https://phabricator.wikimedia.org/T428882#12070498 (10OKarakaya-WMF) Closing this task as we have completed generating suggestions for phase1 (experimental) and phase2 (default-on). W... [13:04:35] 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Editing Suggestions - Increase/refresh LLM generated suggestions. - https://phabricator.wikimedia.org/T428882#12070500 (10OKarakaya-WMF) 05Open→03Resolved [13:08:10] 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 06Traffic: k8s changes needed to allow article topic (and other future isvcs) to use the kserve v2 inference protocol (and gRPC) - https://phabricator.wikimedia.org/T424049#12070508 (10isarantopoulos) 05Open→03Resolved All changes have been re... [13:08:15] (03CR) 10Gkyziridis: [C:03+2] Qwen-model: add ENFORCE_EAGER env var to optionally disable CUDA graph capture [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1306680 (https://phabricator.wikimedia.org/T425680) (owner: 10Gkyziridis) [13:10:29] (03Merged) 10jenkins-bot: Qwen-model: add ENFORCE_EAGER env var to optionally disable CUDA graph capture [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1306680 (https://phabricator.wikimedia.org/T425680) (owner: 10Gkyziridis) [13:13:28] 06Machine-Learning-Team (Q4 FY2025-26): monitoring: View GPU usage per LLM deployment/model - https://phabricator.wikimedia.org/T429236#12070541 (10isarantopoulos) A WIP dashboard is available here https://grafana.wikimedia.org/d/dpzzsnh/ml-infra?orgId=1&from=now-12h&to=now&timezone=utc&var-cluster=000000006&var... [13:13:30] 06Machine-Learning-Team (Q4 FY2025-26): LLM observability - https://phabricator.wikimedia.org/T429597#12070542 (10isarantopoulos) A WIP dashboard is available here https://grafana.wikimedia.org/d/dpzzsnh/ml-infra?orgId=1&from=now-12h&to=now&timezone=utc&var-cluster=000000006&var-namespace=$__all&var-backend=$__a... [13:50:14] 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - Increase/refresh LLM generated suggestions. - https://phabricator.wikimedia.org/T428882#12070719 (10OKarakaya-WMF) We have processed 30K articles in total. We kept suggestions from 14K of them after filtering with the judge prompt [14:01:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [14:01:49] Deployment revertrisk-multilingual-predictor-00002-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - ... [14:01:49] https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-multilingual-predictor-00002-deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [17:42:29] 06Machine-Learning-Team, 10VisualEditor, 10VisualEditor Suggestion Mode, 06Editing-team (Editing-current-Q4-22Jun-3Jul-2026): LLM generated suggestions: attach the suggestion UUID to more events - https://phabricator.wikimedia.org/T430720 (10DLynch) 03NEW [17:42:38] 06Machine-Learning-Team, 10VisualEditor, 10VisualEditor Suggestion Mode, 06Editing-team (Editing-current-Q4-22Jun-3Jul-2026): LLM generated suggestions: attach the suggestion UUID to more events - https://phabricator.wikimedia.org/T430720#12072606 (10DLynch) [17:42:40] 06Machine-Learning-Team, 10VisualEditor Suggestion Mode, 06Editing-team (Editing-current-Q4-22Jun-3Jul-2026), 13Patch-For-Review: Build PoC for integrating initial set of MoS LLM-generated suggestions into Suggestion Mode as `experimental` - https://phabricator.wikimedia.org/T428641#12072607 (10DLynch) [18:57:53] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 10ServiceOps-SharedInfra: Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12072993 (10apaskulin) Thanks for working on this, @Clement_Goubert! [21:06:29] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 10Community Feedback (Growth), 06Growth-Team: AI/ML model update request: Named Entity Recognition for Add-a-Link - https://phabricator.wikimedia.org/T405185#12073746 (10Sdkb)