[02:45:16] (03PS1) 10Kevin Bazira: tts-section-generator: Add ffmpeg to production image [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311599 (https://phabricator.wikimedia.org/T432308) [02:52:43] (03CR) 10Kevin Bazira: [C:03+2] tts-section-generator: Add ffmpeg to production image [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311599 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [02:54:01] (03Merged) 10jenkins-bot: tts-section-generator: Add ffmpeg to production image [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311599 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [04:44:45] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 13Patch-For-Review: [Milestone] Deploy LLM-generated MoS suggestions as experimental (Phase 1) - https://phabricator.wikimedia.org/T428311#12131260 (10ppelberg) [09:04:22] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 06ServiceOps new, 10ServiceOps-SharedInfra, 06MediaWiki-Core-Platform-Team (Radar): Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12131580 (10Jgiannelos) Looks ok: ` curl -v -o /dev/null 'https:... [09:52:17] 06Machine-Learning-Team (Q1 FY2026-27): Establish a benchmarking baseline for LiftWing LLM serving - https://phabricator.wikimedia.org/T431851#12131731 (10OKarakaya-WMF) thank you @achou for working on this. adding a small note: vllm bench serve can return End-to-End Latency metrics by passing `--percentile-met... [11:44:43] (03PS1) 10AikoChou: benchmarks: Add LLM serving benchmarking tooling and V0 baseline [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311835 (https://phabricator.wikimedia.org/T431851) [11:47:55] (03PS2) 10AikoChou: benchmarks: Add LLM serving benchmarking tooling and V0 baseline [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1311835 (https://phabricator.wikimedia.org/T431851) [12:15:33] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work, 13Patch-For-Review: TTS Section Generator Phase 3: containerization with baked NeMo cache, staging deployment, and the timeout and blob-write de-risking spikes - https://phabricator.wikimedia.org/T432308#12132159 (10klausman) - [`admin_ng` changes... [12:24:31] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Establish a benchmarking baseline for LiftWing LLM serving - https://phabricator.wikimedia.org/T431851#12132180 (10achou) Thanks @OKarakaya-WMF. Yes, I'm passing `--percentile-metrics ttft,tpot,itl,e2el`. On why I switched to `vllm-bench` (the standal... [12:29:57] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Establish a benchmarking baseline for LiftWing LLM serving - https://phabricator.wikimedia.org/T431851#12132189 (10achou) **V0 benchmark baseline — `llm-qwen36-27b` (FP8)** First baseline recorded. Run from the ML Lab host with `vllm-bench` against t... [14:41:55] 06Machine-Learning-Team (Q1 FY2026-27): Q1 FY2026-27 Milestone 1: Deliver a V0 LLM platform for Wikimania - https://phabricator.wikimedia.org/T431846#12132590 (10achou) **Update** * Qwen3-27B FP8 (including FP8 KV cache) is now deployed (T431978). * Initial benchmarking baseline for Qwen3-27B FP8 has been recor... [15:10:00] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Build the debian packages for amd node labeller and gpu plugin to pick up the new changes - https://phabricator.wikimedia.org/T423459#12132667 (10elukey) 05Open→03Resolved a:03elukey Got pinged by Aiko on this one,... [15:25:55] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12132718 (10kevinbazira) **Weekly Update:** We completed phase 1 (Contract and Core Library) and phase 2 (Artifacts and Hardening) of the TTS section generator.... [17:25:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [17:25:49] Deployment revertrisk-multilingual-predictor-00001-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - ... [17:25:49] https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-multilingual-predictor-00001-deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [18:30:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [18:30:49] Deployment revertrisk-multilingual-predictor-00001-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - ... [18:30:49] https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-multilingual-predictor-00001-deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas