[07:43:34] (03PS1) 10Kevin Bazira: tts: add per-request timing, RTF logging, and startup warm-up [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309557 (https://phabricator.wikimedia.org/T430536) [08:14:54] (03CR) 10Bartosz Wójtowicz: "LGTM, just have one tiny nit :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309557 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [08:21:47] (03CR) 10Kevin Bazira: tts: add per-request timing, RTF logging, and startup warm-up (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309557 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [08:31:17] (03CR) 10Bartosz Wójtowicz: [C:03+1] tts: add per-request timing, RTF logging, and startup warm-up (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309557 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [08:31:57] (03CR) 10Kevin Bazira: [C:03+1] tts: add per-request timing, RTF logging, and startup warm-up [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309557 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [08:46:24] (03CR) 10Kevin Bazira: [C:03+2] tts: add per-request timing, RTF logging, and startup warm-up [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309557 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [08:50:13] (03Merged) 10jenkins-bot: tts: add per-request timing, RTF logging, and startup warm-up [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309557 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [08:54:23] (03PS1) 10Kevin Bazira: tts: make Wav2Vec2 aligner threads env-tunable (W2V2_THREADS) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309605 (https://phabricator.wikimedia.org/T430536) [09:05:49] (03CR) 10Bartosz Wójtowicz: [C:03+1] "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309605 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:06:34] (03CR) 10Kevin Bazira: [C:03+2] tts: make Wav2Vec2 aligner threads env-tunable (W2V2_THREADS) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309605 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:11:06] (03Merged) 10jenkins-bot: tts: make Wav2Vec2 aligner threads env-tunable (W2V2_THREADS) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309605 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:30:12] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 06Traffic: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12108249 (10BWojtowicz-WMF) The model is now deployed in the `llm` namespace and is exposed via REST Gateway with LiftWingLLM rate limits. Additionally, I've a... [09:32:17] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 06Traffic: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12108253 (10BWojtowicz-WMF) 05Open→03Resolved [11:04:23] (03PS1) 10Kevin Bazira: tts: serialize predict() since pipeline is not thread-safe [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309645 (https://phabricator.wikimedia.org/T430536) [11:08:59] (03CR) 10Kevin Bazira: [C:03+2] tts: serialize predict() since pipeline is not thread-safe [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309645 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [11:10:56] (03Merged) 10jenkins-bot: tts: serialize predict() since pipeline is not thread-safe [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309645 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [12:09:10] (03PS1) 10Bartosz Wójtowicz: qwen36: Allow configuring vLLM KV cache dtype [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309662 [13:05:32] (03PS1) 10Kevin Bazira: tts: overlap alignment with next-chunk synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309672 (https://phabricator.wikimedia.org/T430536) [13:06:25] (03CR) 10CI reject: [V:04-1] tts: overlap alignment with next-chunk synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309672 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [13:07:43] (03PS2) 10Kevin Bazira: tts: overlap alignment with next-chunk synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309672 (https://phabricator.wikimedia.org/T430536) [13:11:46] (03CR) 10Kevin Bazira: [C:03+2] tts: overlap alignment with next-chunk synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309672 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [13:14:10] (03CR) 10Kevin Bazira: [C:03+1] qwen36: Allow configuring vLLM KV cache dtype [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309662 (owner: 10Bartosz Wójtowicz) [13:15:50] (03Merged) 10jenkins-bot: tts: overlap alignment with next-chunk synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309672 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [13:59:10] (03PS1) 10Kevin Bazira: tts: add int16 PCM response encoding (new default), halving payload [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309677 (https://phabricator.wikimedia.org/T430536) [14:05:48] (03CR) 10Kevin Bazira: [C:03+2] tts: add int16 PCM response encoding (new default), halving payload [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309677 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [14:07:20] (03Merged) 10jenkins-bot: tts: add int16 PCM response encoding (new default), halving payload [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309677 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [14:23:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [14:23:49] Deployment revertrisk-multilingual-predictor-00001-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - ... [14:23:54] https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-multilingual-predictor-00001-deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [14:42:37] 06Machine-Learning-Team (Q1 FY2026-27): Q1 FY2026-27 Milestone 1: Deliver a V0 LLM platform for Wikimania - https://phabricator.wikimedia.org/T431846 (10achou) 03NEW [14:51:02] 06Machine-Learning-Team (Q1 FY2026-27): Publish V0 LiftWing LLM documentation on Wikitech - https://phabricator.wikimedia.org/T431847 (10achou) 03NEW [14:52:12] 06Machine-Learning-Team (Q1 FY2026-27), 10Wikidata, 06Wikidata-Omega (Radar/Epics/Stalled): Threshold analysis for WMDE use cases of Wikidata RR model - https://phabricator.wikimedia.org/T429049#12109646 (10karapayneWMDE) [15:01:25] 06Machine-Learning-Team (Q1 FY2026-27): Establish a benchmarking baseline for LiftWing LLM serving - https://phabricator.wikimedia.org/T431851 (10achou) 03NEW [15:06:55] 06Machine-Learning-Team (Q1 FY2026-27): Host Qwen3-14B FP8 as an inference service - https://phabricator.wikimedia.org/T431852 (10achou) 03NEW [15:07:58] 06Machine-Learning-Team (Q1 FY2026-27): Q1 FY2026-27 Milestone 1: Deliver a V0 LLM platform for Wikimania - https://phabricator.wikimedia.org/T431846#12109744 (10achou) [15:08:00] 06Machine-Learning-Team (Q1 FY2026-27): Host Qwen3-14B FP8 as an inference service - https://phabricator.wikimedia.org/T431852#12109743 (10achou) [15:08:22] 06Machine-Learning-Team (Q1 FY2026-27): Q1 FY2026-27 Milestone 1: Deliver a V0 LLM platform for Wikimania - https://phabricator.wikimedia.org/T431846#12109746 (10achou) [15:08:23] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 06Traffic: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12109745 (10achou) [15:08:30] 06Machine-Learning-Team (Q1 FY2026-27): Q1 FY2026-27 Milestone 1: Deliver a V0 LLM platform for Wikimania - https://phabricator.wikimedia.org/T431846#12109748 (10achou) [15:08:31] 06Machine-Learning-Team (Q1 FY2026-27): Establish a benchmarking baseline for LiftWing LLM serving - https://phabricator.wikimedia.org/T431851#12109747 (10achou) [15:08:39] 06Machine-Learning-Team (Q1 FY2026-27): Q1 FY2026-27 Milestone 1: Deliver a V0 LLM platform for Wikimania - https://phabricator.wikimedia.org/T431846#12109750 (10achou) [15:08:40] 06Machine-Learning-Team (Q1 FY2026-27): Publish V0 LiftWing LLM documentation on Wikitech - https://phabricator.wikimedia.org/T431847#12109749 (10achou) [15:23:43] (03PS1) 10Kevin Bazira: tts: add timestamps request parameter (full | proportional | none) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309686 (https://phabricator.wikimedia.org/T430536) [15:27:16] (03CR) 10CI reject: [V:04-1] tts: add timestamps request parameter (full | proportional | none) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309686 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [15:27:41] 06Machine-Learning-Team (Q1 FY2026-27): Q1 FY2026-27 Milestone 1: Deliver a V0 LLM platform for Wikimania - https://phabricator.wikimedia.org/T431846#12109813 (10kevinbazira) [15:32:52] (03PS2) 10Kevin Bazira: tts: add timestamps request parameter (full | proportional | none) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309686 (https://phabricator.wikimedia.org/T430536) [15:36:43] (03CR) 10Kevin Bazira: [C:03+2] tts: add timestamps request parameter (full | proportional | none) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309686 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [15:39:36] (03Merged) 10jenkins-bot: tts: add timestamps request parameter (full | proportional | none) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309686 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [15:44:22] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 06ServiceOps new, 10ServiceOps-SharedInfra: Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12109864 (10Scott_French) Cross-posting from {T431838}: https://gerrit.wikimedia.org/r/1306905 seems to ha... [17:52:01] 06Machine-Learning-Team (Q1 FY2026-27): Migrate TTS inference logic from v0 to v1 (without orchestration, storage, and serving layers) - https://phabricator.wikimedia.org/T430536#12110251 (10kevinbazira) Following the initial deployment in T430536#12093268, we profiled and optimized this isvc as shown below: **... [18:12:17] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12110317 (10kevinbazira) **Weekly Update:** In T430536, we started migrating the TTS inference logic from v0 to v1 (without orchestration, storage, and serving la... [18:28:34] FIRING: KubernetesDeploymentUnavailableReplicas: ... [18:28:34] Deployment revertrisk-multilingual-predictor-00001-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - ... [18:28:39] https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-multilingual-predictor-00001-deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [18:34:02] 06Machine-Learning-Team (Q1 FY2026-27): Q1 FY2026-27 Milestone 1: Deliver a V0 LLM platform for Wikimania - https://phabricator.wikimedia.org/T431846#12110357 (10achou) [18:34:03] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: monitoring: Grafana dashboard for LLM serving on MI300X - https://phabricator.wikimedia.org/T429237#12110356 (10achou) [18:34:07] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: monitoring: View GPU usage per LLM deployment/model - https://phabricator.wikimedia.org/T429236#12110358 (10achou) [18:34:09] 06Machine-Learning-Team (Q1 FY2026-27): Q1 FY2026-27 Milestone 1: Deliver a V0 LLM platform for Wikimania - https://phabricator.wikimedia.org/T431846#12110359 (10achou) [18:35:21] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: monitoring: Grafana dashboard for LLM serving on MI300X - https://phabricator.wikimedia.org/T429237#12110360 (10achou) [18:35:22] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: LLM observability - https://phabricator.wikimedia.org/T429597#12110361 (10achou) [18:35:28] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: monitoring: View GPU usage per LLM deployment/model - https://phabricator.wikimedia.org/T429236#12110362 (10achou) [18:35:29] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: LLM observability - https://phabricator.wikimedia.org/T429597#12110363 (10achou) [21:08:34] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [21:08:34] Deployment revertrisk-multilingual-predictor-00001-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - ... [21:08:34] https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-multilingual-predictor-00001-deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas