[00:50:05] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12336304 (10HNordeenWMF) [02:14:04] FIRING: KubernetesDeploymentUnavailableReplicas: ... [02:14:04] Deployment semantic-highlighting-predictor-00002-deployment in llm at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=llm&var-deployment=semantic-highlighting-predictor-00002-deployment - ... [02:14:04] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [06:14:04] FIRING: KubernetesDeploymentUnavailableReplicas: ... [06:14:04] Deployment semantic-highlighting-predictor-00002-deployment in llm at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=llm&var-deployment=semantic-highlighting-predictor-00002-deployment - ... [06:14:04] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [08:48:36] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Lower down cpu of the applications in llm namespace - https://phabricator.wikimedia.org/T438342#12336852 (10OKarakaya-WMF) 05Open→03Resolved [08:50:29] 06Machine-Learning-Team (Q1 FY2026-27): Increase LLM namespace cpu limit from 150 to 200 - https://phabricator.wikimedia.org/T438448 (10OKarakaya-WMF) 03NEW [08:50:49] 06Machine-Learning-Team (Q1 FY2026-27): Increase LLM namespace cpu limit from 150 to 200 - https://phabricator.wikimedia.org/T438448#12336867 (10OKarakaya-WMF) [08:51:13] 06Machine-Learning-Team (Q1 FY2026-27): Increase LLM namespace cpu limit from 150 to 200 - https://phabricator.wikimedia.org/T438448#12336869 (10OKarakaya-WMF) [09:18:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [09:18:49] Deployment semantic-highlighting-predictor-00002-deployment in llm at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=llm&var-deployment=semantic-highlighting-predictor-00002-deployment - ... [09:18:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [09:22:48] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work, 13Patch-For-Review: Increase LLM namespace cpu limit from 150 to 200 - https://phabricator.wikimedia.org/T438448#12336996 (10OKarakaya-WMF) [09:23:05] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work, 13Patch-For-Review: Increase LLM namespace cpu limit from 150 to 200 - https://phabricator.wikimedia.org/T438448#12337008 (10OKarakaya-WMF) 05Open→03Resolved [09:55:21] 06Machine-Learning-Team, 07Essential-Work: Optimize resource utilization for InferenceServices on LiftWing cluster - https://phabricator.wikimedia.org/T414431#12337148 (10OKarakaya-WMF) I've updated followings: embeddings-staging jina-embeddings jina-embeddings-staging task: https://phabricator.wikimedia.org... [10:02:02] 06Machine-Learning-Team (Q1 FY2026-27), 06Reader Growth Team: [ML TTS] Expose structured sentence, paragraph, and revision metadata for Read-Along synchronization - https://phabricator.wikimedia.org/T438228#12337159 (10isarantopoulos) [10:09:44] 06Machine-Learning-Team (Q1 FY2026-27), 10CirrusSearch, 10Semantic Search, 06Discovery-Search (2026.09.01 - 2026.10.02), 13Patch-For-Review: Create a liftwing service to extract semantic search query highlights - https://phabricator.wikimedia.org/T433872#12337170 (10dcausse) Should be done. @kevinbazira... [11:07:11] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Host Qwen3.8-27B-FP8 on LiftWing - https://phabricator.wikimedia.org/T436644#12337399 (10gkyziridis) === Update === After increasing the cpu quotas in llm namespace by merging/deploying this patch: [[ https://gerrit.wikimedia.org/r/c/o... [11:11:53] (03CR) 10AikoChou: [C:03+1] benchmarks: add llm-qwen38-27b baseline [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1335839 (https://phabricator.wikimedia.org/T437034) (owner: 10Gkyziridis) [11:18:24] (03CR) 10Gkyziridis: [C:03+2] benchmarks: add llm-qwen38-27b baseline [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1335839 (https://phabricator.wikimedia.org/T437034) (owner: 10Gkyziridis) [11:19:58] (03Merged) 10jenkins-bot: benchmarks: add llm-qwen38-27b baseline [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1335839 (https://phabricator.wikimedia.org/T437034) (owner: 10Gkyziridis) [11:53:57] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work: Upgrade airflow-ml to 3.3.x - https://phabricator.wikimedia.org/T435481#12337531 (10isarantopoulos) [11:54:20] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Set up a VM with nginx to store and serve TTS v1 audio files as the CDN origin - https://phabricator.wikimedia.org/T435952#12337534 (10isarantopoulos) 05Declined→03Resolved [11:54:30] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Set up a VM with nginx to store and serve TTS v1 audio files as the CDN origin - https://phabricator.wikimedia.org/T435952#12337536 (10isarantopoulos) 05Resolved→03Declined [12:13:26] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12337571 (10kevinbazira) **Weekly Update:** Storage is resolved and the TTS batch generation run has started. Audio and caption files are now being generated for... [13:16:02] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Add the ML team to the POSIX group `docker` on `ml-build1001` - https://phabricator.wikimedia.org/T393566#12337863 (10Gehel) [13:16:40] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: ml-serve-eqiad: Repartition two MI300X nodes from 24GB to 96GB partitions - https://phabricator.wikimedia.org/T436928#12337877 (10Gehel) [13:17:24] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: monitoring: Grafana dashboard for LLM serving on MI300X - https://phabricator.wikimedia.org/T429237#12337891 (10Gehel) [13:17:29] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: monitoring: View GPU usage per LLM deployment/model - https://phabricator.wikimedia.org/T429236#12337893 (10Gehel) [13:17:54] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Support GPU Partition specification in deployments - https://phabricator.wikimedia.org/T424923#12337906 (10Gehel) [13:18:06] 06Machine-Learning-Team (Q1 FY2026-27), 10Prod-Kubernetes, 06ServiceOps, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), and 3 others: Update istio-cni to 1.29 - https://phabricator.wikimedia.org/T434198#12337904 (10Gehel) [13:20:45] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09): ml-lab: Update pytorch-rocm to a current ROCm-compatible release - https://phabricator.wikimedia.org/T428023#12337979 (10Gehel) [13:20:51] 06Machine-Learning-Team, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Document procedure for updating the underlying distro for model servers (e.g. Bullseye -> Bookworm) - https://phabricator.wikimedia.org/T352189#12337981 (10Gehel) [13:25:17] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12338057 (10Gehel) [13:26:26] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work, 13Patch-For-Review: ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12338079 (10Gehel) [13:27:37] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: LLM observability - https://phabricator.wikimedia.org/T429597#12338116 (10Gehel) [13:27:59] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Upgrade airflow-ml to 3.3.x - https://phabricator.wikimedia.org/T435481#12338126 (10Gehel) [13:45:15] 06Machine-Learning-Team (Q1 FY2026-27), 10Add-Link-Structured-Task, 06Growth-Team: Incorporate user decision signals into Add Link model training - https://phabricator.wikimedia.org/T435831#12338294 (10OKarakaya-WMF) I've collected user decisions (accept/reject/revert): https://gitlab.wikimedia.org/repos/mac... [13:50:18] 06Machine-Learning-Team (Q1 FY2026-27), 10Community Feedback (Growth), 06Growth-Team, 07Epic: [EPIC] Add a Link: improvements requested by communities - https://phabricator.wikimedia.org/T434350#12338321 (10OKarakaya-WMF) Weekly update: - We have collected user decisions dataset including accept/reject/rev... [14:45:43] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Host Qwen3.8-27B-FP8 on LiftWing - https://phabricator.wikimedia.org/T436644#12338622 (10gkyziridis) === Qwen3.8-27b on LiftWing Studio === After the MR was merged I deployed the Qwen3.8 model on LiftWing studio. You can check this in... [15:37:45] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team: Add a link at English Wikipedia: do not suggest a link next to an existing one that would appear to be a single link - https://phabricator.wikimedia.org/T437929#12338901 (10Urbanecm_WMF) @OKarakaya-WMF This seems like something that would be... [16:06:59] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic: TTS MVP within "For you" Data Validation - https://phabricator.wikimedia.org/T438506 (10SNowick_WMF) 03NEW [20:27:46] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12340406 (10Dbrant) (took care of the P0 item) [20:40:43] 06Machine-Learning-Team (Q1 FY2026-27), 10Community Feedback (Growth), 06Growth-Team, 07Epic: [EPIC] Add a Link: improvements requested by communities - https://phabricator.wikimedia.org/T434350#12340465 (10KStoller-WMF)