[06:53:25] (03PS1) 10Kevin Bazira: tts-section-generator: publish index.json for consumers [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1342548 (https://phabricator.wikimedia.org/T436758) [07:07:40] (03CR) 10Ozge: [C:03+1] tts-section-generator: publish index.json for consumers [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1342548 (https://phabricator.wikimedia.org/T436758) (owner: 10Kevin Bazira) [07:36:39] (03CR) 10Kevin Bazira: [C:03+2] tts-section-generator: publish index.json for consumers [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1342548 (https://phabricator.wikimedia.org/T436758) (owner: 10Kevin Bazira) [07:42:46] (03Merged) 10jenkins-bot: tts-section-generator: publish index.json for consumers [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1342548 (https://phabricator.wikimedia.org/T436758) (owner: 10Kevin Bazira) [07:51:59] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12332140 (10isarantopoulos) [08:02:49] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Publish TTS v1 audio files with standard caching at the CDN - https://phabricator.wikimedia.org/T436758#12332166 (10isarantopoulos) @Dbrant the API will be https://analytics.wikimedia.org/published/datasets/ml/tts/experiment-v1/ which can also be cach... [09:44:15] (03Abandoned) 10Ilias Sarantopoulos: docker-compose: Enable Swagger UI docs for all KServe services. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254845 (https://phabricator.wikimedia.org/T332602) (owner: 10Ilias Sarantopoulos) [09:44:52] (03Abandoned) 10Ilias Sarantopoulos: feat(openapi-specs): Expose edit-check spec from the model server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1296565 (https://phabricator.wikimedia.org/T427902) (owner: 10Ilias Sarantopoulos) [11:19:52] (03CR) 10AikoChou: "Thanks for working on this! I tested it locally and it works. I only have one comment, plus a small non-blocking suggestion." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1338218 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [11:32:34] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Host Qwen3.8-27B-FP8 on LiftWing - https://phabricator.wikimedia.org/T436644#12333042 (10gkyziridis) ### Qwen3.8-27B deployed to llm namespace. #### It is serving, but the isvc route is blocked by namespace CPU quotas llm-qwen38-27b... [12:05:14] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work: ml-serve-eqiad: Repartition two MI300X nodes from 24GB to 96GB partitions - https://phabricator.wikimedia.org/T436928#12333191 (10Gehel) a:03DPogorzelski-WMF [13:08:34] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12333559 (10isarantopoulos) a:05isarantopoulos→03klausman thanks for creating this task! re-assigning Tobias as the technical contact for this. ML... [13:08:55] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12333562 (10isarantopoulos) [13:10:22] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12333566 (10klausman) Racking details look good, Rob. I think the power/cooling constraints are already tighter than what we care about placement-wise... [13:13:16] 06Machine-Learning-Team (Q1 FY2026-27): Publish TTS v1 audio files with standard caching at the CDN - https://phabricator.wikimedia.org/T436758#12333575 (10kevinbazira) >>! In T436758#12328674, @Dbrant wrote: > Thanks @kevinbazira! > How does this affect the API that the app will use to fetch the audio and vtt c... [13:13:38] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Lower down cpu of the applications in llm namespace - https://phabricator.wikimedia.org/T438342 (10OKarakaya-WMF) 03NEW [13:19:02] 06Machine-Learning-Team (Q1 FY2026-27): Publish TTS v1 audio files with standard caching at the CDN - https://phabricator.wikimedia.org/T436758#12333617 (10kevinbazira) I have completed the attached publication risk mitigation checklist from the [Data Publications Guidelines](https://foundation.wikimedia.org/wik... [13:19:24] 06Machine-Learning-Team (Q1 FY2026-27): Publish TTS v1 audio files with standard caching at the CDN - https://phabricator.wikimedia.org/T436758#12333622 (10kevinbazira) [13:59:44] 06Machine-Learning-Team (Q1 FY2026-27): Publish TTS v1 audio files with standard caching at the CDN - https://phabricator.wikimedia.org/T436758#12333889 (10kevinbazira) We have started the batch generation as shown below: `bash $ BATCH_GENERATION=/srv/published/datasets/ml/tts/experiment-v1 $ export REQUESTS_CA_... [14:02:55] 06Machine-Learning-Team (Q1 FY2026-27): Publish TTS v1 audio files with standard caching at the CDN - https://phabricator.wikimedia.org/T436758#12333921 (10Dbrant) >>! In T436758#12333575, @kevinbazira wrote: > We are going to run a batch generation that will produce an index under: > https://analytics.wikimedia... [14:13:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [14:13:49] Deployment semantic-highlighting-predictor-00002-deployment in llm at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=llm&var-deployment=semantic-highlighting-predictor-00002-deployment - ... [14:13:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [18:14:04] FIRING: KubernetesDeploymentUnavailableReplicas: ... [18:14:04] Deployment semantic-highlighting-predictor-00002-deployment in llm at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=llm&var-deployment=semantic-highlighting-predictor-00002-deployment - ... [18:14:04] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [22:14:04] FIRING: KubernetesDeploymentUnavailableReplicas: ... [22:14:04] Deployment semantic-highlighting-predictor-00002-deployment in llm at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=llm&var-deployment=semantic-highlighting-predictor-00002-deployment - ... [22:14:04] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [23:38:05] 06Machine-Learning-Team, 10VisualEditor, 10VisualEditor Suggestion Mode, 06Editing-team (FY2026-27 Q1 sprint 6), and 2 others: Announce availability of "experimental" suggestions - https://phabricator.wikimedia.org/T434006#12336236 (10Quiddity) Tech News draft added to https://meta.wikimedia.org/wiki/Tech/...