[05:59:23] (03PS1) 10Kevin Bazira: tts-section-generator: add batch generation script and tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319747 (https://phabricator.wikimedia.org/T433594) [06:15:44] (03CR) 10Kevin Bazira: [C:03+2] tts-section-generator: add batch generation script and tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319747 (https://phabricator.wikimedia.org/T433594) (owner: 10Kevin Bazira) [06:27:05] (03Merged) 10jenkins-bot: tts-section-generator: add batch generation script and tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319747 (https://phabricator.wikimedia.org/T433594) (owner: 10Kevin Bazira) [09:29:28] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Redeploy Liftwing pick up current mirror - https://phabricator.wikimedia.org/T429675#12173745 (10gkyziridis) === Update === Both model servers are updated and deployed using the `amd-pytorch23:2.3.0rocm6.0-3-20260712` base image. - edit... [10:12:02] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Pre-batch hardening for the TTS v1 experiment: generation-identity updates and the batch generation script - https://phabricator.wikimedia.org/T433594#12173856 (10kevinbazira) Workstream B is now complete. We have tested batch generation on ML-Lab u... [10:38:32] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12173914 (10OKarakaya-WMF) attention eager after TORCH_BLAS_PREFER_HIPBLASLT=0 fix. ` (venv) ozge@stat1010:~/repos/wiki/gerrit/inference-services/test/locus... [10:49:25] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12173921 (10OKarakaya-WMF) attention sdpa after TORCH_BLAS_PREFER_HIPBLASLT=0 fix. sdpa is slightly better than eager. ` (venv) ozge@stat1010:~/repos/wiki/... [11:54:36] (03PS1) 10Kevin Bazira: tts: reject over-long text instead of silently dropping words [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319824 (https://phabricator.wikimedia.org/T433594) [11:58:53] (03CR) 10CI reject: [V:04-1] tts: reject over-long text instead of silently dropping words [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319824 (https://phabricator.wikimedia.org/T433594) (owner: 10Kevin Bazira) [12:13:35] (03PS2) 10Kevin Bazira: tts: reject over-long text instead of silently dropping words [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319824 (https://phabricator.wikimedia.org/T433594) [12:24:20] (03CR) 10Kevin Bazira: [C:03+2] tts: reject over-long text instead of silently dropping words [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319824 (https://phabricator.wikimedia.org/T433594) (owner: 10Kevin Bazira) [12:25:59] (03PS1) 10Ozge: embeddings: Remove temporary GPU memory debug logging [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319834 (https://phabricator.wikimedia.org/T432717) [12:26:51] (03CR) 10Ozge: "Final patch in inference services as a clean up." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319834 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [12:30:11] (03Merged) 10jenkins-bot: tts: reject over-long text instead of silently dropping words [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319824 (https://phabricator.wikimedia.org/T433594) (owner: 10Kevin Bazira) [12:42:52] (03CR) 10Kevin Bazira: [C:03+1] embeddings: Remove temporary GPU memory debug logging [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319834 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [12:50:31] (03CR) 10Ozge: [V:03+2 C:03+2] embeddings: Remove temporary GPU memory debug logging [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319834 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [12:52:38] (03Merged) 10jenkins-bot: embeddings: Remove temporary GPU memory debug logging [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319834 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [13:14:27] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work, 13Patch-For-Review: Pre-batch hardening for the TTS v1 experiment: generation-identity updates and the batch generation script - https://phabricator.wikimedia.org/T433594#12174398 (10kevinbazira) Workstream C is complete as shown in the staging te... [13:21:33] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Migrate TTS inference logic from v0 to v1 (without orchestration, storage, and serving layers) - https://phabricator.wikimedia.org/T430536#12174448 (10kevinbazira) 05Open→03Resolved [13:25:53] 06Machine-Learning-Team (Q1 FY2026-27): RR-Wikidata: Decide on Linked Artifact Cache for stream (WME consumption via Hoarde) - https://phabricator.wikimedia.org/T433699 (10gkyziridis) 03NEW [13:36:32] (03PS1) 10AikoChou: benchmarks: Fix empty HOST_HEADER override; document public run [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319848 (https://phabricator.wikimedia.org/T431851) [13:40:37] 06Machine-Learning-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12174538 (10brouberol) [13:40:49] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 5 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12174539 (10brouberol) [13:41:03] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new: codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12174541 (10brouberol) [13:51:43] (03CR) 10Ozge: [C:03+1] benchmarks: Fix empty HOST_HEADER override; document public run [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319848 (https://phabricator.wikimedia.org/T431851) (owner: 10AikoChou) [13:52:03] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12174587 (10kevinbazira) **Weekly Update:** Key things the batch generation run needs from the ML team is built and verified; the run now waits the storage bucket... [14:50:13] 06Machine-Learning-Team (Q1 FY2026-27): Add Wikidata RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T420883#12174854 (10achou) **Update** Enterprise reported a latency regression after the event stream went live on 2026-07-15. Since the event-stream wo... [14:52:13] 06Machine-Learning-Team (Q1 FY2026-27): ml-serve: assess and raise resource quotas to support rolling deployments in revertrisk (and other) namespaces - https://phabricator.wikimedia.org/T426947#12174861 (10achou) While testing the rollout in T420883, I found another deployment constraint that's related to this... [15:02:37] (03CR) 10AikoChou: [C:03+2] "Thanks for the review :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319848 (https://phabricator.wikimedia.org/T431851) (owner: 10AikoChou) [15:03:21] (03Merged) 10jenkins-bot: benchmarks: Fix empty HOST_HEADER override; document public run [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319848 (https://phabricator.wikimedia.org/T431851) (owner: 10AikoChou)