[08:04:50] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-codfw: hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet - https://phabricator.wikimedia.org/T433476 (10klausman) 03NEW [08:08:23] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-codfw: hw troubleshooting: memory errors on ml-serve2004.codfw.wmnet - https://phabricator.wikimedia.org/T433478 (10klausman) 03NEW [08:08:49] 06Machine-Learning-Team (Q1 FY2026-27), 06SRE, 10SRE-swift-storage, 05Goal, 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12166214 (10MatthewVernon) [08:49:37] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12166331 (10OKarakaya-WMF) prod locust latest test results to compare: https://phabricator.wikimedia.org/T418976#11703437 98.90 req/s 77 ms in 98% [09:21:00] (03PS1) 10Ozge: embeddings: Parameterize Locust test for multiple embedding models [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319042 (https://phabricator.wikimedia.org/T432717) [09:33:04] (03CR) 10Kevin Bazira: [C:03+1] embeddings: Parameterize Locust test for multiple embedding models [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319042 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [09:35:07] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12166443 (10OKarakaya-WMF) Actually, the results are much better than I expected and slightly lower than current prod. Jina: - Jina: 99 percentile latency is... [09:52:43] (03CR) 10Ozge: [V:03+2 C:03+2] embeddings: Parameterize Locust test for multiple embedding models [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319042 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [09:54:50] (03Merged) 10jenkins-bot: embeddings: Parameterize Locust test for multiple embedding models [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319042 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:12:52] 06Machine-Learning-Team (Q1 FY2026-27), 06SRE, 10SRE-swift-storage, 05Goal, 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12166592 (10Ladsgroup) >>! In T432944#12163528, @MatthewVernon wrote: >... [10:22:00] (03PS1) 10Ozge: embeddings: Switch Jina embeddings to SentenceTransformers backend - Clean up. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319056 (https://phabricator.wikimedia.org/T432717) [10:22:54] (03CR) 10Ozge: "cleaning up the inference service as jina uses sentences transformers and not vllm." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319056 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:25:42] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12166613 (10OKarakaya-WMF) [10:28:54] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12166615 (10OKarakaya-WMF) [10:29:34] (03CR) 10Kevin Bazira: [C:03+1] embeddings: Switch Jina embeddings to SentenceTransformers backend - Clean up. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319056 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:30:16] (03CR) 10Ozge: [V:03+2 C:03+2] embeddings: Switch Jina embeddings to SentenceTransformers backend - Clean up. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319056 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:32:17] (03Merged) 10jenkins-bot: embeddings: Switch Jina embeddings to SentenceTransformers backend - Clean up. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319056 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [11:18:14] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12166767 (10OKarakaya-WMF) I've removed ml-serve1012 and interesting that the response time has increased to ~5 seconds. I'll set ml-serve1012 to see if this i... [12:57:49] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet - https://phabricator.wikimedia.org/T433476#12167051 (10isarantopoulos) [12:57:58] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: memory errors on ml-serve2004.codfw.wmnet - https://phabricator.wikimedia.org/T433478#12167052 (10isarantopoulos) [12:58:51] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105#12167058 (10isarantopoulos) [13:24:42] 06Machine-Learning-Team (Q1 FY2026-27), 06SRE, 10SRE-swift-storage, 05Goal, 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12167121 (10Ottomata) Hello! Some (no-stupid! :) )questions about the us... [13:26:55] 06Machine-Learning-Team (Q1 FY2026-27), 06SRE, 10SRE-swift-storage, 05Goal, 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12167133 (10Ottomata) ...Speaking of 'renderings', you might want to con... [13:30:35] 06Machine-Learning-Team (Q1 FY2026-27), 06SRE, 10SRE-swift-storage, 05Goal, 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12167146 (10Ottomata) Actually, these comments ^^ would be better placed... [13:30:54] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12167149 (10Ottomata) Hello! Some (no-stupid! :) )questions about the use of `section_id`. I learned in [[ https://docs.google.com/document/d/1jYnfyDVZLm2b7emgDYK... [13:31:26] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12167156 (10Ottomata) ...Speaking of 'renderings', you might want to consider adding a `render_id` somehow in your data and in your response. I doubt that we will... [13:33:09] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12167170 (10Ottomata) Also! Something to keep an eye on: https://docs.google.com/presentation/d/1aUfYkcJqJcxTgvd-GPBz74TX8WuoCZPAC5rY6iTcTaI/edit?slide=id.p7#slid... [13:52:11] 06Machine-Learning-Team (Q1 FY2026-27): Create dataset of article titles for TTS V1 - https://phabricator.wikimedia.org/T432128#12167266 (10Sucheta-Salgaonkar-WMF) Thank you so much Bartosz!!! One thing I should have written in the original ticket is that I think it's better to filter too conservatively for now... [14:23:15] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Update stale documentation - https://phabricator.wikimedia.org/T433502 (10gkyziridis) 03NEW [14:38:14] 06Machine-Learning-Team (Q1 FY2026-27), 07Documentation, 07Essential-Work: Update stale documentation - https://phabricator.wikimedia.org/T433502#12167522 (10Pppery) [15:32:01] 06Machine-Learning-Team (Q1 FY2026-27), 06Infrastructure-Foundations: Move all Machine Learning Docker images under the /ml prefix in the Docker Registry - https://phabricator.wikimedia.org/T428022#12167801 (10elukey) 05Open→03Resolved a:03elukey [16:09:26] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12168053 (10kevinbazira) >>! In T430676#12167149, @Ottomata wrote: > Hello! Some (no-stupid! :) )questions about the use of `section_id`. > > I learned in [[ http... [16:13:30] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12168061 (10kevinbazira) >>! In T430676#12167153, @Ottomata wrote: > ...Speaking of 'renderings', you might want to consider adding a `render_id` somehow in your d... [16:21:02] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12168108 (10kevinbazira) >>! In T430676#12167170, @Ottomata wrote: > Also! Something to keep an eye on: > > https://docs.google.com/presentation/d/1aUfYkcJqJcxTgv... [16:57:52] 06Machine-Learning-Team (Q1 FY2026-27): Create dataset of article titles for TTS V1 - https://phabricator.wikimedia.org/T432128#12168493 (10Sucheta-Salgaonkar-WMF) Ok based on the discussion in the [[ https://wikimedia.slack.com/archives/CVB2EC3KK/p1785332426156929 | Slack thread ]] I think we should use "edite... [16:59:46] 06Machine-Learning-Team (Q1 FY2026-27): Create dataset of article titles for TTS V1 - https://phabricator.wikimedia.org/T432128#12168501 (10Sucheta-Salgaonkar-WMF) [17:06:19] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12168562 (10Ottomata) Thanks Kevin (and your AI friend ;) )! > the index points the app to the latest ready audio for each section. Oh! Looking at your diagram m... [18:34:18] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet - https://phabricator.wikimedia.org/T433476#12168856 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm tobias downtimed the server for me and... [18:35:53] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: memory errors on ml-serve2004.codfw.wmnet - https://phabricator.wikimedia.org/T433478#12168862 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm learned lesson from T433476. same error... [19:57:55] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12169120 (10OKarakaya-WMF) confirmed that the high performance [results](https://phabricator.wikimedia.org/T432717#12166443) are from ml-serve1012 (192GB). The... [20:02:39] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: [Epic] Develop a framework for invalidating edit suggestions - https://phabricator.wikimedia.org/T429426#12169140 (10Isaac) [20:05:53] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: [Epic] Develop a framework for invalidating edit suggestions - https://phabricator.wikimedia.org/T429426#12169147 (10Isaac) To the open questions, I'd also add one about how to handle situations in which it's...