[00:21:13] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 10decommission-hardware, 13Patch-For-Review: decommission ml-cache* - https://phabricator.wikimedia.org/T430654#12097662 (10BTullis) Yes, please feel free to apply the diff to the wikikube and aux clusters. Apologies for the oversight. My possi... [05:03:43] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Enable vLLM Prometheus metrics for LiftWing LLM inference services - https://phabricator.wikimedia.org/T431136#12097900 (10kevinbazira) 05Open→03Resolved [09:15:52] 06Machine-Learning-Team (Q1 FY2026-27), 10Liberica, 10Prod-Kubernetes, 06Traffic, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12098641 (10brouberol) [09:16:03] 06Machine-Learning-Team (Q1 FY2026-27), 10Liberica, 10Prod-Kubernetes, 06Traffic, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12098648 (10klausman) The only remaining section is IPIP for the workers/ingress in codfw+eqiad. I will coordinate with T... [09:19:04] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07Essential-Work: Move Lab machines into analytics net for DL access and switch to homedirs on Ceph - https://phabricator.wikimedia.org/T380279#12098668 (10brouberol) [09:19:05] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07Essential-Work: Move Lab machines into analytics net for DL access and switch to homedirs on Ceph - https://phabricator.wikimedia.org/T380279#12098669 (10brouberol) 05Open→03In progress [09:21:05] 06Machine-Learning-Team (Q1 FY2026-27), 10decommission-hardware, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: decommission ml-cache* - https://phabricator.wikimedia.org/T430654#12098674 (10brouberol) 05Open→03In progress [09:21:15] 06Machine-Learning-Team (Q1 FY2026-27), 10decommission-hardware, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: decommission ml-cache* - https://phabricator.wikimedia.org/T430654#12098672 (10brouberol) [09:24:35] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Possibility to transition ml-serve[2001-2008] and and ml-staging[2001-2002] to 10G - https://phabricator.wikimedia.org/T378714#12098698 (10klausman) 05Open→03In progress [09:24:37] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Possibility to transition ml-serve[2001-2008] and and ml-staging[2001-2002] to 10G - https://phabricator.wikimedia.org/T378714#12098697 (10klausman) [10:20:46] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: [draft] ml-serve: Apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553 (10isarantopoulos) 03NEW [10:21:14] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: [draft] ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12099051 (10isarantopoulos) [10:22:41] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work, 13Patch-For-Review: Set ephemeral-storage requests/limits on ML predictor containers to prevent node disk eviction - https://phabricator.wikimedia.org/T431089#12099052 (10isarantopoulos) [10:24:31] 06Machine-Learning-Team (Q1 FY2026-27): vllm bench serve metrics explained - https://phabricator.wikimedia.org/T431554 (10OKarakaya-WMF) 03NEW [10:25:40] 06Machine-Learning-Team (Q1 FY2026-27): vllm bench serve metrics explained - https://phabricator.wikimedia.org/T431554#12099069 (10OKarakaya-WMF) # vLLM bench serve Benchmark an already-running LLM server with [`vllm bench serve`](https://docs.vllm.ai/en/latest/cli/bench/serve.html). ## Workflow 1. **Start th... [10:26:45] 06Machine-Learning-Team (Q1 FY2026-27): vllm bench serve metrics explained - https://phabricator.wikimedia.org/T431554#12099073 (10OKarakaya-WMF) 05Open→03Resolved [10:42:54] 06Machine-Learning-Team: Article topic cache backfilling using article_topic hive table - https://phabricator.wikimedia.org/T403254#12099140 (10isarantopoulos) [10:42:55] 06Machine-Learning-Team: Evaluate adding caching mechanism for article topic model to make data available at scale - https://phabricator.wikimedia.org/T401778#12099141 (10isarantopoulos) [10:49:53] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2025-26 Goal: Make article topic data available at scale and within SLOs for Year in Review - https://phabricator.wikimedia.org/T392833#12099182 (10isarantopoulos) 05Open→03Resolved a:03isarantopoulos I'm resolving this goal. Congrats... [10:51:55] 06Machine-Learning-Team (Q1 FY2026-27), 06Research, 05Goal, 13Patch-For-Review: Q4 FY2025-26 Goal: Text-to-Speech V0 Prototype - https://phabricator.wikimedia.org/T419288#12099199 (10isarantopoulos) 05Open→03Resolved Resolving this as the prototype is complete and demonstrated in https://wiki-tts.t... [11:02:42] 06Machine-Learning-Team, 05Goal, 07OKR-Work, 13Patch-For-Review: Q1 FY2025-26 Goal: Make article topic data available at scale and within SLOs for Year in Review - https://phabricator.wikimedia.org/T392833#12099282 (10isarantopoulos) [11:04:28] 06Machine-Learning-Team, 06Research, 05Goal, 13Patch-For-Review: Q4 FY2025-26 Goal: Text-to-Speech V0 Prototype - https://phabricator.wikimedia.org/T419288#12099294 (10isarantopoulos) [11:05:07] 06Machine-Learning-Team, 06Product Safety and Integrity, 10WE4.12 Content policy model evaluation, 05Goal: Q2 FY2025-26 Goal: Host a content policy evaluation model on LiftWing - https://phabricator.wikimedia.org/T418267#12099298 (10isarantopoulos) [11:05:42] 06Machine-Learning-Team, 06Product Safety and Integrity, 10WE4.12 Content policy model evaluation, 05Goal: Q2 FY2025-26 Goal: Host a content policy evaluation model on LiftWing - https://phabricator.wikimedia.org/T418267#12099303 (10isarantopoulos) 05Open→03Resolved [11:12:19] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Update kserve Python package to 0.17 across all inference services - https://phabricator.wikimedia.org/T422591#12099320 (10isarantopoulos) [11:33:22] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 07Essential-Work: Support GPU Partition specification in deployments - https://phabricator.wikimedia.org/T424923#12099383 (10BWojtowicz-WMF) Thank you for your inputs @achou @isarantopoulos! It seems we are all agreeing on Option 1, which means tai... [11:42:00] 06Machine-Learning-Team: LLM Generated Editing Suggestions - API changes. - https://phabricator.wikimedia.org/T428740#12099422 (10OKarakaya-WMF) [11:42:01] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q2 FY2025-26 Goal: Generate a list of edit suggestions using machine learning - https://phabricator.wikimedia.org/T409863#12099421 (10OKarakaya-WMF) [11:42:02] 06Machine-Learning-Team: Editing Suggestions - Eval pipeline for enwiki - https://phabricator.wikimedia.org/T425957#12099424 (10OKarakaya-WMF) [11:42:04] 06Machine-Learning-Team: Editing Suggestions - Dataset generation logic from edits - https://phabricator.wikimedia.org/T424347#12099425 (10OKarakaya-WMF) [11:42:05] 06Machine-Learning-Team, 13Patch-For-Review: Editing Suggestions - api - https://phabricator.wikimedia.org/T427794#12099423 (10OKarakaya-WMF) [11:42:07] 06Machine-Learning-Team: Edit suggestions - Precision recall on grammar edit suggestions based on edits. - https://phabricator.wikimedia.org/T421672#12099426 (10OKarakaya-WMF) [11:59:44] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 06Traffic, 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12099545 (10BWojtowicz-WMF) **Update** We've noticed when deploying qwen3-14b model on 2x 24GB VRAM partitions of MI300X - https://phab... [12:12:24] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q2 FY2025-26 Goal: Generate a list of edit suggestions using machine learning - https://phabricator.wikimedia.org/T409863#12099626 (10OKarakaya-WMF) ## Summary - T416869 Edit Suggestions - Dataset for hackathon : We started with running an hackath... [12:25:27] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q2 FY2025-26 Goal: Generate a list of edit suggestions using machine learning - https://phabricator.wikimedia.org/T409863#12099695 (10OKarakaya-WMF) ## Summary - T416869 Edit Suggestions - Dataset for hackathon : We started with running an hackath... [12:47:24] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: `mediawiki.page_change.v1`: negative namespace_id schema validation errors - https://phabricator.wikimedia.org/T421237#12099820 (10Ahoelzl) [12:54:34] (03PS1) 10Kevin Bazira: tts: rebuild Kokoro ONNX session with full graph optimization [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1308631 (https://phabricator.wikimedia.org/T430536) [13:00:51] (03CR) 10CI reject: [V:04-1] tts: rebuild Kokoro ONNX session with full graph optimization [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1308631 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [13:02:13] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q2 FY2025-26 Goal: Generate a list of edit suggestions using machine learning - https://phabricator.wikimedia.org/T409863#12099941 (10OKarakaya-WMF) We processed 30K articles in total based on their edit counts by using gpt-oss-120b model and filter... [13:09:06] 06Machine-Learning-Team (Q1 FY2026-27), 10Liberica, 10Prod-Kubernetes, 06Traffic, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12099970 (10klausman) codfw workers/ingress is now also IPIP. I have run the httpbb tests: `# pwd /srv/deployment/httpbb... [13:13:06] 06Machine-Learning-Team (Q1 FY2026-27), 10Liberica, 10Prod-Kubernetes, 06Traffic, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12100009 (10elukey) @klausman `recommendation-api-ng.discovery.wmnet` is a CNAME to k8s-ingress-ml-serve.discovery.wmnet,... [13:24:04] 06Machine-Learning-Team (Q1 FY2026-27), 06Infrastructure-Foundations: Move all Machine Learning Docker images under the /ml prefix in the Docker Registry - https://phabricator.wikimedia.org/T428022#12100044 (10elukey) Back to this! I created https://gitlab.wikimedia.org/elukey/registry-clone to support the mov... [13:29:47] 06Machine-Learning-Team (Q1 FY2026-27), 10Liberica, 10Prod-Kubernetes, 06Traffic, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12100059 (10klausman) >>! In T420438#12100009, @elukey wrote: > @klausman `recommendation-api-ng.discovery.wmnet` is a CN... [13:41:18] 06Machine-Learning-Team (Q1 FY2026-27), 10Liberica, 10Prod-Kubernetes, 06Traffic, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12100104 (10elukey) @klausman for RR wikidata sometimes it fails and sometimes it doesn't, but I noticed this in the logs... [13:41:36] 06Machine-Learning-Team (Q1 FY2026-27), 10Liberica, 10Prod-Kubernetes, 06Traffic, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12100118 (10klausman) Revertrisk-wikidata was intermittent, so that rules out IPIP being at fault, so only this remains:... [13:49:14] 06Machine-Learning-Team: Article topic cache backfilling using article_topic hive table - https://phabricator.wikimedia.org/T403254#12100149 (10Ottomata) Hm, should we decline or redescribe this task? Now that the lambda is the source of the topic predictions, perhaps a backfill would simply be a slow iteration... [14:08:20] 06Machine-Learning-Team (Q1 FY2026-27), 10Liberica, 10Prod-Kubernetes, 06Traffic, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12100318 (10klausman) Notably, the check works in eqiad (though it does take several seconds): `# httpbb --host inferenc... [14:09:34] 06Machine-Learning-Team (Q1 FY2026-27), 10Liberica, 10Prod-Kubernetes, 06Traffic, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12100320 (10klausman) 05In progress→03Resolved The subject matter of this bug (IPIP migration) is done and things... [14:10:06] (03CR) 10Kevin Bazira: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1308631 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [14:12:59] 06Machine-Learning-Team: httpbb tests from `test_article-descriptions.yaml` fail - https://phabricator.wikimedia.org/T431580 (10klausman) 03NEW [14:59:04] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q2 FY2025-26 Goal: Generate a list of edit suggestions using machine learning - https://phabricator.wikimedia.org/T409863#12100683 (10isarantopoulos) 05Open→03Resolved Resolving this as the generation of the edits has been completed. Thank y... [15:08:50] 06Machine-Learning-Team, 05Goal, 07OKR-Work: Q2 FY2025-26 Goal: Generate a list of edit suggestions using machine learning - https://phabricator.wikimedia.org/T409863#12100742 (10isarantopoulos) [15:43:54] 06Machine-Learning-Team: httpbb tests from `test_article-descriptions.yaml` fail - https://phabricator.wikimedia.org/T431580#12100893 (10BWojtowicz-WMF) Regarding timeouts, our service definitely takes _a lot_ of time to respond (with success!), which results in the 10 second timeout: ` INFO:root:Opening a new A... [16:15:13] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12101035 (10MoritzMuehlenhoff) cumin2003 will need an announcement one day ahead so that people don't run any cookboo... [16:51:23] 06Machine-Learning-Team, 06Editing-team, 06Growth-Team, 10Revise-Tone-Structured-Task: Revise Tone Structured Task: release on remaining wikis supported by Tone check - https://phabricator.wikimedia.org/T426364#12101180 (10DMburugu) [17:56:18] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: [Epic] Develop a framework for invalidating edit suggestions - https://phabricator.wikimedia.org/T429426#12101469 (10dchan) [18:08:05] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: Develop a persistence model for edit suggestions and their state - https://phabricator.wikimedia.org/T431606 (10dchan) 03NEW [18:41:21] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: [Epic] Develop a framework for invalidating edit suggestions - https://phabricator.wikimedia.org/T429426#12101668 (10dchan) [19:26:07] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: Develop a persistence model for edit suggestions and their state - https://phabricator.wikimedia.org/T431606#12101828 (10dchan) [19:27:13] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: Develop a persistence model for edit suggestions and their state - https://phabricator.wikimedia.org/T431606#12101831 (10dchan) [19:27:58] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: Develop a persistence model for edit suggestions and their state - https://phabricator.wikimedia.org/T431606#12101837 (10dchan) [19:30:33] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: [Epic] Develop a framework for invalidating edit suggestions - https://phabricator.wikimedia.org/T429426#12101849 (10dchan) [19:30:34] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: Develop a persistence model for edit suggestions and their state - https://phabricator.wikimedia.org/T431606#12101850 (10dchan) [19:30:46] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: Develop a persistence model for edit suggestions and their state - https://phabricator.wikimedia.org/T431606#12101857 (10dchan) [19:30:48] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: [Epic] Develop a framework for invalidating edit suggestions - https://phabricator.wikimedia.org/T429426#12101856 (10dchan) [22:24:45] 06Machine-Learning-Team, 06Editing-team, 10VisualEditor, 10VisualEditor Suggestion Mode, 07Epic: [Epic] Develop a framework for invalidating edit suggestions - https://phabricator.wikimedia.org/T429426#12102433 (10ppelberg) [23:43:19] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: ORES flagging not working on simplewiki - https://phabricator.wikimedia.org/T431159#12102607 (10Ladsgroup) >>! In T431159#12087949, @HakanIST wrote: > I wasn't able to reproduce this, but Log... [23:47:13] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: ORES flagging not working on simplewiki - https://phabricator.wikimedia.org/T431159#12102625 (10Ladsgroup) 860K rows for ores_classification seems a bit excessive: ` MariaDB [simplewiki]> sel... [23:55:28] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: ORES flagging not working on simplewiki - https://phabricator.wikimedia.org/T431159#12102642 (10Ladsgroup) ` mysql:research@dbstore1007.eqiad.wmnet [simplewiki]> select count(*) from ores_cla... [23:56:23] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: ORES flagging not working on simplewiki - https://phabricator.wikimedia.org/T431159#12102643 (10Ladsgroup) >>! In T431159#12087917, @Johannnes89 wrote: > https://simple.wikipedia.org/wiki/Spe... [23:58:24] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: ORES flagging not working on simplewiki - https://phabricator.wikimedia.org/T431159#12102651 (10Ladsgroup) I probably should run this on all.dblist.