[00:00:44] (03CR) 10Alex Paskulin: "Thanks for the ping! I started looking at this today and will finish up tomorrow" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1330387 (https://phabricator.wikimedia.org/T434682) (owner: 10Gkyziridis) [05:46:03] (03CR) 10CI reject: [V:04-1] build: Updating npm dependencies [extensions/ORES] (REL1_45) - 10https://gerrit.wikimedia.org/r/1316095 (owner: 10Libraryupgrader) [08:15:49] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Update kserve helm chart to include LLMInferenceService - https://phabricator.wikimedia.org/T433973#12264511 (10DPogorzelski-WMF) == LLMInferenceService == A second resource type is available on **ml-staging-codfw**, in the `experimental` namespace:... [08:50:28] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Update kserve helm chart to include LLMInferenceService - https://phabricator.wikimedia.org/T433973#12264577 (10DPogorzelski-WMF) I think it's a good baseline to test and then try to switch to gateway api otherwise it's way too many moving parts at once [09:24:18] 06Machine-Learning-Team (Q1 FY2026-27), 10VisualEditor, 10VisualEditor Suggestion Mode, 06Editing-team (FY2026-27 Q1 sprint 4), 07OKR-Work (DE1 FY2026-27): Exclude NPOV suggestion types from LLM-generated MoS suggestions in the api - https://phabricator.wikimedia.org/T435358#12264660 (10OKarakaya-WMF) a:... [10:54:37] 06Machine-Learning-Team (Q1 FY2026-27), 06Collaboration-Services: Machine learning packages size on GitLab - https://phabricator.wikimedia.org/T435196#12265017 (10isarantopoulos) 05Open→03Resolved a:03isarantopoulos Resolving this. As far as I know the gitlab package registry doesn't have an automati... [10:55:51] 06Machine-Learning-Team (Q1 FY2026-27), 06Collaboration-Services: Machine learning packages size on GitLab - https://phabricator.wikimedia.org/T435196#12265022 (10OKarakaya-WMF) Thank you! Sure, I'll keep an eye. [11:18:44] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Update kserve helm chart to include LLMInferenceService - https://phabricator.wikimedia.org/T433973#12265101 (10isarantopoulos) This sounds awesome, thanks for the summary! So iiuc we can't benefit any of the goodies that llm-d offers (kv cache aware... [12:07:23] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Update kserve helm chart to include LLMInferenceService - https://phabricator.wikimedia.org/T433973#12265230 (10DPogorzelski-WMF) yea exactly, it doesn't hurt but there is no application specific aware routing because we can't support Inference Extens... [13:01:19] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): LLM observability - https://phabricator.wikimedia.org/T429597#12265447 (10Gehel) [13:02:50] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12265481 (10Gehel) [13:03:18] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12265493 (10Gehel) [13:19:54] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): August 2026 Bookworm reboots --- Machine Learning/LiftWing - https://phabricator.wikimedia.org/T434918#12265621 (10Gehel) [13:20:08] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work: Support GPU Partition specification in deployments - https://phabricator.wikimedia.org/T424923#12265627 (10Gehel) [13:20:15] 06Machine-Learning-Team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work: Document procedure for updating the underlying distro for model servers (e.g. Bullseye -> Bookworm) - https://phabricator.wikimedia.org/T352189#12265631 (10Gehel) [13:20:19] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): ml-lab: Update pytorch-rocm to a current ROCm-compatible release - https://phabricator.wikimedia.org/T428023#12265633 (10Gehel) [13:20:25] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): monitoring: View GPU usage per LLM deployment/model - https://phabricator.wikimedia.org/T429236#12265635 (10Gehel) [13:20:31] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): monitoring: Grafana dashboard for LLM serving on MI300X - https://phabricator.wikimedia.org/T429237#12265637 (10Gehel) [13:23:44] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Add the ML team to the POSIX group `docker` on `ml-build1001` - https://phabricator.wikimedia.org/T393566#12265721 (10Gehel) [14:36:01] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work: Support GPU Partition specification in deployments - https://phabricator.wikimedia.org/T424923#12266034 (10DPogorzelski-WMF) 1. How would the taints get applied to our existing nodes?= this is a technical... [16:40:14] 06Machine-Learning-Team (Q1 FY2026-27): Create dataset of article titles for TTS V1 - https://phabricator.wikimedia.org/T432128#12266481 (10Sucheta-Salgaonkar-WMF) Thank you both for following up here!! It sounds like we are good to go re: intratopic diversity, but still need to decide on a next step for the mat... [17:21:21] (03CR) 10Alex Paskulin: [C:03+1] "Tested locally and everything looks correct. Great work! I left two small suggestions." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1330387 (https://phabricator.wikimedia.org/T434682) (owner: 10Gkyziridis)