[00:07:46] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Platform-SRE: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12083858 (10RKemper) (Adding the disk findings from `ml-serve1012` and some context about the calico side of this)  **Where the disk went**:  The pressure...
[04:35:40] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 06Product Safety and Integrity, 10WE4.12 Content policy model evaluation, 05Goal: Q2 FY2025-26 Goal: Host a content policy evaluation model on LiftWing - https://phabricator.wikimedia.org/T418267#12083997 (10kevinbazira)
[06:25:28] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06Traffic, 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12084059 (10gkyziridis) === Timeout ceiling blocks the graphs-on test on 1013/1014 ===   Tried to re-deploy qwen36-27b with `ENFORCE_EAGE...
[07:11:10] <wikibugs>	 (03PS1) 10Kevin Bazira: tts: containerize model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307331 (https://phabricator.wikimedia.org/T430536)
[07:11:33] <wikibugs>	 (03CR) 10CI reject: [V:04-1] tts: containerize model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307331 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[07:43:21] <wikibugs>	 (03CR) 10Bartosz Wójtowicz: [C:03+1] "Thank you for working on it!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1306880 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[07:50:50] <wikibugs>	 (03CR) 10Kevin Bazira: [C:03+2] tts: Add TTS service inference layer [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1306880 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[07:52:00] <wikibugs>	 (03Merged) 10jenkins-bot: tts: Add TTS service inference layer [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1306880 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[07:56:28] <wikibugs>	 (03PS4) 10Kevin Bazira: test: Add TTS model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307057 (https://phabricator.wikimedia.org/T430536)
[08:14:41] <wikibugs>	 (03CR) 10Bartosz Wójtowicz: [C:03+1] "Thank you for adding those! It definitely makes reviewing changes easier when I can see expected test cases for each function." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307057 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[08:28:39] <wikibugs>	 (03CR) 10Kevin Bazira: [C:03+2] test: Add TTS model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307057 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[08:29:15] <wikibugs>	 (03Merged) 10jenkins-bot: test: Add TTS model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307057 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[08:37:27] <wikibugs>	 06Machine-Learning-Team, 07Essential-Work: Set ephemeral-storage requests/limits on ML predictor containers to prevent node disk eviction - https://phabricator.wikimedia.org/T431089 (10gkyziridis) 03NEW
[08:40:43] <wikibugs>	 (03PS2) 10Kevin Bazira: tts: containerize model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307331 (https://phabricator.wikimedia.org/T430536)
[08:45:24] <wikibugs>	 (03PS3) 10Kevin Bazira: tts: containerize model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307331 (https://phabricator.wikimedia.org/T430536)
[08:50:16] <wikibugs>	 (03CR) 10CI reject: [V:04-1] tts: containerize model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307331 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[08:57:14] <wikibugs>	 (03CR) 10Bartosz Wójtowicz: tts: containerize model-server (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307331 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[09:10:21] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26): Redeploy Liftwing pick up current mirror - https://phabricator.wikimedia.org/T429675#12084533 (10kevinbazira) >>! In T429675#12079267, @OKarakaya-WMF wrote: >  I see apps (article-country, revscoring, revertrisk-multilingual, revertrisk-wikidata) are up-to-d...
[09:14:29] <wikibugs>	 (03PS4) 10Kevin Bazira: tts: containerize model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307331 (https://phabricator.wikimedia.org/T430536)
[09:16:59] <wikibugs>	 (03CR) 10Kevin Bazira: "The 2 failing tests (editing_suggestions and edit_check) will be fixed in:" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307331 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[09:18:43] <wikibugs>	 (03CR) 10CI reject: [V:04-1] tts: containerize model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307331 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[09:34:24] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Platform-SRE: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12084671 (10DPogorzelski-WMF) == Investigation: RAM-backed model storage & direct-to-VRAM loading == Summary of findings while looking into the disk-press...
[09:52:22] <wikibugs>	 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12084859 (10Gehel)
[09:52:34] <wikibugs>	 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12084857 (10Gehel)
[09:52:58] <wikibugs>	 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12084863 (10Gehel)
[09:59:06] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Platform-SRE: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12085033 (10DPogorzelski-WMF) In the meantime vg0/kubelet was resized with `lvextend --resizefs -l +100%FREE /dev/vg0/kubelet` on ml-serve100[2-5] which s...
[10:09:04] <wikibugs>	 (03CR) 10Bartosz Wójtowicz: [C:03+1] "got it!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307331 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[10:13:19] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Platform-SRE, 13Patch-For-Review: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12085089 (10DPogorzelski-WMF) a:03DPogorzelski-WMF
[10:23:25] <wikibugs>	 (03CR) 10Kevin Bazira: [C:03+2] tts: containerize model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307331 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[10:26:33] <wikibugs>	 (03CR) 10Kevin Bazira: [V:03+2 C:03+2] tts: containerize model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307331 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira)
[15:26:47] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Add Wikidata RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T420883#12086054 (10achou) **Update:**  Tested the event-stream prediction path on ml-staging-codfw with no issues. Production patc...
[18:29:20] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26): Enable vLLM Prometheus metrics for LiftWing LLM inference services - https://phabricator.wikimedia.org/T431136 (10kevinbazira) 03NEW
[18:47:21] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26): LLM observability - https://phabricator.wikimedia.org/T429597#12086389 (10kevinbazira) >>! In T429597#12079411, @DPogorzelski-WMF wrote: > @kevinbazira some graphs don't have data at the moment so you can ignore those but could you otherwise do a pass on the vllm related g...
[20:16:47] <wikibugs>	 06Machine-Learning-Team, 10VisualEditor, 10VisualEditor Suggestion Mode, 06Editing-team (Editing-current-Q1-2026), 13Patch-For-Review: LLM generated suggestions: attach the suggestion UUID to more events - https://phabricator.wikimedia.org/T430720#12086558 (10VPuffetMichel)
[20:17:09] <wikibugs>	 06Machine-Learning-Team, 10VisualEditor, 10VisualEditor Suggestion Mode, 06Editing-team (Editing-current-Q1-2026), 13Patch-For-Review: Build PoC for integrating initial set of MoS LLM-generated suggestions into Suggestion Mode as `experimental` - https://phabricator.wikimedia.org/T428641#12086566 (10VPuff...
[20:44:04] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Add Wikidata RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T420883#12086643 (10Ottomata) Forgive me if I have forgotten a discussion about this, but!  > EventStreamConfig: add page_revert_ri...