[08:24:30] (03CR) 10AikoChou: [C:03+1] qwen36: Allow configuring vLLM KV cache dtype [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309662 (owner: 10Bartosz Wójtowicz) [08:56:14] 06Machine-Learning-Team (Q1 FY2026-27): Enable FP8 KV Cache for Qwen36-27b - https://phabricator.wikimedia.org/T431978 (10BWojtowicz-WMF) 03NEW [08:57:02] (03PS2) 10Bartosz Wójtowicz: qwen36: Allow configuring vLLM KV cache dtype [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309662 (https://phabricator.wikimedia.org/T431978) [09:03:38] (03CR) 10Bartosz Wójtowicz: [C:03+2] qwen36: Allow configuring vLLM KV cache dtype [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309662 (https://phabricator.wikimedia.org/T431978) (owner: 10Bartosz Wójtowicz) [09:06:03] (03Merged) 10jenkins-bot: qwen36: Allow configuring vLLM KV cache dtype [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1309662 (https://phabricator.wikimedia.org/T431978) (owner: 10Bartosz Wójtowicz) [11:13:17] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12114149 (10hnowlan) graphite2004 doesn't need depooling. centrallog will have some gaps in logs on-disk but that's unavoidable and l... [11:53:36] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Add Wikidata RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T420883#12114289 (10achou) **Update:** proceeding to production. Following up on the capacity concern in T420883#12086054: I've co... [12:06:18] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): [draft] ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12114363 (10klausman) Started reading up on AMD's recommendations here... [12:31:32] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: ORES flagging not working on simplewiki - https://phabricator.wikimedia.org/T431159#12114454 (10Ladsgroup) If it's intermittent, it means some replicas need optimizing to get the stats right.... [12:42:40] (03CR) 10Dreamy Jazz: [C:03+2] Replace deprecated RevisionLookup::getKnownCurrentRevision [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1309794 (https://phabricator.wikimedia.org/T420976) (owner: 10Umherirrender) [13:16:47] (03Merged) 10jenkins-bot: Replace deprecated RevisionLookup::getKnownCurrentRevision [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1309794 (https://phabricator.wikimedia.org/T420976) (owner: 10Umherirrender) [13:28:22] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Implement TTS Section Generator Phase 1: API contract and core library (without transcoding, deployment, and the batch pilot) - https://phabricator.wikimedia.org/T432009 (10kevinbazira) 03NEW [14:40:24] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12115161 (10cmooney) [15:40:20] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 06ServiceOps new, 10ServiceOps-SharedInfra, 06MediaWiki-Platform-Team (Radar): Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12115548 (10Scott_French) The revert of https://gerrit.wikimedia.org/... [15:46:01] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12115582 (10CWilliams-WMF) [15:54:57] 06Machine-Learning-Team, 06Editing-team, 06Growth-Team, 10Revise-Tone-Structured-Task: Revise Tone Structured Task: release on remaining wikis supported by Tone check - https://phabricator.wikimedia.org/T426364#12115615 (10KStoller-WMF) [18:58:22] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12116562 (10BTullis) I have cordoned `dse-k8s-worker2003` ` root@deploy2003:~# kubectl cordon dse-k8s-worker2003.codfw.wmnet node/dse... [21:35:28] 06Machine-Learning-Team, 06App Experience, 06Discovery-Search: Articles on men are appearing as suggestions to users based on a stated interested in women biographies - https://phabricator.wikimedia.org/T431151#12117171 (10TJones) I don't know if the topic model is using Wikipedia Categories, and it may not... [23:32:44] FIRING: LiftWingServiceErrorRate: ... [23:32:44] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-damaging&var-backend=eswiki-damaging-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [23:57:44] RESOLVED: LiftWingServiceErrorRate: ... [23:57:44] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-damaging&var-backend=eswiki-damaging-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate