[09:07:47] 06Machine-Learning-Team (Q1 FY2026-27): RevertRisk-Wikidata: move serving and stream to Linked Artifacts Cache - https://phabricator.wikimedia.org/T433699#12270122 (10achou) [10:18:24] 06Machine-Learning-Team (Q1 FY2026-27), 10Add-Link-Structured-Task, 10Community Feedback (Growth), 06Growth-Team, 13Patch-For-Review: Introduce case sensitivity to Add-a-Link - https://phabricator.wikimedia.org/T434259#12270312 (10OKarakaya-WMF) Production is updated with the suggested change: ` curl... [10:41:38] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work, 13Patch-For-Review: liftwing-openapi-server: Add the LLM endpoints to the OpenAPI spec - https://phabricator.wikimedia.org/T434682#12270362 (10isarantopoulos) [10:54:56] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work, 13Patch-For-Review: Redeploy Liftwing pick up current mirror - https://phabricator.wikimedia.org/T429675#12270395 (10gkyziridis) 05Open→03Resolved [10:55:31] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work, 13Patch-For-Review: logo-detection image build fails: kserve 0.11.2 conflicts with pydantic v2 in shared requirements - https://phabricator.wikimedia.org/T435946#12270400 (10gkyziridis) 05Open→03Resolved [11:16:42] 06Machine-Learning-Team (Q1 FY2026-27), 10Add-Link-Structured-Task, 10Community Feedback (Growth), 06Growth-Team: Introduce case sensitivity to Add-a-Link - https://phabricator.wikimedia.org/T434259#12270447 (10OKarakaya-WMF) 05Open→03Resolved [11:23:02] 06Machine-Learning-Team (Q1 FY2026-27), 10VisualEditor, 10VisualEditor Suggestion Mode, 06Editing-team (FY2026-27 Q1 sprint 4), and 2 others: Exclude NPOV suggestion types from LLM-generated MoS suggestions in the api - https://phabricator.wikimedia.org/T435358#12270452 (10OKarakaya-WMF) npov is removed i... [11:23:50] 06Machine-Learning-Team (Q1 FY2026-27), 10VisualEditor, 10VisualEditor Suggestion Mode, 06Editing-team (FY2026-27 Q1 sprint 4), and 2 others: Exclude NPOV suggestion types from LLM-generated MoS suggestions in the api - https://phabricator.wikimedia.org/T435358#12270454 (10OKarakaya-WMF) 05Open→03Re... [11:45:51] 06Machine-Learning-Team (Q1 FY2026-27): RevertRisk-Wikidata: move serving and stream to Linked Artifacts Cache - https://phabricator.wikimedia.org/T433699#12270497 (10isarantopoulos) > Wikimedia Enterprise — switch to the LAC endpoint, and tell us what freshness they need after an edit. Just noting that in order... [11:49:25] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: Set up a VM with nginx to store and serve TTS v1 audio files as the CDN origin - https://phabricator.wikimedia.org/T435952#12270538 (10isarantopoulos) [11:52:08] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: Set up a VM with nginx to store and serve TTS v1 audio files as the CDN origin - https://phabricator.wikimedia.org/T435952#12270544 (10isarantopoulos) Regarding what is listed above as an open decision; >Open decision: who uploads the files to the VM... [12:01:50] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: Set up a VM with nginx to store and serve TTS v1 audio files as the CDN origin - https://phabricator.wikimedia.org/T435952#12270559 (10Gehel) >>! In T435952#12270544, @isarantopoulos wrote: > Can the ML team have access to the VM? That way we can ite... [12:22:50] FIRING: ORESFetchScoreJobKafkaLag: Kafka consumer lag for ORESFetchScoreJob over threshold for past 1h. ... [12:22:56] - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#Kafka_Consumer_lag_-_ORESFetchScoreJobKafkaLag_alert - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&orgId=1&to=now&var-cluster=main-eqiad&var-consumer_group=cpjobqueue-ORESFetchScoreJob&var-datasource=%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DORESFetchScoreJobKafkaLag [12:24:57] (03PS3) 10Gkyziridis: liftwing-openapi-server: Add the LLM endpoints to the OpenAPI spec [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1330387 (https://phabricator.wikimedia.org/T434682) [12:42:50] RESOLVED: ORESFetchScoreJobKafkaLag: Kafka consumer lag for ORESFetchScoreJob over threshold for past 1h. ... [12:42:50] - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#Kafka_Consumer_lag_-_ORESFetchScoreJobKafkaLag_alert - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&orgId=1&to=now&var-cluster=main-eqiad&var-consumer_group=cpjobqueue-ORESFetchScoreJob&var-datasource=%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DORESFetchScoreJobKafkaLag [12:47:16] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team, 10GrowthExperiments-NewcomerTasks: Don't suggest links for a part of a phrase based on capitalization - https://phabricator.wikimedia.org/T436442#12270663 (10DMburugu) Tagging ML team since this could be a model rule matter. [13:02:13] 06Machine-Learning-Team (Q1 FY2026-27), 10Add-Link-Structured-Task, 06Growth-Team, 10GrowthExperiments-NewcomerTasks: Don't suggest links for a part of a phrase based on capitalization - https://phabricator.wikimedia.org/T436442#12270728 (10isarantopoulos) [14:03:44] FIRING: LiftWingServiceErrorRate: ... [14:03:44] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-goodfaith&var-backend=enwiki-goodfaith-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [14:03:50] FIRING: ORESFetchScoreJobKafkaLag: Kafka consumer lag for ORESFetchScoreJob over threshold for past 1h. ... [14:03:50] - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#Kafka_Consumer_lag_-_ORESFetchScoreJobKafkaLag_alert - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&orgId=1&to=now&var-cluster=main-eqiad&var-consumer_group=cpjobqueue-ORESFetchScoreJob&var-datasource=%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DORESFetchScoreJobKafkaLag [14:26:00] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: ORESFetchScoreJobKafkaLag (1 alert) and LiftWingServiceErrorRate (1 alert) - https://phabricator.wikimedia.org/T436525 (10OKarakaya-WMF) 03NEW [14:26:25] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: ORESFetchScoreJobKafkaLag (1 alert) and LiftWingServiceErrorRate (1 alert) - https://phabricator.wikimedia.org/T436525#12271224 (10OKarakaya-WMF) [14:30:22] 06Machine-Learning-Team (Q1 FY2026-27): Research current best approaches for hosting LLM models - https://phabricator.wikimedia.org/T433580#12271264 (10BWojtowicz-WMF) **3. What are the current problems we face with LLM serving on LiftWing** Expanding on the previous message, there are a couple of major issues... [14:36:12] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: ORESFetchScoreJobKafkaLag (1 alert) and LiftWingServiceErrorRate (1 alert) - https://phabricator.wikimedia.org/T436525#12271293 (10OKarakaya-WMF) We see timeout errors in the mediawiki errors; Jobs fail to be processed. We see timeout errors: https://... [14:38:44] RESOLVED: LiftWingServiceErrorRate: ... [14:38:44] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-goodfaith&var-backend=enwiki-goodfaith-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [15:18:51] RESOLVED: ORESFetchScoreJobKafkaLag: Kafka consumer lag for ORESFetchScoreJob over threshold for past 1h. ... [15:18:51] - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#Kafka_Consumer_lag_-_ORESFetchScoreJobKafkaLag_alert - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&orgId=1&to=now&var-cluster=main-eqiad&var-consumer_group=cpjobqueue-ORESFetchScoreJob&var-datasource=%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DORESFetchScoreJobKafkaLag [16:45:00] 06Machine-Learning-Team (Q1 FY2026-27), 10Add-Link-Structured-Task, 10Community Feedback (Growth), 06Growth-Team, 07User-notice: Introduce case sensitivity to Add-a-Link - https://phabricator.wikimedia.org/T434259#12271962 (10Trizek-WMF) For next Tech News: > The model used with Add a link has been... [16:56:24] (03CR) 10DCausse: add semantic_highlighting (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1322788 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [17:15:31] (03PS4) 10DCausse: Add semantic_highlighting [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1322788 (https://phabricator.wikimedia.org/T433872) [17:15:31] (03PS2) 10DCausse: WIP: semantic_highlighting: move an answer edge to a boundary a reader sees [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1330567 (https://phabricator.wikimedia.org/T433872) (owner: 10Ebernhardson) [17:15:36] (03CR) 10DCausse: Add semantic_highlighting (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1322788 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [17:23:35] 07artificial-intelligence, 06Product Safety and Integrity, 06Security-Team, 07Essential-Work, 07SecTeam-Processed: Create architectural / dependency security review skill for wmf-claude - https://phabricator.wikimedia.org/T435218#12272101 (10ASanford-WMF) [17:48:49] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 10Community Feedback (Growth), 06Growth-Team: Scaling of case sensitivity for Link Suggestions - https://phabricator.wikimedia.org/T436545 (10KStoller-WMF) 03NEW