[07:51:21] (03PS5) 10DCausse: Add semantic_highlighting [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1322788 (https://phabricator.wikimedia.org/T433872) [07:51:21] (03PS3) 10DCausse: WIP: semantic_highlighting: move an answer edge to a boundary a reader sees [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1330567 (https://phabricator.wikimedia.org/T433872) (owner: 10Ebernhardson) [07:51:21] (03PS1) 10DCausse: semantic_highlighting: add answer and score to model output [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1338100 (https://phabricator.wikimedia.org/T433872) [07:53:12] (03CR) 10DCausse: Add semantic_highlighting (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1322788 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [08:03:45] (03CR) 10DCausse: [C:03+1] WIP: semantic_highlighting: move an answer edge to a boundary a reader sees [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1330567 (https://phabricator.wikimedia.org/T433872) (owner: 10Ebernhardson) [08:06:20] 06Machine-Learning-Team (Q1 FY2026-27), 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12301769 (10Gehel) a:05klausman→03RobH [10:15:12] 06Machine-Learning-Team (Q1 FY2026-27), 10Add-Link-Structured-Task, 06Growth-Team: Add-a-Link — add two model features: `link_probability` and `context_w2v` - https://phabricator.wikimedia.org/T437264#12302180 (10OKarakaya-WMF) # `link_probability` and `context_w2v` | Feature | What it asks | Scope | |---|-... [11:11:15] 06Machine-Learning-Team: pytorch image builds are failing - https://phabricator.wikimedia.org/T437427#12302394 (10MoritzMuehlenhoff) [11:28:39] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Add ephemeral-storage to KServe sidecar containers and enforce it on the chart level - https://phabricator.wikimedia.org/T434534#12302483 (10BWojtowicz-WMF) a:03BWojtowicz-WMF [11:57:00] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Resolve issues with python-mwapi dependency on LiftWing - https://phabricator.wikimedia.org/T434763#12302699 (10BWojtowicz-WMF) We discussed the options and want to go with forking `mwapi` and maintaining it ourselves, hosted on Wikimedia GitLab. Main r... [12:36:32] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Update istio-cni to 1.29 - https://phabricator.wikimedia.org/T434198#12302860 (10achou) [12:38:32] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work: Support GPU Partition specification in deployments - https://phabricator.wikimedia.org/T424923#12302868 (10achou) >>! In T424923#12285385, @Gehel wrote: > #machine-learning-team will give the list of part... [12:39:03] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work: Support GPU Partition specification in deployments - https://phabricator.wikimedia.org/T424923#12302874 (10achou) [12:39:04] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: ml-serve-eqiad: Repartition two MI300X nodes from 24GB to 96GB partitions - https://phabricator.wikimedia.org/T436928#12302873 (10achou) [14:14:58] (03CR) 10Ozge: [C:03+1] "thank you for adding this feature." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1338100 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [14:20:51] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Define coding standards for the inference-services repository - https://phabricator.wikimedia.org/T437300#12303446 (10kevinbazira) Thanks for creating this task @BWojtowicz-WMF! As discussed in the ML team meeting, we could benchmark on the main upstre... [14:43:01] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Host a tool-server on LiftWing - https://phabricator.wikimedia.org/T436892#12303613 (10kevinbazira) Thanks to @BTullis for helping with the deployment configs. The tool-server is now deployed in LiftWing staging: ` kevinbazira@deploy1003:~$ kube_env t... [14:53:06] 06Machine-Learning-Team (Q1 FY2026-27): RevertRisk-Wikidata: move serving and stream to Linked Artifacts Cache - https://phabricator.wikimedia.org/T433699#12303706 (10gkyziridis) a:03gkyziridis [14:55:21] (03PS1) 10Gkyziridis: revertrisk_wikidata: Develop KServe v2 and grpc inference protocol [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1338218 (https://phabricator.wikimedia.org/T433699) [15:05:19] (03CR) 10Gkyziridis: "I saw that this patch triggered all the inference-services pipelines which is strange. There is probably an issue on configuration in the " [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1338218 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [15:12:34] (03CR) 10Gkyziridis: "That was probably because I submitted a change in the .pipeline/config.yaml" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1338218 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [15:15:50] FIRING: ORESFetchScoreJobKafkaLag: Kafka consumer lag for ORESFetchScoreJob over threshold for past 1h. ... [15:15:56] - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#Kafka_Consumer_lag_-_ORESFetchScoreJobKafkaLag_alert - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&orgId=1&to=now&var-cluster=main-eqiad&var-consumer_group=cpjobqueue-ORESFetchScoreJob&var-datasource=%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DORESFetchScoreJobKafkaLag [15:30:50] RESOLVED: ORESFetchScoreJobKafkaLag: Kafka consumer lag for ORESFetchScoreJob over threshold for past 1h. ... [15:30:50] - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#Kafka_Consumer_lag_-_ORESFetchScoreJobKafkaLag_alert - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&orgId=1&to=now&var-cluster=main-eqiad&var-consumer_group=cpjobqueue-ORESFetchScoreJob&var-datasource=%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DORESFetchScoreJobKafkaLag [17:18:17] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: RevertRisk-Wikidata: move serving and stream to Linked Artifacts Cache - https://phabricator.wikimedia.org/T433699#12304464 (10Ottomata) > Wikimedia Enterprise — switch to the LAC endpoint Q: I wonder if a better generalized route would be like: `Pu... [18:11:43] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Resolve issues with python-mwapi dependency on LiftWing - https://phabricator.wikimedia.org/T434763#12304691 (10Dzahn) Hi @BWojtowicz-WMF sure, no problem. Just to confirm: You would like to import https://github.com/mediawiki-utilities right? It wou... [19:27:13] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Resolve issues with python-mwapi dependency on LiftWing - https://phabricator.wikimedia.org/T434763#12304997 (10BWojtowicz-WMF) > Just to confirm: You would like to import https://github.com/mediawiki-utilities/python-mwapi right? Yes, exactly this one... [20:15:53] (03PS4) 10Ebernhardson: semantic_highlighting: move an answer edge to a boundary a reader sees [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1330567 (https://phabricator.wikimedia.org/T433872) [20:16:03] (03CR) 10CI reject: [V:04-1] semantic_highlighting: move an answer edge to a boundary a reader sees [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1330567 (https://phabricator.wikimedia.org/T433872) (owner: 10Ebernhardson) [20:16:05] (03PS5) 10Ebernhardson: semantic_highlighting: move an answer edge to a boundary a reader sees [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1330567 (https://phabricator.wikimedia.org/T433872) [20:16:32] (03PS2) 10DCausse: semantic_highlighting: add answer and score to model output [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1338100 (https://phabricator.wikimedia.org/T433872) [20:17:01] (03CR) 10Ebernhardson: [C:03+1] Add semantic_highlighting [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1322788 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse)