[00:52:44] RESOLVED: LiftWingServiceErrorRate: ... [00:52:44] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-damaging&var-backend=itwiki-damaging-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [02:17:52] FIRING: [3x] SLOMetricAbsent: revertrisk-la-availability - https://slo.wikimedia.org/?search=revertrisk-la-availability - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [06:17:52] FIRING: [3x] SLOMetricAbsent: revertrisk-la-availability - https://slo.wikimedia.org/?search=revertrisk-la-availability - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [07:38:44] (03CR) 10Nikerabbit: [C:03+2] Update dependencies [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1245360 (owner: 10KartikMistry) [07:39:50] (03CR) 10Nikerabbit: [C:03+2] Cache update: randomize sleep time after failure [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1240745 (owner: 10Sbisson) [07:41:36] (03CR) 10CI reject: [V:04-1] Cache update: randomize sleep time after failure [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1240745 (owner: 10Sbisson) [07:41:38] (03CR) 10CI reject: [V:04-1] Update dependencies [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1245360 (owner: 10KartikMistry) [10:17:53] FIRING: [3x] SLOMetricAbsent: revertrisk-la-availability - https://slo.wikimedia.org/?search=revertrisk-la-availability - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [10:22:40] (03CR) 10Gkyziridis: [C:03+2] Revertrisk-multilingual: Add predictions to events stream. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1238685 (https://phabricator.wikimedia.org/T415892) (owner: 10Gkyziridis) [10:27:21] (03Merged) 10jenkins-bot: Revertrisk-multilingual: Add predictions to events stream. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1238685 (https://phabricator.wikimedia.org/T415892) (owner: 10Gkyziridis) [11:32:34] (03CR) 10AikoChou: "I did an initial pass and have one question to better understand the implementation :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1245307 (https://phabricator.wikimedia.org/T418493) (owner: 10Bartosz Wójtowicz) [11:37:04] (03PS5) 10Bartosz Wójtowicz: article-topics: Add outlink cache adapter for outlink-topic-model [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1245307 (https://phabricator.wikimedia.org/T418493) [11:37:50] (03CR) 10Bartosz Wójtowicz: article-topics: Add outlink cache adapter for outlink-topic-model (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1245307 (https://phabricator.wikimedia.org/T418493) (owner: 10Bartosz Wójtowicz) [13:13:29] 06Machine-Learning-Team, 06Data-Engineering, 10Event-Platform: Emit article quality predictions as a stream and expose in EventStreams API. - https://phabricator.wikimedia.org/T417794#11662794 (10achou) We likely need a new event schema for this use case. The schema that Lift Wing has been using assumes [[ h... [13:56:29] (03CR) 10KartikMistry: "recheck" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1240745 (owner: 10Sbisson) [14:17:53] FIRING: [3x] SLOMetricAbsent: revertrisk-la-availability - https://slo.wikimedia.org/?search=revertrisk-la-availability - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [15:09:51] (03PS6) 10Bartosz Wójtowicz: article-topics: Add outlink cache adapter for outlink-topic-model [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1245307 (https://phabricator.wikimedia.org/T418493) [15:13:09] 06Machine-Learning-Team: Build and push images to the docker registry from ml-lab - https://phabricator.wikimedia.org/T394778#11663260 (10DPogorzelski-WMF) 05Open→03Resolved [15:14:51] 06Machine-Learning-Team, 06Data-Engineering, 10Event-Platform: Emit article quality predictions as a stream and expose in EventStreams API. - https://phabricator.wikimedia.org/T417794#11663262 (10Ottomata) > The schema that Lift Wing has been using assumes classification outputs and no more. The article qual... [16:58:46] 06Machine-Learning-Team, 10EditCheck, 06Growth-Team, 10Revise-Tone-Structured-Task, and 3 others: LiftWing edit-check:predict model is 404ing - https://phabricator.wikimedia.org/T418173#11664119 (10ppelberg) 05Open→03Resolved a:03ppelberg [17:51:43] 06Machine-Learning-Team: Incident: 2026-02-23 ml-serve - https://phabricator.wikimedia.org/T418722#11664583 (10Aklapper) [18:17:53] FIRING: [3x] SLOMetricAbsent: revertrisk-la-availability - https://slo.wikimedia.org/?search=revertrisk-la-availability - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [18:30:19] 06Machine-Learning-Team, 06Wikimedia Enterprise: Test liftwing wikidata revert risk API for scale and latency - https://phabricator.wikimedia.org/T409388#11664793 (10FNavas-foundation) Decision - given the better results, we're going to move forward with productionizing at the current standard. We expect our r... [22:17:53] FIRING: [3x] SLOMetricAbsent: revertrisk-la-availability - https://slo.wikimedia.org/?search=revertrisk-la-availability - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent