[03:39:04] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment revertrisk-multilingual-predictor-default-00031-deployment in revertrisk at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [05:35:05] 06Machine-Learning-Team, 10Semantic Search, 05Goal: Q2 FY2025-26 Goal: Semantic Search - Embeddings Service for MVP - https://phabricator.wikimedia.org/T412338#11510873 (10Sucheta-Salgaonkar-WMF) [05:36:41] 06Machine-Learning-Team, 10Semantic Search, 05Goal, 07OKR-Work: Q2 FY2025-26 Goal: Semantic Search - Embeddings Service for MVP - https://phabricator.wikimedia.org/T412338#11510875 (10Sucheta-Salgaonkar-WMF) [05:49:05] 10Lift-Wing, 06Machine-Learning-Team, 10Wikidata, 06Wikimedia Enterprise, and 3 others: Q2 FY2025-26 Goal: Host Wikidata Revert Risk model on LiftWing - https://phabricator.wikimedia.org/T406179#11510890 (10kevinbazira) **Weekly Update:** - The Wikimedia Enterprise team conducted load tests to simulate the... [07:39:04] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment revertrisk-multilingual-predictor-default-00031-deployment in revertrisk at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [08:02:21] 06Machine-Learning-Team, 05Goal, 07OKR-Work: Q1 FY2025-26 Goal: Make article topic data available at scale and within SLOs for Year in Review - https://phabricator.wikimedia.org/T392833#11511095 (10BWojtowicz-WMF) **Weekly Update** //Current State// * This task is being picked up again this year. * We are... [09:03:49] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment revertrisk-multilingual-predictor-default-00031-deployment in revertrisk at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [09:08:49] RESOLVED: [2x] KubernetesDeploymentUnavailableReplicas: Deployment revertrisk-multilingual-predictor-default-00031-deployment in revertrisk at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [09:33:15] (03CR) 10Nik Gkountas: [C:03+2] Guard against: 'NoneType' object has no attribute 'keys' [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1224716 (owner: 10Sbisson) [09:34:44] (03Merged) 10jenkins-bot: Guard against: 'NoneType' object has no attribute 'keys' [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1224716 (owner: 10Sbisson) [10:08:13] 06Machine-Learning-Team, 05Goal, 07OKR-Work: Q1 FY2025-26 Goal: Task generation engine for Revise Tone task - https://phabricator.wikimedia.org/T408341#11511462 (10achou) **Weekly Report** Progress update on the hypothesis for the week, including if something has shipped: - Working on post-delivery optimiza... [10:24:36] 10Lift-Wing, 06Machine-Learning-Team, 10Wikidata, 07OKR-Work: Optimize revertrisk-wikidata inference service to achieve ~500ms latency target - https://phabricator.wikimedia.org/T414060#11511531 (10kevinbazira) I tested the multi-worker processing in the rr-wikidata model-server within a local container th... [10:54:16] (03PS1) 10AikoChou: revise-tone-task-generator: Guard against empty html to mwparserfromhtml.Article [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1225510 (https://phabricator.wikimedia.org/T412210) [10:59:49] (03PS1) 10Kevin Bazira: revertrisk-wikidata: support multi-worker processing [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1225514 (https://phabricator.wikimedia.org/T414060) [11:06:00] (03CR) 10Bartosz Wójtowicz: [C:03+1] "Thank you for tackling those :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1225510 (https://phabricator.wikimedia.org/T412210) (owner: 10AikoChou) [11:13:30] (03CR) 10AikoChou: [C:03+2] revise-tone-task-generator: Guard against empty html to mwparserfromhtml.Article [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1225510 (https://phabricator.wikimedia.org/T412210) (owner: 10AikoChou) [11:18:22] (03CR) 10Gkyziridis: [C:03+1] "Thank you for working on this one." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1225514 (https://phabricator.wikimedia.org/T414060) (owner: 10Kevin Bazira) [11:20:37] (03CR) 10Kevin Bazira: [C:03+2] revertrisk-wikidata: support multi-worker processing [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1225514 (https://phabricator.wikimedia.org/T414060) (owner: 10Kevin Bazira) [11:23:01] (03Merged) 10jenkins-bot: revise-tone-task-generator: Guard against empty html to mwparserfromhtml.Article [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1225510 (https://phabricator.wikimedia.org/T412210) (owner: 10AikoChou) [11:23:02] (03Merged) 10jenkins-bot: revertrisk-wikidata: support multi-worker processing [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1225514 (https://phabricator.wikimedia.org/T414060) (owner: 10Kevin Bazira) [13:53:09] 06Machine-Learning-Team, 05Goal, 07OKR-Work: Q1 FY2025-26 Goal: Make article topic data available at scale and within SLOs for Year in Review - https://phabricator.wikimedia.org/T392833#11512287 (10Ottomata) Thank you for these status updates! They are very helpful! [15:01:26] 06Machine-Learning-Team, 06Data-Engineering, 10Event-Platform: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399#11512600 (10Ottomata) I just reread this task and comments to try to remember our past intentions and how to m... [15:28:06] 10Lift-Wing, 06Machine-Learning-Team, 10Wikidata, 06Wikimedia Enterprise, and 3 others: Q2 FY2025-26 Goal: Host Wikidata Revert Risk model on LiftWing - https://phabricator.wikimedia.org/T406179#11512730 (10gkyziridis) ==== Update ==== >>! In T406179#11510890, @kevinbazira wrote: > **Weekly Update:** > -... [16:22:20] (03PS1) 10Gkyziridis: revertrisk-multilingual: Use the same bookworm base image as in revertrisk-language-agnostic. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1225603 (https://phabricator.wikimedia.org/T411786) [16:55:48] 06Machine-Learning-Team, 06Data-Engineering, 10Event-Platform: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399#11513204 (10Ottomata) >>> interwiki prefixes are local to the source wiki >> Maybe we can add a project (or do... [17:32:39] 06Machine-Learning-Team, 06Data-Engineering, 10Event-Platform: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399#11513365 (10Ottomata) Are a page's links a property of the page or a property of the revision? Links are in... [18:30:33] 06Machine-Learning-Team, 06Data-Engineering, 10Event-Platform: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399#11513622 (10Ottomata) Relevant: https://www.mediawiki.org/wiki/Manual:Domain_events/Hierarchy#Page_Rendering_E... [21:34:21] 06Machine-Learning-Team, 06Growth-Team, 10Revise-Tone-Structured-Task: Export all current (wiki_id, page_id) data from ml_cache.page_paragraph_tone_scores (Cassandra) - https://phabricator.wikimedia.org/T414385 (10Eevans) 03NEW [21:51:36] 06Machine-Learning-Team, 06Growth-Team, 10Revise-Tone-Structured-Task: Export all current (wiki_id, page_id) data from ml_cache.page_paragraph_tone_scores (Cassandra) - https://phabricator.wikimedia.org/T414385#11514493 (10Eevans) p:05Triage→03Medium So as a Rule of Thumb: We should not be relying on the...