[07:31:14] morning folks [07:46:39] 10Lift-Wing, 10Machine-Learning-Team: Move Revert-risk multilingual model from staging to production - https://phabricator.wikimedia.org/T333124 (10elukey) ` elukey@stat1004:~$ time curl https://api.wikimedia.org/service/lw/inference/v1/models/revertrisk-multilingual:predict -X POST -d '{"rev_id": 123456, "lan... [07:46:53] revert risk multi-lingual exposed via api gateway! [07:57:29] Morning! [07:57:32] And: nice! [07:58:22] Also works fine from my home machine [07:59:23] we just need the docs on the api portal and we are done [07:59:26] Latency is an additional 200ms or so. [08:00:14] (as in end-to-end latency, which is of course a very complex beast) [08:07:08] thank Luca!! \o/ [08:07:29] I'll add the docs on the api portal! [08:12:22] super [08:48:08] 10Machine-Learning-Team, 10ORES: Build a brand-new GUI for ORES - https://phabricator.wikimedia.org/T207071 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that something is missing, b... [08:48:53] 10Machine-Learning-Team, 10ORES, 10UI-Standardization, 10Epic: Make the ORES UIs better - https://phabricator.wikimedia.org/T195607 (10elukey) [08:49:27] 10Machine-Learning-Team, 10ORES, 10patch-welcome: Extend the GUI for ORES - https://phabricator.wikimedia.org/T107723 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that something i... [08:49:33] 10Machine-Learning-Team, 10ORES: Implement selective purging of model scores in varnish - https://phabricator.wikimedia.org/T148999 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that... [08:52:27] 10Machine-Learning-Team, 10ORES, 10Upstream: Make ORES dependency solving upstreamable - https://phabricator.wikimedia.org/T201657 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel tha... [08:52:42] 10Machine-Learning-Team, 10ORES, 10Scap: scap support for multiple services - https://phabricator.wikimedia.org/T185255 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that something... [08:52:45] 10Machine-Learning-Team: Investigate separating k8s-level users between our k8s and thr ServiceOps k8s - https://phabricator.wikimedia.org/T277492 (10klausman) 05Open→03Resolved a:03klausman Yes, I think we can close this. I'd even say that having different user configs in codfw vs eqiad is an antipattern. [08:53:02] 10Machine-Learning-Team, 10drafttopic-modeling: Fit more topic models into ORES - https://phabricator.wikimedia.org/T249520 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that somethi... [08:53:04] 10Machine-Learning-Team, 10drafttopic-modeling: Build article topic model for Swedish Wikipedia - https://phabricator.wikimedia.org/T252599 (10elukey) [08:53:09] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10articlequality-modeling, and 2 others: ORES deployment - Spring 2021 - https://phabricator.wikimedia.org/T278723 (10elukey) [08:53:11] 10Machine-Learning-Team, 10ORES: [Discuss] Future ORES architecture - https://phabricator.wikimedia.org/T226193 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that something is missin... [08:53:42] 10Machine-Learning-Team, 10ORES, 10Epic: Using TensorFlow in ORES - https://phabricator.wikimedia.org/T161375 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that something is missin... [08:58:55] 10Machine-Learning-Team: Shut down and deconfigure NLLB setup on AWS - https://phabricator.wikimedia.org/T337369 (10klausman) 05Open→03In progress p:05Triage→03Medium [09:05:23] 10Machine-Learning-Team: Shut down and deconfigure NLLB setup on AWS - https://phabricator.wikimedia.org/T337369 (10klausman) Instances have been shutdown and S3 has been cleared of all but the latest checkpoint etc. Still configured: - Endpoint configset - Lambda - API GW I will take care of those at the end... [09:05:46] 10Machine-Learning-Team, 10MinT: Shut down and deconfigure NLLB setup on AWS - https://phabricator.wikimedia.org/T337369 (10Pginer-WMF) [09:10:20] 10Machine-Learning-Team: Move the kserve custom helm chart to the upstream one - https://phabricator.wikimedia.org/T327241 (10klausman) This should probably until we have updated to kserve 0.11 (T337213), I will tackle this after that. [09:10:30] 10Machine-Learning-Team: Move the kserve custom helm chart to the upstream one - https://phabricator.wikimedia.org/T327241 (10klausman) a:03klausman [09:10:32] 10Machine-Learning-Team, 10ORES: Document differences between features in Python and API - https://phabricator.wikimedia.org/T199485 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel tha... [09:10:56] 10Machine-Learning-Team, 10ORES-Support-Checklist: Document and share operational details of ores-support-checklist - https://phabricator.wikimedia.org/T222271 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Ple... [09:11:07] 10Machine-Learning-Team, 10ORES: ORES query with many statistics results in 503 - https://phabricator.wikimedia.org/T232855 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that somethi... [09:11:36] 10Machine-Learning-Team, 10MediaWiki-Vagrant: Clean up ORES vagrant role - https://phabricator.wikimedia.org/T181850 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that something is m... [09:11:58] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence: Package dictionaries better for ORES models - https://phabricator.wikimedia.org/T217343 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-ope... [09:12:02] 10Machine-Learning-Team, 10ORES: Clean up 500 errors in ORES service - https://phabricator.wikimedia.org/T210575 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that something is missi... [09:12:09] 10Machine-Learning-Team, 10ORES, 10Infrastructure-Foundations, 10SRE, 10Puppet: Clean up puppet & configs for ORES - https://phabricator.wikimedia.org/T142002 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES... [09:12:25] 10Machine-Learning-Team, 10ORES: Dogfooding ORES - https://phabricator.wikimedia.org/T195239 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that something is missing, but we'd prefer... [09:12:34] 10Machine-Learning-Team, 10ORES: ORES worker icinga message not specific enough - https://phabricator.wikimedia.org/T181536 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that somethi... [09:12:45] 10Machine-Learning-Team, 10ORES, 10SRE: Clean up redundant ORES celery_workers defaults - https://phabricator.wikimedia.org/T186734 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel th... [09:16:47] 10Machine-Learning-Team, 10ORES, 10Documentation, 10User-AKlapper: Update docs that ORES will be replaced by Lift Wing - https://phabricator.wikimedia.org/T305963 (10elukey) This should be prioritized :) Now we have https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, and in its intro there is a... [09:17:15] 10Machine-Learning-Team, 10ORES, 10Documentation, 10User-AKlapper: Update docs that ORES will be replaced by Lift Wing - https://phabricator.wikimedia.org/T305963 (10elukey) We should probably add a banner in multiple pages pointing users to the Lift Wing Wikitech page, thoughts? [09:17:23] 10Machine-Learning-Team, 10ORES, 10Documentation, 10User-AKlapper: Update docs that ORES will be replaced by Lift Wing - https://phabricator.wikimedia.org/T305963 (10elukey) a:05calbon→03None [09:17:56] did a quick cleanup of old tasks :D [09:18:00] more to come in the next days [09:25:56] elukey: o/ [09:26:16] aiko: o/ [09:26:51] I found a problem with the regex in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/922073. It doesn't match "revertrisk-language-agnostic" because (\w+) only matches "language" [09:27:16] we can change the regex to revertrisk-(\w+-*\w+), so it matches one more "-" symbol [09:27:34] aiko: ahhh interesting yes, we can do it when we publish the next one! [09:27:38] would it be ok? [09:27:46] didn't think about that use case [09:29:05] we already moved both models to production, didn't we? [09:31:50] yep we have revertrisk-language-agnostic and revertrisk-multilingual in prod [09:32:28] I'd recommend `-[\w-]+:predict` instead, so we can have even more dashes, just in case. But that's nitpicking. [09:33:01] Oh, and add () to capture the right group for \1 [09:40:25] ([\w-]+) looks good. but I agree let's wait for the next publish coz I haven't gotten the model card for the revertrisk-language-agnostic [09:40:56] Roger, I'll prep a change right now and we can then review/publish it when the time comes [09:43:38] klausman: o/ thanks! [09:48:27] 10Machine-Learning-Team: Fix Regular Expression in API GW config for revert risk - https://phabricator.wikimedia.org/T337378 (10klausman) [09:51:20] Also reminds me to once more sing the praise of regex101.com [10:02:29] Is there a strict schedule for the publications or is it just "whenever we have something to publish"? [10:07:52] the latter, but if you want you can make the change now and deploy it once Hugh reviews it [10:08:09] it was just to avoid another round of api-gateway deployments, but if you have time go ahead :) [10:10:36] (03PS2) 10Elukey: WIP - Add read-only cache support to ores-legacy [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/922561 (https://phabricator.wikimedia.org/T337287) [10:10:38] (03PS1) 10Elukey: ores-legacy: simplify test in test_liftwing.py [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/922809 (https://phabricator.wikimedia.org/T337287) [10:12:02] (03PS3) 10Elukey: WIP - Add read-only cache support to ores-legacy [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/922561 (https://phabricator.wikimedia.org/T337287) [10:12:17] (03CR) 10CI reject: [V: 04-1] ores-legacy: simplify test in test_liftwing.py [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/922809 (https://phabricator.wikimedia.org/T337287) (owner: 10Elukey) [10:12:42] (03PS4) 10Elukey: WIP - Add read-only cache support to ores-legacy [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/922561 (https://phabricator.wikimedia.org/T337287) [10:20:34] elukey: roger, will do so after lunch [10:30:28] * elukey lunch [10:33:57] same [12:47:58] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10ML-Governance, 10Documentation: Use citation template on English Wikipedia ORES model cards - https://phabricator.wikimedia.org/T337391 (10kevinbazira) [13:54:14] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10ML-Governance, 10Documentation: Use citation template on English Wikipedia ORES model cards - https://phabricator.wikimedia.org/T337391 (10kevinbazira) The citation transclusion template has been added to 5/6 English Wikipedia ORES model cards... [13:56:18] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10ML-Governance, 10Documentation: Use citation template on English Wikipedia ORES model cards - https://phabricator.wikimedia.org/T337391 (10kevinbazira) [16:18:14] * elukey afk! [18:00:02] https://api.wikimedia.org/wiki/API_reference/Service/Lift_Wing/Get_revertrisk_multilingual_prediction :D [18:36:56] Nice! Hopefully I can submit the API GW change tomorrow and the rr-l-a model will work then as well [21:44:37] (03CR) 10Ladsgroup: [C: 03+2] Handle possible null statistics on SpecialORESModels [extensions/ORES] - 10https://gerrit.wikimedia.org/r/919927 (https://phabricator.wikimedia.org/T329304) (owner: 10Umherirrender) [21:46:40] (03Merged) 10jenkins-bot: Handle possible null statistics on SpecialORESModels [extensions/ORES] - 10https://gerrit.wikimedia.org/r/919927 (https://phabricator.wikimedia.org/T329304) (owner: 10Umherirrender)