[07:04:08] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 13th round of wikis - https://phabricator.wikimedia.org/T308138 (10kevinbazira) Model evaluation has been completed and below are the backtesting results: | | Precision@0.5 | Recall@0.5 |novwiki | 0.88 | 0.61 |nqowiki... [07:46:53] hello folks [07:48:35] 10Machine-Learning-Team, 10API-Portal: Add documentation about LiftWing to the API Portal - https://phabricator.wikimedia.org/T325759 (10elukey) Since it was asked over IRC: Lift Wing documentation can be found in https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing#Usage, this task is to copy the sam... [07:50:27] Bon Giorno! [07:50:36] kalimera :) [08:13:40] * elukey running some errands [09:33:57] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES: Migrate ORES MediaWiki Extension to LiftWing - https://phabricator.wikimedia.org/T332200 (10elukey) This is probably a duplicate of https://phabricator.wikimedia.org/T319170, let's decide what ticket to keep open and close the other one :) [09:37:15] 10Machine-Learning-Team: Migrate ORES clients to LiftWing - https://phabricator.wikimedia.org/T312518 (10isarantopoulos) [09:37:17] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES: Migrate ORES MediaWiki Extension to LiftWing - https://phabricator.wikimedia.org/T332200 (10isarantopoulos) 05Open→03Declined Closed as it is a duplicate one [09:38:21] elukey: closed the duplicate task. Had missed the existence of the previous task [09:40:18] 10Machine-Learning-Team: Investigate procuring and installing two GPUs on Lift Wing - https://phabricator.wikimedia.org/T327923 (10elukey) a:05calbon→03None [09:43:54] 10Machine-Learning-Team, 10Research-Backlog, 10Section-Level-Image-Suggestions, 10Structured-Data-Backlog: Productionize section alignment model training - https://phabricator.wikimedia.org/T325316 (10elukey) a:05calbon→03None [09:44:46] 10Machine-Learning-Team, 10artificial-intelligence, 10SRE, 10serviceops-radar: New Service Request 'open_nsfw' - https://phabricator.wikimedia.org/T250110 (10elukey) a:05calbon→03None [09:46:23] 10Machine-Learning-Team, 10artificial-intelligence, 10SRE, 10serviceops-radar: New Service Request 'open_nsfw' - https://phabricator.wikimedia.org/T250110 (10elukey) This task needs a bit more clarification, we already have an experimental model server for nsfw content. Putting back in "Unsorted" status so... [09:48:55] 10Machine-Learning-Team, 10ContentTranslation, 10Wikimedia Enterprise: Run NLLB-200 model in a new instance - https://phabricator.wikimedia.org/T321781 (10elukey) [09:49:01] 10Machine-Learning-Team, 10Wikimedia Enterprise: Write/polish documentation for NLLb200 on AWS - https://phabricator.wikimedia.org/T324468 (10elukey) 05Open→03Declined Closing this task since we are hopefully moving off AWS for this model. We can re-open the task in case it will be needed in the future. [09:49:12] 10Machine-Learning-Team, 10ContentTranslation, 10Wikimedia Enterprise: Run NLLB-200 model in a new instance - https://phabricator.wikimedia.org/T321781 (10elukey) [09:49:36] 10Machine-Learning-Team, 10ContentTranslation, 10Wikimedia Enterprise: Add monitoring+alerting for NLLB200 AWS service - https://phabricator.wikimedia.org/T324467 (10elukey) 05Open→03Declined Closing this task since we are hopefully moving off AWS for this model. We can re-open the task in case it will b... [09:52:40] 10Machine-Learning-Team, 10Analytics-Radar, 10Data-Engineering-Icebox, 10Patch-For-Review: Upgrade ROCm to 4.5 - https://phabricator.wikimedia.org/T295661 (10elukey) Upstream already reached 5.x, we should probably upgrade to a more recent version as well to keep up and have better support (especially if w... [09:54:01] 10Machine-Learning-Team, 10ORES: Review pre-cached wikis for ORES - https://phabricator.wikimedia.org/T281713 (10elukey) 05Open→03Declined We are deprecating ORES as part of T312518. [09:56:01] 10Machine-Learning-Team, 10ORES: ORES deployment repos not mirroring regular git changes anymore - https://phabricator.wikimedia.org/T299664 (10elukey) 05Open→03Declined We are moving away from git to store models with Lift Wing (in favor of Swift). [10:02:44] 10Machine-Learning-Team, 10Epic: Migrate Content Translation Recommendation API to Lift Wing - https://phabricator.wikimedia.org/T308164 (10elukey) @calbon @kevinbazira should we keep this task open? If so, what are the next steps and/or subtasks? [10:14:17] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES: Migrate ORES MediaWiki Extension to LiftWing - https://phabricator.wikimedia.org/T332200 (10Aklapper) @isarantopoulos: In such cases, please feel free to {nav icon=anchor,name=Edit Related Tasks... > Close As Duplicate} in the upper right corner.... [10:14:40] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES: Move backend of ORES MediaWiki extension to Lift Wing - https://phabricator.wikimedia.org/T319170 (10Aklapper) [10:14:45] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES: Migrate ORES MediaWiki Extension to LiftWing - https://phabricator.wikimedia.org/T332200 (10Aklapper) [10:20:39] klausman: o/ when you have time (even next week) could you please create a task related to TrainWing? Maybe a special tag is needed, etc.. [10:20:51] Will do [10:21:03] (even something related to producing a proposal etc.., it is sufficient) [10:21:05] thanks :) [11:32:10] * elukey lunch [12:08:02] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10MatthewVernon) [12:27:02] 10Lift-Wing, 10Machine-Learning-Team: Update revert-risk multilingual model - https://phabricator.wikimedia.org/T332392 (10achou) [15:49:06] o/ is there any more info about train wing at the moment? [16:00:16] desperation, sadness and tears :D [16:00:25] this is what it will cost us to deploy it :D [16:00:31] I don't know more :D [16:00:54] jokes aside, I think that we don't have a lot of ideas yet, nobody worked on kubeflow before [16:01:11] and there is the issue of accessing data using kerberos from hadoop [16:16:30] ack [16:18:03] I can help in some parts. I've deployed it both minikube and kind and I've helped upgrade it on k8s (not initial deployment) [16:18:28] the official docs suggest a pretty dirty way to deploy it which is not recommended [16:23:31] (03CR) 10Ilias Sarantopoulos: [C: 03+2] ores-legacy: add LW error messages and exceptions to response [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/897910 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [16:26:36] (03Merged) 10jenkins-bot: ores-legacy: add LW error messages and exceptions to response [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/897910 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [16:35:16] (03PS9) 10Ilias Sarantopoulos: Add logging for FastAPI app [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) [16:55:17] elukey: I'll be merging the logging patch if that is ok with you. I added all except the response size and I refactored a bit moving most of logging to the decorator so main is much clearer [17:01:30] yes yes please! [17:01:55] I am logging off in a bit, have a nice weekend folks! [17:04:31] have a nice weekend! [17:04:39] (03CR) 10Ilias Sarantopoulos: [C: 03+2] Add logging for FastAPI app [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [17:05:41] (03Merged) 10jenkins-bot: Add logging for FastAPI app [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [17:21:34] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES: Move backend of ORES MediaWiki extension to Lift Wing - https://phabricator.wikimedia.org/T319170 (10isarantopoulos) It seems to me that there are two main changes that will enable the transition to Lift Wing: - ORESService.php: as @elukey notes above we...