[05:04:22] 10Machine-Learning-Team: Deploy new fawiki articlequality model to ORES and LiftWing - https://phabricator.wikimedia.org/T319373 (10kevinbazira) Both ORES and LiftWing deployments were completed successfully. The new fawiki articlequality model prediction can be seen on: 1. ORES: https://ores.wikimedia.org/v3/... [07:40:49] morning :) [08:07:04] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Observability-Logging, and 2 others: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10fgiunchedi) >>! In T319214#8400080, @Volans wrote: > @fgiunchedi @elukey I seeing some strange beha... [08:31:00] 10Lift-Wing, 10Machine-Learning-Team, 10Research: Upload new outlinks topic model to LiftWing - https://phabricator.wikimedia.org/T322881 (10achou) The new outlinks topic model has been deployed to production. I'm going to mark this as RESOLVED. Summarize the steps: 1. Check if we need to change the model... [08:33:29] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Connect Outlink topic model to eventgate - https://phabricator.wikimedia.org/T315994 (10achou) This task has been done. I'm going to mark this as RESOLVED. We'll follow up DE team's new page state stream, and open other tasks when needed. Some notes:... [08:36:01] (03CR) 10Elukey: Refactor revscoring model servers (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/856520 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [08:47:02] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Observability-Logging, and 2 others: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10Volans) >>! In T319214#8401405, @fgiunchedi wrote: > Yes very much possible, I believe the lag has... [09:03:57] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Observability-Logging, and 2 others: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10Volans) And now (before the merge of the above patch) the data is back in sync. Hence it looks to m... [09:10:44] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Observability-Logging, and 2 others: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10elukey) The other actor that plays a role in this pipeline is Druid: https://grafana.wikimedia.org/... [09:27:42] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Observability-Logging, and 2 others: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10elukey) Another thing that we should discuss is how many Kafka partitions of `webrequest_{upload,te... [09:58:13] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Observability-Logging, and 2 others: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10fgiunchedi) >>! In T319214#8401493, @Volans wrote: > And now (before the merge of the above patch)... [10:01:20] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Observability-Logging, and 2 others: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10Volans) Ok, after refreshing a bit the query in this [[ https://superset.wikimedia.org/superset/exp... [10:15:53] 10Machine-Learning-Team, 10Patch-For-Review: Add new syntax directive to blubber.yaml files to enable users to directly use docker build with blubber.yaml. - https://phabricator.wikimedia.org/T322006 (10isarantopoulos) For M1 (apple silicon) processors docker build will not work with this directive However by... [10:20:00] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Deploy revert-risk-model to production - https://phabricator.wikimedia.org/T321594 (10achou) @MunizaA The reason of the socket connection error seems to be the limited file opening on Linux according to the [[ https://medium.com/ntossapo/what-should-w... [10:21:40] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Deploy revert-risk-model to production - https://phabricator.wikimedia.org/T321594 (10achou) The revertrisk model has been deployed to production. I'm going to mark this as RESOLVED. We'll open other tasks for new model deployment when needed. Summar... [10:28:00] morning :) [11:51:58] * elukey lunch! [12:10:32] 10Machine-Learning-Team: ORES doesn't support fawiki family template - https://phabricator.wikimedia.org/T314302 (10kevinbazira) Support for sfn templates was added to fawiki articlequality, the model was retrained (T317531), evaluated (T317531#8362584), and deployed (T319373) to ORES and LiftWing. @Mojtaba-kd,... [13:05:43] 10Machine-Learning-Team, 10Patch-For-Review: Test ML model-servers with Benthos - https://phabricator.wikimedia.org/T320374 (10achou) Hi @Isaac the issue has been solved, but the task hasn't been updated. I'm doing it now. Thanks for the heads up. After adding more logging, we found MW API sometimes returned... [14:00:32] (03PS2) 10Ilias Sarantopoulos: ci: add new syntax directive for blubber files [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/856880 (https://phabricator.wikimedia.org/T322006) [14:05:04] Morning all [14:10:41] o/ [14:22:04] 10Machine-Learning-Team, 10Patch-For-Review: Test ML model-servers with Benthos - https://phabricator.wikimedia.org/T320374 (10Isaac) > We have changed the code so it doesn't raise a 400 HTTPError exception in this case! Great to hear! And I'd always wondered about batchcomplete and now I understand (T84977#93... [14:22:43] o/ [14:57:35] Where can I find model binaries in order to test them locally wiht kserve? e.g. I am trying to test article quality. Is this the place ?https://github.com/wikimedia/articlequality/tree/master/models [14:59:10] isaranto: o/ you can fetch them from the git repos, but I can also show you how to fetch them from our Swift endpoint [14:59:21] basically where/how Lift wing fetches them [14:59:30] and how to download them etc.. [14:59:34] do you have time? [14:59:42] always! [15:01:53] gimme 5 mins and then we can jump into a call [15:02:23] alrighty [15:04:34] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Research: Proposal: deprecate the mediawiki.revision-score stream in favour of more streams like mediawiki-revision-score- - https://phabricator.wikimedia.org/T317768 (10Ottomata) @Isaac, for sake of continuity, let's have this discussion over on... [15:16:31] isaranto: ah snap I have a meeting at :30 of half an hour, sigh [15:17:31] isaranto: if you are ok to meet in ~ 45 mins I can explain the whole thing, in the meantime if you need to test the model please get it from github [15:17:34] or [15:17:36] on stat100x [15:17:39] sudo s3cmd -c /etc/s3cmd/cfg.d/ml-team.cfg get s3://wmf-ml-models/articlequality/enwiki/20211022183902/model.bin [15:17:50] and then you can scp it to your laptop [15:17:54] (03CR) 10AikoChou: [C: 03+1] "something I noticed today :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/856520 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [15:18:31] Ok ping me in 45’ then.thanks for the tips [15:22:21] basically we have an s3 endpoint backed by OpenStack Swift [15:22:46] clear enough :) [15:50:00] 10Machine-Learning-Team: ORES doesn't support fawiki family template - https://phabricator.wikimedia.org/T314302 (10calbon) Thanks @kevinbazira! Thanks @Mojtaba-kd ! [16:04:05] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "As you understand my review is not extensive. However I like the idea of having a single (parent) model class. Nice work!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/856520 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [16:12:02] isaranto: ready! [16:12:19] meet.google.com/bao-rzgx-ixb [17:05:12] * elukey afk!