[06:22:46] good morning folks :) [06:24:21] I am going to quickly test the new events.py code in staging, adn then I'll roll it out [06:29:26] elukey: hey!! any news about the wikilabels databases? [06:33:41] taavi: o/ not yet, I think that klausman is working on it, he'll read later on in the morning :) [06:35:18] the enwiki-goodfaith test in staging went fine, filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/840792/ [07:22:40] 10Machine-Learning-Team, 10Data-Engineering, 10observability: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10elukey) >>! In T319214#8280625, @gmodena wrote: > This looks really interesting, especially for ease of deployment. @elukey do you know if `http_client` calls are... [07:50:21] staging updated [07:50:36] testing enwiki-goodfaith with benthos, looks very good [07:55:51] (03PS1) 10Elukey: Import HTTPStatus where missing in the Python shared code [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/840836 [07:56:53] another bug! --^ :D [08:05:24] 10Machine-Learning-Team: Test ML model-servers with Benthos - https://phabricator.wikimedia.org/T320374 (10elukey) [08:05:43] 10Machine-Learning-Team: Test ML model-servers with Benthos - https://phabricator.wikimedia.org/T320374 (10elukey) First errors reported and fixed were related to the `events.py` module: https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/840095/ [08:06:02] (03PS2) 10Elukey: Import HTTPStatus where missing in the Python shared code [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/840836 (https://phabricator.wikimedia.org/T320374) [08:06:47] ah weird it seems that the MW api can also return [08:06:48] 'query': {'badrevids': {'1109212803': {'revid': 1109212803, 'missing': ''}}}} [08:44:53] 10Machine-Learning-Team, 10Data-Engineering, 10observability: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10BTullis) I'm in favour of further experiments with benthos, given that it appears to be so simple run and and so flexible. We might think of Benthos as the //Swis... [08:49:12] I am wondering if https://github.com/wikimedia/ores/blob/master/ores/wsgi/util.py#L178-L205 is the code that decides what revision-create events are good for ORES, and what to discard [08:49:42] mmm maybe no [08:56:34] 10Machine-Learning-Team, 10Data-Engineering, 10observability: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10elukey) I had a chat with Filippo last week and it shouldn't be too difficult to package/deploy Benthos somewhere. We could create a Debian package and deploy it t... [08:56:59] aiko: o/ [08:57:02] morning :) [08:57:21] if you have a min for https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/840836 later on (super simple) [08:57:22] good morning Luca! [08:57:49] it prevents kserve-container to log what revision-create event leads to [08:58:05] 'query': {'badrevids': {'1109212803': {'revid': 1109212803, 'missing': ''}}}} [08:58:15] (03CR) 10AikoChou: [C: 03+1] Import HTTPStatus where missing in the Python shared code [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/840836 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [08:58:18] I am wondering if ORES filters these somehow [08:58:19] aiko: <3 [08:58:27] (03CR) 10Elukey: [C: 03+2] Import HTTPStatus where missing in the Python shared code [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/840836 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [08:59:13] aiko: ah also your outlink transformer change is ready and deployed in staging if you want to test [09:01:16] elukey: I'm wondering what wiki_db is for the revid 1109212803 in the event. do we know? [09:02:43] thanks! I'll test the outlink transformer later :) [09:05:42] aiko: were you able to check how 1109212803 [09:05:45] looks like? [09:05:55] I tried using mwapi but got the error msg [09:08:53] (03Merged) 10jenkins-bot: Import HTTPStatus where missing in the Python shared code [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/840836 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [09:09:35] I also tried mwapi https://en.wikipedia.org/w/api.php?action=query&revids=1109212803&format=json&formatversion=2 and it returned the same badrevids msg. [09:12:58] looks like that revision was on a page that has been deleted [09:19:15] taavi: thanks! [09:20:06] I am going to wait for the new docker images to be published, and then I'll try to see in the logs the revision-create events triggering the problem [09:20:53] with the fix we should return a nice error msg stating that it was not possible to preprocess features from MWAPI, just wondering if we should do something when a badrevid happens or not [09:41:16] I was testing outlink transformer. the response time for the MW API call was reduced from 2702.58ms to 490.53ms. That's super nice [09:47:06] taavi: WL is at the back of my mind, but I'mmout sick today, so no progress [09:52:50] aiko: wow! [10:03:14] klausman: rest and take it easy! [10:13:22] 10Machine-Learning-Team: Test ML model-servers with Benthos - https://phabricator.wikimedia.org/T320374 (10elukey) Sometimes a given revision-create event leads to a mediawiki api response like `{'badrevids': {'1109212905': {'revid': 1109212905, 'missing': ''}}}}`. In our case the `preprocess` function handles i... [10:36:21] * elukey afk! lunch [10:38:59] klausman: ok, happy to hear that it's still in the roadmap [10:39:05] take care! [11:13:06] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Use non-blocking HTTP calls to get outlinks for Outlinks topic model - https://phabricator.wikimedia.org/T311043 (10achou) > Currently MW API returns 50 links each call, determined by the parameter gpllimit, not sure what is the maximum value we can set. Th... [11:14:59] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Align ORES prediction output with Lift Wing's one (for revscoring models) - https://phabricator.wikimedia.org/T318932 (10achou) a:03achou [11:31:54] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Align ORES prediction output with Lift Wing's one (for revscoring models) - https://phabricator.wikimedia.org/T318932 (10achou) Current status: - changed the `model.py` server of **editquality** to return output that aligned the ORES prediction output. -... [12:45:07] 10Machine-Learning-Team, 10ORES, 10Advanced Mobile Contributions, 10Growth-Team, and 3 others: 'Highlight likely problem edits' preference doesn't select any filters in mobile web - https://phabricator.wikimedia.org/T318683 (10kostajh) Tagging #readers-web-backlog in case they have ideas about this. AFAIK... [14:29:14] (03PS1) 10AikoChou: Remove directories and scripts that are not used in production [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841141 [14:32:48] 10Machine-Learning-Team, 10Data-Engineering, 10observability: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10Ottomata) > We might think of Benthos as the Swiss army knife of stream processing, compared with the CNC milling machine of stream processing provided by Flink.... [14:41:40] (03CR) 10Elukey: "Thanks a lot for the cleanup! LGTM, left some comments to see if we can remove more." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841141 (owner: 10AikoChou) [15:38:03] * elukey bbiab [16:48:36] 10Machine-Learning-Team: Test ML model-servers with Benthos - https://phabricator.wikimedia.org/T320374 (10elukey) Posted https://github.com/benthosdev/benthos/discussions/1506 to figure out how to hit multiple models from the same Benthos config. If this is possible it should enable us to have a bigger set of t... [17:05:29] 10Machine-Learning-Team: Test ML model-servers with Benthos - https://phabricator.wikimedia.org/T320374 (10elukey) Example of revisions leading to a MW API `badrevids` error: ` { "$schema": "/mediawiki/revision/create/1.1.0", "meta": { "uri": "https://en.wikipedia.org/wiki/List_of_Battle%27s_Involving_P... [17:09:31] going afk for the evening, have a nice rest of the day folks [17:10:17] (leaving benthos to hit the staging lift wing endpoint for the night) [17:10:38] (or maybe not, let's avoid issues, stopping it :)