[06:27:38] good morning :) [07:11:42] going to rollout the new docker images to the revscoring pods [07:11:47] on ml-serve-* clusters [07:53:49] 10Machine-Learning-Team, 10Observability-Logging: Logging spam from revscoring deploys - https://phabricator.wikimedia.org/T320468 (10fgiunchedi) [07:54:20] 10Machine-Learning-Team, 10Observability-Logging: Logging spam from revscoring deploys - https://phabricator.wikimedia.org/T320468 (10fgiunchedi) See also {T300707} cc @elukey [08:03:31] we'd need to decide what is the best HTTP return code for preprocess() errors due to MW API's badrevids [08:04:04] if we alarm on HTTP 500 then we may get a lot of false positives in the future [08:04:30] maybe in this specific case a HTTP 400 is more appropriate [08:05:23] yeah I am inclined to change the code to reflect this [08:14:04] * elukey prepare patch [08:20:02] (03PS1) 10Elukey: extractor_utils.py: handle badrevids errors from the MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841449 (https://phabricator.wikimedia.org/T320374) [08:43:16] morning! :) [08:43:29] (03PS2) 10Elukey: extractor_utils.py: handle badrevids errors from the MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841449 (https://phabricator.wikimedia.org/T320374) [08:43:34] aiko: o/ [08:44:24] (03CR) 10Elukey: "Tested locally on Docker, looks working nicely :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841449 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [08:48:57] * elukey afk for a bit [09:16:35] (03CR) 10AikoChou: [C: 03+1] "Looks good to me! only two nits :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841449 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [09:42:04] (03PS3) 10Elukey: extractor_utils.py: handle badrevids errors from the MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841449 (https://phabricator.wikimedia.org/T320374) [09:42:54] (03CR) 10Elukey: extractor_utils.py: handle badrevids errors from the MW API (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841449 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [09:54:12] 10Machine-Learning-Team, 10Patch-For-Review: Test ML model-servers with Benthos - https://phabricator.wikimedia.org/T320374 (10elukey) I was able to configure Benthos with the following to hit en and zh wiki in Lift Wing staging: ` pipeline: threads: 1 processors: - bloblang: | root = match {... [09:54:15] aiko: --^ [09:54:34] (03CR) 10AikoChou: [C: 03+1] extractor_utils.py: handle badrevids errors from the MW API (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841449 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [09:56:05] (03PS4) 10Elukey: extractor_utils.py: handle badrevids errors from the MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841449 (https://phabricator.wikimedia.org/T320374) [09:56:18] (03CR) 10Elukey: extractor_utils.py: handle badrevids errors from the MW API (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841449 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [10:04:50] elukey: that's cool it supports regular expression [10:04:56] e [10:05:32] also pattern matching! [10:05:32] elukey: does it support multiple http_clients in the output section? so we can hit different models like damaging and reverted model [10:05:34] (03CR) 10Ladsgroup: [C: 03+2] Make use of the ?? syntax where it makes sense [extensions/ORES] - 10https://gerrit.wikimedia.org/r/838877 (owner: 10Thiemo Kreuz (WMDE)) [10:05:53] (03CR) 10Ladsgroup: [C: 03+2] Use PHPUnit's convenience shortcuts where possible [extensions/ORES] - 10https://gerrit.wikimedia.org/r/838876 (owner: 10Thiemo Kreuz (WMDE)) [10:07:22] or can the url be written as 'https://inference-staging.svc.codfw.wmnet:30443/v1/models/${! json("revision_create_event.database") }-(goodfaith|damaging|reverted):predict'? [10:09:11] aiko: IIUC only one sink/output is supported, but you can workaround it https://www.benthos.dev/docs/components/outputs/about [10:09:32] in our case it may also be ok to have multiple streams configured in the same way, if we template benthos in the right way [10:10:13] do we consider goodfaith/damaging/reverted different "models" ? [10:10:26] if so we'll have different revisions-score streams in theory [10:14:09] yeah I think goodfaith/damaging/reverted are different models [10:15:35] but goodfaith and damaging are regarded as more advanced and they support more wikis [10:15:50] yep yep [10:15:59] you can see it in the doc https://www.mediawiki.org/wiki/ORES#Edit_quality [10:15:59] so far Benthos seems to be a good choice [10:16:20] let's see what DE thinks about it [10:16:30] it could be an alternative to Flink [10:16:38] (I mean having both at the same time) [10:20:22] sounds good to me :D [10:22:27] (03CR) 10Elukey: [C: 03+2] extractor_utils.py: handle badrevids errors from the MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841449 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [10:25:50] (03Merged) 10jenkins-bot: Make use of the ?? syntax where it makes sense [extensions/ORES] - 10https://gerrit.wikimedia.org/r/838877 (owner: 10Thiemo Kreuz (WMDE)) [10:26:26] (03Merged) 10jenkins-bot: Use PHPUnit's convenience shortcuts where possible [extensions/ORES] - 10https://gerrit.wikimedia.org/r/838876 (owner: 10Thiemo Kreuz (WMDE)) [10:30:31] * elukey lunch! [10:31:10] (03Merged) 10jenkins-bot: extractor_utils.py: handle badrevids errors from the MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841449 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [11:11:53] (03CR) 10AikoChou: Remove directories and scripts that are not used in production (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841141 (owner: 10AikoChou) [11:24:56] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Cleanup the liftwing/inference-services repository - https://phabricator.wikimedia.org/T320494 (10achou) [11:28:07] (03PS2) 10AikoChou: Remove directories and scripts that are not used in production [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841141 (https://phabricator.wikimedia.org/T320494) [11:32:46] --^ submit a new patch that removed test/benchmark directory [12:02:14] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Cleanup the liftwing/inference-services repository - https://phabricator.wikimedia.org/T320494 (10achou) I had a look at the files in `test/benchmark/` directory. There are some load test job configs using [[ https://github.com/tsenart/... [12:47:07] ah interesting [12:47:08] revscoring.errors.TextDeleted: TextDeleted: Text deleted (datasource.revision.parent.text) [12:47:38] this is returned in editquality's feature_values = list(self.extractor.extract(rev_id, self.model.features)) [12:51:24] aiko: qq (if you have time) - I see various functions in our codebase called "fetch_modeltype_features" [12:51:46] that are a wrapper around the revscoring's extractor basicaly [12:51:50] *basically [12:52:05] would it make sense to centralize them in revscoring_utils.py? [12:52:20] sorry, extrator_utils.py [12:54:11] 10Machine-Learning-Team: Test ML model-servers with Benthos - https://phabricator.wikimedia.org/T320374 (10elukey) Interesting exception raised: ` Traceback (most recent call last): File "/opt/lib/python/site-packages/tornado/web.py", line 1704, in _execute result = await result File "/o... [12:54:25] reported the full stack trace in --^ [13:02:04] 10Machine-Learning-Team, 10Data-Engineering, 10observability: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10elukey) >>! In T319214#8305260, @Ottomata wrote: >> >> Do you have any specific requirements in mind? If so I can try to test them :) > Our main use cases is makin... [13:20:03] 10Machine-Learning-Team, 10Data-Engineering, 10observability: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10Ottomata) The tricky thing about async calls in streams, is that the ordering of the events might get all messed up, as the calls will evaluate in an undetermined... [13:23:29] Morning all! [13:39:30] morning chrisalbon ! [13:41:30] morning! I'm just catching up on the IRC messages since Friday [13:43:26] (03CR) 10Kevin Bazira: [C: 03+2] "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841141 (https://phabricator.wikimedia.org/T320494) (owner: 10AikoChou) [13:45:04] (03PS3) 10AikoChou: Remove directories and scripts that are not used in production [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841141 (https://phabricator.wikimedia.org/T320494) [13:49:16] kevinbazira: o/ [13:49:24] the patch was still under discussion :( [13:51:34] elukey: o/ sorry, it looked like Aiko and I had given our position. I didn't know the discussion was not done yet. My bad 🙏 [13:52:20] Thanks Kevin! Next time you can give +1, so we know you approve it. Because if you give +2, the CI will start merging the code. [13:52:59] kevinbazira: exactly yes, let's use the rule of not adding +2s for code created by others, only +1s [13:53:05] then everybody can +2 their code etc.. [13:53:16] so we'll know exactly the right time to merge [13:53:19] Yeah, I sure will. [13:53:21] no issue this time don't worry! [13:53:25] :) [13:53:41] Good lesson for all of us [13:59:35] big thing for me today: update the lift wing comms material and get it on wiki. tltaylor put a ton of notes and edits so I think a new draft is within reach https://docs.google.com/document/d/1nAEZUzt0sKzL5DkrS52T9j5cE_O3CnyftZ5Q4H3PLYQ/edit?pli=1#heading=h.qiqnwzwc3u3r [14:34:24] 10Machine-Learning-Team, 10ORES, 10MediaWiki-Core-Preferences, 10Moderator-Tools-Team (Kanban): When ORES quality filters are selected in mobile web, entries should be highlighted - https://phabricator.wikimedia.org/T314026 (10eigyan) Greetings @Jdlrobson, based on your last feedback we have taken some tim... [14:46:24] 10Machine-Learning-Team, 10ORES, 10MediaWiki-Core-Preferences, 10Moderator-Tools-Team (Kanban): When ORES quality filters are selected in mobile web, entries should be highlighted - https://phabricator.wikimedia.org/T314026 (10Samwalton9) 05Open→03Stalled [14:48:23] 10Machine-Learning-Team, 10ORES, 10Advanced Mobile Contributions, 10Growth-Team, and 3 others: 'Highlight likely problem edits' preference doesn't select any filters in mobile web - https://phabricator.wikimedia.org/T318683 (10Samwalton9) 05Open→03Stalled [15:03:50] * elukey taking a break [16:32:13] * elukey afk! [16:32:19] have a good rest of the day folks! [17:05:57] 10Machine-Learning-Team: Test ML model-servers with Benthos - https://phabricator.wikimedia.org/T320374 (10achou) @elukey Is this exception raised by a deleted page (`badrevids` error)? To your question - yeah I think it is good to centralize them in extrator_utils.py, because the function "fetch_modeltype_feat... [17:25:45] (03PS4) 10AikoChou: Remove directories and scripts that are not used in production [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/841141 (https://phabricator.wikimedia.org/T320494)