[00:24:56] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Investigate what would be required to include countries in ORES articletopic modeling - https://phabricator.wikimedia.org/T301671 (10Tgr) We only need Argentina for the March 11 deadline, not all countries, right? >>! In... [09:59:20] kevinbazira_: o/ [09:59:37] elukey: o/ [09:59:43] do we support things like https://ores.wikimedia.org/v3/scores/frwiki/186357639/damaging?features in liftwing? [09:59:50] namely the extra features output [10:00:13] I had a chat with Joseph and I think people use it (IIUC it is the revscoring feature extractor) [10:01:32] Ummm ... currently we don't seem to be supporting this in LW. [10:02:50] We could discuss in the tech meeting, if there is consensus to add it then we'll add it. [10:03:19] I can open a task in the meantime, thanks :) [10:04:06] Great. Thanks! [10:31:35] I also have an idea to improve the kserve-inference chart for the revscoring use case [10:31:59] we could have a separate yaml to simplify the isvc config and avoid some repetion [10:32:02] *repetition [11:20:16] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Investigate what would be required to include countries in ORES articletopic modeling - https://phabricator.wikimedia.org/T301671 (10kostajh) [11:20:37] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Investigate what would be required to include countries in ORES and accessible via a search keyword - https://phabricator.wikimedia.org/T301671 (10kostajh) [11:20:48] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Investigate what would be required to include countries in ORES and accessible via a search keyword - https://phabricator.wikimedia.org/T301671 (10kostajh) [11:21:13] created https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/762790 and https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/762791 [11:21:37] this is an idea about how to simplify the current revscoring settings, without adding too many conditionals to the main service.yaml [11:21:54] we can use this config for the current models, and deprecate it once they will not be used [11:23:50] 10Lift-Wing, 10Machine-Learning-Team: Support (or not) the ORES augmented feature output in liftwing - https://phabricator.wikimedia.org/T301766 (10elukey) [11:25:42] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Investigate what would be required to include countries in ORES and accessible via a search keyword - https://phabricator.wikimedia.org/T301671 (10kostajh) >>! In T301671#7708303, @EBernhardson wrote: > We talked about thi... [13:52:11] kserve 0.8 released - https://github.com/kserve/kserve/releases/tag/v0.8.0 [15:02:23] Morning all [15:03:18] This is day 8 [15:03:24] of me [15:03:37] solo parenting [15:04:14] keep going Chris, you should be almost done right? (10 days) [15:06:38] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10elukey) >>! In T300195#7708977, @Halfak wrot... [15:09:08] Yep almost done! Day 10 is the end [15:24:53] 10Machine-Learning-Team, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install ml-serve100[5-8] - https://phabricator.wikimedia.org/T294949 (10elukey) [15:33:59] chrisalbon: as FYI we have the stream of scores generated by precache calls from changeprop on Hadoop/Hive, event.mediawiki_revision_score [15:34:20] so the dataset can be queried etc.. [15:36:23] From ores or lift wing? [15:37:30] chrisalbon: it is currently done from ORES scores, and people rely on it afaics.. So we'll need to replace ORES with Liftwing, maybe refactoring a bit the pipeline to avoid the changeprop hack [15:37:39] (we could collaborate with Data Engineering on this) [15:38:41] Cool, good to know we have one thing to build in the mvp [15:42:18] not sure if it can be done for the MVP, it would be great but it is a rather complex thing [15:42:32] I'll try to open a task to summarize it all [15:44:27] Okay great thanks [15:51:14] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/change-propagation/+/refs/heads/master/sys/ores_updates.js [15:51:24] this is the special code in change-prop for ores [15:54:07] o/ [15:54:17] nice find elukey! [15:54:51] ive actually never looked at the ores changeprop code before lol [15:55:36] o/ [15:56:15] accraze: it listens for a revision create event, and hits ores with a precache URI, then it sends the score as event to eventgate (mediawiki.revision_score) [15:56:51] ideally, as Andrew was saying the other day, ORES/LiftWing should send the event to eventgate [15:57:58] good find. What are the advantages to our users with eventgate as opposed to say, a scoring cache, mediawiki db query, raw API hit etc? [15:59:02] Miriam's use case is one example, namely having scores datasets to use as feature when training [15:59:30] So if someone wanted all the scores from an article over time they could get this from eventgate? [16:00:14] chrisalbon: eventgate is only an http endpoint that receives event, validates them (via special schemas) and stores the event in kafka [16:00:37] then any consumer can pull from kafka and populate different things [16:00:57] the DE folks pull from the mediawiki.revision-score topic in kafka and store the data on HDFS (with some refinement) [16:01:39] ah, and we could work with them to make a publicly accessible historical score database (not a requirement, just a thought) [16:01:41] the mediawiki db query is something different, it is used (IIUC) only for special events called ORES FetchScoreJob (or similar) [16:02:34] chrisalbon: yep there was a big discussion in the past to produce regular dumps of scores $somewhere [16:02:45] elukey: for training, we can use scores, but also embeddings, I think we can consider both as "predictions" from a netowrk. Another use case is to use scores (say, article topics) for large-scale analysis, as we do in the research team or product analytics. [16:03:11] okay cool, got it. I am just trying to build in my head the different access points for the scores and their use cases. [16:03:57] ^ yeah we should document the access points for scores or make a diagram or something [16:04:32] that would be good, it'd help us figure out how we should prioritize building things out [16:05:12] we can probably use a wikipage [16:05:17] Additionally, the mediawiki.revision-score is exposed publicly via EventStreams [16:05:34] https://stream.wikimedia.org/?doc#/streams/get_v2_stream_mediawiki_revision_score [16:05:39] https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams [16:05:49] https://stream.wikimedia.org/v2/ui/#/?streams=mediawiki.revision-score [16:07:46] accraze: one thing that we could do in LiftWing's revscoring code is to have a special endpoint/URI that if called, sends a score as HTTP POST event to eventgate [16:08:11] in this way changeprop should simply call us [16:08:48] ^ yep that would be pretty straight forward [16:09:03] we could build a separate stream called mediawiki.revision-score-liftwing or similar, and compare results for a while [16:09:26] could look into knative eventing too eventually but that might be a bit heavy [16:09:45] yeah and we also have an old version of knative, so it may not be great [16:09:52] ahhh right :( [16:10:00] I thought about evening too, maybe it could be part of a next iteration [16:11:20] accraze: did you see my task about the ?feature thing? Are we going to support it? [16:11:33] didn't know that it was available until today [16:12:46] yep! just saw it.... unless im missing something that would be trivial to include in the response, as we already have the `feature_values` available in a variable [16:13:17] super.. would it be a different api-gateway URI, or maybe an extra parameter? [16:13:35] like "include_features: bool" [16:13:48] or return_features, whatever [16:14:05] hmmm yeah id say lets do an extra parameter in input.json for now [16:15:51] the only model class that might be tricky is articlequality since we aren't using the extractor directly in the isvc code [16:16:07] i gotta think about that one a bit [16:21:24] accraze: ack, do you want me to open a task? [16:23:24] should we just use https://phabricator.wikimedia.org/T301766? [16:26:00] accraze: right sorry too many tasks in mind, I'll add a comment to it :) [16:26:19] cool! [16:26:32] lots of tasks :) [16:27:19] 10Lift-Wing, 10Machine-Learning-Team: Support (or not) the ORES augmented feature output in liftwing - https://phabricator.wikimedia.org/T301766 (10elukey) After a chat with Andy, it seems that an easy way to go could be to add another input parameter that states whether or not the full features output is need... [16:29:24] I am wondering if we should have a task for things that are still missing before reaching MVP [16:29:35] like LiftWingMVP or similar [16:29:45] (if we don't have one already, in case I forgot about it) [16:33:17] 10Lift-Wing, 10Machine-Learning-Team: Support (or not) the ORES augmented feature output in liftwing - https://phabricator.wikimedia.org/T301766 (10ACraze) The feature names are stored in the model instance and then we get the feature values back from the extractor, so we will need to combine them together bef... [16:35:23] elukey: i was just looking at https://phabricator.wikimedia.org/T272917 to see what else is needed for mvp [16:35:49] i started on the 4xx http response yesterday, should have a cr up later today too [16:36:00] \o/ [16:36:28] ok if we want to keep using that task let's see if anything is missing from its subtasks [16:37:05] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Lift Wing proof of concept - https://phabricator.wikimedia.org/T272917 (10elukey) [16:37:07] 10Lift-Wing, 10Machine-Learning-Team: Support (or not) the ORES augmented feature output in liftwing - https://phabricator.wikimedia.org/T301766 (10elukey) [16:39:49] elukey: i also started messing around with using asyncio on the preprocess call to mwapi, still no luck yet tho [16:40:56] not entirely sure that will give us the speed up we are looking for [16:42:07] not sure either due to how tornado is configured and the heavy models :( [16:45:03] ah accraze, relevant to your deployment-charts changes https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/762790 [16:47:05] ohhh nice! [16:47:38] should we merge that first? [16:48:13] that will cut down on boilerplate for sure [16:48:42] exactly yes, I was wondering what's best [16:48:57] it duplicates a little the kserve-inference code, but it simplifies our final config for revscoring models [16:49:32] accraze: basically like https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/762791/2/helmfile.d/ml-services/revscoring-editquality/values.yaml [16:50:36] yeah that works for me! [16:50:40] +1'd [16:51:01] ok lemme see if the deployment is a no-op [16:54:44] (03PS1) 10Halfak: Removes importlib_resources that was picked up in python 3.7 [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/762868 [16:55:21] (03PS2) 10Halfak: Removes importlib_resources that was picked up in python 3.7 [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/762868 (https://phabricator.wikimedia.org/T300195) [16:57:21] accraze: all good, just deployed (with your change of the new image) [16:57:30] so you can simplify your changes now if you want [16:59:32] mmm I see [16:59:33] File "model-server/model.py", line 24, in preprocess [16:59:33] rev_id = inputs.get("rev_id") [16:59:33] AttributeError: 'list' object has no attribute 'get' [17:00:21] my input.json contains { "rev_id": 132421 } [17:00:25] has anything changed? [17:01:17] hmmm nothing other than us using the `preprocess` method this time for feature extract, previously everything was in predict [17:01:25] lemme see whats going on [17:01:54] (03CR) 10Elukey: [V: 03+2 C: 03+2] Removes importlib_resources that was picked up in python 3.7 [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/762868 (https://phabricator.wikimedia.org/T300195) (owner: 10Halfak) [17:05:03] (03PS1) 10Elukey: Bump the submodule wheel to include the latest changes [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/762872 (https://phabricator.wikimedia.org/T300195) [17:07:23] elukey: i see what the issue is.... we were calling preprocess directly from `predict` method when it is already called from the `KFModel.__call__` already [17:07:30] will fix today [17:07:53] https://github.com/kserve/kserve/blob/release-0.7/python/kserve/kserve/kfmodel.py#L52 [17:09:11] or maybe we just put everything back in predict method for now, might be easiest [17:09:33] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) I'm not sure this change alone will... [17:13:34] accraze: ack np! [17:14:10] (03CR) 10Elukey: [V: 03+2 C: 03+2] Bump the submodule wheel to include the latest changes [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/762872 (https://phabricator.wikimedia.org/T300195) (owner: 10Elukey) [17:17:50] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10elukey) ` Processing ./submodules/wheels/typ... [17:26:22] * elukey afk! [17:26:29] have a nice day folks :) [17:26:40] seeya elukey! [17:28:23] night elukey! [18:39:14] (03PS1) 10Halfak: Removes old unused packages. [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/762894 (https://phabricator.wikimedia.org/T300195) [18:39:34] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) I'm trying to work through these now... [18:40:22] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) I was able to start the server and l... [18:43:55] (03PS1) 10Halfak: Updates frozen-requirements after wheel cleanup. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/762901 (https://phabricator.wikimedia.org/T300195) [18:48:50] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) I included an update to the `frozen-... [19:22:53] (03PS1) 10Accraze: editquality: handle http bad request [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/762933 (https://phabricator.wikimedia.org/T300270) [19:37:34] (03PS1) 10Accraze: editquality: fix double preprocess call [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/762937 (https://phabricator.wikimedia.org/T301412) [20:32:33] (03CR) 10Elukey: [C: 03+1] "LGTM, left a comment related to strings and constants, but I'll leave you and Kevin decide what's best." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/762937 (https://phabricator.wikimedia.org/T301412) (owner: 10Accraze)