[07:59:52] hello folks [08:00:32] from https://github.com/kserve/kserve/issues/2292#event-6922695273 in theory knative 1.0 is not strictly required for kserve 0,8 [08:00:36] but of course suggested :) [08:00:50] they said that we can test it and see, maybe we could do it on staging first [08:07:30] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Send score to eventgate when requested - https://phabricator.wikimedia.org/T301878 (10elukey) >>! In T301878#8042200, @Ottomata wrote: > Looks good. My only worry is that these making these new streams now, and planning to re... [08:10:46] I am going to open a task for the kserve's upgrade [08:17:12] 10Machine-Learning-Team: Upgrade ml clusters to kserve 0.8 - https://phabricator.wikimedia.org/T311982 (10elukey) [08:45:30] elukey: It's July 4th, at least the Americans are out :) [08:46:31] klausman: yep I know :) [08:46:42] but in our team it means only Chris :) [09:29:18] That sounds like a good approach, sequestering away the calls, in a manner [09:29:35] that stage would be a black box where we can do whatever we want [09:32:02] I added all to https://phabricator.wikimedia.org/T309623#8048160 [09:32:12] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Test async preprocess on kserve - https://phabricator.wikimedia.org/T309623 (10elukey) I think that I got, more or less, how the ORES feature injection works (https://www.mediawiki.org/wiki/ORES/Feature_injection#Feature_injection:_play... [09:35:26] so if we add the cassandra cache approach + this feature injection one, we should be able to have a good set of performances for the cached and uncached score use case [09:35:47] (and in theory also decide more freely how many items our LRU should keep, TTL, etc..) [09:35:50] does it make sense? [09:44:52] yeah, sounds good. [09:45:29] It would also allow us to instrument things a bit better, probably. I'd especially want to know that cache hit rate of "organic" queries. [09:46:25] And a histogram of queried-revisions age. I.e. what is to 99% (and 90 and 75 and 50) of revision ages. [09:47:17] Or to put it another way: how often do people care about revisions that are older than a week, month, quarter, yesr. [10:05:24] hey Luca, I don't understand that "We call the MW API asynchronously from preprocess, fetching what we need." - you mean we don't use api extractor, we call the mwapi directly? but I think it won’t give us the feature values we want [10:05:56] I think api extractor does something to transform the result from mwapi (e.g. article content, user info) to the list of feature value that models need. We can't get the feature values just from mwapi [10:06:32] Or did I misunderstand your point? [10:08:19] And for your question about if the feature list is consistent for a given model, from I see in https://github.com/wikimedia/editquality/tree/master/editquality/feature_lists [10:08:54] I think the answer is yes for a given model and wiki project [10:13:21] aiko: o/ what I meant was that we can call the mw api asyncronously to fetch the feature values that we need, and then we inject them in the api-extractor [10:13:57] it is fine if we have to go through it, the important bit is that the http calls are made elsewhere [10:14:05] (so they are not blocking) [10:14:12] does it make sense? [10:17:53] or maybe we figure out what features we need to inject in the cache to avoid the http calls from mwapi [10:18:47] if possible [10:20:13] It would also allow us to parallelize http calls [10:21:05] well make them async [10:21:06] (if they can't be, because they all depend on each other, it might also be a point where the MWAPI is re-examined, to see if what we need can be fetched in a single call, or at least fewer calls to reduce RTT) [10:21:28] we also need to limit the scope of changes of revscoring [10:22:03] (and even if that is not possible, we should examine if we're re-using the HTTP connection to avoid doing the whole TWHS every time) [10:22:56] aiko: another alternative could be to add to the api extractor's signature a "http_cache" parameter, that we use to skip or not http calls [10:23:10] it could be very simple, like the feature cache one [10:23:49] so when we get to lines like https://github.com/wikimedia/revscoring/blob/773f9cd8029de7ef5c7713addd2f6661bce94b4e/revscoring/extractors/api/extractor.py#L242 we could simply check in cache or make the http call [10:24:22] it should be a small change in revscoring, by default we'd keep the default behavior [10:24:28] err the current behavior [10:26:37] yeah it should be a small change to https://github.com/wikimedia/revscoring/blob/master/revscoring/extractors/api/extractor.py, I don't see "session.get" used elsewhere [10:26:56] I have to go now but I'll try to write something this evening [10:27:09] (please all comment if it seems a bad idea, I'll read later :) [12:08:44] elukey: my doubt is if we can fetch the feature values that you inject in the cache from calling the mw api. As I understand it, the api extractor calls mw api to fetch raw data like article content, and then transforms this data to feature values [12:09:03] I agree we need to figure out what features we need to inject in the cache. Maybe it could be the raw data like article content and the api extractor will return the feature value we want without calling mwapi. [13:13:54] aiko: yeah I get your point, maybe we should just add something like "http_cache" [13:14:03] and inject raw content [13:14:13] to let the extractor to do the job [13:20:34] Opened https://github.com/wikimedia/ores/pull/362 for the ores repo, just to clear out some Dependabot alerts (no-op) [13:22:45] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Test async preprocess on kserve - https://phabricator.wikimedia.org/T309623 (10elukey) Reporting a summary of what has been discussed over IRC. The extractor calculates most of the above features, so one way forward could be to instruct... [15:30:47] 10Lift-Wing, 10SRE-swift-storage, 10Machine-Learning-Team (Active Tasks): Create Swift account for readonly access to ML models - https://phabricator.wikimedia.org/T311628 (10elukey) The new mlserve:ro account has been added, but if I try to use s3cmd with the new credentials I get an error: ` elukey@stat10... [15:33:37] going afk folks, have a nice rest of the day! [18:28:44] 10Machine-Learning-Team: Spike. Try to ML models distributted in jupyter notebooks with dask - https://phabricator.wikimedia.org/T243089 (10JArguello-WMF)