[06:16:48] 10ORES, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: revscoring feature extraction error for wikitext papes in Wikidata - https://phabricator.wikimedia.org/T302851 (10elukey) Yep sorry forgot a few details, nice :) Before finishing let's expand https://wikitech.wikimedia.org/wiki/ORES/Deployme... [07:11:25] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Set up the ml-cache clusters - https://phabricator.wikimedia.org/T302232 (10elukey) To keep archives happy - I am having a chat with Eric over email about this cluster and its future usage. The AQS cassandra cluster should bec... [07:38:43] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Load test the Lift Wing cluster - https://phabricator.wikimedia.org/T296173 (10elukey) @achou I am curious about the running processes inside the pod after we use kserve - what do you see if you run `ps -aux | grep python` and `ps -eLf | grep python`? My und... [08:58:19] (03PS1) 10Elukey: editquality: upgrade revscoring to 2.11.4 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/799275 (https://phabricator.wikimedia.org/T309102) [08:59:07] no idea if I am missing something but for editquality it seems relatively easy to migrated to revscoring 2.11.4 [08:59:28] let's see what CI says [08:59:34] Trying not to hope too much, lest Mr. Murphy hears it ;) [08:59:59] pip was happy about it, hopefully no big surprises :) [09:11:42] mmm CI doesn't really say anything [09:11:49] not sure if it is a loong job or not [09:11:56] jenkins doesn't show much for our jobs [09:16:34] ah I see https://integration.wikimedia.org/ci/job/trigger-inference-services-pipeline-editquality/118/ [09:16:37] super long [09:21:59] ah nice CI said +2 [09:22:33] added the team for review :) [09:22:38] I have an errand to make, ttl! [10:07:36] (03CR) 10Kevin Bazira: [C: 03+2] "LGTM" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/799275 (https://phabricator.wikimedia.org/T309102) (owner: 10Elukey) [10:09:56] (03CR) 10Klausman: [C: 03+1] editquality: upgrade revscoring to 2.11.4 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/799275 (https://phabricator.wikimedia.org/T309102) (owner: 10Elukey) [10:16:08] (03Merged) 10jenkins-bot: editquality: upgrade revscoring to 2.11.4 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/799275 (https://phabricator.wikimedia.org/T309102) (owner: 10Elukey) [12:50:21] thanks a lot for the merge and the reviews Kevin/Tobias :) [12:50:41] I filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/799328 to apply the new image to enwiki-goodfaith, will check later to see if the CI diff is consistent [12:50:47] going to the vet in a bit :) [13:15:23] <- lunch (yeah, I know, late) [13:47:06] revscoring 2.11.4 deployed for enwiki-goodfaith, all working! [13:47:09] \o/ [14:00:20] ah TIL https://pypi.org/project/articlequality/ [14:00:29] this depends on revscoring 2.8.x [14:07:14] ah ok so we pinned an old git sha id [14:07:28] version 4.2 should work, but it is a jump ahead of a lot of commit [14:07:30] *commits [14:15:58] ah no so 0.4.2, last release, was in 2020 - https://github.com/wikimedia/articlequality/commit/55c6ed60135af9d4d3629d9adcef0779db2fdf43 [14:16:01] sigh [14:16:14] so we should probably release 0.4.3 to get the revscoring compatibility out [14:16:29] there is already a commit for the bump in revscoring, but we should verify the new commits] [14:18:23] 10Machine-Learning-Team: Release articlequality 0.4.3 to Pypi - https://phabricator.wikimedia.org/T309205 (10elukey) [14:19:10] 10ORES, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: revscoring feature extraction error for wikitext papes in Wikidata - https://phabricator.wikimedia.org/T302851 (10elukey) [14:19:12] 10Machine-Learning-Team: Release articlequality 0.4.3 to Pypi - https://phabricator.wikimedia.org/T309205 (10elukey) [14:20:24] 10Machine-Learning-Team: Release articlequality 0.4.3 to Pypi - https://phabricator.wikimedia.org/T309205 (10elukey) [14:20:31] 10ORES, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: revscoring feature extraction error for wikitext papes in Wikidata - https://phabricator.wikimedia.org/T302851 (10elukey) [14:29:41] elukey: https://docs.python.org/3.7/library/asyncio-eventloop.html#asyncio.loop.run_in_executor [14:30:04] elukey: I think we can use this to run preprocess async and no need to change mwapi [14:30:17] a concurrent.futures.ThreadPoolExecutor is set with max_asyncio_workers when kserve starts, so we can use that in preprocess [14:30:34] https://github.com/kserve/kserve/commit/c10e6271897d7fd058f5618d5e0e70b31496f64c#diff-bca095090a28f3f36b6612d4d819794995d10962b69b8907b2fdf86df4fe5bbdR102 [14:35:41] morning all [14:36:43] so it would be like loop = asyncio.get_running_loop() [14:36:47] inputs[self.FEATURE_VAL_KEY] = await loop.run_in_executor(None, self._fetch_articlequality_text, rev_id) [14:37:10] o/ morning Chris! :) [14:56:23] aiko: o/ yeah I think something along those lines, but I was wondering if we could use https://github.com/kserve/kserve/blob/release-0.7/python/kserve/kserve/kfmodel.py#L67 too [14:56:43] I mean IIRC we extend KFModel right? [14:57:30] predict uses it like https://github.com/kserve/kserve/blob/release-0.7/python/kserve/kserve/kfmodel.py#L149 [14:58:00] so maybe if preprocess was turned "async" we could swap the mwapi code with that bit? [14:58:26] it would, in theory, cause the preprocess to be ran into the tornado IO loop [14:58:33] meanwhile with Ray workers we could run the model code [14:58:40] and have thing separated [14:58:45] (not sure if it makes sense or not) [14:59:01] in the future we'd keep the same structure for transformers as well [14:59:04] all asybc [14:59:06] *async [15:58:50] logging off for today folks! [15:58:55] have a nice evening :) [15:59:08] (evening/afternoon!) [15:59:16] you too Luca! :) [17:13:17] 10Machine-Learning-Team: Release articlequality 0.4.3 to Pypi - https://phabricator.wikimedia.org/T309205 (10achou) a:03achou [19:51:26] I. DECLARE. SOFTWARE. CAPITALIZATION. BANKRUPTCY.