[06:25:37] Good morning! [07:00:15] (03CR) 10Kevin Bazira: [C: 03+1] "Thank you for adding the message, it's showing up. I have also printed dataframe results as you advised and below is the response:" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993078 (https://phabricator.wikimedia.org/T355394) (owner: 10Ilias Sarantopoulos) [07:05:39] (03CR) 10Ilias Sarantopoulos: [C: 03+2] locust: save separate results file per model [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993078 (https://phabricator.wikimedia.org/T355394) (owner: 10Ilias Sarantopoulos) [07:13:46] (03CR) 10Ilias Sarantopoulos: [V: 03+2 C: 03+2] locust: save separate results file per model [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993078 (https://phabricator.wikimedia.org/T355394) (owner: 10Ilias Sarantopoulos) [07:14:14] (03CR) 10Ilias Sarantopoulos: [V: 03+2 C: 03+2] locust: save separate results file per model (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993078 (https://phabricator.wikimedia.org/T355394) (owner: 10Ilias Sarantopoulos) [07:25:27] (03PS1) 10Ilias Sarantopoulos: fix: upgrade pip in Makefile [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/994883 [07:26:34] klausman: can you check the make command with the following patch https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/994883? [07:27:52] with python 3.9 [07:28:16] we should be checking and adding support for more recent python versions for sure [08:20:26] (03PS4) 10Kevin Bazira: Makefile: add support for article-descriptions [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993481 (https://phabricator.wikimedia.org/T356176) [08:22:19] (03CR) 10Kevin Bazira: Makefile: add support for article-descriptions (033 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993481 (https://phabricator.wikimedia.org/T356176) (owner: 10Kevin Bazira) [08:28:31] (03CR) 10Kevin Bazira: [C: 03+1] fix: upgrade pip in Makefile [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/994883 (owner: 10Ilias Sarantopoulos) [09:08:23] klausman: I've rolled out the latest Java 11 secutity updates on ml-cache, can you please take care of the rolling restart? [09:15:24] will do [09:15:49] isaranto: giving that a spin today [09:22:42] moritzm: is there phab ticket I can mention in the reason field? [09:24:37] no, we don't open Phab tasks for most software updates, but you can e.g. refer to DSA 5604 (https://lists.debian.org/debian-security-announce/2024/msg00011.html) [09:25:02] thanks! [09:44:43] morning! [09:45:19] isaranto: ah, there's the rub: my system Python is 3.11, not 3.9. So I doubt and outdated pip is the problem. But I'll try it [09:45:23] Morning Aiko! [09:48:54] Hey Tobias &Aiko ! [09:50:24] klausman: RRML is not supported via makefile yet. how did you run it? what trouble did you encounter? [09:50:45] I just replaced LA with ML in the Makefile and hoped :D [09:51:03] IIRC, I had to install additional requirements, but I can get the specifics in a pastebin in a moment [09:51:10] (03CR) 10AikoChou: [C: 03+1] fix: upgrade pip in Makefile [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/994883 (owner: 10Ilias Sarantopoulos) [09:53:45] klausman: RRML has different requirements.txt and model url [09:54:12] The Model URL I fixed, but I wasn't sure which req.txt to use/where to find it [09:54:57] (03CR) 10Klausman: [C: 03+1] fix: upgrade pip in Makefile [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/994883 (owner: 10Ilias Sarantopoulos) [09:55:47] ah, revert_risk_model/model_server/multilingualrequirements.txt [09:56:01] klausman: need to change DEP_DIR to multilingual [09:56:04] yess [09:57:23] trying that now. [09:58:16] and of course the kserve version in req.txt doesn't support Python 3.11 %) [09:59:00] Giving 0.11.2 a go. [10:01:24] It works! \o/ [10:04:39] Cool! [10:05:57] moritzm: both eqiad and codfw Cassandras restarted [10:07:03] niceeee!! [10:08:54] ack, thanks [10:32:05] (03CR) 10AikoChou: Makefile: add support for article-descriptions (033 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993481 (https://phabricator.wikimedia.org/T356176) (owner: 10Kevin Bazira) [10:40:27] Ok, that took some rewriting, due to the way the async preproc works. But I now haven an even uglier POC that's returning cached results for RRML in ~1.5ms \o/ [10:41:32] \o/ [10:57:14] https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/995001 If people want to have a look. I don't think an actual code review is needed, since this is just an exploration [11:02:54] But it shows one interesting wrinkle: since preprocess is run asynchronously, we may waste that computation time even if we have a response cache. The trick is to cache the preprocess result as well, and return early. I don't know of any way of "canceling" the preprocess thread from the predict method. [11:24:04] I think we can figure out something or have two separate versions [11:24:36] If the preprocess function doesn't exist it goes to the predict directly [11:27:05] But on the other hand the best thing is to cache both steps [11:28:05] In the future it may be useful as well. Caching the preprocess mimic the behavior of a feature store [11:29:05] Yeah, I think all three approaches (caching only one of the two, or both) can be a good fit depending on the model/preproc latencies. [11:29:42] Currently there is only one nev var for the cache (all or nothing), but naturally, this could be made more fine-grained [11:29:53] anyway, /me lunch now [12:17:24] (03CR) 10Ilias Sarantopoulos: [V: 03+2 C: 03+2] fix: upgrade pip in Makefile [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/994883 (owner: 10Ilias Sarantopoulos) [13:59:20] (03PS1) 10Ilias Sarantopoulos: locust: add article_descriptions load tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/995039 [14:00:03] the above is WIP! please disregard for now [14:18:27] Morning all [14:26:32] Morning Chris! [14:50:09] hi Chris! [15:45:35] klausman: when you have sometime (can be tomorrow) could you deploy this change in admin_ng ns? https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/994117 [15:46:16] Vielen Dank! [15:58:16] Logging off folks, have a nice evening/rest of day [15:59:19] Actually I'll be online but not in front of a PC for the next hour (watching the staff meeting on the go) :) [16:05:33] sure, will do in a sec [16:08:24] isaranto: and done [16:10:36] Super! Well try it later. Thanks! [18:33:44] (03PS1) 10Umherirrender: Add return type hint to functions in ServiceWiring.php [extensions/ORES] - 10https://gerrit.wikimedia.org/r/995075 [19:59:38] (03CR) 10DannyS712: [C: 03+2] Add return type hint to functions in ServiceWiring.php [extensions/ORES] - 10https://gerrit.wikimedia.org/r/995075 (owner: 10Umherirrender) [20:23:20] (03Merged) 10jenkins-bot: Add return type hint to functions in ServiceWiring.php [extensions/ORES] - 10https://gerrit.wikimedia.org/r/995075 (owner: 10Umherirrender)