[06:29:17] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 9th round of wikis - https://phabricator.wikimedia.org/T308134 (10kevinbazira) [06:45:55] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 9th round of wikis - https://phabricator.wikimedia.org/T308134 (10kevinbazira) Model evaluation has been completed and below are the backtesting results: | | Precision@0.5 | Recall@0.5 |hawiki | 0.68 | 0.46 |hakwiki... [07:36:24] hello folks! [07:49:21] https://github.com/kserve/kserve/releases/tag/v0.10.0 [07:49:22] lol [07:51:22] there are some "security" patches but it seems all not super urgent [07:51:25] at least afaics [08:45:35] 10Machine-Learning-Team: Automate the procedure to bootstrap minikube on the ML-Sandbox and to share it by multiple users - https://phabricator.wikimedia.org/T305447 (10elukey) One thing that we discussed recently was to have a single VM for each user under the machine learning cloud vps tag. In this way every u... [08:52:36] Hey! And I was curious when 0.10 would come out but there was no specific timeline [08:53:16] 10Machine-Learning-Team: Define SLI/SLO for Lift Wing - https://phabricator.wikimedia.org/T327620 (10elukey) [08:53:41] o/ [08:53:52] opened a specific task to work on SLI/SLOs --^ [08:54:41] need to change quickly the tires of my car (it is snowing today and I haven't done it yet :D) so I hope to be back in 30/40 mins! [08:58:56] (03PS2) 10Ilias Sarantopoulos: Deployment script examples [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/881899 [09:58:34] isaranto: IIRC when I tested the changes, I got the "Expected 96 from C header, got 80 from PyObject" when bumping scipy [09:59:02] in this case it seems coming from gensim, but there may be a tie [10:03:46] this is the error I get. I remember running into it again during the upgrade but dont recall how I resolved it. I'm working on it , however I read that it may be resolved by upgrading numpy [10:05:16] I'd try to downgrade scipy for a quick test, maybe I misremember but it may work [10:23:26] ah snap revscoring 2.11.9 depends on scikit-learn>=1.0.2 [11:15:42] isaranto: I am ready to deploy the changeprop's staging change to test goodfaith in ml-staging. Planning to do it this afternoon, is it ok? [11:15:50] the pods in there look good afaics [11:16:10] (worst case we disable changeprop's stream) [11:16:31] ok! [11:16:48] there is the issue with some drafttopic pods I am working on [11:16:55] python dependency hell :D [11:24:48] regarding scipy we also have this from dependabot https://github.com/wikimedia/revscoring/pull/539 [11:25:42] if we use dependabot pull requests we'll need to also bump the versions. I'll check if it is easy to add it automatically [11:27:25] I think that it will be easier to upgrade python along with kserver as the numpy restrictions are lifted [11:31:47] isaranto: I am a bit scared of those dependabot pull requests, they seem a little bit too aggressive for what we want (namely bump deps when needed etc..) [11:31:55] but I can be convinced otherwise [11:32:11] anyway, going afk for lunch, lemme know later on if you need help in testing/debuggin! [11:32:17] lemme check cause we want only the secutiry ones [11:32:23] (and thanks a lot for the work!) [11:32:24] *security ones [11:35:53] I'm cool thanks! enjoy lunch! [12:02:43] (03CR) 10Kevin Bazira: [V: 03+1] "LGTM" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/881899 (owner: 10Ilias Sarantopoulos) [12:10:54] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 9th round of wikis - https://phabricator.wikimedia.org/T308134 (10kevinbazira) The conclusion on the backtesting results is that most of the languages look fine besides: - hywwiki has a low precision (0.63) and recal... [14:13:06] (03CR) 10Elukey: [C: 04-1] Deployment script examples (034 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/881899 (owner: 10Ilias Sarantopoulos) [14:16:03] (03CR) 10Elukey: [C: 04-1] Deployment script examples (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/881899 (owner: 10Ilias Sarantopoulos) [14:18:07] 10Machine-Learning-Team: Upgrade ml clusters to kserve 0.9 - https://phabricator.wikimedia.org/T325528 (10isarantopoulos) There is an issue/blocker on upgrading the python kserve package to 0.9.0 that has to do with its dependencies. Let me explain the chain of dependencies: - `kserve==0.9.0` depends on `kub... [14:20:50] 10Machine-Learning-Team: Upgrade ml clusters to kserve 0.9 - https://phabricator.wikimedia.org/T325528 (10elukey) @isarantopoulos could you please send a pull request for yamlconf? We know the owner and they were quick in the past to release new versions :) [14:23:39] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10Research: Establish process for periodically refreshing link recommendation models - https://phabricator.wikimedia.org/T327212 (10kostajh) @kevinbazira I believe this is something for #machine-learning-team to decide on (cc @KStoller-WMF), perhaps in co... [14:29:24] 10Machine-Learning-Team: Upgrade ml clusters to kserve 0.9 - https://phabricator.wikimedia.org/T325528 (10isarantopoulos) Sure! Just opened a PR [[ https://github.com/halfak/yamlconf/pull/7 | https://github.com/halfak/yamlconf/pull/7 ]] [14:38:20] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team: Fix Armenian sentence tokenization bug in the link recommendation algorithm - https://phabricator.wikimedia.org/T327371 (10kostajh) @kevinbazira I see this is "Watching" on your team's board; is this something that @MGerlach and Research might work on? [14:54:29] (03CR) 10Ilias Sarantopoulos: Deployment script examples (034 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/881899 (owner: 10Ilias Sarantopoulos) [14:54:51] (03PS3) 10Ilias Sarantopoulos: Deployment script examples [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/881899 [14:58:35] (03CR) 10Elukey: Deployment script examples (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/881899 (owner: 10Ilias Sarantopoulos) [14:59:49] (03PS1) 10Ilias Sarantopoulos: ci: add pre-commit checks in all images [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/882658 (https://phabricator.wikimedia.org/T325198) [15:01:00] (03PS2) 10Ilias Sarantopoulos: ci: add pre-commit checks in all images [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/882658 (https://phabricator.wikimedia.org/T325198) [15:05:56] (03CR) 10CI reject: [V: 04-1] ci: add pre-commit checks in all images [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/882658 (https://phabricator.wikimedia.org/T325198) (owner: 10Ilias Sarantopoulos) [15:09:25] (03PS3) 10Ilias Sarantopoulos: ci: add pre-commit checks in all images [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/882658 (https://phabricator.wikimedia.org/T325198) [16:12:01] (03PS8) 10Ilias Sarantopoulos: ci: add pre-commit checks in all images [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/882658 (https://phabricator.wikimedia.org/T325198) [16:13:38] ok so the new config for changeprop is in staging, fixed some errors aaand.. no message registered on liftwing staging [16:13:41] sigh [16:16:09] * isaranto pats Luca on the back [16:22:00] 10Machine-Learning-Team, 10Patch-For-Review: Create a pre-commit hook for inference-services repo - https://phabricator.wikimedia.org/T325198 (10isarantopoulos) Added these hooks to all the images hosted in the inference-services repo. If one wants to install the pre-commit hooks in order to run these locally... [16:25:10] (03CR) 10Elukey: [C: 03+1] "Nice work!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/882658 (https://phabricator.wikimedia.org/T325198) (owner: 10Ilias Sarantopoulos) [16:28:59] Morning all! [16:29:03] 10Machine-Learning-Team, 10Patch-For-Review: Upgrade python from 3.7 to 3.9 in docker images - https://phabricator.wikimedia.org/T325657 (10isarantopoulos) The [[ https://github.com/halfak/yamlconf/pull/7 | PR ]] has been merged and yamlconf has been updated [16:29:18] hey Chris! [16:29:40] just as I was heading out.. [16:30:11] I opened the last patch for the pre-commit hooks for all the images [16:30:25] going to continue with the python upgrade tomorrow [16:30:27] cu folks! [16:30:38] o/ [16:41:42] Night! [16:41:56] Yeah I'm late today because my kids don't have school so its chaos [16:43:17] (03PS1) 10Ilias Sarantopoulos: (WIP) - feat: revscoring kserve upgrade to 0.9 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/882689 (https://phabricator.wikimedia.org/T325528) [16:43:57] still here 😄 [16:44:17] good kind of chaos though right 🤗 [16:45:37] elukey: I opened a draft patch as WIP where I am testing kserve 0.9.0. Sorry for jumping on your task but it seems that unless numpy is upgraded some models won't play with python 3.9 (and numpy can only be updated if we update kserve) [16:45:46] more tomorrow! [16:46:13] please jump anytime, glad that we can do both together :) [16:46:58] does that remove the weird 80-bytes error that we were discussing this morning? [16:47:31] we'll see. didn't have time to check yet. building a model as we speak [16:47:43] but I got to go in a bit [16:48:23] super thanks! [16:49:37] are GCP IPs banned from this channel? [16:49:58] derenrich: hi! What do you mean with GCP? [16:50:06] we didn't ban anything specific IIRC [16:50:24] elukey: i can't seem to join this channel using my normal jumpbox which is hosted on google cloud [16:51:28] anyways i'll just ask what i was coming here to ask: does anyone know of regularly updated trained embeddings for wikipedia/wikidata items? is this something the Wikimedia ML team owns? [16:53:41] derenrich: I'd ask to the Research team! [16:53:53] elukey: where do they live? [16:54:35] #wikimedia-research [16:54:38] thanks [17:23:17] (03PS2) 10Ilias Sarantopoulos: (WIP) - feat: revscoring kserve upgrade to 0.10 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/882689 (https://phabricator.wikimedia.org/T325528) [17:23:26] still here again [17:23:27] hehe [17:23:52] 0.10?? :D [17:24:01] elukey: the error went away! I had to upgrade kserve to 0.10.0 that allows numpy above 1.21... [17:24:45] * elukey cries in a corner [17:24:51] yeah let's discuss it tomorrow. It is the only way I could make it work..I think I've built the image so many times today.. [17:25:20] yes yes we can bump the control plane to 0.10 on a later stage in theory, and/or set 0.10 only for revscoring [17:38:38] 10Machine-Learning-Team: Investigate if the mediawiki.revision-score stream can be broken down into multiple ones with ChangeProp - https://phabricator.wikimedia.org/T327302 (10elukey) After getting some help from Hugh (thanks!) I was able to have some rules processed by Changeprop in staging: https://grafana.w... [17:42:21] logging off! o/