[15:59:33] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10articlequality-modeling: Build article quality model for Dutch Wikipedia - https://phabricator.wikimedia.org/T223782 (10Halfak) Sorry for the late response. The holiday weekend in the US (memorial day) had me out of my usual flow. For... [16:14:36] morning [16:14:59] was just thinking we should eventually setup jenkins-bot for the inference-services repo [16:15:16] at the very least it could verify and merge on +2 [16:29:28] accraze: o/ I still didn't check the inference-services repo, but I suspect that we may need to deploy our services via helm eventually [16:29:36] so via something like deployment-charts [16:29:46] aha! that is good to know [16:30:49] how are we deploying right now? kubectl apply -f ? [16:30:54] i did some more reading on blubber/deployment-pipeline/docker-pkg/etc. yesterday and was wondering how it will all fit in together with our inference-services [16:31:12] elukey: yep that's correct just kubectl apply -f .... [16:31:32] there is a repo called "deployment-charts" where I am working on to add the istio mess ehm the istio beauty\ [16:31:47] lolol [16:31:56] and it uses helm to deploy services from the deployment host to a cluster [16:32:27] in theory when we set up istio etc.. then we should be able to leverage that repo for our services [16:32:36] ah ok so we'll likely need to make a chart for each service then [16:32:39] so you and Kevin will be able to deploy as well if needed [16:32:53] yes something like that, we'll need to figure out a solution [16:33:07] (we are deviating a bit from the standard service deploy) [16:33:18] haha yeah that doesn't sound too bad [16:33:51] the more we keep close to SRE the more we can reuse etc.. [16:34:16] now I am wondering how kfserving fits into this [16:34:38] I'll check the inference service repo asap to get a better idea, but for the moment I think it is fine to keep working on it [16:35:14] I mentioned this to allow you to avoid spending a ton of time on CI if we discard the repo eventually [16:36:30] haha yeah that's been on my mind a bit, have been holding off on stuff for the repo due to everything in flux right now [16:40:50] but yeah i agree, we should keep close to the SRE processes as much as possible [16:41:09] the helm deploy experience is very nice once it is set up [16:58:46] accraze: I checked the Docker files in the repo (quickly, will do a more in depth analysis) and the only thing that concerns me is the pip install part, namely how to translate it into docker-pkg [16:59:19] as for the base image, if possible I'd use https://docker-registry.wikimedia.org's bullseye not ubuntu [16:59:28] (or even bullseye if needed) [17:00:09] less surprises when we'll have to docker-pkg them [17:01:50] ahh good point, i'll give that a try later today [17:05:42] accraze: ah nice I see that pip can be used inside docker-pkg images, but it uses the one provided by debian [17:06:12] so overall I think that translating what you and Kevin are doing to docker-pkg for the internal registry should be relatively easy [17:06:18] nice, yeah that should be fine (as long as it's pip3) [17:06:41] I am not sure about the the model.py part, but we'll figure it out [17:07:10] yeah im sure there's a way to copy that file into the image [17:07:44] yes yes there is but not sure where it is best to put model.py, since we'll likely want it in a separate repo etc.. [17:08:00] but these are details [17:08:08] ok to proceed with the testing, all good :) [21:04:13] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Production images for ORES/revscoring models - https://phabricator.wikimedia.org/T279004 (10ACraze) I was talking with @elukey today and he mentioned that we should begin using base images from the WMF docker registry where we can. This means, the production... [21:23:36] 10Lift-Wing, 10artificial-intelligence, 10revscoring, 10Machine-Learning-Team (Active Tasks): Create generic revscoring inference service - https://phabricator.wikimedia.org/T283526 (10ACraze) @kevinbazira I've been thinking about how to structure our repo with the generic revscoring image and all the serv... [23:03:56] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10articlequality-modeling: Build article quality model for Dutch Wikipedia - https://phabricator.wikimedia.org/T223782 (10RonnieV) Halfak, thanks for you elaborated answer. I will dive into the log after a good night of sleep. A quick react... [23:13:29] oof trying to swap our base image for the revscoring inference service from ubuntu to bullseye from the wmf docker registry [23:15:13] scipy seems to still have issues supporting python3.9, which is the only available python3 pkg for bullseye at the moment [23:16:11] going to try to manually downgrade to python3.8.5 using ppa or something