[08:35:17] :-p [11:02:36] hi folks, I have a request for a brainbounce for anybody interested :) So my team is writing a little python/fast-api app to run on k8s that will help us deprecating ORES. The app is very simple - it shows the same interface as ores.wikimedia.org but behind the scenes it calls Lift Wing. [11:03:10] The app will be deployed on Lift Wing itself, but we think it is best to avoid the Istio/Knative/etc.. complexity [11:03:34] and use what ServiceOps provides (so basically we'll need a sidecar with istio anyway to contact inference.wikimedia.orgetc..) [11:04:12] the idea is to keep "the istio proxy mesh" only for kserve, since it is tightly coupled with it, and leave the rest with the serviceops standards [11:04:55] we'll have two things to maintain but I feel that it would be worse to try to standardize more Istio as mesh (as opposed than a simple gateway service) [11:05:01] does it make sense? Lemme know :) [11:07:21] elukey: do you think you could build on the generic chart scaffolding for that? That is maintained by serviceops anyways so it might not even add real overhead to you maintenance wise [11:09:20] jayme: yes yes this is the idea, the only thing that we'd need is to have a localhost:port combination for inference.w.o in the sidecar [11:11:22] elukey: yeah. But that's really just about adding a listener to https://gerrit.wikimedia.org/g/operations/puppet/+/refs/heads/production/hieradata/common/profile/services_proxy/envoy.yaml then [11:12:34] jayme: ok perfect! I'll do it :) [11:22:34] sounds like a valid path to me. I am a little bit shy on the details, but I don't think it will harm to go down that path. It's well trodden one. And apparently kserve might not be yet a good fit for that service ? [11:23:51] akosiaris: kserve comes with a lot of constraints about how to write python code, it assumes that you have a preprocess/predict function(s) to implement, a model binary, etc.. so well tailored for model-servers use cases [11:24:33] the task with all the details is https://phabricator.wikimedia.org/T330414 [11:25:05] damn, I need to catch up on Kserve soon. [11:25:07] I had no idea [11:27:30] yeah I know, a ton of complexity for a very narrow use case :( [11:31:25] akosiaris: an alternative road could be to use "only" knative/istio-mesh without kserve, to have autoscaling/tls-proxy/etc.. [11:31:39] but it comes with a lot of more configurations etc.. [11:31:59] not sure if it is worth it, especially when we have a good common standard (this is why I wanted to discuss the use case) [13:53:05] FYI, I synced up with Olja regarding the trainings. After the last ping, enough people have expressed interest that we probably don't need to open it up to sre-at-large