[15:55:42] o/ [15:58:42] o/ [16:08:21] working on upgrading articlequality images to use kservev0.7.0 today [16:09:32] there's a weird edge case with the model-server related to running py3.7, some deps (like dataclasses) are not supported or have moved to built-in modules [16:09:56] the other model-servers should be a bit more straight-forward though [16:14:46] let me know if you need help/brainbounce/reviews/etc.. [16:18:27] will do! [16:19:01] elukey: what all did you do to get ml-sandbox cluster to run, did you just copy over .kube and .minikube dir to your home dir? [16:19:22] trying to help kevinbazira debug it lol [16:21:07] we did something similar but knative and kserve configs seem to be missing [16:22:26] so the last times I took a shortcut and did sudo -u accraze blabla [16:22:30] that worked nicely :D [16:22:40] LOL [16:22:49] when I copied the files I had to replace your username in some of the configs [16:22:58] there must be a better way to share minikube [16:23:47] I did replace the username but still no luck [16:24:21] what errors do you get ? [16:24:31] I can try to connect and sudo as you [16:24:55] none of the kserve stack is installed after copying [16:26:24] I see the same thing with both users if I sudo [16:26:24] in theory we should be able to move the .minikube dir to root user home dir to share, although we'd need to run minikube on a hypervisor (kvm, virtualbox) instead docker [16:27:03] did you need to run minikube start? [16:27:23] nope, I sudoed as you [16:27:28] and kevin [16:27:29] ahhhh [16:27:36] what pods do you see? [16:27:50] i think kevinbazira might have created an additional cluster ...? [16:28:34] if I run minikube ssh I see only the pods listed, no kserve [16:28:37] kevinbazira@ml-sandbox:~$ kubectl apply -f enwiki-goodfaith-service.yaml [16:28:37] error: unable to recognize "enwiki-goodfaith-service.yaml": no matches for kind "InferenceService" in version "serving.kserve.io/v1beta1" [16:28:45] The error I get ^^^ [16:28:59] yeah it makes sense, there seems to be no istio/knative/kserve installed [16:29:34] oh shoot yeah lol i just checked my pods and no kserve installed [16:30:06] i think we may have overwritten something somewhere during the debugging/copying [16:30:14] yeah I see [16:30:15] 2021-11-30 12:29:45 minikube start --kubernetes-version=v1.16.15 [16:30:30] no worries though, i have a script that will get us back up [16:30:35] ack :) [16:34:19] actually i just gonna set this to run on kvm and try to store .minikube in the root user home this time so we can share [17:28:28] going afk! will read later to see if there are questions for minikube etc.. :) [17:28:46] cool, see ya later elukey! [18:16:51] ok ml-sandbox is back up, i ran into issues w/ kvm so went with docker again, going to look into sharing the .minikube dir now [18:56:03] ok minikube is running as root and i've got enwiki-goodfaith and enwiki-articlequality running on namespace kserve-test [18:56:36] feels good to have all our custom wmf kserve stack running in dev now ;) [18:57:22] now to move forward with the kserve v0.7.0 upgrade for our images [20:46:40] \o/ [21:09:52] i've been trying to get the new articlequality transformer running on ml-sandbox, for some reason the isvc SERVICE_HOSTNAME does not resolve, but when we use the explicit pod name as SERVICE_HOSTNAME (i.e. enwiki-goodfaith-predictor-default.kserve-test.example.com) it works [21:10:54] when i curl the isvc url as SERVICE_HOSTNAME -- I get a 503 Serivce Unavailable [21:12:05] so we can technically hit our older model-servers, but we can't hit the full inference service like `request->transformer->predictor->result` [21:12:50] i'm guessing this is due to our cluster-local-gateway config somewhere [22:40:48] yeah this is so weird, i can hit pods just fine but it's like the top level isvc abstraction isn't working