[15:14:20] morning ll [15:14:22] all [15:29:06] o/ [15:41:41] I rearranged phab. Either I am a genius or a monster [16:24:11] I am trying to make everything work without cert-manager, not working so far sigh [16:38:12] ooophf [16:42:40] elukey: is it due to the kfserving controller? [16:44:21] accraze: o/ yep there is a failure in validating the certificate when the webhooks are used [16:44:42] I am using the self-signed.sh script [16:44:54] ahh i see [16:45:35] it says [16:45:36] Error from server: error when creating "service.yaml": conversion webhook for serving.kubeflow.org/v1alpha2, Kind=InferenceService failed: Post "https://kfserving-webhook-server-service.kfserving-system.svc:443/convert?timeout=30s": x509: certificate signed by unknown authority [16:46:02] I checked the webhooks configs to see if they had a stale CA etc.. [16:46:08] but it looks the one that the script creates [16:46:56] there was also a previous problem about SANs not added, but I fixed the script (it was missing a little argument0 [16:55:30] my guess it it is still looking for cert-manager due to an annotation on some resource somewhere [16:58:22] yes there are some cert-manager-related CRDs [16:58:51] (in the kfserving.yaml config) [17:00:01] This is not deep learning work promised to me by all the industry hype. Where are the robots and shit. [17:57:35] accraze: so I tried to inspect the mutating/validating webhook configs in my minikube, they mention cert-manager but it seems only an annotation [17:58:16] but the theory is good - I suspect that the kfserving manager uses the self signed cert, but the k8s api don't use the ca.crt correctly for some reason [17:58:20] and fails to validate it [17:58:54] yeah there's got to be something relying on cert-manager further into the stack [18:01:13] or there is a resource that is using something like `apiVersion: cert-manager.io/v1alpha2` somewhere [18:03:21] in theory what self-signed.sh does it to override those thigns [18:03:25] *things [18:06:34] ah yeah looking at the script now, i see what you mean [18:24:04] * elukey dinner, will keep battling tomorrow with kfserving :D [22:02:49] hmmm seems like our revscoring inference service has difficulties running the articlequality models due to python versions [22:03:22] *generic revscoring image [22:04:18] we might need to use multi-stage builds depending on whether the inference service will be of type editquality, articlequality, drafttopic etc.. [22:37:03] makes sense