[10:17:02] aaaahhhhhh I found the probleeeemmmmm [10:17:31] so there is another webhook that is not patched by the self-signed-ca.sh script, namely a conversion webhook [10:17:48] from v1alphaX to v1beta1, since I am testing kfserving 0.5.1 [10:18:29] so it wasn't patched with the custom CA, and it failed to create the example service since I was using v1alpha2 [10:18:53] I switched to the beta api (with some adjustments) and it worked [10:21:57] Revision "enwiki-goodfaith-predictor-default-4sbmf" failed with message: Unable to fetch image "accraze/kfserving-custom-model": failed to resolve image to digest: Get "https://index.docker.io/v2/": x509: certificate signed by unknown authority. [10:22:02] * elukey cries in a corner [10:41:55] * elukey lunch [12:08:27] opened https://github.com/kubeflow/kfserving/issues/1657 [12:19:53] the last issue is then how to trust docker.io, this is probably due to the custom CA certs [15:00:50] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10articlequality-modeling: Build article quality model for Dutch Wikipedia - https://phabricator.wikimedia.org/T223782 (10Halfak) The output of https://ores.wikimedia.org/v3/scores/nlwiki/123125/articlequality is pure JSON and links are not... [15:13:59] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10articlequality-modeling: Build article quality model for Dutch Wikipedia - https://phabricator.wikimedia.org/T223782 (10RonnieV) Hi @Halfak, Would it then be possible to show the link, like https://nl.wikipedia.org/w/index.php?oldid=12312... [15:15:14] I made it work using kubectl -n knative-serving edit configmap config-deployment and adding registriesSkippingTagResolving: index.docker.io but it is not really a fix [15:15:26] but the issue seems to be in knative, not kfserving [15:15:38] so probably not related to the self-signed-ca.sh [15:17:07] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10articlequality-modeling: Build article quality model for Dutch Wikipedia - https://phabricator.wikimedia.org/T223782 (10Halfak) I see. You're asking to include the "weighted sum" measure in the JSON output? [15:18:52] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10articlequality-modeling: Build article quality model for Dutch Wikipedia - https://phabricator.wikimedia.org/T223782 (10RonnieV) Yes, that is the right name of the value. [15:38:52] made it work again without cert-manager! [15:39:08] * elukey dances [15:39:11] coffee [15:53:54] elukey: that is great news! [16:04:48] elukey++ [16:20:10] <3 [16:20:13] accraze: morningggg [16:20:56] did you see my comments about the v1beta1 vs v1alpha2 api version for the InferenceService ? [16:21:05] I am totally lost, should we prefer beta ? [16:21:39] I recall that you mentioned something like that in the past but my limited brain works with a simple LRU and I may have dropped it :D [16:28:11] haha I like the LRU analogy [16:28:36] yeah I'm starting to think we should prefer beta, there are plans to deprecate alpha in the next release [16:29:04] that being said, all our current inference services are v1alpha2 [16:29:58] I made this task so we don't forget to upgrade them: https://phabricator.wikimedia.org/T284115 [16:31:29] i marked it as low priority since alpha services get converted to beta automatically via webhook in KF1.3, but maybe we should do this sooner if it affects TLS? [16:33:10] accraze: I mentioned the problem in the github issue so I think that they'll update the script to patch the webhook that converts the api, but yes if possible I'd use somethin called beta rather than alpha :D [16:39:54] yeah that's a good point [16:43:28] it looks like the cluster kevin is working on is running kfserving v0.3, which does not have the v1beta1 api yet. [16:43:55] i'll see about upgrading that and then will rewrite the inference service specs to use v1beta1 [16:44:12] ah nice yes! [16:44:19] I am using 0.5 in minikube [16:47:51] ok cool that's what i'm on as well, will target 0.5 [16:53:08] I am wondering what is a possible upgrade path [16:53:28] just kubectl apply new-giant-yaml-and-pray-for-the-best.yaml ? [17:02:19] lol yeah i think so [17:10:01] hopefully helm will help [17:30:59] so I am checking the kfserving.yaml file [17:31:21] and a gigantic part of those 16k lines seems to be the api conversion webhook [17:31:24] lol [17:34:14] anyway, logging off, ttl! [17:38:01] lol that would make sense! [17:38:05] have a good one elukey [17:49:35] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Move inference services to v1beta1 api - https://phabricator.wikimedia.org/T284115 (10ACraze) p:05Low→03Medium a:03ACraze After talking with @elukey, it seems the conversion webhook is not patched by the self-signed-ca.sh (see: T280661) script in KFSer... [23:23:16] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Move inference services to v1beta1 api - https://phabricator.wikimedia.org/T284115 (10ACraze) [23:26:56] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Move inference services to v1beta1 api - https://phabricator.wikimedia.org/T284115 (10ACraze) [23:26:58] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Lift Wing proof of concept - https://phabricator.wikimedia.org/T272917 (10ACraze) [23:40:39] 10Lift-Wing, 10artificial-intelligence, 10revscoring, 10Machine-Learning-Team (Active Tasks): Create generic revscoring inference service - https://phabricator.wikimedia.org/T283526 (10ACraze) @kevinbazira Just adding notes here after digging into all the different model classes (articlequality, drafttopic... [23:41:37] hmmm it seems upgrading kfserving inside of the minikf distribution is a bit more challenging than applying the new yaml file [23:42:55] it might be easier to just upgrade that sandbox from kfv1.1 to kfv1.3, will look a bit more into it tomorrow