[03:16:03] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Platform Engineering: IndexPager::buildQueryInfo (contributions page unfiltered) LEFT JOIN ores_classification needs tuning - https://phabricator.wikimedia.org/T284888 (10Ladsgroup) [09:41:36] lovely, for some reason I never hit the following before now [09:41:37] https://github.com/knative/serving/blob/release-0.18/pkg/reconciler/revision/resources/queue.go#L174 [09:42:02] we don't use google's ko to build images, and there is an hardcoded path for the revision code [09:42:22] trying to build a special version of our docker image to see if it works [11:28:40] It would be rather annoying having to patch that in [11:30:10] klausman: worked around it via https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/700179 [11:43:14] * elukey lunch [13:25:43] so now my minikube setup seems to have all pods working, but for some reason the ingress setup seems to be not working [13:25:51] at least this is what I am seeing from the logs [13:25:53] mmmmm [13:26:16] may be another problem related to knative's helm chart, never seen this before [14:45:18] interesting, I used kfserving's upstream yamls on a fresh install, same issue [14:45:21] ingress not configured [14:46:10] and in the past the same stack worked [14:46:28] maybe it is the modified kfserving yaml [15:05:21] nope even that one seems fine [15:56:09] o/ [15:56:37] elukey: is it the internal gateway that you are having issues with? [15:57:16] yeah! [15:57:19] accraze: o/ [15:57:31] I just recalled that you had a similar issue right? [15:57:45] but I checked the istio annotation in service.yaml [15:58:09] I am using the istioctl released by upstream, that may have some differences [15:58:32] (we have now a debian package with istioctl versions to deploy on the deployment hostS) [15:58:48] what does the url you are using look like? [16:00:37] I am not even using an URL since the SR [16:00:44] SERVICE_HOSTNAME comes back empty [16:01:01] and I see ingress errors n kubectl describe inferenceservice enwiki-goodfaith [16:01:11] revisions are up, all pods up [16:01:28] Type: PredictorRouteReady [16:01:28] Last Transition Time: 2021-06-17T15:04:44Z [16:01:28] Message: Ingress has not yet been reconciled. [16:01:28] Reason: IngressNotConfigured [16:01:28] Status: Unknown [16:01:30] Type: Ready [16:01:34] can you share the script you are using to hit the service? [16:01:47] err... wait [16:02:19] SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3) [16:03:00] that might actually be an auth error... do you see a 302 show up anywhere? [16:03:21] nono the above paste is from kubectl [16:03:44] for example [16:03:45] kubectl get ksvc [16:03:45] NAME URL LATESTCREATED LATESTREADY READY REASON [16:03:49] enwiki-goodfaith-predictor-default http://enwiki-goodfaith-predictor-default.default.example.com enwiki-goodfaith-predictor-default-9mx78 enwiki-goodfaith-predictor-default-9mx78 Unknown IngressNotConfigured [16:04:25] ah is this deployed in the default namespace? [16:04:57] it is yes, but it worked in the past [16:05:28] oh interesting i was never able to get it running in the default [16:05:41] it seems a weird istio issue [16:05:53] yeah for sure [16:06:22] going to wipe everything and try with the istioctl built manually [16:06:23] so weird [16:15:17] it does seem like an auth issue we had on the sandbox clusters where istio was blocking the call from queue_proxy to the kfserving-container pod [16:22:17] but in that case I'd expect to see some horrors in the queue-proxy logs [16:23:13] ah yeah good point [16:29:21] (I had 2 hours of dealing with it earlier on this is why I know it :D) [16:37:17] going to log off earlier today, ttl! [16:37:18] o/ [16:37:44] ah accraze, maybe we could set up a meeting next week (whoever wants to join is invited) to discuss blubber? [16:37:51] just as brainstorm [16:39:17] yep that's a good idea, maybe we can just do the wednesday meeting? [16:43:19] sure as you prefer! [16:44:32] (that'll give me time to read up on it a bit more lol) [16:45:40] been thinking through how we migrate the editquality models from ORES to Lift Wing, I think blubber and helm will be crucial [16:48:06] but yeah we'll brainstorm next week, have a good weekend! [16:48:16] you too! [16:55:09] ./istio-1.6.14/bin/istioctl analyze [16:55:09] Warn [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run 'kubectl label namespace default istio-injection=enabled' to enable it, or 'kubectl label namespace default istio-injection=disabled' to explicitly mark it as not needing injection [16:55:13] Warn [IST0107] (Deployment enwiki-goodfaith-predictor-default-6htsj-deployment.default) Misplaced annotation: sidecar.istio.io/inject can only be applied to Pod [16:55:16] * elukey cries in a corner [16:55:29] I totally forgot that I had to do it in the past tests to make default working [16:55:42] istioctl++ btw [16:56:27] ah no but the error seems to persist [16:57:16] anyway :) [17:42:14] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10articlequality-modeling: Build article quality model for Dutch Wikipedia - https://phabricator.wikimedia.org/T223782 (10Psingh07) Hey! @Ciell I have added the samples of all the classes on @Halfak [[ https://nl.wikipedia.org/wiki/Gebruike... [18:50:36] Great work all, lets use the Wednesday meeting for it