[06:58:15] <elukey>	 good morning
[06:59:46] <elukey>	 there are a couple of things left to do before being able to bootstrap the first version of kfserving on ml-serve
[06:59:58] <elukey>	 1) configure TLS for the istio ingress gateway
[07:00:31] <elukey>	 2) complete the kfserving's helm chart (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/700470) with TLS settings
[07:00:56] <elukey>	 for 1) it should be, IIUC, something like inference-service.wikimedia.org, or similar
[07:01:23] <elukey>	 for 2) it is the certificate for the webhook service
[07:02:24] <elukey>	 these are the SANs added to the self-signed cert (from the upstream's script)
[07:02:27] <elukey>	                 DNS:kfserving-webhook-server-service, DNS:kfserving-webhook-server-service.kfserving-system, DNS:kfserving-webhook-server-service.kfserving-system.svc, DNS:kfserving-webhook-server-service.kfserving-system.svc.cluster, DNS:kfserving-webhook-server-service.kfserving-system.svc.cluster.local
[07:02:56] <elukey>	 that are a little weird, trying to figure out if/how they can be changed
[09:53:21] <elukey>	 ok now, in the middle of istio checks, I found out a weird thing
[09:53:35] <elukey>	 if I apply a manifest via istioctl 1.9.5 all works
[09:53:54] <elukey>	 but then if I want to re-apply the same with a little change, istioctl tells me that the version deployed is 1.9.0
[09:54:00] <elukey>	 and that it needs to upgrade
[09:54:03] <elukey>	 that is weird
[10:05:41] <elukey>	 I think that the istio pilot binary has a /version endpoint, so istioctl probes it and it gets 1.9.0
[10:05:52] <elukey>	 so there is probably an istio image build parameter missing
[10:23:44] <elukey>	 ahhh it is missing a VERSION env variable
[10:23:47] * elukey cries in a corner
[10:23:52] <elukey>	 will update the cr
[10:33:32] * elukey lunch
[13:08:42] <klausman>	 !log starting update of etcd machines in codfw
[13:08:49] <klausman>	 Ah.
[13:09:16] <elukey>	 we can add it if we want!
[13:15:47] <klausman>	 Would it be "write-only" or also spam the loggings of everyone else here?
[13:26:36] <elukey>	 it would be like the other !log in the chans, they are all aggregated in logs and available via tools like https://sal.toolforge.org/production
[14:09:01] <klausman>	 Ah, roger.
[14:09:19] <klausman>	 So with codfw about done, do you think I should wait with eiqad or do that rightaway
[14:13:48] <elukey>	 nothing is running on the clusters so you can go ahead anytime
[14:14:15] <klausman>	 Alrighty. Might as well get it over with
[14:56:23] <elukey>	 one thing that I am trying to test now (on minikube) is if we can drop the istio ingress gateway, and keep only the cluster local one (exposing its ports via nodeport0
[14:56:36] <elukey>	 so we could have an LVS in front of it
[14:56:53] <elukey>	 but then we'd also need to have TLS termination on the cluster local gw
[14:57:41] <elukey>	 the cluster local gateway is used by knative
[14:58:03] <elukey>	 and in recent version it was replaced by a knative "native" gateway (that maps onto an istio one IIUC)
[14:58:24] <elukey>	 I am trying to understand if we need both at the moment for our LVS setting 
[15:02:52] <elukey>	 otherwise we could have the istio cluster local gw using internal kubernetes ips etc..
[15:03:03] <elukey>	 and the istio ingress gateway exposed via NodePort
[15:16:35] <klausman>	 Make sense (I think ;))
[15:16:43] <klausman>	 Unrelatedly, updates and reboots done.
[15:16:53] <elukey>	 nice :)
[15:17:10] <elukey>	 so istioctl 1.9.5 and 1.6.14 is deployed on deploy1001
[15:17:29] <elukey>	 there is a code review for the new docker images (1.9.5(
[15:17:45] <elukey>	 after that, if we find a good config, we'll be able to probably deploy :)
[15:17:55] <elukey>	 knative's helm chart is basically done
[15:18:07] <elukey>	 kfserving one is WIP, we need to find how to add TLS certs
[15:18:48] <elukey>	 I am pretty sure that we'll find something interesting to debug when applying all the code to prod
[15:18:53] <elukey>	 0 chance that it will work fine
[15:29:17] <klausman>	 Still, great work, Luca!
[15:29:37] <elukey>	 <3
[15:29:46] <klausman>	 I'm gonna take a quick break and run some errands
[15:39:43] <chrisalbon>	 morning all
[15:49:34] <elukey>	 o/
[15:50:51] <wikibugs>	 10Machine-Learning-Team, 10Analytics: Configure the Hadoop cluster to use the GPUs available on some workers - https://phabricator.wikimedia.org/T276791 (10Ottomata)
[15:55:43] <klausman>	 Yo, Chris!
[15:56:23] <klausman>	 (upside of vaccination going well in CH: more people out&about. Downside: crammed trams and buses)
[15:58:24] <chrisalbon>	 Yeah I wasn't happy when car traffic suddenly reappeared
[15:59:51] <klausman>	 CH is approaching 2.5M people fully vaccinated, and 6.5M with the first vaccine (out of 8.5M)
[16:00:18] <chrisalbon>	 nice!
[16:00:26] <klausman>	 I suspect within a month, we'll be close to 6-7M fully vaccinated and just about everyone having had at least one shot
[16:01:01] <klausman>	 There's of course th long tail of those who can't be vaccinated (or don't want to be)
[16:01:08] <chrisalbon>	 Yeah, that is where San Francisco is now, 81% over 12 years old with at least one shot.
[16:02:27] <klausman>	 I'd suspect SF to be more vaccine-friendly on average than other parts of the US
[16:02:45] <klausman>	 All those biotech startups ;)
[16:07:43] <chrisalbon>	 lol yes
[18:28:57] * elukey afk o/