[06:20:23] good morning! [06:20:34] so I found an interesting log on the kube api server [06:20:40] (it was alarming for high latency) [06:20:55] failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: context deadline exceeded [06:27:57] so the validation webhook should be called by the kubeapi when a new istio config is processed by istiod [06:28:11] and in this case istiod is trying to create a wrong one, to test that everything works (and it doesn't) [06:28:46] the kube api fails to call the istiod.istio-system.svc, that kinda makes sense [06:28:55] need to figure out how it is supposed to work [06:43:39] seems https://github.com/istio/istio/issues/19532 [08:54:00] hi elukey o/ [08:54:07] hey! [08:54:37] have a minute to discuss make files? [08:55:46] here is the makefile https://github.com/wikimedia/drafttopic/blob/master/Makefile [08:58:03] I can run "make word2vec/euwiki-20201201-learned_vectors.50_cell.10k.kv" to execute one target https://github.com/wikimedia/drafttopic/blob/master/Makefile#L406 [08:59:33] Do you know of a way I can run one command to execute multple targets? e.g "make word2vec/*" to run all targets that start with word2vec/* [09:09:11] (sorry got called on the phone, 2 mins and I'll be here :) [09:11:31] that's fine... please take your time :) [09:18:42] kevinbazira: so what I know is that you can list multiple targets in one line (like make a b c ..) but I don't recall the use of glob/wildcard [09:20:09] yeah ... considering I would like to add this command to a docker file, I was trying to see whether there is a better solution than listing multiple targets in one line [09:21:09] but thanks, let me keep digging ... there might be a way that we are not yet aware of :) [09:21:12] we could add a specific makefile target listing all the word2vec targets [09:21:27] so you'll be able to do make word2vec [09:21:29] or similar [09:21:34] it could be an optipn [09:22:02] are you suggesting we add the target inside the Makefile? [09:27:00] oh ok ... I get it. This would require us to edit the Makefile. I was hoping it didn't have to get to this :/ [09:27:07] it could be an option yes, if needed, not sure if there is a better way [09:27:10] yeah :( [09:27:43] Thanks elukey for sparing sometime to help with my issue :) [09:31:36] <3 [09:31:44] anytime! not a great help :( [09:41:30] 10Machine-Learning-Team, 10SRE, 10serviceops: Add the possibility to deploy calico on kubernetes master nodes - https://phabricator.wikimedia.org/T285927 (10elukey) [09:41:52] 10Machine-Learning-Team, 10SRE, 10serviceops: Add the possibility to deploy calico on kubernetes master nodes - https://phabricator.wikimedia.org/T285927 (10elukey) [09:42:39] this is the problem with the istio deployment --^ [09:42:42] 10Machine-Learning-Team, 10SRE, 10serviceops, 10Kubernetes: Add the possibility to deploy calico on kubernetes master nodes - https://phabricator.wikimedia.org/T285927 (10JMeybohm) [09:44:37] 10Machine-Learning-Team, 10SRE, 10serviceops, 10Kubernetes: Add the possibility to deploy calico on kubernetes master nodes - https://phabricator.wikimedia.org/T285927 (10JMeybohm) [09:52:10] 10Machine-Learning-Team, 10SRE, 10serviceops, 10Kubernetes: Add the possibility to deploy calico on kubernetes master nodes - https://phabricator.wikimedia.org/T285927 (10JMeybohm) I don't like the idea of having another way of how calico-node is run (it's already complex enough). Because of that I'll sugg... [09:58:31] 10Machine-Learning-Team, 10SRE, 10serviceops, 10Kubernetes: Add the possibility to deploy calico on kubernetes master nodes - https://phabricator.wikimedia.org/T285927 (10elukey) Definitely, it seems a good way to proceed. The only concern that I have is that our kube masters are lightweight VMs (1 virtual... [10:18:56] 10Machine-Learning-Team, 10SRE, 10serviceops, 10Kubernetes: Add the possibility to deploy calico on kubernetes master nodes - https://phabricator.wikimedia.org/T285927 (10JMeybohm) Yeah, maybe. Calico-node runs with a memory limit of 400Mi and CPU requests if 350m but the other components will also take up... [11:53:03] elukey: I ended up using the "make a b c" option but since I didn't want to hard code "a b c" in the dockerfile, I had to search for strings that matched the targets we needed. This is the final command I used: make $(grep '^word2vec/.*' Makefile | sed 's/:.$//') [11:53:59] kevinbazira: makes sense, even if it feels that there must be a better way [11:54:44] Yeah, sure does. Will definitely implement it in case I come across it :) [13:31:59] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Lift Wing proof of concept - https://phabricator.wikimedia.org/T272917 (10elukey) [13:32:14] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Install KFServing standalone - https://phabricator.wikimedia.org/T272919 (10elukey) [13:32:34] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Install Knative on ml-serve cluster - https://phabricator.wikimedia.org/T278194 (10elukey) 05Open→03Stalled We are currently try to deploy layer by layer in production, following this order: * istio * knative-serving * kfserving * inference servi... [13:32:48] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Install KFServing standalone - https://phabricator.wikimedia.org/T272919 (10elukey) 05Open→03Stalled We are currently try to deploy layer by layer in production, following this order: * istio * knative-serving * kfserving * inferen... [13:51:06] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Install Istio on ml-serve cluster - https://phabricator.wikimedia.org/T278192 (10elukey) I tried to deploy the istio config outlined in https://gerrit.wikimedia.org/r/697938 to ml-serve-eqiad and these are the problems that came up: * istiod seems to... [13:51:16] summarized the current issues with istio in --^ [15:50:55] Yeah sure @accraze [16:56:29] * elukey afk! [17:18:32] 10Machine-Learning-Team, 10SRE, 10serviceops, 10Kubernetes, 10Patch-For-Review: Add the possibility to deploy calico on kubernetes master nodes - https://phabricator.wikimedia.org/T285927 (10herron) p:05Triage→03Medium