[14:46:33] <wikibugs>	 10Machine-Learning-Team, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:(Need By: TBD) rack/setup/install ml-train100[1-4] - https://phabricator.wikimedia.org/T291579 (10elukey) Sorry @Jclark-ctr some other discussions started with Service Ops about networking, we still need to reach a quorum about final settings (vlan,...
[16:05:09] <accraze>	 o/
[16:07:30] <elukey>	 accraze: o/
[16:08:18] <elukey>	 reporting the same question in here - I am planning (after a chat with DE) to name the ml-train k8s worker node hosts "dse-k8s-worker100[1-4]
[16:08:28] <elukey>	 to reflect the fact that the cluster will be shared
[16:08:41] <accraze>	 sounds good to me!
[16:08:42] <elukey>	 is the name ok? (I'd need to unblock dcops, they are configuring the nodes)
[16:09:28] <accraze>	 yeah that name is fine imo
[16:10:30] <elukey>	 perfect :)
[16:10:49] <elukey>	 I'll add the final naming to the task later on, in case somebody wants to add their opinion
[19:04:19] <wikibugs>	 10Machine-Learning-Team, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:(Need By: TBD) rack/setup/install ml-train100[1-4] - https://phabricator.wikimedia.org/T291579 (10elukey)
[19:05:42] <wikibugs>	 10Machine-Learning-Team, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:(Need By: TBD) rack/setup/install ml-train100[1-4] - https://phabricator.wikimedia.org/T291579 (10elukey) @Jclark-ctr thanks for the patience! We'd like to call the nodes `dse-k8s-worker100[1-4]`, let me know if this is viable (`dse` in this case me...
[19:06:04] <wikibugs>	 10Machine-Learning-Team, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:(Need By: TBD) rack/setup/install dse-k8s-worker100[1-4] - https://phabricator.wikimedia.org/T291579 (10elukey)
[22:25:09] <accraze>	 ok getting a bit closer with our new ml-sandbox deployment, got our wmf images running the kserve stack
[22:25:41] <accraze>	 just deployed a basic sklearn-iris inference service to test, but getting a 503 service unavailable
[22:25:58] <accraze>	 when i try to curl and retrieve a prediction
[22:26:53] <accraze>	 i feel like i've seen this error before....
[22:43:40] <accraze>	 checked the kserve-controller-manager logs, seeing the IngressReconciler updating external service - things look correct there
[23:11:23] <accraze>	 i'm guessing it's related to how i setup istio ingress gateway