[10:16:42] <wikibugs>	 10Lift-Wing, 10serviceops, 10Kubernetes, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Discussion: dedicated directory in the deployment-chart repository for ML services - https://phabricator.wikimedia.org/T286791 (10elukey) Current status - part of the helmfile private dir refactoring is d...
[10:37:08] * elukey lunch!
[14:19:36] <chrisalbon>	 Morning all!
[14:19:45] <chrisalbon>	 Stiiiillll sick, but I'm on the mend
[14:37:21] <elukey>	 morning!
[14:44:12] <chrisalbon>	 Afternoon elukey! My apologies for being not being so active these last two days. Colds are terrible.
[14:45:04] <elukey>	 chrisalbon: no apologies needed :)
[14:55:10] <ottomata>	 https://docs.lakefs.io/integrations/kubeflow.html
[14:55:18] <ottomata>	 versioned data in object stores
[15:03:00] <elukey>	 wait a min - what is lakefs?
[15:25:23] <chrisalbon>	 I assume it is like dat, versioned data
[15:42:17] <ottomata>	 elukey:  first i heard of it was today too
[15:42:27] <elukey>	 :D
[15:42:39] <ottomata>	 it looks like a nice abstraction versioning and organizing data in object stores
[15:42:39] <elukey>	 I was trying to picture where it would sit in our stack
[15:42:48] <ottomata>	 shared-data platform
[15:45:29] <chrisalbon>	 Ignorant infrastructure question: The main use for this in my head would be historic data (i.e. what did the data look like at the time the model was trained) however I'd think an event-based approach would give the same functionality
[15:45:49] <chrisalbon>	 I guess that wasn't a question, more of an ignorant statement
[15:46:00] <accraze>	 lol
[15:46:04] <accraze>	 (also good morning)
[15:46:08] <chrisalbon>	 morning!
[15:46:30] <ottomata>	 chrisalbon:  where are models planned to be stored atm?  
[15:48:05] <ottomata>	 for serving?
[15:48:10] <accraze>	 ottomata: we are storing model binaries on thanos swift rn, but plan to target the M.O.S.S. cluster in the future
[15:48:14] <ottomata>	 (very ignorant about kubeflow)
[15:48:43] <ottomata>	 right so....versioned / branchable model storage on swift?
[15:48:48] <accraze>	 yup!
[15:49:10] <ottomata>	 was wondering if lakefs would be nice there, allowing you to version  models like in git, but store in swift
[15:49:18] <chrisalbon>	 ohhhhh
[15:49:43] <ottomata>	 deployment and rollback support might be easier to build
[15:50:57] <ottomata>	 elukey:  
[15:50:57] <ottomata>	 https://app.hopin.com/events/apachecon-2021-home/sessions/0b7e697b-d4d5-4ac4-8cdb-9fd691a18396
[15:51:30] <ottomata>	 Alluxio Data Orchestration for Machine Learning
[15:51:31] <ottomata>	 happenign now
[16:03:42] <accraze>	 dang, i shoulda registered for apachecon, looks like some pretty interesting talks
[16:12:22] <ottomata>	 its free you can propb just sign up now
[16:25:39] <wikibugs>	 10Machine-Learning-Team, 10DC-Ops, 10ops-eqiad: (Need By: TBD) rack/setup/install ml-train100[1-4] - https://phabricator.wikimedia.org/T291579 (10RobH)
[16:26:27] <wikibugs>	 10Machine-Learning-Team, 10DC-Ops, 10ops-eqiad: (Need By: TBD) rack/setup/install ml-train100[1-4] - https://phabricator.wikimedia.org/T291579 (10RobH)
[16:27:57] <chrisalbon>	 Thanks elukey for giving the green light for the servers
[16:28:08] <chrisalbon>	 MOAR SERVERS!
[16:29:13] <elukey>	 ottomata: I am planning to watch the recordings when they'll be available, there are a couple about hadoop that are interesting too!
[21:28:57] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): API Gateway Integration - https://phabricator.wikimedia.org/T288789 (10ACraze) I updated the API spec again after more discussion with the team. We will be matching the KFServing Data Plane v1 API instead of the Predict v2 API. Also we will be using v0 versi...
[23:43:04] <accraze>	 wow - it's official, kfserving is now kserve: https://github.com/kserve/kserve
[23:44:11] <accraze>	 but the README still says kfserving ;)