[10:18:55] <klausman>	 elukey: sorry for writing novels in your patch reviews :D
[10:21:57] <elukey>	 klausman: nono the opposite, more comments are good :)
[10:23:47] <elukey>	 I am waiting others to comment for the golang image since it is more complex that I anticipated, not sure if the os-split dirs still make  sense or not
[10:23:55] <elukey>	 we'll see what SRE prefers
[10:24:08] <elukey>	 I am trying to keep as much compatibility as what they have done so far
[10:28:21] <klausman>	 Aye
[10:28:52] <klausman>	 a) don't break stuff if you don't have to b) keep it simple c) make Lift Wing work, is the order of priorities, I'd wager :)
[10:34:58] <wikibugs>	 10Jade: Replace Content::getNativeData() calls with TextContent::getText() in Jade - https://phabricator.wikimedia.org/T283676 (10Aklapper)
[10:35:05] <wikibugs>	 10Jade: Replace Content::getNativeData() calls with TextContent::getText() in Jade - https://phabricator.wikimedia.org/T283676 (10Aklapper) p:05Triage→03Lowest
[11:42:19] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Install Istio on ml-serve cluster - https://phabricator.wikimedia.org/T278192 (10Theofpa) > I understand that we'll not have envoy proxy related metrics, but I'd expect to see some metrics about how istiod works (pilot metrics maybe) and possibly also...
[11:42:54] <klausman>	 elukey: I think Guiseppe's proposal re: image names for the Go image is a great idea.
[11:57:20] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team, 10artificial-intelligence, 10revscoring: Create generic revscoring inference service - https://phabricator.wikimedia.org/T283526 (10kevinbazira) **>>FILES<<**  On the KFv1.1 sandbox, I have been able to create a generic revscoring inference service using the files belo...
[13:28:17] <elukey>	 klausman: yes it makes sense, will update the code review in sec
[16:26:13] <accraze>	 klausman: thanks for the additional code review on the inference-services, it's very nice to have more eyes on stuff
[16:27:42] <elukey>	 accraze: o/ 
[16:27:46] <accraze>	 in the past it was mostly just me & kevin lol forgot what it's like to have more team members
[16:28:03] <elukey>	 one question about STORAGE_URI: does it need more sidecar/configs/horrors to kfservign?
[16:28:15] <accraze>	 elukey: hey hey
[16:29:26] <accraze>	 good question, both kevinbazira & I are pulling images directly from a public s3 bucket right now with no additional config outside of setting an env var
[16:30:09] <accraze>	 the only other config we might need is a secret for thanos swift. i have a WIP patchset for that here: 
[16:30:11] <accraze>	 https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/693217
[16:30:28] <elukey>	 accraze: we have all code reviews lined up for our images in https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/695173 (there is a chain of them)
[16:30:43] <elukey>	 as soon as we have them on our docker registry we should start using them if possible
[16:30:54] <accraze>	 ^ whoa this is awesome!!
[16:30:55] <elukey>	 I'll also try to add istio 1.10
[16:31:48] <accraze>	 yeah i think the next thing i really want to do is run is to try running our services on all of our own images
[16:32:24] <accraze>	 lol need more coffee
[16:33:33] <elukey>	 super
[16:43:30] <accraze>	 elukey: is there any documentation about how to make a new base image and push to our registry?
[16:45:21] <accraze>	 also i did some experimentation with external transformers yesterday for the outlinks topic model
[16:45:48] <accraze>	 might have run into a TLS issue with the local gateway, but plan on digging into it a bit more today
[16:45:49] <elukey>	 ah yes I wanted to ask you about the transformers.. are those separate containers?
[16:46:42] <accraze>	 yep -- separate containers. I have a patchset up: https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/694756
[16:47:36] <accraze>	 it seems to work when i use the public ingress, but when I use the local cluster address it runs into trouble
[16:48:31] <elukey>	 ah it is a custom image from you okok
[16:48:53] <elukey>	 with local cluster address you mean the cluster local gateway?
[16:49:04] <accraze>	 ^ yeah that's the one
[16:49:08] <elukey>	 (ETOOMANY names in my head sorry)
[16:49:14] <elukey>	 :D
[16:49:16] <accraze>	 LOL that was me yesterday
[16:49:42] <elukey>	 for the base images, I am not sure if we have some docs
[16:49:48] <elukey>	 but we use a tool called docker-pkg
[16:50:01] <elukey>	 and the production-images gerrit repo
[16:50:12] <elukey>	 it is basically a nice wrapper for docker build etc..
[16:51:03] <elukey>	 for the moment let's use your dockerhub account for testing, once the istio/knative/etc.. images are code reviewed and built we can look into that
[16:51:08] <elukey>	 does it work??
[16:51:41] <accraze>	 perfect! i think kevinbazira is using his account too for now
[16:52:17] <accraze>	 once we know the other images are good to go, then we can start talking about how to handle our custom images in production
[16:54:29] <accraze>	 i'll just say, injecting the model binaries from storage into a serving container has been really nice so far
[17:08:24] <wikibugs>	 10artificial-intelligence: Qualitative data collection for ores bias analysis - https://phabricator.wikimedia.org/T225441 (10Groceryheist) 05Open→03Resolved
[17:08:26] <wikibugs>	 10ORES, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Find out what tools are used for making reverts on the ores-enabled wikis. - https://phabricator.wikimedia.org/T225134 (10Groceryheist)
[17:10:29] <wikibugs>	 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10Growth-Team, 10Growth-Team-Filtering: Why are there three Q-marks (???) in threshholds in Special:ORESModels? - https://phabricator.wikimedia.org/T221870 (10Groceryheist) 05Open→03Stalled
[17:10:58] <wikibugs>	 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10Growth-Team, 10Growth-Team-Filtering: Why are there three Q-marks (???) in threshholds in Special:ORESModels? - https://phabricator.wikimedia.org/T221870 (10Groceryheist) 05Stalled→03Resolved a:03Groceryheist
[17:11:00] <wikibugs>	 10ORES, 10Research ideas: Analyze the effects of ORES deployments on counter-vandalism behavior - https://phabricator.wikimedia.org/T200898 (10Groceryheist) 05Open→03Resolved
[17:11:54] <wikibugs>	 10ORES, 10artificial-intelligence, 10editquality-modeling, 10Epic, 10Machine-Learning-Team (Active Tasks): ORES bias analysis - https://phabricator.wikimedia.org/T224901 (10Groceryheist)
[17:11:59] <wikibugs>	 10ORES, 10artificial-intelligence, 10editquality-modeling: Fit models for revert prediction - https://phabricator.wikimedia.org/T224902 (10Groceryheist) 05Open→03Declined This project ended up going a different direction.
[17:23:03] <elukey>	 accraze: one thing that it will probably be needed is a k8s egress rule to fetch the model from swift (nothing big but let's remember that)
[17:23:48] <wikibugs>	 10Machine-Learning-Team, 10SRE, 10Release-Engineering-Team (Radar): Contact number of some WMDE staff should be avalible to SRE/RelEng - https://phabricator.wikimedia.org/T210721 (10WMDE-leszek) 05Open→03Resolved wikidata-emergency@wikimedia.de email address is meant to be used as a means of reaching WMD...
[17:24:03] <klausman>	 accraze: ping me any time for code reviews if you like more eyes on things
[17:52:34] * elukey afk!
[18:13:03] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team, 10artificial-intelligence, 10revscoring: Create generic revscoring inference service - https://phabricator.wikimedia.org/T283526 (10ACraze) @kevinbazira this is great news! glad to hear the generic container approach is working so far. I have uploaded the enwiki-damagi...
[18:42:05] <wikibugs>	 10Lift-Wing, 10artificial-intelligence, 10Machine-Learning-Team (Active Tasks): Create a tutorial for deploying a model on toolforge - https://phabricator.wikimedia.org/T281317 (10Isaac) FYI -- Research has two templates for this that might be of use: * API on Cloud VPS: https://github.com/wikimedia/research...