[09:17:12] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Integrate cert-manager/issuer in ml-serve clusters - https://phabricator.wikimedia.org/T298976 (10elukey) Cleaned up all config and certs related to inference.discovery.wmnet (on the puppet private repo and on k8s secrets etc..). The Pu... [09:18:05] inference.discovery.wmnet's puppet cert cleaned up! [10:08:23] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Integrate cert-manager/issuer in ml-serve clusters - https://phabricator.wikimedia.org/T298976 (10elukey) The remaining puppet-based certificates are: * kserve-webhook-server-service.kserve.svc.cluster.local * istio-egressgateway.istio... [11:28:11] * elukey lunch [15:24:04] 10Machine-Learning-Team, 10ORES, 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Search (Current work): Estimate how many Wikidata items have low/no ORES score - https://phabricator.wikimedia.org/T288262 (10AKhatun_WMF) The analysis is done here (for Q-ids): [[ https://wikitech.wikimedia.org/wiki/User:... [15:32:56] o/ [15:33:02] o/ [15:45:50] Morning! Again, sorry about yesterday [15:46:53] o/ [15:49:06] no worries chrisalbon :) [15:50:43] +1 :) [15:52:53] almost done with the draftquality-transformer, one last CR to get the pipelines to publish the image to wmf registry [15:53:08] still need to figure out why the transformer can't talk to the predictor on the new ml-sandbox [15:57:49] it's gotta be related to cluster-local-gateway [15:58:39] anyways, i have the ml-sandbox install script (and literate config) here: https://gitlab.wikimedia.org/accraze/ml-sandbox-cfg [16:13:48] Can you walk me through the sandbox sometime next week? [16:17:05] chrisalbon: for sure! would be good to get more eyes on it [16:17:38] in theory, you could even use the install script to have a minikube cluster running the wmf kserve stack on your machine [16:19:36] hopefully this will make onboarding a bit easier too [16:21:49] Yeah that is my current interest, I'll find some time next week. Thanks! [16:48:48] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10articlequality-modeling: Build article quality model for Dutch Wikipedia - https://phabricator.wikimedia.org/T223782 (10Halfak) Thanks @ACraze! It looks like I no longer have permission to manually mirror changes into the gerrit model rep... [16:51:12] 10Machine-Learning-Team, 10ORES: ORES deployment repos not mirroring regular git changes anymore - https://phabricator.wikimedia.org/T299664 (10Halfak) [16:58:13] (03PS1) 10Halfak: Adds hiwiki editquality models to config [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/755731 [16:59:42] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10articlequality-modeling: Build article quality model for Dutch Wikipedia - https://phabricator.wikimedia.org/T223782 (10Halfak) FYI, here is the config change. https://gerrit.wikimedia.org/r/c/mediawiki/services/ores/deploy/+/755731 It is... [16:59:54] (03PS2) 10Halfak: (WIP) Adds hiwiki editquality models to config [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/755731 [17:06:08] the inference.discovery.wmnet endpoint is finally up! [17:06:41] \o/ [17:13:04] nice one elukey [17:15:55] whew! Good to hear [17:28:20] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Create a LB service for inference.discovery.wmnet - https://phabricator.wikimedia.org/T289835 (10elukey) The inference endpoint is up, but there is still an error with the monitoring: ` PROBLEM - LVS inference eqiad port 30443/tcp - Inference ML serv... [17:29:43] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Integrate cert-manager/issuer in ml-serve clusters - https://phabricator.wikimedia.org/T298976 (10elukey) After a chat with John in https://gerrit.wikimedia.org/r/755651 it seems that for our use case we can use the discovery intermedi... [17:34:16] added next steps for all tasks in progress :) [17:34:17] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): API Gateway Integration - https://phabricator.wikimedia.org/T288789 (10elukey) Next steps for this task: 1) Wait for the per-service rate-limit deployment T295956. 2) Wait for the update of the TLS CA certs trusted by the api-gateway pods T299550 3) Add the... [17:34:29] logging off for today, have a good day/evening folks! [17:35:22] awesome, thanks elukey! have a good one :)