[06:30:13] <elukey>	 good morning!
[06:30:19] <elukey>	 helm3 deployments seems to work!
[06:30:23] <elukey>	 \o/
[09:10:16] <wikibugs>	 10Machine-Learning-Team, 10Observability-Logging: logstash indexing failures reported for "knative-serving" - https://phabricator.wikimedia.org/T292174 (10elukey) ` {   "level": "warn",   "ts": "2021-09-30T11:20:50.892Z",   "logger": "activator",   "caller": "net/revision_backends.go:285",   "msg": "Failed pro...
[09:21:08] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Lift Wing proof of concept - https://phabricator.wikimedia.org/T272917 (10elukey)
[09:21:31] <wikibugs>	 10Lift-Wing, 10serviceops, 10Kubernetes, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Discussion: dedicated directory in the deployment-chart repository for ML services - https://phabricator.wikimedia.org/T286791 (10elukey) 05Open→03Resolved a:03elukey This has been implemented, and...
[09:22:03] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Lift Wing proof of concept - https://phabricator.wikimedia.org/T272917 (10elukey)
[09:22:53] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Install KFServing standalone - https://phabricator.wikimedia.org/T272919 (10elukey) 05Open→03Resolved a:03elukey
[09:25:16] <wikibugs>	 10Machine-Learning-Team, 10Observability-Logging: logstash indexing failures reported for "knative-serving" - https://phabricator.wikimedia.org/T292174 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Thank you @elukey for taking a look. I think the error was a type conflict, where before the last deploym...
[10:18:25] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team: Add network policies to the ML k8s clusters - https://phabricator.wikimedia.org/T289834 (10elukey) A lot of ports and complexity, but overall this should happen:  * all pods shouldn't have rules for outgoing traffic (same as we do for the main cluster) * the istio gateway...
[10:46:49] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Growth-Scaling, 10Growth-Team: Establish processes for running the dataset pipeline - https://phabricator.wikimedia.org/T276438 (10kostajh) a:03kevinbazira >>! In T276438#7334245, @kostajh wrote: >>>! In T276438#7329170, @kevinbazira wrote: >> @MGerlach is transitio...
[10:49:42] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Growth-Team: Define SLIs/SLOs for link recommendation service - https://phabricator.wikimedia.org/T278083 (10kostajh) >>! In T278083#7022330, @MMiller_WMF wrote: > @kostajh -- I've not been through a process to determine SLOs before, but I can express what I think would...
[14:10:05] <elukey>	 next step is to write down firewall rules about traffic between pods in the cluster, this one may be a little complex due to all the pods that we have
[15:11:12] <elukey>	 I also created a patch for a new group called "deploy-ml-services" so that people allowed to deploy to ml-serve will be added there
[15:11:17] <elukey>	 (intially only us)
[15:28:43] <elukey>	 aaaand also fixed the last pending problem with puppet and knative/kfserving
[16:14:43] <accraze>	 o/
[16:14:59] <accraze>	 elukey: nice job! glad to hear helm3 deployments are working :)
[16:21:27] <elukey>	 \o/
[16:34:02] <elukey>	 going afk, have a good day/weekend folks!
[16:45:23] <accraze>	 see ya elukey! great work this week :)
[21:11:37] <wikibugs>	 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10articlequality-modeling: Build article quality model for Dutch Wikipedia - https://phabricator.wikimedia.org/T223782 (10Halfak) I'm running into some issues with the wikilabels updates.  Looks like some of our deployment code has gotten ol...
[22:07:40] <wikibugs>	 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10articlequality-modeling: Build article quality model for Dutch Wikipedia - https://phabricator.wikimedia.org/T223782 (10Halfak) I was able to get the campaign loaded!  See https://labels.wmflabs.org/ui/nlwiki/  This will require 3 labels b...