[15:05:44] <inflatador>	 working on getting our new service into production on wikikube and just curious...do any non-service-ops-owned services require special handling during a cluster upgrade? ref https://phabricator.wikimedia.org/T342149#9230869
[15:18:17] <elukey>	 hello folks, tried to refactor the k8s high latency alert: https://gerrit.wikimedia.org/r/c/operations/alerts/+/964025
[15:19:18] <elukey>	 (changing the dashboard link to be more precise)
[15:19:28] <elukey>	 if you want to review it please go ahead :)
[15:20:17] <elukey>	 inflatador: in the ML case it is just deploying our custom stack (istio/knative/etc..)
[15:21:03] <elukey>	 istio configs are deployed on all clusters via istioctl, a binary that takes specific yaml manifest as inputs, and we store them in the custom.d dir in deployment-charts
[15:21:31] <elukey>	 in the case that you pointed out we'll need to add the specific command in the upgrade procedure (probably)
[15:21:36] <elukey>	 (not sure if it answers)
[15:22:51] <dcausse>	 I think the question is how do you bring the cluster down, do you undeploy individual services (via helmfile destroy) or simply "turn off" k
[15:22:53] <dcausse>	 8s
[15:23:38] <elukey>	 dcausse: ah ok! So in the past, IIRC, we wiped etcd, that meant resetting the cluster 
[15:23:40] <inflatador>	 https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/k8s/upgrade-cluster.py looking at the cookbook now too
[15:23:43] <elukey>	 there is a cookbook about it
[15:24:04] <elukey>	 (of course after depooling etc..)
[15:24:16] <dcausse>	 if the services are just depooled and not "undeployed" then I think we should be good
[15:24:19] <elukey>	 specific helmfile destroy commands will need to be done before the upgrade, if any
[15:25:39] <inflatador>	 sounds like we'd be OK, but if there's a way to step thru this on a test cluster let us know...
[15:26:30] <dcausse>	 yes would be nice to have a way to "simlute" this so we make sure the service recovers properly after deploying
[15:26:37] <dcausse>	 s
[15:26:48] <dcausse>	 s/simlute/simulate/
[15:27:16] <elukey>	 mmm reading through the phab task, it seems that the flink operator is a little flaky in this regard
[15:27:42] <elukey>	 the next upgrade is very far from now (we need to migrate away from PSP and change the container engine before that)
[15:27:49] <elukey>	 we can probably discuss with Janis this use case
[15:28:09] <elukey>	 (going afk for today folks, have a nice weekend!)
[15:28:39] <dcausse>	 enjoy!
[15:28:43] <inflatador>	 thanks, enjoy
[21:04:50] <inflatador>	 We just deployed a new service on wikikube-staging, cirrus-streaming-updater ( https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/964071/ ). 
[21:08:13] <inflatador>	 well...we can't actually get it to deploy with helmfile. The namespace looks fine, just no resources are being created with an apply. If anyone has time next week to look at it, reach out to myself d-causse or e-bernhardson . Thanks for your time