[15:11:13] heads-up elukey running https://wikitech.wikimedia.org/wiki/Kubernetes/Add_a_new_service#Deploy_changes_to_helmfile.d/admin_ng on dse8-kse-eqiad to apply https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/891577 [15:16:45] hmm no elukey in here? And the helmfile apply is not working. Looks like he's busy with other things per #wikimedia-sre [15:21:19] inflatador: what's going wrong? [15:21:23] error pasted here: https://phabricator.wikimedia.org/P44885 . [15:22:32] claime ^^ . looks like there are other changes besides the one I wanted to apply [15:22:42] Hmm [15:22:44] something related to cfssl it appears [15:23:17] it's not urgent, but if you do have any ideas LMK [15:29:34] inflatador: we are upgrading dse-k8s to 1.23 :) [15:29:44] cfssl is not there, this is why you get the errors [15:30:26] elukey OK, so you're in the middle of the upgrade right now? [15:30:32] yes [15:31:13] elukey ACK np, hit me up when finished if that works for you [15:32:09] inflatador: sure :) Please ping me/Ben/Tobias before deploying next time [15:32:57] elukey will do, thanks again! [17:31:55] elukey: o/ we were going to re-deploy mw page content enrichment on dse k8s in an hour or so. i see deploy maybe is still ongoing? [17:31:59] should we not? [17:33:51] ottomata: o/ the cluster is not ready yet, we had issues with hosts in row E/F that failed to reimage.. I'll try to get it done tomorrow, there is still some work to do to finish the bootstrap [17:34:04] is it a problem? And/or blocking etc..? [17:35:10] we have value streams a demo scheduled for tomorrow, were hoping to show the thing live, but we don't have to. oh gabriele is not in this room [17:40:18] ottomata: you don't need istio right? If so I just deployed cfssl, so in theory the error that inflatador saw should not re-happen [17:40:40] ah but I guess that mw enrichment is different, it is not the rdf-updater okok [17:40:53] then you can try now, I don't guarantee that all works but afaics nothing is wrong :) [17:43:38] elukey: that's fine! and if you do more stuff and take it down that is fine too. [17:43:43] we'll give it a go, if it doesn' work no worries [17:44:54] ack :) [18:04:02] fixed also the istio deployment (in case it is needed) [18:40:17] k! i dunno what is needed i just do helmfile apply :) [20:57:21] elukey: aye, there must be some istio thing i need [20:57:32] while trying to re deploy the fliink-operator [20:57:32] i got [20:58:03] Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "..." network for pod "flink-kubernetes-operator-...": networkPlugin cni failed to set up pod "flink-kubernetes-operator-..." network: pods "flink-kubernetes-operator-..." is forbidden: User "istio-cni" cannot get resource "pods" in API group "" in the namespace "flink-operator" [20:58:38] the command i used to to deployo is [20:58:45] kube_env admin dse-k8s-eqiad [20:59:01] root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# helmfile -e dse-k8s-eqiad -l name=flink-operator apply