[07:50:34] sooo all clusters on 1.23? [07:50:55] at this point we can definitely say "congrats" for the amazing work to jayme :) [07:51:25] yes, all clusters are now on 1.23. And congrats to both of you for a job well done. [07:51:53] 🎉 cheers elukey! [07:54:05] side note - all k8s etcd clusters are now using PKI to manage their tls certs, thanks to btullis' work :) [07:56:16] I planned to go around and delete the cergen certs for those from private puppet today [07:59:29] will start with the staging ones + manual puppet runs because I'm a wee bit scared :) [07:59:50] definitely :) [08:01:16] akosiaris: do you plan on taking care of cleaning up leftovers from the ip-pool change? [08:03:52] jayme: yup, doing so now [08:05:15] nice! Feel free to resolve https://phabricator.wikimedia.org/T326617 after [08:19:30] elukey: I did not kill wikikube aparently ... you okay with me proceeding with ml etcd clusters? [08:20:16] also: did you get puppet-private mails for my changes? 🤔 I did not [08:34:28] jayme: +1! [08:34:52] took the liberty.. :) all certs gone and puppet ran on all etcd nodes [09:34:38] elukey: thanks for the review. I'll merge https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/895336/ now. Would you be so kind to deploy and check some kserve/knative afterwards? [09:36:04] sure! [09:37:41] nice. The updated charts should be around in a minute [09:43:32] admin_ng showed no diff for wikikube,dse,aux and ml-staging [09:43:56] (which was expected) [09:45:53] okok [09:46:19] I am in the middle of an upgrade on ml-staging and then I'll apply also changes to ml-serve, so I should be able to do kserve/knative today [09:46:33] ack [11:11:53] if anyone familiar with the k8s cookbooks could have a quick sanity check of this would be great: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/895208/ [11:32:54] {{done}} [11:32:58] <3 [11:33:17] TIL about type hints for class attributes [11:33:35] you can type hint pretty much anything nowadays :D [11:39:17] I am not 100% sure if the readability improves with type hints outside function signatures, but surely worth to use them [11:41:24] it's actually required to run mypy, because it infers the type from the first declaration of the variables and in some case it's not able to or will just error out aftewards [11:41:45] because of a different type assigned to it [11:42:04] in most cases it's able to autodetect it, not always [11:46:23] the last patch in the series is adding mypy checks to the cookbooks repo in a non-mandatory way (if there are they are checked, if not all good) and so I had to fix all outstanding errors :) [12:20:43] istio is.. weird. https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/895748 [13:56:00] FYI, going to try the spark-operator deployment to dse-k8s now'ish [13:57:23] oh boy! [14:02:57] oh, yeah! godspeed! un*fortunately I have to leave for a doc appt. [14:11:20] ottomata: There is a minor update to the flink-operator scheduled to go out as well. Are you happy for us to deploy it? [14:11:39] https://www.irccloud.com/pastebin/K41btXff/ [14:12:06] btullis: go for it, that is part of work jayme is doing [14:12:28] There's a call here, if you'd like to join us https://meet.google.com/bxn-tqiw-ono but no need if you're busy. [14:13:01] Thanks. [14:43:32] First hurdle [14:43:34] https://www.irccloud.com/pastebin/IuF3cptM/ [14:45:29] That is when deploying the changes to the namespaces. [15:07:11] We have passed that error, but the next one is this: [15:07:14] Error: UPGRADE FAILED: an error occurred while rolling back the release. original upgrade error: cannot patch "deploy" with kind RoleBinding: RoleBinding.rbac.authorization.k8s.io "deploy" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"ClusterRole", Name:"deploy-sparkapplications"}: cannot change roleRef: cannot patch "deploy" with kind RoleBinding: [15:07:15] RoleBinding.rbac.authorization.k8s.io "deploy" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"ClusterRole", Name:"deploy-sparkapplications"}: cannot change roleRef [15:07:41] Error: UPGRADE FAILED: an error occurred while rolling back the release. original upgrade error: cannot patch "deploy" with kind RoleBinding: RoleBinding.rbac.authorization.k8s.io "deploy" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"ClusterRole", Name:"deploy-sparkapplications"}: cannot change roleRef: cannot patch "deploy" with kind RoleBinding: [15:07:42] RoleBinding.rbac.authorization.k8s.io "deploy" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"ClusterRole", Name:"deploy-sparkapplications"}: cannot change roleRef [15:07:56] Dammit, that was supposed to be a paste. Sorry. [16:42:00] Oh. Namespaces deployed successfully. Proceeding again. [17:03:11] Deployed successfully. Proceeding to test. [23:05:21] are secrets only avaliable as helmfile values? Or is there some way to create a secret resource from our private repo?