[08:36:42] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10MoritzMuehlenhoff) Let's bump to v1.23.14 before migrating? That version fixes two new security issues: CVE-20... [08:41:40] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Import istio 1.1x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T322193 (10MoritzMuehlenhoff) Excellent timing, also 1.15.3 fixes a security vulnerability: https://github.com/istio/istio/security/advisories/GHSA-6c6p-h79f-g6p4... [09:17:33] _joe_: yes, I know [09:17:37] bit that does not mean we have metrics for it [09:17:39] *but [09:25:21] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Switch to cgroup v2 and systemd as cgroup driver for docker and kubelet - https://phabricator.wikimedia.org/T313473 (10JMeybohm) 05Open→03Resolved [09:25:27] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [09:26:20] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) >>! In T307943#8388861, @MoritzMuehlenhoff wrote: > Let's bump to v1.23.14 before migrating? That ver... [10:01:56] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Import istio 1.1x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T322193 (10elukey) Applying istioctl to the main manifest returns the following error: ` [...] affinity: nodeAffinity: requiredDurin... [10:19:42] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Import istio 1.1x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T322193 (10elukey) Tried the httpbin example on minikube without sidecar injection, everything works as expected. The sidecar injection use case will need more test... [10:19:52] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Import istio 1.1x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T322193 (10elukey) [10:26:20] hello folks [10:26:56] I am going to test (manually) an istio config on staging-codfw [10:27:22] I found some little issues with the current config on 1.15.3, if those can be backported then we'll not need any change when we'll upgrade k8s [10:27:48] <_joe_> jayme: here it is in all its glory: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/855668 [10:28:07] <_joe_> tested on termbox and cxserver and the conversion worked out of the box [10:28:26] <_joe_> this is a basic conversion just to get rid of common_templates [10:28:31] <_joe_> but still [10:28:40] <_joe_> I thought it would be much more manual [10:31:36] ...not sure I want to click that link :) [10:33:19] can it convert eventgate? :D [10:38:43] (of course the patch doesn't work entirely, so I added a comment to our configs in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/855967) [10:39:17] (istio upgrade task done afaics \o/) [10:50:27] <_joe_> elukey: the error is due to https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/855967/1/custom_deploy.d/istio/aux-k8s/config.yaml#33 [10:50:35] <_joe_> that isn't valid yaml [10:52:03] <_joe_> you have an extra : [10:54:14] _joe_ thanks! Are you implying that I am not a good yaml engineer? :D [10:54:33] <_joe_> elukey: only excellent yaml engineers generate invalid yaml [10:54:39] ahahhaha [10:54:45] <_joe_> bad yaml engineers generate valid yaml that has invalid content [10:54:46] I think he's implying he is a good yaml validator :) [10:55:01] <_joe_> in fact, it's somewhat challenging to craft a string that yaml parsers won't guzzle [11:07:31] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [11:07:33] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: Metric name changes with Kubernetes v1.23 - https://phabricator.wikimedia.org/T322919 (10JMeybohm) [11:07:49] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: Metrics changes with Kubernetes v1.23 - https://phabricator.wikimedia.org/T322919 (10JMeybohm) [11:13:05] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Import pause container image >= 3.5 (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T322920 (10elukey) [11:16:20] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [13:16:20] <_joe_> jayme: around? [13:16:28] yes [13:16:35] <_joe_> I need help figuring out what to do with the mess scap made on the mw-* deployments [13:16:55] give me a minute to git commit my stugg [13:16:59] *stuff [13:17:44] <_joe_> sure [13:23:40] ok. What's up? [13:24:47] <_joe_> so first of all scap was running 8 helmfile apply in parallel [13:24:55] uh, nice [13:25:02] <_joe_> that seems not to have shocked k8s too much, but [13:25:15] <_joe_> it also ran 8 helmfile test in parallel [13:25:23] <_joe_> which I didn't expect [13:26:18] <_joe_> in the meantime at least the test pods stopped spawning [13:28:22] <_joe_> now, that moment caused issues on mw-web [13:28:29] <_joe_> which is the largest deployment [13:28:47] <_joe_> now I tried to re-run a deployment, and it seems stuck with 2 old pods [13:29:01] <_joe_> there are messages about quota resources but they're old [13:29:41] quick Q: the 8 parallel runs are one for each mw-* namespace, right? [13:30:07] so it's not 8 times the same thing [13:30:10] <_joe_> yes [13:30:19] <_joe_> no thankfully no :P [13:30:38] ah, okay. I was wondering about the "that seems not to have shocked k8s too much" initially :D [13:30:58] so in theory it should have been 9 paralel runs? [13:31:16] 2 for each ns apart from mw-debug which only has 1? [13:31:31] <_joe_> no [13:31:39] <_joe_> let's talk later about the details [13:31:58] ok. Next Q: Only mw-web is broken? [13:32:03] <_joe_> now it seems this has created tens of test containers in Error state in all the others [13:32:10] <_joe_> and yes only mw-web is broken [13:32:33] <_joe_> I suppose somehow the namespace is too small for a rolling upgrade like helm does by default? [13:32:52] <_joe_> so I'll just wait for the rollback to end [13:32:56] <_joe_> or time out [13:33:09] <_joe_> and recreate the release with a smaller number of replicas [13:33:23] <_joe_> it's just strange because this didn't happen before when I ran helmfile apply by hand [13:33:44] <_joe_> so I'm not 100% sure what's going on there [13:34:21] there should be room for 90 CPUs and 100Gi of RAM per NS [13:34:41] currently it's at 66 CPUs, 40Gi RAM [13:35:33] <_joe_> which is way too high for 10 pods, heh [13:35:36] <_joe_> but still [13:35:49] <_joe_> it should have room for 4 more pods [13:36:04] <_joe_> and take a look at codfw if you want, which I didn't touch [13:37:46] in fact I only see 8 pods [13:38:05] (in eqiad) [13:38:56] <_joe_> it should be 8+2 [13:39:01] <_joe_> main + canary [13:39:12] <_joe_> but yeah, what happened is that the main release failed to apply [13:39:20] <_joe_> and helm rolled it back [13:39:26] <_joe_> I just figured it out [13:40:19] <_joe_> ok so the problem is running helm without a selector I'd say [13:40:58] helmfile you mean? [13:41:17] <_joe_> yes [13:41:19] <_joe_> sorry [13:41:52] not sure I understand what problem that creates [13:44:57] <_joe_> so looking at the events [13:45:05] AIUI you can fit 10 pods in one namespace in parallel [13:45:17] <_joe_> it seems like helm tries to first create all the new pods [13:45:22] <_joe_> then kill the old ones [13:45:29] <_joe_> because it's failing again [13:46:00] <_joe_> ah no it finally managed [13:47:21] you have a maxSurge of 25%. That number pods will be created in addition to the number of replicas [13:49:11] <_joe_> by both the releases [13:49:13] terminating pods do not count as available but still count towards the resource quota, so there might be even more pods running than replicas + maxSurge depending on how long the old ones take to shut down [13:49:27] <_joe_> which means 4 new pods at a time? [13:49:35] <_joe_> ok [13:49:48] <_joe_> so we need to either reduce maxSurge or raise the caps in that NS [13:50:08] yes. or do releases one-by-one [13:50:18] <_joe_> which we should [13:50:23] <_joe_> anyways [13:50:31] yes. It makes sense to do canary first :-) [13:50:32] <_joe_> that was one of the bugs we encountered [13:53:22] but still: with 2 replicas for canaries and 8 for main the namespace quota is pretty tight [13:53:56] if the resource limits are not different for canaries - I have not checked [15:06:18] elukey: do you remember why ml-serve is configured with kubelet_ipv5=false? [15:06:29] haha, kubelet_ipv6 :) [15:07:14] to be precise, it does not have kubelet_ipv6=true, and the key defaults to false [15:08:29] jayme: I don't recall why to be honest [15:08:45] maybe it is because we never really used ipv6 pools right? [15:08:57] I think we can try to turn it true [15:13:36] pod wise we use them actually [15:13:51] no need to turn it on immediately, I was just curious [15:14:06] dse and aux oviously copied from you :) [15:15:00] do we use ipv6 for pods? [15:15:03] I'm just currently working on migrating the ipv6 frankenstein mode we have to full ipv6 dualstack in puppet and fell into the trap thinking that all clusters are equal in that regard [15:15:11] this is news for me, I thought it was only ipv4 [15:15:38] ah ok so this might explain - maybe at the time we wondered "do we want the frankestein for ml-serve?" [15:16:33] it depends on the definition of "use" I guess. All pods do have ipv6 ips (in wikikube) [15:22:41] interesting, I thought it wasn't enabled [15:23:27] we can simplify and roll out ipv6 everywhere actually, I guess that gradually over time pods will get their ipv6 address [15:23:44] or do we need a complete restart of pods? [15:23:59] aux and dse are easy to change, ml-serve should be as well [15:24:06] so we keep one standard and that's it [15:24:45] I dont know what is needed for a gradual rollout tbh. But we can enable it with the reimage to v1.23 [15:29:12] definitely yes, if it is an option and if it doesn't impact the puppet refactor +1 (badly I mean) [15:32:50] yeah, should be fine [15:56:37] netbox search isn't really...intuitive [15:57:11] that combined with everyone coming up with their own scheme for description of networks is a tad painful :)