[07:05:37] <_joe_> bblack: sorry about that, cp6011 had puppet disabled for weeks so I failed repeatedly to reenable puppet there [07:05:47] <_joe_> so at some point I excluded it from my cumin runs [07:06:02] <_joe_> with the result that when you reenabled puppet, I actually disabled it [07:13:15] good morning! Today's reimage menu [07:13:21] - kubernetes2015 https://gerrit.wikimedia.org/r/c/operations/puppet/+/771422 [07:13:35] - kubernetes2016 https://gerrit.wikimedia.org/r/c/operations/puppet/+/771423/1 [07:15:54] after these --^ the codfw cluster is completed (modulo 200[1-4] hosts that are still on stretch but that will be decommed [07:27:04] (I need to run some errands then I'll start, if anybody could review the above I'd be grateful :) [08:47:05] jayme, _joe_: for the k8s module for spicerack, there are a couple of pending comments on the CR. But given that we're not blocked by buster anymore, we can merge+deploy+test it anytime. [08:47:25] <_joe_> volans: yes, ENOTIME [08:48:17] for the comments if you want I can help and update the PS myself, but for the testing I'll need one of you to chime in ;) [09:49:16] draining + reimaging 2015 (thanks for the reviews) [10:38:54] 10serviceops: Test running php7.2 and php7.4 in parallel on the beta cluster - https://phabricator.wikimedia.org/T295578 (10JMeybohm) [10:57:17] reimaging 2016 [11:26:27] aaand 2016 done, so the codfw cluster is on bullseye! [11:28:23] hurray \o/ [11:36:28] jayme: in the meantime we could also do the eqiad vms, what do you think? [11:36:53] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Move kubernetes workers to bullseye and docker to overlayfs - https://phabricator.wikimedia.org/T300744 (10elukey) [11:38:26] fine by me. Those won't benefit from the manager (I always imagine that in the layout of the godfather) adding new nodes anyways [11:47:01] exactly yes [14:05:16] jayme: https://gerrit.wikimedia.org/r/c/operations/puppet/+/771600/ and chained (the usual boilerplate stuff for the eqiad vms, no hurry, when you have time) [14:10:13] elukey: the fiters in netboot are pretty confusing :) [14:10:58] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10akosiaris) >>! In T293728#7734355, @elukey wrote: > Important note - we had to modify grub's config to add `systemd.unified_cgroup_hierarchy=0` (the kubelets d... [14:11:29] jayme: I have only simplified them after the last reimages, I can reduce the noise if you want [14:11:38] yeah it's a first match wins thing, it can be confusing [14:11:55] I guess we will simplify them after the migration to bullseye + overlay [14:12:13] and we will also be able to get rid of the puppet lvm module after all of that [14:12:46] which is nice cause it's the only puppet module that is incompatibly licensed to the rest of the modules (upstream or in-house) that we have [14:12:49] that I actually don't find confusing (first match wins), but it does make intentions a bit less clear. I'm all for leaving it like this until all nodes are reimaged, though [14:14:18] done :) [14:14:49] +1ed [14:16:03] since we are all here - after a chat with Janis, I am going to work on operations/debs/istio to have a calico-like deb repository. It will produce the istioctl deb and the istio-cni deb, that we (as ML) will deploy on our clusters (together with some other cni + helmfile config of course) [14:16:44] let me know if there is any issue with it, otherwise I am going to work on it in a bit :) [14:18:29] (I know that Janis is looking forward to review my next patches) [15:00:40] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host kubernetes1018.eqiad.wmnet with OS bullseye [15:00:56] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host kubernetes1019.eqiad.wmnet with OS bullseye [15:28:34] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install parse100[01-24] - https://phabricator.wikimedia.org/T299573 (10Cmjohnson) @akosiaris I can spread the other 3 between B and D if that works better for you? [15:30:52] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host kubernetes1020.eqiad.wmnet with OS bullseye [15:31:10] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host kubernetes1021.eqiad.wmnet with OS bullseye [15:31:55] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host kubernetes1022.eqiad.wmnet with OS bullseye [16:15:04] 10serviceops, 10Prod-Kubernetes: cert-manager created multiple CertificateRequest objects for the same next revision - https://phabricator.wikimedia.org/T304092 (10JMeybohm) p:05Triage→03Medium [16:15:20] 10serviceops, 10Prod-Kubernetes: cert-manager created multiple CertificateRequest objects with the same next revision - https://phabricator.wikimedia.org/T304092 (10JMeybohm) [16:44:27] 10serviceops, 10Prod-Kubernetes: cert-manager created multiple CertificateRequest objects with the same certificate-revision - https://phabricator.wikimedia.org/T304092 (10JMeybohm) [17:04:33] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host kubernetes1018.eqiad.wmnet with OS bullseye executed with erro... [17:05:26] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host kubernetes1018.eqiad.wmnet with OS bullseye [17:07:03] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host kubernetes1019.eqiad.wmnet with OS bullseye executed with erro... [17:07:21] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host kubernetes1019.eqiad.wmnet with OS bullseye [17:08:23] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host kubernetes1021.eqiad.wmnet with OS bullseye executed with erro... [17:08:41] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host kubernetes1021.eqiad.wmnet with OS bullseye [17:08:52] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host kubernetes1022.eqiad.wmnet with OS bullseye executed with erro... [17:09:08] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host kubernetes1022.eqiad.wmnet with OS bullseye [17:09:41] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host kubernetes1020.eqiad.wmnet with OS bullseye executed with erro... [17:09:56] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host kubernetes1020.eqiad.wmnet with OS bullseye [17:30:37] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host kubernetes1018.eqiad.wmnet with OS bullseye completed: - kuber... [17:33:51] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host kubernetes1020.eqiad.wmnet with OS bullseye completed: - kuber... [17:35:43] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host kubernetes1019.eqiad.wmnet with OS bullseye completed: - kuber... [17:36:47] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host kubernetes1021.eqiad.wmnet with OS bullseye completed: - kuber... [17:37:42] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host kubernetes1022.eqiad.wmnet with OS bullseye completed: - kuber... [18:09:57] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Create a partman config for kubernetes masters - https://phabricator.wikimedia.org/T299634 (10elukey) 05Open→03Resolved a:03elukey [18:10:01] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: kube-apiserver need to reach webhooks running inside of the cluster - https://phabricator.wikimedia.org/T290967 (10elukey) [22:21:09] 10serviceops, 10SRE, 10envoy: Refactor envoy max_requests_per_connection from Cluster to HttpProtocolOptions - https://phabricator.wikimedia.org/T304124 (10RLazarus) [22:26:35] 10serviceops, 10SRE, 10envoy: Refactor envoy max_requests_per_connection from Cluster to HttpProtocolOptions - https://phabricator.wikimedia.org/T304124 (10RLazarus) 05Open→03Stalled p:05Triage→03Low [22:26:39] 10serviceops, 10SRE, 10Traffic, 10envoy, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus) [22:35:16] 10serviceops, 10SRE, 10Traffic, 10envoy, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus)