[06:57:12] 10SRE-tools, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 3 others: Create a cookbook for depooling one or all services from one kubernetes cluster - https://phabricator.wikimedia.org/T260663 (10JMeybohm) >>! In T260663#6924279, @JMeybohm wrote: > The cookbook does not seem to work (tried du... [09:34:49] 10SRE-tools, 10Spicerack: IcingaHosts.wait_for_downtimed() does not honor dry_run - https://phabricator.wikimedia.org/T315537 (10JMeybohm) p:05Triage→03Medium [11:40:10] fyi im going to do some reboots of sretest1002 [12:19:56] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Link from lsw1-e1-eqiad to lsw1-f2-eqiad down - https://phabricator.wikimedia.org/T315052 (10cmooney) @Cmjohnson @Jclark-ctr can we get someone to visit the DC urgently to look at this? I'm concerned we have a link down between core devices for o... [13:47:59] 10SRE-tools, 10Spicerack: spicerack.dnsdisc.Discovery should not allow pooling active/passive services in both datacenters - https://phabricator.wikimedia.org/T315560 (10JMeybohm) p:05Triage→03Medium [13:54:59] 10SRE-tools, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 4 others: Create a cookbook to perform a rolling reboot of a kubernetes cluster - https://phabricator.wikimedia.org/T260661 (10JMeybohm) [13:58:24] 10SRE-tools, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 4 others: Create a cookbook to perform a rolling reboot of a kubernetes cluster - https://phabricator.wikimedia.org/T260661 (10JMeybohm) Reboot of staging clusters and codfw (batchsize 1, took ~3.25 hours) went smoothly without any al... [13:58:32] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10Spicerack, and 2 others: Spicerack cookbooks TODO list - https://phabricator.wikimedia.org/T203943 (10JMeybohm) [13:58:36] 10SRE-tools, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 4 others: Create a cookbook to perform a rolling reboot of a kubernetes cluster - https://phabricator.wikimedia.org/T260661 (10JMeybohm) 05Open→03Resolved