[08:52:24] 10SRE-tools, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 2 others: Write a cookbook to set a k8s cluster in maintenance mode - https://phabricator.wikimedia.org/T277677 (10elukey) a:03elukey [10:05:08] 10puppet-compiler, 10Infrastructure-Foundations: pcc facts import not working as expected - https://phabricator.wikimedia.org/T325053 (10jbond) [12:26:51] 10SRE-tools, 10Infrastructure-Foundations: cookbooks: sre.hosts.reboot-single update to support disabled puppet - https://phabricator.wikimedia.org/T325153 (10jbond) p:05Triage→03Medium [14:09:57] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): Spicerack: Load cookbooks from multiple directories - https://phabricator.wikimedia.org/T325168 (10fnegri) [14:11:04] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (Kanban): Spicerack: Load cookbooks from multiple directories - https://phabricator.wikimedia.org/T325168 (10fnegri) [14:11:20] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (Kanban): Spicerack: Load cookbooks from multiple directories - https://phabricator.wikimedia.org/T325168 (10fnegri) [14:16:43] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (Kanban): WMCS Cookbook Automation Q2 tracking task - https://phabricator.wikimedia.org/T319401 (10fnegri) [14:17:08] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (Kanban): Spicerack: Load cookbooks from multiple directories - https://phabricator.wikimedia.org/T325168 (10fnegri) 05Open→03In progress p:05Triage→03Medium [14:17:36] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (Kanban): Spicerack: Load cookbooks from multiple directories - https://phabricator.wikimedia.org/T325168 (10fnegri) a:03Volans [14:59:29] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, 10serviceops: Agree strategy for Kubernetes BGP peering to top-of-rack switches - https://phabricator.wikimedia.org/T306649 (10JMeybohm) With {T270191} I've changed the zone of k8s ganeti workers to to their respective ganeti cluster and g... [15:06:02] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (Kanban): WMCS Cookbook Automation Q2 tracking task - https://phabricator.wikimedia.org/T319401 (10fnegri) 05Open→03In progress [16:42:37] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (Kanban): WMCS Cookbook Automation Q2 tracking task - https://phabricator.wikimedia.org/T319401 (10fnegri) [16:44:18] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (Kanban): WMCS Cookbook Automation Q2 tracking task - https://phabricator.wikimedia.org/T319401 (10fnegri) [16:44:34] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (Kanban): WMCS Cookbook Automation Q2 tracking task - https://phabricator.wikimedia.org/T319401 (10fnegri) [16:46:28] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): Decide sudoers rules for users without global root - https://phabricator.wikimedia.org/T325067 (10nskaggs) As someone without global root who has been a test case in the past for this, allowing wmcs* cookbook runs for a subset of user... [16:48:01] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (Kanban): WMCS Cookbook Automation Q2 tracking task - https://phabricator.wikimedia.org/T319401 (10fnegri) [16:54:20] moritzm, XioNoX: we have a quick question for you about how the cloudcuminNNNN hosts should SSH to Openstack VMs. Originally we thought of doing a double ProxyJump via the prod bastions, but that requires to add a new user there (as we don't want the cloudcumin* hosts to be able to SSH to the prod bastions as root). [16:54:56] jbond just came up with an alternative idea that is to instead use squid on the webproxies to allow ssh traffic from the cloudcumin hosts without the need to add a user there. [16:55:05] Do you have any insta-thoughts on this? [17:01:35] volans: I think that could work [17:02:11] volans: I didn't know squid supported SSH, we need to be extra careful on the possible limitations [17:02:30] squid becoming of some kind of poor man's NAT [17:03:31] it wouldn;t need to neccesarily support ssh ii would just proxt the tcp connection at layer 4 similar to how tls connections are proxied, the nat comment still stands though [17:09:28] ok, fair. Same, we need to make sure there aren't any limitations that might bite us in the future and how ssh behaves with it [17:10:34] so you would prefer the add user way for now? [17:15:04] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): Decide sudoers rules for users without global root - https://phabricator.wikimedia.org/T325067 (10Volans) Indeed, I agree that we might need later on some more fine-tuned way to authorize things. That said the new cloudcumin setup wil... [17:17:30] the the squid way supported out of the box by ssh or it needs to be wrapped in some L4 tunnels to squid? [17:19:41] XioNoX: it seems easy from https://fedoramagazine.org/configure-ssh-proxy-server/ [17:19:47] but I have zero experience on squid [17:21:23] XioNoX: we would need to using something like "ProxyCommand nc --proxy webproxy:8080 %h %p", instead of "ProxyJump bastion" [21:40:14] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Create Quality of Service design for WMF internal networks - https://phabricator.wikimedia.org/T316358 (10cmooney) @jbond I've uplaoded a separate patch (above) that makes a stab and working this closer to how we discussed earlier. It defi...