[03:24:46] 10Puppet, 10Beta-Cluster-Infrastructure, 10Infrastructure-Foundations: Beta mwmaint puppet runs fail with "Resource type not found: Profile::Lvs::Classes" - https://phabricator.wikimedia.org/T322901 (10Tgr) [03:51:12] 10Puppet, 10Infrastructure-Foundations, 10Beta-Cluster-reproducible: Beta mwmaint puppet runs fail with "Resource type not found: Profile::Lvs::Classes" - https://phabricator.wikimedia.org/T322901 (10Tgr) `modules/profile/types/lvs/classes.pp` is physically not present on deployment-puppetmaster04. Which wou... [09:10:17] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox, and 2 others: Fix rule violation in the lvs balancer role - https://phabricator.wikimedia.org/T264132 (10jbond) it would be useful to understand why theses changes where reverted to avoid issues in the future. [09:46:08] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin2002 for host cloudvirt2002-dev.codfw.wmnet wi... [10:52:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin2002 for host cloudvirt2002-dev.codfw.wmnet with O... [11:42:47] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin2002 for host cloudvirt2002-dev.codfw.wmnet wi... [11:51:38] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin2002 for host cloudvirt2002-dev.codfw.wmnet with O... [11:54:10] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin2002 for host cloudvirt2002-dev.codfw.wmnet wi... [12:37:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin2002 for host cloudvirt2002-dev.codfw.wmnet with O... [12:46:00] hi netops folks! [12:46:11] we set up the new LVS host yesterday; thanks to topranks for his help [12:46:22] on Monday, we will move lvs4005 to lvs4008 [12:46:36] that requires updating the static fallback routes [12:46:52] route 198.35.26.96/28 { next-hop 10.128.0.15 [12:46:59] this points to lvs4005, we will then need to point it to lvs4008 [12:47:13] if you can tell me how to do it, I am happy taking care of it for this and then other refreshes coming up, including in eqsin [12:47:21] essentially the plan is: [12:47:29] - update the static fallback routes [12:47:35] - move traffic to lvs4008 [12:47:37] - decom 4005 [12:51:24] (I do have cr access) [12:56:49] sukhe: great stuff! [12:56:57] In general changing the statics is something like this: [12:56:58] https://phabricator.wikimedia.org/P39250 [12:57:20] Unfortunately it's not automated via homer right now so it's a manual change. [12:57:45] I'm more than happy to take a look Monday anyway, just in case there are any issues [12:57:48] so feel free to ping me [13:18:13] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [13:30:41] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin2002 for host cloudvirt2003-dev.codfw.wmnet with OS bullseye [14:07:40] thanks topranks! [14:07:54] I will follow the paste on Monday and let you know if there are any issues [14:12:38] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin2002 for host cloudvirt2003-dev.codfw.wmnet with OS bullseye completed:... [14:47:04] 10SRE-tools, 10DBA, 10Data-Persistence-Backup, 10Infrastructure-Foundations, 10User-Kormat: Revert workaround for cumin output verbosity on RemoteExecution (CuminExecution) abstraction - https://phabricator.wikimedia.org/T282775 (10Marostegui) What's the status of this task? [14:55:17] 10SRE-tools, 10DBA, 10Data-Persistence-Backup, 10Infrastructure-Foundations, 10User-Kormat: Revert workaround for cumin output verbosity on RemoteExecution (CuminExecution) abstraction - https://phabricator.wikimedia.org/T282775 (10jcrespo) I coded RemoteExecution initially for the backup library. But I... [14:57:10] 10SRE-tools, 10DBA, 10Data-Persistence-Backup, 10Infrastructure-Foundations, 10User-Kormat: Revert workaround for cumin output verbosity on RemoteExecution (CuminExecution) abstraction - https://phabricator.wikimedia.org/T282775 (10Marostegui) Thanks for the update, so it is still a valid task :) [15:08:29] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [17:24:48] 10Puppet, 10Infrastructure-Foundations: Consider alternative configuration managment tooling - https://phabricator.wikimedia.org/T321874 (10jhathaway) > Let me elaborate a little more on my experience in deployment-prep: > * I created a cloud server with cloud-init and my cloud public key, but was permanently... [17:27:11] 10netops, 10Infrastructure-Foundations: Migrate row E/F network aggregation to dedicated Spine switches - https://phabricator.wikimedia.org/T322937 (10cmooney) p:05Triage→03Medium [17:38:31] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10cmooney) @Jclark-ctr bit of a heads up I'm hoping to get the migration kicked off for those Juniper Spine devices now that we've got the lic...