[00:38:58] 10Mail, 10Infrastructure-Foundations, 10MediaWiki-Email, 10MW-1.35-notes, and 6 others: Incorrect email headers using PHP mail in PHP 8.0 - https://phabricator.wikimedia.org/T320864 (10Reedy) [10:22:01] 10CFSSL-PKI, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10serviceops, and 2 others: Update cfssl-issuer to cert-manager 1.8.x - https://phabricator.wikimedia.org/T310486 (10JMeybohm) 05Open→03Resolved [11:23:47] hello netops folks! I wanted to ask for some help in configuring the new LVS hosts, lvs4008-10 [11:24:15] specifically the Netbox side of things, including the switch configuration and things like VLAN tagging [11:24:46] I am happy taking care of the others if someone could run down even one host for me but I want to make sure we get it correct, including the ordering of the setup [11:25:09] I have been told last time it was topranks whose help was invaluable in setting these up :) [11:26:00] if there is documentation to read, I am happy to do that as well [11:26:03] thanks! [11:43:03] sukhe: hey, I’ve an errand to run but I’ll take a look in an hour or so when I’m back? [11:43:21] Should be simple enough. If there is some documentation that’s worth adding we can look at that too [11:43:21] topranks: of course, thanks, not urgent please! [11:43:40] I am very happy to help in any way so as to make it easier for all of us too :P [12:24:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet - https://phabricator.wikimedia.org/T296411 (10aborrero) [12:26:06] 10netops, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): connect 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10aborrero) 05Stalled→03Declined It seems we'll be working on {T297596} instead. [12:26:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet - https://phabricator.wikimedia.org/T296411 (10aborrero) [12:31:25] toprank.s: heading out for breakfast + family stuff. I will be back online at around 9AM so in 1.5hours [13:52:20] sukhe: drop me a line when you're back, I've made the netbox changes and pushed the config to the switch, so I think you should be set to continue to deploy those. [14:16:40] hi topranks! [14:16:42] here and checking [14:17:46] topranks: https://netbox.wikimedia.org/dcim/devices/4456/interfaces/ we were talking about this yesterday [14:18:12] eno1 should be deleted? [14:18:47] yeah I was looking at that [14:19:13] I added a trunk to the public vlan in ulsfo to the other interface [14:19:38] which I think is all that is needed, that link is already connected to private vlan, and with change it can also reach back-ends on the private vlan [14:19:54] unlike say, codfw or eqiad, there is no other 'row' that needs a second interface [14:20:09] so eno1 does look like it's not needed - from my understanding anyway [14:20:49] In terms of deleting from Netbox we should talk to rob / dc-ops and get the cable removed if it's there, until that's physically removed we should leave it documented in Netbox [14:20:49] XioNoX said the same, so there's a confirmation at least :) [14:21:03] +1 [14:21:33] ok so first the cable gets removed then we delete from Netbox [14:21:40] I guess I can pick some other host to get started then [14:21:57] we can still move forward with the provisioning [14:22:03] as the switch port is disabled [14:22:06] ok... well we can "disable" the interfaces in Netbox in the interim also [14:22:26] also the puppetdb import would re-import eno1 if it's physically on the host, just without attributes [14:22:53] volans: yeah, the interface exsits, it's just the cable that is superfluous [14:23:25] k [14:23:37] also there is something weird with the eno1 being 1G and the switch port being 10G [14:24:33] good point [14:25:04] that should be a netbox report too :) or Custom Model Validation [14:27:24] topranks: since you did this last time, do you remember if there is a particular order of things too? [14:27:40] like should I push the updated homer change with the new lvs host and IP and then provision the host? [14:27:45] the other way around? does it matter? [14:27:57] since it's LVS, just wanted to be sure, if you remember what you did last time with brandon [14:28:09] sukhe: I'm not 100% sure, I think we need to probably do it in the order we've done it now [14:28:23] yeah that's OK [14:28:31] thanks for the help, much appreciated [14:28:37] not surprisingly, I was quite lost with this :) [14:28:37] i.e. manually change the switch port to trunk mode as I did, then you can push the updated homer change [14:29:14] ok, seems similar to what we have done in the past [14:29:16] XioNoX, volans: I believe the 1G speed on that is because the interface is down, puppet facts show the speed as "-1" [14:29:17] (for other stuff) [14:29:29] which means the import script leaves it as default, 1G [14:29:52] if it came up, puppet would report 10G, and import script would set it to that [14:29:54] topranks: I looked with ethtool yesterday and it says 1G [14:30:13] usually the onboard interfaces are named enoX and at 1G [14:30:21] vs. expansion cards 10G [14:30:50] ok yeah was gonna check ethtool too [14:32:11] Switch side / cable was probably manually added then? Just an error selecting 10G? [14:32:43] I'd bet on entry mistake, yeah [14:41:25] XioNoX: bit of an issue allocating IPs for the new LVS in ulsfo [14:41:31] no free IPs in the public subnet [14:41:32] https://netbox.wikimedia.org/ipam/prefixes/13/ip-addresses/ [14:42:00] There are 2 "reserved for infra" though, and the old LVS will be decommissioned when the new ones are ready, so we can maybe juggle some stuff around [14:42:53] yeah I guess it's because of the migration? [14:44:07] makes sens to use the reserved ones [14:44:08] yeah 3 new LVS going in, replacing 3 existing. [14:45:05] lvs4005, 6, 7 going and 8, 9, 10 coming (ulsfo hardware refresh) [14:45:38] but [5-7] are still racked because it's LVS so we can't take the existing ones out without the new ones commissioned [15:33:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin2002 for host cloudgw2002-dev.codfw.wmnet with... [16:01:27] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [16:01:41] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [16:04:00] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin2002 for host cloudgw2002-dev.codfw.wmnet with OS...