[07:14:32] <wikibugs>	 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10ayounsi) About the Ganeti, as I understand it, the issue is that they have .svc. in their fqdn, while not an SVC IP.  I agree that changing the IP to so...
[07:53:39] <wikibugs>	 10netbox, 10Infrastructure-Foundations, 10SRE: Evaluate Nautobot fork of Netbox and decide whether to use. - https://phabricator.wikimedia.org/T288515 (10Majavah)
[08:02:31] <volans>	 FYI I'm grabbing sretest1001 and 1002 for a last reimage test
[09:00:15] <wikibugs>	 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10akosiaris) >>! In T270071#7404620, @ayounsi wrote: > About the Ganeti, as I understand it, the issue is that they have .svc. in their fqdn, while not an...
[10:46:06] <jbond>	 volans: fyi i rolled out the new ipaddress6 fact .  the only nodes to change where the gitlab servers which now have the correct value 
[10:46:21] <volans>	 and mx1001 I hope :D
[10:48:04] <jbond>	 yes mx now has the correct one as well
[10:49:21] <volans>	 great
[10:49:25] <volans>	 thanks a lot
[10:51:42] <jbond>	 np
[12:09:52] <wikibugs>	 10netops, 10Infrastructure-Foundations: Eqiad Expansion - LVS Connectivity Options - https://phabricator.wikimedia.org/T292630 (10cmooney)
[12:10:03] <wikibugs>	 10netops, 10Infrastructure-Foundations: Eqiad Expansion - LVS Connectivity Options - https://phabricator.wikimedia.org/T292630 (10cmooney) p:05Triage→03High
[12:39:02] <wikibugs>	 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10fgiunchedi) As far as Prometheus goes the `svc` CNAME in PoPs is there for symmetry reasons with eqiad/codfw (where prometheus is LVS'd), so prometheus...
[13:51:53] <wikibugs>	 10Puppet, 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Jelto) I rolled out the puppetised changes to `gitlab2001` (gitlab-replica). Apart from a [minor ssl fix](https://gerrit.wiki...
[14:20:13] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Eqiad Expansion - LVS Connectivity Options - https://phabricator.wikimedia.org/T292630 (10cmooney) @wiki_willy One option that Dell do seem to have are the Mellanox ConnextX-3 and ConnectX-4 QSFP+ based cards.  With the right module we can in theory do 4x10G off...
[14:24:13] <topranks>	 Infra Foundations people... in relation to the above task I'm investigating options for LVS connectivity to the new Eqiad rows.
[14:24:37] <topranks>	 TL;DR is that the existing LVS don't have free PCI slots for additional NICs to connect them the same way as current rows.
[14:25:01] <topranks>	 There are some options here, but one obvious one is to use denser NICs.
[14:25:22] <topranks>	 Either 4xSFP+, or 1xQSFP+ NICs, either of which will give 4x10G off each NIC.
[14:25:38] <topranks>	 Problem is Broadcom don't seem to make such a card, the options are Intel or Mellanox.
[14:25:58] <topranks>	 I'm not sure what the feelings on those are?  I know we standardize on BCM and are comfortable with the drivers etc.
[14:26:08] <topranks>	 Wondering if I should discount the possibility completely or not.
[14:54:52] <volans>	 make sure to get also brandon's feedback on this topic ;)
[14:59:12] <topranks>	 Yep I've asked him as well.  I know there is a lot of tuning etc. on the LVS
[14:59:14] <topranks>	 thanks.
[15:27:49] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Eqiad Expansion - LVS Connectivity Options - https://phabricator.wikimedia.org/T292630 (10BBlack) I'm pretty sure you're right about the 2x PCIe limitation on these servers, unfortunately.  What I'm not so sure about, is whether the (currently BIOS-disabled) "on...
[15:36:32] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Eqiad Expansion - LVS Connectivity Options - https://phabricator.wikimedia.org/T292630 (10BBlack) Also note that lvs1013-16 are ~4 years old now.  I don't think they were scheduled for refresh this year, but they probably would be next year (and by then we would...
[17:17:58] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Eqiad Expansion - LVS Connectivity Options - https://phabricator.wikimedia.org/T292630 (10cmooney) Thanks for the detailed response @BBlack   > And yes, the tunnel option is also risky/complex/painful, but we'll have to weigh that against all the above. Without...
[19:39:57] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Eqiad Expansion - LVS Connectivity Options - https://phabricator.wikimedia.org/T292630 (10cmooney) > I know in the R720s, R730s etc the "on-board" network ports are on a daughterboard which is swappable.  From everything I've seen online this isn't the case with...
[21:06:01] <wikibugs>	 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10ops-codfw: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10Papaul) We are seeing this issue because all those hosts are running an old firmware version for the IDRAC. Upgrading...