[08:51:30] 10Puppet, 10Infrastructure-Foundations: Search broken on puppetboard - https://phabricator.wikimedia.org/T304484 (10ayounsi) p:05Triage→03Low [11:07:38] 10Puppet, 10Infrastructure-Foundations: Search broken on puppetboard - https://phabricator.wikimedia.org/T304484 (10Volans) Yes that seems a typo upstream, IIRC I reported that to John a while ago, not sure if it was fixed upstream by now. [11:14:43] 10netops, 10Infrastructure-Foundations: IPv6 BFD Sessions Failing from Bird (Anycast VMs) to Juniper QFX in drmrs - https://phabricator.wikimedia.org/T304501 (10cmooney) p:05Triage→03Low [11:25:58] 10netops, 10Infrastructure-Foundations: IPv6 BFD Sessions Failing from Bird (Anycast VMs) to Juniper QFX in drmrs - https://phabricator.wikimedia.org/T304501 (10cmooney) For comparisons sake the session from cr1-codfw to doh2001 are up, and using multi-hop mode. These are similar to drmrs in that one side is... [11:44:10] 10Puppet, 10Infrastructure-Foundations: Search broken on puppetboard - https://phabricator.wikimedia.org/T304484 (10jbond) 05Open→03Resolved a:03jbond I have deployed an update which has fixed this, please reopen if i missed something [13:35:18] 10netbox, 10Infrastructure-Foundations, 10IPv6, 10User-jbond: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK) - https://phabricator.wikimedia.org/T253173 (10fgiunchedi) [13:41:02] 10netbox, 10Infrastructure-Foundations, 10IPv6, 10User-jbond: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK) - https://phabricator.wikimedia.org/T253173 (10ayounsi) FYI, the full list up to date is on https://netbox.wikimedia.org/extras/reports/network.Network/ [14:20:02] 10netops, 10Infrastructure-Foundations, 10SRE: IPv6 BFD Sessions Failing from Bird (Anycast VMs) to Juniper QFX in drmrs - https://phabricator.wikimedia.org/T304501 (10cmooney) Thinking about this further I think it works from the CRs because the peering is from the local public/private subnet to the loopbac... [18:24:49] 10netops, 10Infrastructure-Foundations: Unify loopback filters between CR routers and L3 switches - https://phabricator.wikimedia.org/T304553 (10cmooney) p:05Triage→03Medium [18:25:07] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Patch-For-Review: drmrs: initial geodns configuration - https://phabricator.wikimedia.org/T304089 (10BBlack) PT was pretty smooth, ES likely to be later today, closer to when their daily traffic cycle begins to trend downwards. [18:32:36] 10netops, 10Infrastructure-Foundations, 10Patch-For-Review: Unify loopback filters between CR routers and L3 switches - https://phabricator.wikimedia.org/T304553 (10cmooney) The diff in the above patch is relatively straightforward on the CRs, basically the same as you can see looking at the policy changes:... [18:36:04] 10netops, 10Infrastructure-Foundations, 10Patch-For-Review: Unify loopback filters between CR routers and L3 switches - https://phabricator.wikimedia.org/T304553 (10cmooney) In drmrs the changes are more involved, and it's a little harder to read the homer diff. Ultimately they are only small and shouldn't... [18:46:57] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10Jclark-ctr) [18:47:30] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10Jclark-ctr) [18:48:44] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10Jclark-ctr) @cmooney i have connected spine switches to scs and updated netbox [19:20:44] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unify loopback filters between CR routers and L3 switches - https://phabricator.wikimedia.org/T304553 (10cmooney) To clarify the 'port' isn't an option on QFX even for UDP, although it allows you to define a term with that. So I've changed... [19:41:36] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10cmooney) @jclark-ctr super thanks for that! I'll open a task and start planning how we take care of the move. [20:01:33] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10cmooney) @Jclark-ctr I'm not getting any output on port 20 or 29 of the scs-f8. Are the two Junipers powered on? If not can you double c...