[08:34:45] 10Traffic, 10SRE, 10Patch-For-Review: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [08:58:22] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10dcaro) [09:10:37] vgutierrez: hey, I am going to change the netmask on IP 10.2.2.87 from /24 to /32 in Netbox [09:11:34] or, actually looking at it, this IP is listed twice in Netbox right now: [09:11:35] https://netbox.wikimedia.org/ipam/ip-addresses/?q=10.2.2.87 [09:11:56] our netbox coherance report is alerting on the one with the /24 netmask, as others in the range are defined as /32 [09:12:38] (don't worry for now what netmask you are using on the server) [09:13:01] My thinking is we need to remove one of those IP entries, the /24 one I expect. [09:14:05] It's not clear to me if there is just a duplicate entry, or if actually it's being used in error in two separate places? [09:15:54] I see h.erron allocated the "pyrra.svc.eqiad.wmnet" one, which makes me think perhaps this isn't "same thing documented twice" but rather a genuine duplicate allocation? [09:22:00] Uh... [09:22:09] That's definitely an issue [09:22:24] I allocated 10.2.2.87 on netbox but not for pyrra [09:25:17] * vgutierrez double checking [09:25:44] https://netbox.wikimedia.org/ipam/ip-addresses/13774/ [09:25:50] ^^ that's mine [09:26:13] I totally missed the /24 prefix, sorry about that [09:27:02] and yes, it's a duplicated allocation [09:29:15] cc elukey, klausman ^^^ [09:29:37] (I saw a similar discussion on a CR) [09:30:57] yeah I was following up, we want to allocate .88 [09:33:57] vgutierrez: all good [09:35:12] I guess we just need a new IP for one of the services, not sure which is easier to change [09:35:33] the mask should be /32 on both when that's done. thanks! [09:37:13] = [09:37:21] happy to drop mine if needed [09:37:42] as it's only used in a few static routes in lvs1013-1016 [09:41:14] Yeah whichever is easiest, if yours isn't difficult to do then we can assign a new one [09:41:52] We can just click 'edit' on the link you pasted above and change it to 10.2.2.89/32, which is free [09:41:57] then run dns cookbook [09:42:59] it doesn't look like it is urgent as the 10.2.2.87 is not in the CR routing tables at all right now [09:43:13] so I don't think the duplication is breaking anything, so not urgent [10:15:55] since we are getting radio silence from the other side I'll take care of it from my side [10:16:52] topranks: https://netbox.wikimedia.org/ipam/ip-addresses/13774/ looking good now? :) [10:27:59] vgutierrez: yep looks perfect thanks [10:28:08] please see -sre :) [10:47:00] Hello. I have a quick DNS question. Reference: https://phabricator.wikimedia.org/T346165#9219409 - I'm getting inconsistent case and zero-padding for internal IPv6 address resolution. [10:47:02] Have we seen this before? [10:48:03] They're coming from `dnslookup::query` functions in puppet. The data is consistent, but the formatting is not and it's causing puppet to report unnecessary changes. Thanks. [11:45:20] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10dcaro) [11:55:06] 10Traffic, 10SRE, 10Patch-For-Review: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [12:03:14] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Cabling for Eqiad racke E5-7 and F5-7 - https://phabricator.wikimedia.org/T334231 (10ayounsi) [12:05:44] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox Juniper report - https://phabricator.wikimedia.org/T306238 (10ayounsi) Asked Juniper about their timeline on getting this setup. [12:06:29] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, and 2 others: Netflow/pmacct: use forwardingStatus - https://phabricator.wikimedia.org/T331707 (10JAllemandou) [12:35:22] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10netbox: Avoid ghost hosts on the network - https://phabricator.wikimedia.org/T306007 (10ayounsi) I think here the only/best option is to reduce the time delta between when a server is connected and when switch port is configured (line `Run the sr... [13:05:15] 10Traffic, 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-esams: Q4/Q1:knams racking elevations & planning - https://phabricator.wikimedia.org/T331886 (10ayounsi) 05Open→03Resolved a:03RobH I believe this is all done. [14:04:24] 10Traffic, 10SRE, 10Patch-For-Review: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [14:16:29] 10Traffic, 10SRE: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [14:36:03] 10Traffic, 10SRE: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [14:47:38] btullis: unsure about the zero-padding but the case differences may be related to this: [14:48:05] https://www.theregister.com/2023/01/19/google_dns_queries/ [14:49:44] TL;DR resolvers should return answers with same case as the query, some clients randomise the case when querying to decrease chance of a successful spoofed response [14:50:02] I am unsure of puppet dnsquery does but it might explain [14:52:59] topranks: Interesting, thanks. [15:07:07] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools: Improve Homer output when Juniper device rejects config - https://phabricator.wikimedia.org/T328747 (10ayounsi) 05Open→03Resolved Homer 0.6.4 released. [15:17:11] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10VRiley-WMF) cp1112 - D 2. U 1. CableID 20220171 port 21 cp1113 - D 4. U 29 CableID 230304500241 port 6 cp1114 - D 4. U 38 CableID 230304500243 port 8 cp1115 - D 7. U 20 CableID 2303045... [15:35:10] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10RobH) [18:16:56] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Remove static routes for ns[01] and replace their announcements with bird - https://phabricator.wikimedia.org/T348041 (10ssingh) [18:17:09] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Remove static routes for ns[01] and replace their announcements with bird - https://phabricator.wikimedia.org/T348041 (10ssingh) [18:17:15] 10Traffic, 10SRE, 10Patch-For-Review: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10ssingh) [18:17:35] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Remove static routes for ns[01] and replace their announcements with bird - https://phabricator.wikimedia.org/T348041 (10ssingh) p:05Triage→03Medium [18:31:58] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Remove static routes for ns[01] and replace their announcements with bird - https://phabricator.wikimedia.org/T348041 (10ssingh) We can and probably should have a backup static routes for each of `ns[01]` but it can be to a single host instead of al... [19:52:33] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Remove static routes for ns[01] and replace their announcements with bird - https://phabricator.wikimedia.org/T348041 (10BBlack) Looks about right to me! [20:17:30] 10Traffic, 10SRE, 10Patch-For-Review: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [21:49:20] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10netbox: Avoid ghost hosts on the network - https://phabricator.wikimedia.org/T306007 (10wiki_willy) ++ @Papaul , who's going to dig around a bit and provide some feedback