[00:41:50] 10netops, 10Infrastructure-Foundations: Netbox DNS changes are not updating - https://phabricator.wikimedia.org/T315630 (10Papaul) [07:03:04] 10netops, 10Infrastructure-Foundations, 10SRE: Netbox DNS changes are not updating - https://phabricator.wikimedia.org/T315630 (10cmooney) Hi @Papaul My apologies that's due to me, as soon as I can get https://gerrit.wikimedia.org/r/c/operations/dns/+/824572 merged I'll sort it. [09:08:50] 10netops, 10Infrastructure-Foundations, 10SRE: Occasional high ICMP probe response from codfw to cr2-drmrs - https://phabricator.wikimedia.org/T315645 (10cmooney) p:05Triage→03Low [09:09:15] 10netops, 10Infrastructure-Foundations, 10SRE: Occasional high ICMP probe response from codfw to cr2-drmrs - https://phabricator.wikimedia.org/T315645 (10cmooney) a:03cmooney [09:26:12] 10netops, 10Infrastructure-Foundations, 10SRE: Netbox DNS changes are not updating - https://phabricator.wikimedia.org/T315630 (10cmooney) 05Open→03Resolved Ok got the +1 and merged. All is good now. ` cmooney@cumin1001:~$ dig +short A kubernetes2024.codfw.wmnet @ns0.wikimedia.org 10.192.48.87 cmooney... [09:29:55] 10netops, 10Infrastructure-Foundations, 10SRE: Occasional high ICMP probe response from codfw to cr2-drmrs - https://phabricator.wikimedia.org/T315645 (10cmooney) [11:47:12] 10netops, 10Infrastructure-Foundations, 10SRE: Occasional high ICMP probe response from codfw to cr2-drmrs - https://phabricator.wikimedia.org/T315645 (10cmooney) Ok so I got some results back. Firstly 10,000 pings to cr1-drmrs from bast2002, starting at 08:14 UTC. Average RTT was 118ms, worst was 154ms: `... [11:54:54] 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE: Allow jumbo frames between cloud hosts in production realm - https://phabricator.wikimedia.org/T315446 (10dcaro) That seemed to do the trick yes! Thanks! [13:55:57] 10netops, 10Infrastructure-Foundations, 10SRE: Netbox DNS changes are not updating - https://phabricator.wikimedia.org/T315630 (10Papaul) @cmooney thank you my changes are now on the DNS server. [15:13:28] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10Sustainability (Incident Followup), 10cloud-services-team (Kanban): Puppet labs/private.git data loss incident affecting some projects - https://phabricator.wikimedia.org/T254491 (10Krinkle) [15:25:52] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic: Users of Jio ISP (India, AS 55836) unable to reach Wikimedia sites - https://phabricator.wikimedia.org/T260449 (10Krinkle) [15:26:03] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Sustainability (Incident Followup): clean up workaround and measurements put in place during Jio RPKI error - https://phabricator.wikimedia.org/T260452 (10Krinkle) [16:10:30] 10netops, 10Infrastructure-Foundations, 10SRE: Overlay VRF / VXLAN traffic failure between lsw1-f2-eqiad and lsw1-f3-eqiad - https://phabricator.wikimedia.org/T315038 (10cmooney) p:05High→03Low Thanks yep case opened with JTAC now will keep it open to document any information they may provide. [17:44:57] 10netops, 10Infrastructure-Foundations, 10Observability-Metrics, 10SRE, and 2 others: LibreNMS seemingly not collecting data for many ports after migration to netmon1003 - https://phabricator.wikimedia.org/T314972 (10CDanis) Looks like this is resolved...? [19:43:30] 10netops, 10Infrastructure-Foundations, 10Observability-Metrics, 10SRE, and 2 others: LibreNMS seemingly not collecting data for many ports after migration to netmon1003 - https://phabricator.wikimedia.org/T314972 (10andrea.denisse) 05Open→03Resolved