[06:44:58] 10netops, 10Infrastructure-Foundations, 10SRE, 10decommission-hardware, 10ops-eqiad: Decommission asw-c-eqiad - https://phabricator.wikimedia.org/T208734 (10ayounsi) 05Resolved→03Open One report is alerting with: > asw-c1-eqiad connected console ports attached to unracked device asw-c1-eqiad: consol... [07:01:44] XioNoX: there's a notification from Lumen about an upcoming deep sea operation, they're gonna start working on circuit BDFS2448 on August 25th [07:02:15] how long is the maintenance? [07:02:22] maximum 13 days (!) [07:02:27] https://groups.google.com/u/0/a/wikimedia.org/g/ops-maintenance/c/_MtAXMz40Z0 [07:03:42] that's no fun [07:06:42] XioNoX: do you want me to open a ticket to track this, or in general is there anything else I can do other than adding the maintenance to the calendar? [07:10:09] ema: I added it to https://phabricator.wikimedia.org/T288503#7280774 [07:10:27] yeah, putting it on the calendar is the only thing we can do for now [07:10:28] thx [07:10:49] ack [08:25:21] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Unable to load en.wikipedia.org from 84.19.61.192/26 - https://phabricator.wikimedia.org/T279503 (10A189605) Apologies for the delay in responding. We are taking this up with our ISP as we do not see any traffic coming back from them when capturing... [09:24:16] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Unable to load en.wikipedia.org from 84.19.61.192/26 - https://phabricator.wikimedia.org/T279503 (10cmooney) The best we can probably do is share some screenshots from our Netflow collectors, showing outbound traffic to your range. Please note our... [14:03:28] 10Traffic, 10SRE, 10Patch-For-Review: Deploy Wikidough: Experimental DNS-over-HTTPS (DoH) public resolver - https://phabricator.wikimedia.org/T252132 (10ssingh) [14:04:00] 10Traffic, 10SRE: Offer Wikidough as an anycasted service - https://phabricator.wikimedia.org/T283027 (10ssingh) 05Open→03Resolved Marking this as resolved as we are now offering Wikidough from all our PoPs as an anycasted service. Thanks to @bblack, @ayounsi, and @cmooney for all their help! [14:11:56] sukhe: wikimedia-dns.org is pure anycast, no geodns or anything like that, right? I'm somehow routed from Finland to eqiad even though my ping time to esams is like ~100ms shorter :/ [14:24:56] majavah: yep, pure anycast. what does "kdig wikipedia.org @wikimedia-dns.org +tls-ca +nsid" say for you? (package knot-dnsutils). you should be hitting esams, so NSID should either doh3001 or doh3002 from Finland (before we get into traceroutes) [14:25:23] sukhe: doh1001 [14:26:04] that's ... interesting. how is your stub configured like? [14:27:55] stub? [14:28:10] possibly stupid question, but no VPNs right? :) [14:29:19] nope, just my normal home connection [14:32:55] please do share the output of 'mtr -zwb wikimedia-dns.org' in a DM, if you are comfortable. thanks! [14:36:28] 10:27:55 < majavah> stub? [14:37:06] I was curious how have you configured Wikidough in your stub resolver (like Firefox) but it doesn't matter because if the output of the kdig above is doh1001, you are hitting that irrespective of your local configuraiton [14:41:07] sukhe: https://phabricator.wikimedia.org/P17020 [14:44:49] majavah: thanks! [14:45:13] very interesting indeed [14:46:14] sukhe: and edited it to also include traceroute to doh3001, if that's interesting or helpful at all [14:48:45] it is, thanks. I will check and message you when I have some more information as to why this is happening [14:50:16] from a host in Finland (and just to confirm if something else is not broken), I am hitting esams from there, as I should [15:36:48] 10netops, 10Infrastructure-Foundations: Link failure between mr1-eqiad and asw2-a8-eqiad Aug 13th 2021 - https://phabricator.wikimedia.org/T288834 (10cmooney) [15:41:15] 10netops, 10Infrastructure-Foundations: Link failure between mr1-eqiad and asw2-a8-eqiad Aug 13th 2021 - https://phabricator.wikimedia.org/T288834 (10cmooney) [16:39:09] 10netops, 10Infrastructure-Foundations, 10SRE: Link failure between mr1-eqiad and asw2-a8-eqiad Aug 13th 2021 - https://phabricator.wikimedia.org/T288834 (10cmooney) Seems I was incorrect about how the switches manage logs. It keeps older log files similar to a typical unix system, gzips them up. Logs from... [16:42:51] 10netops, 10Infrastructure-Foundations, 10SRE: Link failure between mr1-eqiad and asw2-a8-eqiad Aug 13th 2021 - https://phabricator.wikimedia.org/T288834 (10cmooney) Logs from mr1-eqiad: ` Aug 13 14:28:53 mr1-eqiad rpd[1456]: RPD_OSPF_NBRUP: OSPF neighbor 208.80.154.204 (realm ospf-v2 ge-0/0/1.401 area 0.0.... [16:55:08] 10netops, 10Infrastructure-Foundations: Traffic Engineering for Anycast Ranges - https://phabricator.wikimedia.org/T288843 (10cmooney) [16:57:58] 10netops, 10Infrastructure-Foundations: Traffic Engineering for Anycast Ranges - https://phabricator.wikimedia.org/T288843 (10cmooney)