[07:53:26] 10netops, 06Infrastructure-Foundations, 06SRE: magru network setup - https://phabricator.wikimedia.org/T362421#9807801 (10ayounsi) Before advertising ns2, we need to do some traffic engineering. Telxius being part of Spain's main ISP, Telefonica ES prefers magru to drmrs : See https://w.wiki/A6qH {F53575207}... [07:58:36] cdanis: I'm curious about Chile (and overall west coast), does traffic cross through the land, or does it take the scenic oceanic route ? [09:26:24] 10netops, 06Infrastructure-Foundations, 06SRE: magru network setup - https://phabricator.wikimedia.org/T362421#9808055 (10cmooney) +1 sounds like a good idea. Nice we have some limited scope to experiment with the DoH ranges before pulling the plug on ns2. FWIW I think these would be the ones to use with E... [10:08:32] 06Traffic, 06Infrastructure-Foundations, 06SRE: Slowly ramping up traffic to the Brazil data center (magru) and related geo-maps - https://phabricator.wikimedia.org/T359054#9808106 (10cmooney) >>! In T359054#9807307, @CDanis wrote: > Adding the 3rd transit link in magru **greatly** improved the latency for m... [10:32:52] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Problem re-imaging hosts on row-wide vlan on EVPN switches - https://phabricator.wikimedia.org/T365204#9808187 (10cmooney) Pcap of DHCP request from contint2002 here: {F53586857} [10:35:19] 10netops, 06Infrastructure-Foundations, 06SRE: magru network setup - https://phabricator.wikimedia.org/T362421#9808194 (10ayounsi) Cogent is a bit surprising, from EU or the US they route to magru. `lines=15 Fri May 17 10:29:23.898 UTC BGP routing table entry for 185.71.138.0/24 Versions: Process... [10:42:19] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Problem re-imaging hosts on row-wide vlan on EVPN switches - https://phabricator.wikimedia.org/T365204#9808229 (10cmooney) One observation is that the NAK's are unique in so far as they are sent from 208.80.153.33 (Switch IRB int IP) to 255.255.25... [10:49:32] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Problem re-imaging hosts on row-wide vlan on EVPN switches - https://phabricator.wikimedia.org/T365204#9808246 (10cmooney) Also I didn't see in the dhcpd docs and way to constrain the generation of NAKs in response to invalid REQUEST messages. [... [10:59:47] 10netops, 06Infrastructure-Foundations, 06SRE: magru network setup - https://phabricator.wikimedia.org/T362421#9808254 (10cmooney) >>! In T362421#9808194, @ayounsi wrote: > They might prefer going through EdgeUno once we add the prepending to Novvacore, so the same change would be needed there as well. It's... [12:52:47] XioNoX: I'm still messing with the plot but the 3rd transit was a big improvement for both Paraguay and Chile as well [13:17:13] 10netops, 06Infrastructure-Foundations, 06SRE: magru network setup - https://phabricator.wikimedia.org/T362421#9808627 (10ayounsi) The Telxius community doesn't seem to be of any effect so far, I'll wait for their reply, maybe they changed or need to be enabled on their side first. I'll look at the other pro... [13:24:56] 10netops, 06Infrastructure-Foundations, 06SRE: Support Anycast GW on EVPN switches without unique IP - https://phabricator.wikimedia.org/T350579#9808649 (10cmooney) Just a note on this, I only discovered this document after the task: https://www.juniper.net/documentation/us/en/software/nce/nce-216-evpn-... [13:28:56] 06Traffic: Craft geo-maps file to create lowest-latency routes from south america - https://phabricator.wikimedia.org/T363722#9808676 (10CDanis) The 3rd transit was also of great help to Chile, and probably Peru (although sample size there is a bit small). {F53600438} [13:29:12] 06Traffic, 06Infrastructure-Foundations, 06SRE: Slowly ramping up traffic to the Brazil data center (magru) and related geo-maps - https://phabricator.wikimedia.org/T359054#9808679 (10CDanis) The 3rd transit was also of great help to Chile, and probably Peru (although sample size there is a bit small). {F53... [13:29:44] btw I don't really understand the difference between T359054 and T363722, should they be merged? [13:29:47] T363722: Craft geo-maps file to create lowest-latency routes from south america - https://phabricator.wikimedia.org/T363722 [13:31:09] I think the intended distinction was: T359054 is for the intial turning up and T363722 was for the rest of the geo-maps [13:31:37] I think your updates make sense for both in a way but probably T363722 is better? [13:31:53] and thanks for running it again <3 [13:32:35] sukhe: this is so fun [13:32:47] I'm doing something close to real science right now [13:32:48] :) [13:34:08] next step after "real science" is p-hacking right? :) [14:20:36] 06Traffic, 06Movement-Insights: Disable Chrome Private Prefetch Proxy - https://phabricator.wikimedia.org/T364126#9808854 (10akosiaris) >>! In T364126#9806926, @BBlack wrote: > >>>! In T364126#9805638, @akosiaris wrote: >> * They do have a lot of presence all over the world. Presence we don't have currently a... [15:01:30] 06Traffic, 06Movement-Insights: Disable Chrome Private Prefetch Proxy - https://phabricator.wikimedia.org/T364126#9809001 (10BBlack) What a fun deep-dive! :) >>! In T364126#9808854, @akosiaris wrote: >> In the case of a mispredicted prefetch to upload, we waste resources sending an image that is ultimately d... [15:21:07] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE, 13Patch-For-Review: Problem re-imaging hosts on row-wide vlan on EVPN switches - https://phabricator.wikimedia.org/T365204#9809120 (10cmooney) Re-reading the man page for dhcpd.conf it seems that pontentially changing the 'authoritative' stateme... [15:46:54] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE, 13Patch-For-Review: Problem re-imaging hosts on row-wide vlan on EVPN switches - https://phabricator.wikimedia.org/T365204#9809234 (10cmooney) From what I can tell the 'authoritative' statement only controls NAK generation. I think we're hittin... [15:53:31] 06Traffic, 06Movement-Insights: Disable Chrome Private Prefetch Proxy - https://phabricator.wikimedia.org/T364126#9809242 (10akosiaris) >>! In T364126#9809001, @BBlack wrote: > > - If I'm digging in raw webrequest data for upload.wikimedia.org fetches to answer some question about our image traffic: I can c... [19:14:46] 10netops, 06Infrastructure-Foundations, 10ops-eqiad: partial power outage for lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365289 (10CDanis) 03NEW [19:14:58] 10netops, 06Infrastructure-Foundations, 10ops-eqiad: partial power outage for lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365289#9810003 (10CDanis) p:05Triage→03High [19:16:45] 06Traffic, 06Data-Persistence, 06SRE, 10SRE-swift-storage, and 6 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914#9809948 (10Jdlrobson) [21:01:21] 06Traffic: Craft geo-maps file to create lowest-latency routes from south america - https://phabricator.wikimedia.org/T363722#9810239 (10CDanis) Latest results: magru is a clear win for BR, AR, CL, PY, UY, BO This adds BO to the "clear win" set. I am guessing this is another consequence of the 3rd transit link... [21:01:27] 06Traffic, 06Infrastructure-Foundations, 06SRE: Slowly ramping up traffic to the Brazil data center (magru) and related geo-maps - https://phabricator.wikimedia.org/T359054#9810241 (10CDanis) Latest results: magru is a clear win for BR, AR, CL, PY, UY, BO This adds BO to the "clear win" set. I am guessing...