[07:53:20] 10netops, 10Infrastructure-Foundations, 10SRE: CRs ECMP traffic to LVS VIPs despite higher MED on backup route - https://phabricator.wikimedia.org/T348446 (10ayounsi) Maybe prepending the AS on the backup LVS is easier to do than expected? i though PyBal's development had stopped, but seeing @Vgutierrez 's [... [08:24:37] 10netops, 10Infrastructure-Foundations, 10SRE: CRs ECMP traffic to LVS VIPs despite higher MED on backup route - https://phabricator.wikimedia.org/T348446 (10Vgutierrez) >>! In T348446#9252677, @ayounsi wrote: > Maybe prepending the AS on the backup LVS is easier to do than expected? > i though PyBal's devel... [09:43:46] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10ayounsi) >>! In T348837#9250190, @jhathaway wrote: >[...] Evidently UDP encapsulation may have performance benefits because routers are tuned to support it [...] At the endpoints,... [12:03:41] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10cmooney) Regarding the UDP encapsulation it's an interesting idea, and is a reminder that currently our switches distribute flows based on source internet IP, which gives us lots o... [12:05:40] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Consolidate Automation Templates for DC Switches - https://phabricator.wikimedia.org/T312635 (10cmooney) [12:07:47] Hello, we're looking to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/962250 . Were adding 3 new druid servers that have been setup and brought into service in the druid_public cluster. This change adds them to the druid-public-broker VIP. requesting a review and some help with PyBal. [12:46:12] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977 (10cmooney) p:05Triage→03Low [12:52:52] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977 (10cmooney) [12:59:48] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10ayounsi) Regarding MTU. We MUST NOT need to fragment any v4 packet. And MUST reduce the need of IPv6 PMTUD as much as possible. There are 2 main options: 1/ increase the MTU on a... [13:09:43] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10BBlack) Could we take the opposite approach with the MTU fixup for the tunneling, and arrange the host/interface settings on both sides (the LBs and the target hosts) such that the... [13:16:41] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10cmooney) > There are 2 main options: > 2/ Decrease the MSS on the realservers (any host were a tunnel can terminate) > In a TCP handshake each side tells its peer what its MSS is,... [13:28:01] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10cmooney) >>! In T348837#9253673, @BBlack wrote: > If per-route MTU can usefully be set higher than base interface MTU, this seems trivial, While you can set MTUs on a route, afa... [13:58:53] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10Vgutierrez) >>! In T348837#9253425, @cmooney wrote: > Regarding the UDP encapsulation it's an interesting idea, and is a reminder that currently our switches distribute flows based... [14:21:36] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10BBlack) >>! In T348837#9253720, @cmooney wrote: > The one thing you may not be able to control with mtu/advmss on a route is traffic to the local subnet, as that route is added by... [14:37:23] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10Vgutierrez) BTW This is also the approach recommended by Katran >>! In T348837#9253591, @ayounsi wrote: > So my preference here would go to option (2) limit the MSS on the relevant... [14:46:32] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10BBlack) One potential issue with relying solely on MSS reduction is that, obviously, it only affects TCP. For now this is fine, as long as we're only using LVS (or future liberica... [14:48:18] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10Vgutierrez) for QUIC there are ongoing efforts like https://datatracker.ietf.org/doc/draft-pskim-passive-probing-pmtud/ [14:48:35] 10Traffic, 10Content-Transform-Team-WIP, 10RESTBase, 10RESTBase Sunsetting, and 7 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365 (10MSantos) [14:48:54] 10Traffic, 10Content-Transform-Team-WIP, 10RESTBase, 10RESTBase Sunsetting, and 7 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365 (10MSantos) [14:49:13] 10Traffic, 10Content-Transform-Team-WIP, 10RESTBase, 10RESTBase Sunsetting, and 7 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365 (10MSantos) p:05Triage→03High [14:49:21] 10Traffic, 10Content-Transform-Team-WIP, 10RESTBase, 10RESTBase Sunsetting, and 7 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365 (10MSantos) [14:49:51] 10Traffic, 10Content-Transform-Team-WIP, 10RESTBase, 10RESTBase Sunsetting, and 7 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365 (10MSantos) [14:51:12] bblack, vgutierrez: interesting, looks like they just hardcode a max packet size: https://groups.google.com/a/chromium.org/g/proto-quic/c/uKWLRh9JPCo [14:56:36] 10netops, 10Infrastructure-Foundations, 10SRE: CRs ECMP traffic to LVS VIPs despite higher MED on backup route - https://phabricator.wikimedia.org/T348446 (10cmooney) >>! In T348446#9252677, @ayounsi wrote: > Maybe prepending the AS on the backup LVS is easier to do than expected? > i though PyBal's developm... [15:01:21] XioNoX: it kinda make sense... they cannot rely on UDP after all being fire and forget [15:01:44] yeah [15:02:46] Hi all! [15:03:11] and of course they always have TCP:443 as a fallback [15:03:14] "Currently it's a static value chosen client side, which was based on some empirical testing. All handshake packets must be padded to the full size in both directions, so if the MTU is too large, the handshake fails and Chrome falls back just like it would if UDP was blocked entirely." [15:03:20] duesen: hi :) [15:03:38] Is there any reason we may run into problems with returning partial URLs (no domain/port/protocol) in Location headers of redirects returned from rest.php? [15:04:49] Could there be issues with varnish or ATS? This would affect permanent (cacheable) redirects as well as temporary redirects. [15:05:28] I'm not worried about the actual clients. But I want to be sure that our caching infra won't choke on this kind of thing :) [15:05:34] nothing wrong regarding RFC 9110 (HTTP semantics) [15:05:55] Well, Location was required to be absolut until 2014. [15:06:22] per RFC 2616. [15:06:34] But that has been obsolete for nearly a decade. [15:06:48] I'm just wondering if some component perhaps didn't get the memo... [15:07:36] Context: I want to simplify this patch, and make relative redirects unconditional: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/965545 [15:08:39] Varnish does some magic indeed [15:09:05] context being T134464 [15:09:05] T134464: Make RB ?redirect=false cache-efficient - https://phabricator.wikimedia.org/T134464 [15:10:40] vgutierrez: e.g. https://docs.trafficserver.apache.org/en/latest/admin-guide/files/remap.config.en.html says: "‘redirect-URL’ is a redirection URL specified according to RFC 2616 " [15:11:12] we don't use map_with_referer [15:12:21] yea... still... scary... [15:12:38] that magic hack, that should be for index.php only. Or is it for all endpoints? [15:13:19] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ssingh) [15:13:51] duesen: "^/api/rest_v1/.*[?&]redirect=" [15:14:15] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/varnish/templates/text-frontend.inc.vcl.erb#271 [15:14:45] I think we've done relative redirects before, I don't think they cause issues [15:14:59] but that's just my best recollection! :) [15:15:44] vgutierrez: this is intersting... we'll likely want to do the same thing for rest.php, once it is being used more. We have just added support for wiki redirects there, and my question is directly related to them. [15:16:05] bblack: I take that to mean that we should try and see ;) [15:16:31] these endpoints are not used by anything yet, so if this doesn't work, it wouldn't be catastrophic [15:17:23] duesen: if it doesn't work it's a bug on some CDN component, so go ahead :) [15:20:10] 10Traffic, 10API Platform, 10MediaWiki-REST-API, 10serviceops: Use relative URLs in redirects emitted by rest.php - https://phabricator.wikimedia.org/T349001 (10daniel) [15:20:25] 10Traffic, 10API Platform, 10MediaWiki-REST-API, 10serviceops: Use relative URLs in redirects emitted by rest.php - https://phabricator.wikimedia.org/T349001 (10daniel) [15:21:11] bblack, vgutierrez : I filed a ticket, can you put a quick comment there? https://phabricator.wikimedia.org/T349001 [15:21:11] that whole X-RB-NOREDIR thing is truly just a perf hack. Nothing would break if we removed it entirely, and honestly if it's not perf-important traffic to us, maybe we should. [15:24:26] 10Traffic, 10API Platform, 10MediaWiki-REST-API, 10serviceops, 10Patch-For-Review: Use relative URLs in redirects emitted by rest.php - https://phabricator.wikimedia.org/T349001 (10Vgutierrez) This shouldn't break anything at the CDN layer [last famous words]. If it does we should fix it so please be bol... [15:41:17] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Change cloud-instance-transport vlan subnets from /30 to /29 - https://phabricator.wikimedia.org/T348140 (10dcaro) a:03dcaro [15:42:51] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Change cloud-instance-transport vlan subnets from /30 to /29 - https://phabricator.wikimedia.org/T348140 (10dcaro) [19:04:28] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10jhathaway) >>! In T348837#9253425, @cmooney wrote: > Either way, katran/liberica uses IPIP, so having the option for GUE in IPVS doesn't solve that problem if we hit it. I think w... [19:16:34] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10jhathaway) >>! In T348837#9254127, @BBlack wrote: > One potential issue with relying solely on MSS reduction is that, obviously, it only affects TCP. For now this is fine, as long... [23:38:26] 10Traffic, 10SRE, 10Patch-For-Review, 10SRE Observability (FY2023/2024-Q2): Remove legacy ELK LVS entries - https://phabricator.wikimedia.org/T299700 (10lmata)