[01:10:35] 10Traffic, 10envoy, 10serviceops: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus) p:05Triage→03Medium [08:16:15] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10ayounsi) a:05ayounsi→03RobH > Please note the above diagram has a mistake, showing both routers connecting to PP:15/16 when cr1:xe-0/... [10:51:02] 10netops, 10Infrastructure-Foundations, 10SRE: Configuration of New Switches Eqiad Rows E-F - https://phabricator.wikimedia.org/T299758 (10cmooney) [10:54:19] 10netops, 10Infrastructure-Foundations, 10SRE: Configuration of New Switches Eqiad Rows E-F - https://phabricator.wikimedia.org/T299758 (10cmooney) Just to update we've had console access for most of this week and configuration / testing is under way. Will submit CRs when config is ready. [10:54:34] 10netops, 10Infrastructure-Foundations, 10SRE: Validate EVPN/VXLAN configuration for Juniper QFX Platform - https://phabricator.wikimedia.org/T294115 (10cmooney) 05Open→03Resolved Closing this task. Have discussed with @ayounsi and we are broadly in agreement on next steps for the Eqiad expansion. Furt... [11:45:35] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10cmooney) Actually one thing that is outstanding I believe is to confirm the cable IDs? **Inter-Switch Links** I documented the inter-switc... [15:07:10] vgutierrez: sorry to disturb you on a Friday but have you had a chance to look at T300161 yet? [15:07:10] T300161: Serve redirect wikimediastatus.net --> www.wikimediastatus.net - https://phabricator.wikimedia.org/T300161 [15:08:12] hmm that's tricky [15:08:28] yeah [15:08:32] our prod environment doesn't support anything besides dns-01 [15:08:35] got it [15:08:48] maybe we need (another) external service :/ [15:09:13] (tbh I was very surprised that our registrar couldn't just take care of this) [15:09:29] well, you're going to run into some variant of the same problem, it's just a question of what the solution is [15:09:57] what I meant is that lots of registrars offer simple HTTP redirect services because of the CNAME/apex problem [15:10:10] who hosts the DNS itself? [15:10:12] and indeed does, but, not in combination with 'real' records being served from the zone too [15:10:14] MM [15:10:18] ah ok' [15:10:42] could we handle DNS for the domain? [15:10:45] so, you could use a 3rd paty DNS-host separate from the registrar [15:10:58] vgutierrez: I think that would defeat the independence goal :) [15:11:02] ^ [15:11:08] duh! [15:11:12] sorry :) [15:11:27] then ncredir wouldn't be a valid solution either, right? [15:11:45] sort of :) [15:12:02] I don't think we need as strong a guarantee for a redirect (especially if we serve a permanent redir) as we do for the domain itself [15:12:04] I think the idea is that all links/canonical-refs would be on www, and this is just to catch manual bad typing [15:12:29] but: I'd argue that if you make the redirect reliably work in good times, people will still manage to be caught out by relying on it bad times, one way or another. [15:12:50] fair [15:13:01] do you have any recommendations for an external DNS hoster? [15:13:03] you maybe could have _acme_challenge.wikimediastatus.net as a CNAME to some domain hosted on our production DNS infra (say wikimediastatus-acme-challenge.wikimedia.org) and have acme-chief update that [15:13:25] cdanis: I'm not really up to speed on dns hosters (in terms of which have what feature-sets at what prices, etc) [15:14:37] but you'd think there would be a cheap one that could host the apex themselves with an LE-based cert and still CNAME out the www. [15:14:43] yeah [15:15:50] the one that springs to mind, from unrelated contact, is dnsmadeeasy [15:15:56] but I don't know their featureset [15:16:19] I'd prefer any hoster we already have a business relationship with, or, that was cheap enough to just do something that doesn't involve a contract [15:16:44] https://dnsmadeeasy.com/services/anamerecords [15:17:03] I'm actually not sure if we can get ANAME working here well, sadly [15:17:11] ^ apparently they do "ANAME" (which is most likely that they basically lookup and cache your CNAME-target's IP for a short while and serve it as the apex address) [15:17:13] Atlassian supports only serving one domain at a time [15:17:41] looks like dnsmadeeasy does support serving HTTP redirects, though [15:18:05] yeah but you can avoid that level of complexity entirely [15:18:21] their dumbed-down explanation seems to match my expectations [15:19:15] heh, ok [15:19:16] what I mean is that the statuspage site itself will serve a cert for / answer queries for exactly one of either www.wikimediastatus.net or wikimediastatus.net [15:19:33] that's... very limiting :) [15:19:39] yes [15:20:04] I haven't asked Atlassian about custom setups yet because I'm a bit wary of that breaking at some point if they do something manual in their infra [15:20:46] do they offer some other integrated standard service at atlassian that could just do the redirecting part? [15:20:56] I'll ask about that, that's a good thought [15:21:02] not that that would necessarily be "better" in most sense, but might reduce dependency creep [15:21:04] yeah [15:25:33] the reason I happened to have dnsmadeeasy on my mind, is they're part of Tiggee, one of the other parts of that entity is a place called Constellix, which offers a geodns service: https://constellix.com/products/geodns [15:26:03] and they apparently use gdnsd somewhere under the hood (probably with lots of other nifty things wrapped around it), because they send bug reports and recently uploaded a nice patch contribution! :) [15:26:33] haha, neat [15:44:15] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10Vgutierrez) [15:46:32] vgutierrez: bblack: thanks for the discussion, for now I've opened a support request with Atlassian asking about them serving both names [15:46:41] and we can figure out the DNS part from there, depending on how they answer [15:46:51] if we can just switch to using ANAME at some other hoster, that sounds ideal [15:53:48] just keep in mind "ANAME" and similar concept (some call is ALIAS or such), they're non-standard and vary by implementor, so you always have to check the details a bit on how each one does it (but generally, when used at an apex for this kind of purpose, they're resolving+possibly-caching on their backend somehow) [15:53:55] yeah [18:32:55] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) >>! In T300277#7658774, @ayounsi wrote: >> Please note the above diagram has a mistake, showing both routers connecting to PP:15/16... [18:54:48] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10wiki_willy) Hi @cmooney - here's the doc that @Jclark-ctr put together when running the cables for the inter-switch links. Some of the cabl... [19:56:48] 10Domains, 10Traffic, 10SRE, 10WMF-Communications, 10wikimediafoundation.org: Project Unseen campaign URL redirect - https://phabricator.wikimedia.org/T300398 (10Varnent) [22:47:16] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) Submitted the revised document (using numbered steps) to Interxion via ticket CS0433959. I listed @wiki_willy, @ayounsi, & @cmoone...