[09:25:46] 10netops, 10Infrastructure-Foundations, 10SRE: Link failure between mr1-eqiad and asw2-a8-eqiad Aug 13th 2021 - https://phabricator.wikimedia.org/T288834 (10ayounsi) Looks like the whole of FPC8 failed, I opened JTAC case 2021-0816-0128. Good thing you saved the logs as they now fully rolled over. [09:35:20] 10netops, 10Infrastructure-Foundations, 10SRE: Link failure between mr1-eqiad and asw2-a8-eqiad Aug 13th 2021 - https://phabricator.wikimedia.org/T288834 (10cmooney) Thanks @ayounsi yeah I was looking there that seems to be the case. Logs from a host connected also seem to confirm the entire switch died: `... [09:41:03] 10netops, 10Infrastructure-Foundations, 10SRE: Switch failure: asw2-a8-eqiad Aug 13th 2021 - https://phabricator.wikimedia.org/T288834 (10cmooney) [09:42:35] 10netops, 10Infrastructure-Foundations, 10SRE: Switch failure: asw2-a8-eqiad Aug 13th 2021 - https://phabricator.wikimedia.org/T288834 (10cmooney) [09:43:33] 10Traffic, 10Anti-Harassment: Enable automatic redirection to the mobile version of votewiki - https://phabricator.wikimedia.org/T288938 (10phuedx) [09:47:15] 10Traffic, 10Anti-Harassment, 10SRE: Enable automatic redirection to the mobile version of votewiki - https://phabricator.wikimedia.org/T288938 (10phuedx) @Jdlrobson noted that the "Mobile view" link at the bottom of every page has been blanked out by setting [[ https://vote.wikimedia.org/wiki/MediaWiki:Mobi... [09:50:51] 10netops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover, 10User-fgiunchedi: Record traffic flows in and out of eqiad during switchover - https://phabricator.wikimedia.org/T286038 (10ayounsi) @fgiunchedi anything left to do for netops or is it ok to close the task? [10:00:58] 10netops, 10Infrastructure-Foundations, 10SRE: Traffic Engineering for Anycast Ranges - https://phabricator.wikimedia.org/T288843 (10ayounsi) Some thoughts: * We need to find the good balance between config complexity and low latency for users, otherwise it's going to be a cat and mouse game, fixing special... [10:09:09] 10Traffic, 10Anti-Harassment, 10SRE: Enable automatic redirection to the mobile version of votewiki - https://phabricator.wikimedia.org/T288938 (10phuedx) @dom_walden noted that after he'd jumped from one wiki to votewiki, the cookies returned by `https://vote.wikimedia.org/wiki/Secure:SpecialPoll/login/{$el... [10:19:41] 10netops, 10Infrastructure-Foundations, 10SRE: Traffic Engineering for Anycast Ranges - https://phabricator.wikimedia.org/T288843 (10cmooney) Agreed we need to balance complexity and usefulness. A few points: - I think it's too complex to consider doing this for peers at IXPs. - I would only anticipate... [12:59:26] 10netops, 10Infrastructure-Foundations, 10SRE: Traffic Engineering for Anycast Ranges - https://phabricator.wikimedia.org/T288843 (10ssingh) (Thanks Cathal for filing this task!) >>! In T288843#7284411, @ayounsi wrote: > Some thoughts: > [...] > @ssingh what's the timeline for Wikidough? So we know how to p... [15:29:40] XioNoX: one last maintenance notification overlap before the end of my clinic duty shift :) [15:29:53] XioNoX: Lumen 21744629 and Telia PWIC207863.1 tomorrow [16:16:03] ema: which circuits are they for? [16:17:17] XioNoX: Lumen 442550293, Telia IC-308844 [16:17:53] ema: all good! [16:17:54] thx! [16:17:59] nice, thanks for checking! [16:45:08] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Switch failure: asw2-a8-eqiad Aug 13th 2021 - https://phabricator.wikimedia.org/T288834 (10ayounsi) p:05Triage→03High a:03Cmjohnson JTAC pointed out that the switch failure matches with a VCP (to the backup spine) going down: ` ayounsi@asw2... [17:13:12] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Switch failure: asw2-a8-eqiad Aug 13th 2021 - https://phabricator.wikimedia.org/T288834 (10ayounsi) p:05High→03Low a:05Cmjohnson→03ayounsi Keeping it a bit for monitoring, will close if no more interfaces errors. [17:37:44] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Join ARIN waiting list to request additional IPv4 resources. - https://phabricator.wikimedia.org/T288342 (10Andrew) @aborrero is on holiday for a bit -- I'd like to hear from him as well but here's my current thought: With our... [18:02:27] 10Traffic, 10Anti-Harassment, 10SRE: Enable automatic redirection to the mobile version of votewiki - https://phabricator.wikimedia.org/T288938 (10Niharika) p:05Triage→03High @BBlack @mark Hello. Sorry to put undue pressure here but the Board election is due to start in two days and we want to test this... [18:08:13] 10Traffic, 10Anti-Harassment, 10SRE: Enable automatic redirection to the mobile version of votewiki - https://phabricator.wikimedia.org/T288938 (10jrbs) >>! In T288938#7284392, @phuedx wrote: > @Jdlrobson noted that the "Mobile view" link at the bottom of every page has been blanked out by setting [[ https:/...