[06:27:22] 10netops, 10Infrastructure-Foundations: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ayounsi) p:05Triage→03High [06:28:41] 10netops, 10Infrastructure-Foundations: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ops-monitoring-bot) ===== Automated diagnostic for Netbox interface ID cr2-eqiad:xe-4/1/3 --- **Interface cr2-eqiad:xe-4/1/3** - admin-status: up - ⚠️ oper-status... [06:33:38] 10netops, 10Infrastructure-Foundations, 10SRE: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ayounsi) From Lumen diagnostic tool: > SERVICE ALARMS NEEDS ATTENTION We have detected equipment alarms. Further Investigation is required. > Ticket ID... [09:02:08] 10netops, 10Infrastructure-Foundations, 10SRE: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ayounsi) > I am seeing an issue on our SUBSEA portion of the circuit. I am engaging my SUBSEA group at this time. [10:26:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox, 10Patch-For-Review: Represent sub-interface and bridge device assocations in Netbox - https://phabricator.wikimedia.org/T296832 (10jbond) >>! In T296832#8205878, @Volans wrote: >>>! In T296832#8143427, @cmooney wrote: >> For a bit of context the ab... [11:40:58] 10netops, 10Infrastructure-Foundations, 10SRE: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ayounsi) > LUMEN Subsea group performed a cold reset on a card in Bude England to restore service. I am seeing traffic up at this time. [11:41:27] 10netops, 10Infrastructure-Foundations, 10SRE: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ops-monitoring-bot) ===== Automated diagnostic for Netbox interface ID cr2-eqiad:xe-4/1/3 --- **Interface cr2-eqiad:xe-4/1/3** - admin-status: up - oper-... [11:42:29] 10netops, 10Infrastructure-Foundations, 10SRE: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ayounsi) 05Open→03Resolved a:03ayounsi [13:10:21] XioNoX: ^^ nice, TTFB went back to normal values [13:31:00] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox, 10Patch-For-Review: Represent sub-interface and bridge device assocations in Netbox - https://phabricator.wikimedia.org/T296832 (10Volans) Great, thanks for checking it @jbond [15:53:19] 10Traffic: Varnish SLI is impacted by external components performance|behavior - https://phabricator.wikimedia.org/T317051 (10Vgutierrez) [15:53:34] 10Traffic: Varnish SLI is impacted by external components performance|behavior - https://phabricator.wikimedia.org/T317051 (10Vgutierrez) p:05Triage→03Medium [16:44:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox, 10Patch-For-Review: Netbox: use FHRP Groups feature - https://phabricator.wikimedia.org/T311218 (10ayounsi) [16:44:59] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox, 10Patch-For-Review: Netbox: use FHRP Groups feature - https://phabricator.wikimedia.org/T311218 (10ayounsi) [17:38:58] 10Traffic, 10affects-Kiwix-and-openZIM: HTTP 500 against api.php?action=parse API on tr.wikipedia.org - https://phabricator.wikimedia.org/T317011 (10Aklapper) Not sure which project tags to add when it comes to caching layers (?), assuming #Traffic. Cannot reproduce the problem here (central Europe): ` $:acko... [17:41:44] 10Traffic, 10affects-Kiwix-and-openZIM: HTTP 500 against api.php?action=parse API on tr.wikipedia.org - https://phabricator.wikimedia.org/T317011 (10Kelson) > Cannot reproduce the problem here (central Europe): Me neither now, suprisingly, because this is a recurring problem with this very same article over m... [20:53:16] 10Traffic, 10SRE, 10affects-Kiwix-and-openZIM: HTTP 500 against api.php?action=parse API on tr.wikipedia.org - https://phabricator.wikimedia.org/T317011 (10Platonides) I suspect it's a timeout at Varnish level, and it then got cached somewhere. I was always getting 200, even when asking every dc: ` for dc i... [21:25:21] 10Traffic, 10MediaWiki-Page-history, 10Regression: History pages' caches not being invalidated after edits - https://phabricator.wikimedia.org/T317064 (10Legoktm) [21:32:26] 10Traffic, 10MediaWiki-Page-history, 10SRE, 10Regression: History pages' caches not being invalidated after edits - https://phabricator.wikimedia.org/T317064 (10Legoktm)