[06:27:22] 10netops, 10Infrastructure-Foundations: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ayounsi) p:05Triage→03High [06:28:40] 10netops, 10Infrastructure-Foundations: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ops-monitoring-bot) ===== Automated diagnostic for Netbox interface ID cr2-eqiad:xe-4/1/3 --- **Interface cr2-eqiad:xe-4/1/3** - admin-status: up - ⚠️ oper-status... [06:33:36] 10netops, 10Infrastructure-Foundations, 10SRE: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ayounsi) From Lumen diagnostic tool: > SERVICE ALARMS NEEDS ATTENTION We have detected equipment alarms. Further Investigation is required. > Ticket ID... [08:10:37] 10netbox, 10Infrastructure-Foundations: ganeti-netbox-sync.py fails in dry-mode - https://phabricator.wikimedia.org/T314794 (10Volans) That's correct. The problem with the dry-run in this script is that we have different cascading objects. If we don't create a new say cluster group we can't create a cluster, e... [08:25:32] 10netbox, 10Infrastructure-Foundations: ganeti-netbox-sync.py fails in dry-mode - https://phabricator.wikimedia.org/T314794 (10ayounsi) Sure, that sounds cleaner, maybe with a message to let the user know it's normal in dry-run mode. [09:02:06] 10netops, 10Infrastructure-Foundations, 10SRE: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ayounsi) > I am seeing an issue on our SUBSEA portion of the circuit. I am engaging my SUBSEA group at this time. [10:12:34] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: Upgrade pynetbox - https://phabricator.wikimedia.org/T310745 (10ayounsi) [10:14:59] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: Upgrade pynetbox - https://phabricator.wikimedia.org/T310745 (10ayounsi) [10:15:20] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: Upgrade pynetbox - https://phabricator.wikimedia.org/T310745 (10ayounsi) [10:26:17] 10netbox, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Represent sub-interface and bridge device assocations in Netbox - https://phabricator.wikimedia.org/T296832 (10jbond) >>! In T296832#8205878, @Volans wrote: >>>! In T296832#8143427, @cmooney wrote: >> For a bit of context the ab... [10:32:30] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: Pynetbox: enable threading - https://phabricator.wikimedia.org/T311486 (10ayounsi) [10:33:22] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: Pynetbox: enable threading - https://phabricator.wikimedia.org/T311486 (10ayounsi) `cumin1001:~$ time sudo cookbook -d sre.dns.netbox "test"` Before: 2m54.708s After: !!1m12.567s!! [11:40:56] 10netops, 10Infrastructure-Foundations, 10SRE: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ayounsi) > LUMEN Subsea group performed a cold reset on a card in Bude England to restore service. I am seeing traffic up at this time. [11:41:26] 10netops, 10Infrastructure-Foundations, 10SRE: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ops-monitoring-bot) ===== Automated diagnostic for Netbox interface ID cr2-eqiad:xe-4/1/3 --- **Interface cr2-eqiad:xe-4/1/3** - admin-status: up - oper-... [11:42:27] 10netops, 10Infrastructure-Foundations, 10SRE: Lumen link between cr2-eqiad and cr2-esams down - Sept 2022 - https://phabricator.wikimedia.org/T317009 (10ayounsi) 05Open→03Resolved a:03ayounsi [11:54:19] topranks, jbond, volans, there are some local changes on netbox-dev2002:/srv/deployment/netbox-extras (hiera_export.py and interface_automation.py) are you still working on them or is it ok to bring them on par with prod? [11:55:06] XioNoX: from my PoV its fine to wipe them out and sync with prod [11:55:15] Good to scrap mine - from interface_automation - I wasn’t messing with the other one [11:55:22] Thanks [11:57:17] XioNoX: no WIP from my end [12:04:53] cool [13:30:58] 10netbox, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Represent sub-interface and bridge device assocations in Netbox - https://phabricator.wikimedia.org/T296832 (10Volans) Great, thanks for checking it @jbond [13:34:09] XioNoX: can I delete a ganeti cluster on netbox-next? [13:34:17] to test the patch for the dry-run [13:34:57] volans: sure, can you enable threading too? [13:36:07] sure [15:19:53] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10User-jbond: Add a deprecated flag to admin groups - https://phabricator.wikimedia.org/T248161 (10jbond) 05Open→03Resolved implmented [16:44:16] 10netbox, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Netbox: use FHRP Groups feature - https://phabricator.wikimedia.org/T311218 (10ayounsi) [16:44:57] 10netbox, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Netbox: use FHRP Groups feature - https://phabricator.wikimedia.org/T311218 (10ayounsi) [17:55:10] 10netbox, 10Infrastructure-Foundations: ganeti-netbox-sync.py fails in dry-mode - https://phabricator.wikimedia.org/T314794 (10Volans) 05Open→03Resolved This is now solved.