[07:27:20] 10netops, 10Infrastructure-Foundations, 10ops-codfw: asw-a-codfw management interface unreachable - https://phabricator.wikimedia.org/T330048 (10ayounsi) p:05Triage→03High [09:20:04] 10netops, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10Jelto) [09:23:58] 10netops, 10Infrastructure-Foundations, 10SRE: Standardize VRRP group IDs - https://phabricator.wikimedia.org/T260363 (10ayounsi) 05Open→03Declined We're slowly moving away from VRRP. The benefits of renumbering them all is not worth the time, especially as we removed the custom field in favor or {T311218}. [10:42:26] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Move management routers ssh port - https://phabricator.wikimedia.org/T277438 (10ayounsi) Before we merge/deploy any of those changes, Rancid and [[ https://github.com/wikimedia/operations-software-homer/blob/de281c32054862799dbf8102ed627d7d... [11:54:03] 10netops, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10Clement_Goubert) [14:22:55] 10netbox, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Issues converting services from active/passive to active/active - https://phabricator.wikimedia.org/T330084 (10jbond) [14:25:42] 10netbox, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Patch-For-Review: Issues converting services from active/passive to active/active - https://phabricator.wikimedia.org/T330084 (10jbond) p:05Triage→03High [14:46:22] 10netbox, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Patch-For-Review: Issues converting services from active/passive to active/active - https://phabricator.wikimedia.org/T330084 (10jbond) p:05High→03Medium lowering priority @Vgutierrez confirmed there are no immediate issues with dns. They a... [15:41:15] 10netops, 10Infrastructure-Foundations, 10SRE, 10observability: Investigate Juniper structured logs - https://phabricator.wikimedia.org/T250703 (10ayounsi) 05Open→03Declined The need never was very strong, and that would be a pain to integrate with ECS. > The good news is that at least 1 person in the... [16:39:15] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack: wait_for_optimal() should ignore acked alerts - https://phabricator.wikimedia.org/T319277 (10Volans) Spicerack v6.2.1 ships with this new feature, and it works as expected. One small improvements that we should add below. I think we should add support f... [16:42:58] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack: spicerack dnsdisc.Discovery attempts to query depooled/disabled dns auth servers - https://phabricator.wikimedia.org/T329773 (10Volans) 05Open→03Resolved p:05Triage→03Medium a:03Volans Spicerack v6.2.1 was deployed with the above fix (see [[ h... [16:50:42] Hello :) I'm not fully aware of the impact of the switch maintenance in codfw. But is it expected that management SSH in codfw is down on a lot of hosts? We just stumbled about that in alertmanager https://alerts.wikimedia.org/?q=%40state%3Dactive&q=alertname%3DManagementSSHDown Manual tests confirmed management SSH beeing down. [16:52:09] jelto: could be related to T330048 ? [16:52:09] T330048: asw-a-codfw management interface unreachable - https://phabricator.wikimedia.org/T330048 [16:53:38] although they seem to be in different racks [16:53:59] ah yes! sounds like the right task, thanks