[08:30:18] <wikibugs>	 10netbox, 10Infrastructure-Foundations: Netbox: define strategy to track standard server configurations - https://phabricator.wikimedia.org/T284614 (10cmooney) Thanks Riccardo, that sums the call up well I thinl.  Overall this is something DC-Ops need so I'm happy for them to define it, but I'll share my think...
[09:27:21] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[10:09:23] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney)
[10:10:21] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[10:37:58] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney)
[11:06:12] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney)
[11:06:38] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[11:13:57] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney)
[11:14:15] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[11:16:40] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10cmooney)
[11:17:14] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney)
[11:41:57] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney)
[11:45:57] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[12:14:24] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney)
[12:14:34] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10cmooney)
[12:35:45] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney)
[12:36:09] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[12:37:55] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[12:39:16] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[12:43:26] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney)
[12:44:28] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney)
[12:47:53] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[14:24:51] <wikibugs>	 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-codfw: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10RobH)
[14:26:21] <wikibugs>	 10netops, 10DC-Ops, 10ops-codfw: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10RobH)
[14:27:46] <wikibugs>	 10netops, 10DC-Ops, 10ops-codfw: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10RobH) Please note I'm not putting this request into CyrusOne until after Arzhel confirms they are ready for this step.
[14:33:03] <wikibugs>	 10netops, 10DC-Ops, 10ops-codfw: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10cmooney) First log of an issue was this sent from the master switch in the virtual-chassis:      Jul 16, 2021 @ 13:15:55.000 %-SNMP_TRAP_LINK_DOWN: ifIndex 927, ifAdminStatus up(1), ifOperStatus down(2),...
[14:52:01] <wikibugs>	 10netops, 10DC-Ops, 10SRE, 10ops-codfw: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10RobH) I've opened support ticket 2022508 with cyrunsone to have them use remote hands to powercycle this.       > The switch is a Juniper QFX5100-48S-6Q, labeled asw-a2-codfw, located in U26 (re...
[14:57:32] <wikibugs>	 10netops, 10DC-Ops, 10SRE, 10ops-codfw, 10Wikimedia-Incident: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10Majavah)
[15:29:42] <wikibugs>	 10netops, 10DC-Ops, 10SRE, 10ops-codfw, 10Wikimedia-Incident: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10RobH) Remote hands has completed the powercycle of the switch (via removing all power cables).  Both before and after power removal, all LEDs are illuminated, which is no...
[15:39:00] <wikibugs>	 10netops, 10DC-Ops, 10SRE, 10ops-codfw, 10Wikimedia-Incident: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10RobH) a:03Papaul I'll attempt to summarize the IRC discussion.  @ayounsi, @cmooney, and myself discussed how it is likely safer to let a single switch sit broken over t...
[15:41:47] <wikibugs>	 10netops, 10DC-Ops, 10SRE, 10ops-codfw, 10Wikimedia-Incident: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10RobH)
[16:11:01] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney)
[16:15:33] <wikibugs>	 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[19:53:21] <wikibugs>	 10Mail, 10Infrastructure-Foundations, 10Znuny, 10User-jbond: Clean up OTRS/Znuny addresses handles by gsuite - https://phabricator.wikimedia.org/T284145 (10Keegan) >>! In T284145#7134174, @jbond wrote: > @Keegan I notice you working on the [[ https://phabricator.wikimedia.org/T280392 |  orts/znuny migratio...