[08:22:04] 10netops, 10Infrastructure-Foundations, 10ops-eqsin: asw1-eqsin: VC mastership change - https://phabricator.wikimedia.org/T323094 (10ayounsi) p:05Triage→03High [08:28:35] 10netops, 10Infrastructure-Foundations, 10ops-eqsin: asw1-eqsin: VC mastership change - https://phabricator.wikimedia.org/T323094 (10ayounsi) @RobH I couldn't find any open task so I opened this one, in the future please make sure a task is opened as soon as any issue happens. Can you also let us know the im... [08:47:05] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqsin: asw1-eqsin: VC mastership change - https://phabricator.wikimedia.org/T323094 (10ayounsi) JTAC case 2022-1115-586910 opened. [09:11:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqsin: asw1-eqsin: VC mastership change - https://phabricator.wikimedia.org/T323094 (10Volans) I would also like to understand why eqsin was not immediately depooled when it happened. The patch was already ready on Gerrit. There was [[ https://grafana.wi... [10:49:06] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Update CAS to 6.6 - https://phabricator.wikimedia.org/T311235 (10MoritzMuehlenhoff) [13:24:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqsin: asw1-eqsin: VC mastership change - https://phabricator.wikimedia.org/T323094 (10ayounsi) Note that we will need to move the mastership back to `asw-0604-eqsin` to keep everything standardized. For that, better depool the site. [13:51:51] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic: Consider lowering IPv6 TCP MSS - https://phabricator.wikimedia.org/T283058 (10ayounsi) 05Open→03Declined We haven't seen any issue related to this so closing the task. [16:13:38] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqsin: asw1-eqsin: VC mastership change - https://phabricator.wikimedia.org/T323094 (10RobH) >>! In T323094#8395125, @Volans wrote: > I would also like to understand why eqsin was not immediately depooled when it happened. > The patch was already ready o... [16:17:46] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqsin: asw1-eqsin: VC mastership change - https://phabricator.wikimedia.org/T323094 (10RobH) Order of operations, all of this was at or just before 5AM UTC. * Jin racks new hosts, and plugs in cp5032 - the port was NOT setup and the server was not in ne... [16:41:20] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqsin, 10Wikimedia-Incident: asw1-eqsin: VC mastership change - https://phabricator.wikimedia.org/T323094 (10jcrespo) [17:20:35] 10netops, 10Infrastructure-Foundations, 10SRE: Migrate row E/F network aggregation to dedicated Spine switches - https://phabricator.wikimedia.org/T322937 (10cmooney)