[04:58:25] legoktm: gotcha, thanks [04:58:46] legoktm: do you want to close the above task or leave it open? [04:58:58] I think we need to leave it open [04:59:37] The current spicerack in the Git repo is broken, I just live hacked cumin1001 [04:59:58] sounds good :) [05:00:10] not the repo breakage, but leaving the task open [05:00:32] :p [05:00:44] Going to give the buffer pools another go just now [05:01:40] Cool, I was going to ask if there's any other pre-switchover steps we need to check [05:03:20] I am going to double check replication between codfw -> eqiad again (although stevie did a check a few days ago, it doesn't hurt!) [08:35:01] legoktm: note that while right now x2 is the exception, eventually all "core" dbs will have the same behaviour (when multi-dc finally happens, any day now, ™) [15:06:35] removed old s5 backup sources from orchestrator as they are no longer there [19:18:08] 10DBA, 10SRE, 10Datacenter-Switchover: When switching DCs, update pc hosts in tendril - https://phabricator.wikimedia.org/T266723 (10Legoktm) I've documented this as a manual step going forward: https://wikitech.wikimedia.org/w/index.php?title=Switch_Datacenter&type=revision&diff=1917204&oldid=1917111 [19:45:03] 10DBA, 10SRE, 10Datacenter-Switchover: switchdc should automatically downtime "Read only" checks on DB masters being switched - https://phabricator.wikimedia.org/T285803 (10Legoktm) [20:22:09] 10DBA, 10observability, 10Datacenter-Switchover: "User 'prometheus' has exceeded the 'max_user_connections' resource" during DC switchover - https://phabricator.wikimedia.org/T285807 (10Legoktm) [20:38:32] 10DBA, 10observability, 10Datacenter-Switchover: "User 'prometheus' has exceeded the 'max_user_connections' resource" during DC switchover - https://phabricator.wikimedia.org/T285807 (10Marostegui) 05Open→03Invalid That timestamp isn't from the DC switch, that's from two weeks ago when the host got overl... [20:44:23] 10DBA, 10observability, 10Datacenter-Switchover: "User 'prometheus' has exceeded the 'max_user_connections' resource" during DC switchover - https://phabricator.wikimedia.org/T285807 (10Legoktm) ...I should've looked more closely, my bad. Also no idea why it ended up in the etherpad :p