[04:42:23] <marostegui>	 I am going to start switching s8 eqiad primary
[05:22:10] <marostegui>	 Done
[08:36:04] <marostegui>	 arnaudb: can you schedule s1 primary switchover for next week? maybe tuesday?
[08:37:08] <arnaudb>	 sure!
[08:38:13] <marostegui>	 thanks!
[08:38:18] <marostegui>	 that way we can finish the mariadb upgrades
[08:38:22] <arnaudb>	 marostegui: it triggers a 500 error while trying to schedule for s1 eqiad → I think it's due to a hieradata issue 
[08:38:37] <marostegui>	 arnaudb: it is because the candidate master is depooled now, due to schema change
[08:38:42] <marostegui>	 I will let you know when it is back in the pool
[08:39:05] <arnaudb>	 ack, thanks, will put a placeholder instead of a ticket id 
[08:45:43] <marostegui>	 I am going to reimage db1215 - zarcillo
[08:45:48] <marostegui>	 There will be some prometheus alerts 
[08:46:01] <arnaudb>	 😱
[09:07:37] <arnaudb>	 will go for T366241
[09:07:37] <stashbot>	 T366241: Switchover s2 master (db2207 -> db2204) - https://phabricator.wikimedia.org/T366241
[09:07:47] <marostegui>	 good
[09:10:33] <arnaudb>	 uh not a good idea, Amir1 was running a schema change :/
[09:11:03] <arnaudb>	 oh and zarcillo is down haha
[09:12:14] <arnaudb>	 anyway, it failed on the first host of the pool which was depooled https://phabricator.wikimedia.org/P63657
[09:12:29] <arnaudb>	 I'll revert the candidate master's former weight
[09:13:43] <marostegui>	 Ok
[09:13:54] <marostegui>	 Double check orchestrator to make sure everything is ok
[09:14:18] <arnaudb>	 orchestrator depends on zarcillo's db
[09:14:26] <arnaudb>	 which is currently unavailable
[09:14:30] <arnaudb>	 https://orchestrator.wikimedia.org/web/cluster/alias/s2
[09:16:21] <Amir1>	 yeah
[09:16:45] <marostegui>	 ah yes I forgot haha
[09:16:59] <marostegui>	 It is coming back now
[09:16:59] <Amir1>	 I'm running a schema change on s2 and it will take until Monday or so
[09:17:19] <Amir1>	 marostegui: you know the drill at this point, when I can have some private time with the old s8 master?
[09:17:25] <marostegui>	 Amir1: nope
[09:17:29] <marostegui>	 Amir1: Probably not till tomorrow
[09:17:36] <Amir1>	 💔
[09:18:02] <marostegui>	 zarcillo is back
[09:18:42] <Amir1>	 Thanks
[09:18:57] <Amir1>	 did everything work about the switchover with the new replication user?
[09:19:09] <marostegui>	 yeah all fine
[09:19:32] <Amir1>	 \o/
[09:21:29] <arnaudb>	 I'll go for T366242 then x)
[09:21:32] <stashbot>	 T366242: Switchover s3 master (db2205 -> db2127) - https://phabricator.wikimedia.org/T366242
[09:21:40] <marostegui>	 ok
[09:52:37] <marostegui>	 arnaudb: let me know when I can operate the old s3 codfw master :)
[09:53:18] <arnaudb>	 sure, will run my schema update just before, its short!
[09:53:27] <marostegui>	 yeah no worries, just when you are done let me know
[09:53:32] <arnaudb>	 sure
[09:53:54] <Amir1>	 I have schema changes too, let me know :D
[09:54:01] <marostegui>	 Amir1: get in the queue
[09:54:23] <Amir1>	 no it's a stack, LIFO
[09:54:46] <marostegui>	 You wish
[09:54:54] <Amir1>	 xD
[09:55:31] <marostegui>	 arnaudb: s1 candidate is now being repooled, so SwitchOver web should be working again if you want to generate s1 eqiad primary switchover ticket and patches
[09:57:18] <arnaudb>	 thanks!
[10:38:45] <marostegui>	 arnaudb: you done with it?
[10:39:10] <arnaudb>	 oh sorry I forgot to hl you, yes i am!
[10:39:18] <marostegui>	 thanks I will take it
[13:46:18] <arnaudb>	 afk for about 25min :)
[14:14:16] <arnaudb>	 back