[08:41:16] <jynus>	 I'm going to restart, os is again acting up
[12:30:28] <marostegui>	 what is the status with test-s4 cluster? I see it very weirdly on orchestrator
[12:30:32] <marostegui>	 Amir1: are you still working with it?
[12:30:48] <Amir1>	 nope
[12:30:56] <Amir1>	 let me check if I messed up something
[12:30:59] <marostegui>	 then I think we need to leave it in a consistent state
[12:31:23] <Amir1>	 probably the semi-sync got messed up
[12:31:39] <marostegui>	 I see lag on db1124, no master, and db1133 not replicating
[12:32:22] <Amir1>	 sigh, I think semi-sync broke it
[12:32:29] <Amir1>	 let me try to fix it
[12:32:34] <Amir1>	 sorry I missed this
[12:32:42] <marostegui>	 make sure it has db1124 as master, db1125 and db1133 as slaves
[12:32:59] <Amir1>	 sure
[12:35:59] <Amir1>	 marostegui: when I was testing, db1125 was the master and db1124 and db1133 were the replicas. 
[12:37:01] <Amir1>	 I for now, make db1125 the master and then let you move the toplogy if you want to 
[12:37:11] <marostegui>	 Yeah, whatever was the previous state was ok
[12:44:18] <Amir1>	 https://orchestrator.wikimedia.org/web/cluster/alias/test-s4 now only db1133 left
[12:45:54] <Amir1>	 marostegui: okay back to normal. It seems the biggest reason was that somehow db1125 (master) got its mysql daemon shut down. I don't remember doing that nor the cookbook should do it either. 
[12:46:02] <marostegui>	 thanks
[12:46:17] <Amir1>	 so that part is a mystery to me