[19:12:31] <AntiComposite>	 Krinkle, looks like cvn-app9 has gone down, taking a bunch of bots with it. Can you take a look?
[19:15:26] <Krinkle>	 AntiComposite: looking
[19:27:30] <Krinkle>	 cvn-app9: connect failed: No route to host
[19:27:36] <Krinkle>	 looks like a partial wmcs outage
[19:27:40] <Krinkle>	 nothing I can do unfortunately
[19:43:30] <TheresNoTime>	 have you tried turning it off and on again
[19:55:40] <Krinkle>	 OK, the observed universe has resumed for our VMs
[19:55:53] <Krinkle>	 the bots should spin up momentarily. I've restarted them just in case.
[19:56:49] <Krinkle>	 TheresNoTime: unfortunately the issue was not a lack of power, but a lack of time. the universe appears to have stood still from the server's perspective as the cloud host was down.
[19:57:20] <TheresNoTime>	 ah so you could say... TheresNoTime?
[20:03:08] <Krinkle>	 indeed. E_THERES_NO_TIME 
[20:38:44] <AntiComposite>	 Krinkle, CVNBot5 and 20 are still down btw
[20:50:53] <Krinkle>	 cvn.cvn+ 21462  1.6  0.9 1022564 79292 ?       Sl   Mar22 363:43 /usr/bin/mono /srv/cvn/services/cvnbot/CVNBot5/CVNBot.exe
[20:51:11] <Krinkle>	 cvn.cvn+  2321  1.1  0.6 1020020 49120 ?       Sl   Feb05 1031:58 /usr/bin/mono /srv/cvn/services/cvnbot/CVNBot20/CVNBot.exe
[20:51:22] <Krinkle>	 they're on app8
[20:51:24] <Krinkle>	 killed now 
[20:51:26] <Krinkle>	 I guess they ghosted
[20:59:16] <Krinkle>	 https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/T6RRXCZ3LPJ6TYEVRALBGILFEU3BHXQR/
[20:59:28] <Krinkle>	 acknowledgement as of 40min ago
[21:02:52] <AntiComposite>	 yeah they were down all of yesterday