[12:55:23] <dcaro>	 !log paws Out of space on nfs, looking
[12:55:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[13:01:23] <dcaro>	 !log paws removed the files pending deletion as the systemd timer seems to be failing, that cleared some space
[13:01:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[17:01:17] <Damianz>	 Hi, webservices appear to be down, getting "no healthy upstream" for multiple tools though the pods are up and healthy (no recent changes). First alert was about 25min ago but it seems some requests are getting through (alerts cleared, but things are just spinning for me)
[17:02:08] <dcaro>	 we are having a huge peak in traffic, haproxies are flapping
[17:02:28] <dcaro>	 !status unstable
[17:03:18] <Damianz>	 Ok, at long as you're looking at it. I'll wait for hahaproxy to be haproxy again
[17:11:06] <dcaro>	 thanks
[17:34:30] <dcaro>	 !log tools rebooting haproxy-8
[17:34:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:54:10] <dcaro>	 things seem to be calming down, I reduced considerably the connection rates on the haproxies
[17:54:19] <dcaro>	 (also the global traffic went down)
[17:54:41] <dcaro>	 I'm wrapping/documenting the changes I did, will clear the status in a bit, see if this stabilizes
[18:08:36] <Damianz>	 Just checking, but it shouldn't be affecting internal stuff? Getting "ComponentsClientError: The components service seems to be down – please retry in a few minutes." intermittently but consistently 
[18:11:05] <dcaro>	 I'll check, it should not be affecting it no
[18:13:10] <dcaro>	 it might affect actually, should be ok now, let me try to fix better
[18:23:09] <dcaro>	 Damianz: have you seen any more errors on that?
[18:44:34] <dcaro>	 !status ok
[18:44:46] <dcaro>	 should be ok now, still keeping an eye, please report any issues
[19:36:43] <legoktm>	 dcaro: I think there's issues again?
[19:46:25] <Damianz>	 dcaro: just tried to create a deployment again and it fails with the same, then on another try it went through.. the components grafana dashboard looks like metrics are missing so maybe there is a wonky pod. I managed to get the change done now so not super urgent.
[19:47:25] <Damianz>	 web services still seem dodgy, getting flapping alerts as of a couple of min ago
[20:13:38] <dcaro>	 looking