[10:13:48] <RhinosF1>	 !help users reporting ssh and other stuff on beta slow (also not sure if related but tools-checker went off)
[10:13:48] <wm-bot>	 If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban
[10:34:08] <dcaro>	 RhinosF1: yes sorry, we are doing a ceph intervention that went the non-expected way
[10:34:53] <RhinosF1>	 dcaro: np, we managed to get beta back working
[10:35:59] <dcaro>	 things should be more or less stable right now, though might slow down a bit in a few min (we are adding a node to the cluster, but with a new network setup that showed trickier, and triggered a whole re-shuffling of data around the cluster)
[10:36:41] <RhinosF1>	 dcaro: np, should we keep an eye out incase of any more crashes?
[10:36:43] <dcaro>	 that extra load made VMs io too slow
[10:37:08] <dcaro>	 we are looking also, but an extra pair of eyes is always welcome :)
[10:38:06] <RhinosF1>	 np np
[10:38:28] <dcaro>	 btw. where did the users report issues? (making sure I'm in the right channels)
[10:39:02] <RhinosF1>	 dcaro: -releng
[10:39:15] <dcaro>	 I'm there xd /me looks
[10:39:15] <RhinosF1>	 TheresNoTime was first
[10:39:51] <RhinosF1>	 the beta DB went offline for a bit so it did the expected and locked itself just in case
[11:18:28] <RhinosF1>	 dcaro: i assume that's ceph again
[11:26:36] <dcaro>	 yep, we are still doing things, what are you seeing?
[11:26:36] <RhinosF1>	 dcaro: CI started being very upset. beta is also loosing edits for some reason at the moment.
[12:06:56] <dcaro>	 RhinosF1: jbond things should be a bit better now, can you verify?
[12:42:55] <RhinosF1>	 dcaro: yes believe so
[13:05:16] <jbond>	 dcaro: sorry was in an interview checking now
[13:07:08] <jbond>	 dcaro: lgtm
[13:57:30] <andrewbogott>	 !log admin decommissioning cloudcontrol1003 + cloudcontrl1004. I backed up $home in case anyone needs their files.
[13:57:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[14:25:47] <RhinosF1>	 **fyi** s2 replication to cloud world is broken
[14:40:23] <RhinosF1>	 ^ has been fixed
[16:55:01] <taavi>	 !log tools restart puppetdb on tools-puppetdb-1, crashed during the ceph issues
[16:55:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:16:19] <bd808>	 !log tools.stashbot Added Cwhite as co-maintainer; removed 20after4 as co-maintainer
[17:16:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[18:27:14] <bd808>	 If your tools use the externallinks tables from the wiki replicas, check out T312666 which is proposing some schema changes to reduce the size of this table across the wikis. Constructive feedback is welcome!
[18:27:14] <stashbot>	 T312666: Remove duplication in externallinks table - https://phabricator.wikimedia.org/T312666