[06:37:38] <marostegui>	 This is being fun https://phabricator.wikimedia.org/P52505 XD
[06:38:52] <marostegui>	 https://media.tenor.com/gTg8ZSZMR6YAAAAC/scaler-create-impact.gif
[07:34:58] <jynus>	 db1128 failed again? Or at least I think it crashed? but it has 10.4?
[07:37:41] <marostegui>	 Yeah, I have been debuging it now with Marko
[07:37:44] <marostegui>	 I think we know what it is
[07:41:49] <jynus>	 did you see https://phabricator.wikimedia.org/T346233#9163481 ? I was told it was you who asked for it.
[07:42:18] <marostegui>	 I asked for a way to report when it'd be time to create more tables
[07:42:28] <marostegui>	 I didn't mind which way we decide how to measure it, so that's cool
[07:42:42] <marostegui>	 I don't want to end forgetting about when we need to create new tables
[07:42:55] <marostegui>	 So if long running backups is the unit we want to use to measure when that time has arrived, that's cool with me!
[07:43:19] <jynus>	 yeah, so I was asking if that seemed reasonable- that was will also help me for my work
[07:43:30] <jynus>	 *that way
[07:43:45] <marostegui>	 yeah, that works for me. Maybe just add a message so we don't forget that it is about that it might be time to create new tables
[07:43:56] <marostegui>	 Otherwise imagine in 2 years we receive that message and we'd be like: what is this for? 
[07:44:01] <jynus>	 yeah, of course, text is not final
[07:44:07] <marostegui>	 great!
[07:54:52] <Amir1>	 have fun debugging that :D
[08:52:08] <Amir1>	 moritzm: we can't do any db maint anymore until a couple of days after switchover so if you want to reboot cumin1001, that's your chance :P
[08:52:47] <moritzm>	 oh, thanks for the hint!
[08:53:16] <moritzm>	 jynus: would a cumin1001 reboot today work for backups and if so, at what time?
[08:53:30] <jynus>	 let me make sure they had finished
[08:53:55] <jynus>	 they have indeed, so green light for me
[08:57:55] <moritzm>	 ack, thanks. I'll check what else is currently running there and will sync with the -sre channeö
[09:48:10] <marostegui>	 multi master is now enabled everywhere
[10:02:13] <moritzm>	 ok to reboot dborch1001 now?
[10:02:24] <marostegui>	 yeah
[10:05:51] <Amir1>	 reboot all the things
[10:07:58] <moritzm>	 I'll wait a bit until the current parsoid thing has calmed down
[10:14:03] <moritzm>	 going ahead in 5m
[10:27:05] <moritzm>	 Orchestrator is back
[10:27:12] <marostegui>	 excellent!
[10:49:57] <marostegui>	 wow pagelinks in enwiki is 169GB
[10:56:53] <Amir1>	 yup. it will be less than half in a couple of months :D
[12:42:07] <marostegui>	 https://phabricator.wikimedia.org/T345509#9166433
[12:54:53] <Emperor>	 ouch
[12:58:21] <marostegui>	 And sometimes people wonder we we test so much...
[12:58:29] <marostegui>	 *why
[13:03:24] <Amir1>	 marostegui: do you want me to run optimize table on all tables in all replicas?
[13:03:34] <Amir1>	 it's gonna take a while but doable
[13:05:09] <marostegui>	 no, let's not do that for now
[13:05:22] <Amir1>	 noted
[13:05:54] <marostegui>	 maybe we can on candidate masters though