[10:16:36] marostegui, Amir1: any objection to me schema-changing s5? [10:18:53] mm, looks like neither of you have anything relevant running on any of the cumin hosts [10:18:55] hi all, any dba around that can help finding out the process to recover the mysql replication after a disk out of space issue on the master? [10:19:35] (trying to search on wikitech, but not finding too much, might end up just stopping the slave, copying over the data, and starting again kind-of-thing) [10:19:53] has the primary recovered ok? [10:22:36] yes, the primary is up and running [10:23:10] the secondary is stuck complaining about a broken binlog event [10:25:21] how much data are we talking about? [10:26:16] ~2.7TB [10:26:22] oof. [10:27:21] yep :S [10:27:40] can you paste the full output of `show slave status` somewhere? [10:27:51] (to be clear: i don't know if i can help, but i seem to be the only one around :)) [10:29:16] there you go: https://paste.toolforge.org/view/3081aeaf [10:29:57] how large is log.275246 on the primary? (in bytes) [10:30:40] 15499264 [10:30:49] (~15MB) [10:31:42] mmm [10:31:56] ok. first thing i would try is simply `stop slave; start slave`, and see what happens. [10:32:12] done, still stuck [10:32:51] ok. so my best guess is that the primary does not have a complete binlog [10:33:08] in which case there's no way to recover, and you have to repopulate the data [10:33:26] that was my fear yep [10:35:43] having caught up on email, manual is not around for the moment, so i have to recommend biting the bullet and going with that approach. sorry, wish i could have better news [10:37:03] okok, no problem, I'll try to dig up any info on how we did it in the past and well, go at it xd [14:26:14] I just woke up [14:26:44] dcaro: I might be able to help, do you still need to fix the replication? [14:27:44] kormat: I'm finalizing the bot that reports maint work [14:34:42] thank Amir1, replication is running, just skipped the event itself for now [14:35:29] dcaro: yup, that would do the trick :) [15:42:42] kormat: you don't need to run the schema change on s8 and s4. Those don't have flaggedrevs enabled [15:44:28] and size of the table went from 30GB to 11GB ^^ [15:44:42] (in dewiki) [16:30:21] Amir1: oh, great thanks. marked as such in the task.