[08:30:52] <jynus>	 replication broke on m5
[08:32:31] <jynus>	 so I am not crazy, someone is modifying dump grants
[08:40:06] <jynus>	 what is going on???? https://phabricator.wikimedia.org/P60740
[08:40:51] <jynus>	 that is for db2135, the codfw master
[08:40:59] <jynus>	 which should not have dumping grants in the first place
[08:42:19] <jynus>	 is there any script automating grants?
[08:43:16] <jynus>	 marostegui: I am lost, how to proceed?
[08:44:12] <arnaudb>	 oh 
[08:44:17] <arnaudb>	 it's me
[08:44:33] <jynus>	 but I am guessing not manual/intended?
[08:44:39] <arnaudb>	 indeed
[08:44:46] <arnaudb>	 I muste have grepped a bit too much on the grants I'm trying to fix
[08:44:51] <arnaudb>	 it explains why I was having trouble :/
[08:44:52] <jynus>	 your script may have a bug :-D
[08:45:26] <jynus>	 also, grants for the dump user should never exist on the masters or regular slaves
[08:45:44] <arnaudb>	 ah
[08:46:10] <jynus>	 that is why the template has those separate on puppet with dump-
[08:46:33] <jynus>	 in any case, you don't have to handle that- just ping me and I will do it
[08:47:01] <arnaudb>	 thanks jynus, did I broke dump somehow?
[08:47:10] <jynus>	 not dumps, production broke
[08:47:21] <arnaudb>	 😬
[08:47:25] <jynus>	 replication broke
[08:47:47] <marostegui>	 yeah
[08:47:47] <jynus>	 let's start by fixing that
[08:47:49] <marostegui>	 I am going to fix that
[08:48:01] <jynus>	 did you ran anyhing other than grant? to skip it
[08:48:17] <jynus>	 then I will help cleaning up the grants
[08:48:25] <jynus>	 but waiting for manuel
[08:48:34] <marostegui>	 I am going to skip those specific transactions related to grants
[08:48:36] <jynus>	 don't want to touch the same server twice
[08:48:44] <jynus>	 marostegui: I will let you do that
[08:48:53] <jynus>	 let's sync when you are done
[08:49:05] <marostegui>	 yep, I will get back to you, there are quite a few
[08:50:02] <arnaudb>	 I've dropped/created databases but I was struggling with grants before this
[08:50:17] <marostegui>	 yeah, right now there are so many striker grants on m5 that I am not sure which ones are correct anymore
[08:50:29] <arnaudb>	 you can drop all that looks like toolsbeta
[08:50:46] <arnaudb>	 as it was the ones I was trying to create
[08:50:54] <jynus>	 you really like toolsbeta :-D `strikertoolsbetatoolsbetatoolsbetatoolsbeta_toolsbeta`
[08:51:07] <arnaudb>	 :D
[08:51:43] <jynus>	 after that, let's meet for a mysql grant and revoke primer (if you want)
[08:51:54] <jynus>	 a session
[08:52:36] <arnaudb>	 after I pick back mariadbtest to create myself a sandbox, avoiding this kind of event 
[08:52:40] <jynus>	 in any case, you don't need to handle dump grants, I do
[08:52:42] <arnaudb>	 but gladly jynus thanks!
[08:54:42] <marostegui>	 jynus: I think I have fixed everything, and dropped the NOT dump grants. So can you go ahead and check which dump users have to be and drop those that aren't needed on the master? (and slaves)
[08:54:57] <jynus>	 yeah, I will do a cleanup now
[08:55:06] <jynus>	 if you finish touching it?
[08:56:02] <jynus>	 in general, there should not be dump hosts on live production ones, with the exception of es* hosts
[08:56:09] <jynus>	 *dump grants
[08:57:50] <marostegui>	 jynus: I am fully done
[08:58:50] <arnaudb>	 thanks guys <3
[08:59:07] <marostegui>	 arnaudb: don't worry about it, mistakes happens :)
[09:01:51] <jynus>	 the masters are now cleanup, I will now handle the replicas
[09:07:26] <jynus>	 I cleaned up db1217
[09:07:34] <jynus>	 db2160 next
[09:21:33] <jynus>	 so both puppet/modules/profile/templates/mariadb/grants/production-m5.sql.erb and dumps-{eqiad,codfw}-m5.sql.erb need update
[09:21:46] <jynus>	 can you help with the first one, arnaudb?
[09:21:53] <jynus>	 I will handle the dump grants
[09:22:53] <arnaudb>	 yep
[09:23:27] <marostegui>	 it is probably easier to revert https://gerrit.wikimedia.org/r/c/operations/puppet/+/1018408 and start from scratch 
[09:24:05] <marostegui>	 Or maybe not, I am not following closely for now :)
[10:54:48] <jynus>	 sorry, got distracted by outage
[10:54:55] <jynus>	 will go back to you soon
[11:04:45] <Amir1>	 marostegui arnaudb: Planning to start a schema change on s8, there is anything on the map for today but double checking just in case
[11:05:21] <marostegui>	 not from my side
[11:05:22] <arnaudb>	 looks good to me!
[11:06:28] <Amir1>	 awesome
[16:10:29] <cdanis>	 marostegui: some good news for you
[16:12:50] <cdanis>	 https://phabricator.wikimedia.org/T360029#9723385
[16:20:30] <marostegui>	 Oh nice