[07:05:49] marostegui: thanks for the advice [08:39:41] marostegui: I'm thinking of splitting "Blocked" column of Blocked-on-schema-changes tag to two columns: 1- Things are blocked on DBAs, e.g. primary switchover 2- Things that are blocked on the person requesting schema changes (e.g. clarification on something, clean up of data, etc.) [08:40:12] once we do a switchover, we can check which ones now can be considered done [08:40:30] Amir1: If something is tagged with blocked-on-schema change, there should not be anything from the outside blocking it. That tag is supposed to be used when a schema change patch was merged and everything is ready to be deployed in prod [08:40:48] if there's something blocked on the outside, we should probably removed that tag and let it be with either Schema-change and DBA or both [08:41:07] hmm, ok then [08:41:33] otherwise it can be very confusing I think [08:41:44] yeah [08:41:45] and we might end up with another DBA workboard [10:34:41] marostegui: `drop user if exists 'root'@'10.192.48.16';` - running that against all primaries should do the trick, right? (with repl enabled) [10:34:59] (for removing the cumin2001 grants) [10:35:13] kormat: yep, the if exists should prevent weird stuff on replication [10:35:26] that's my favourite kind of weird stuff [11:00:50] relogging at the last minute with google... [13:13:52] afk [13:38:12] marostegui: er, crap. i have an s6 primary switch scheduled for tomorrow morning :( [13:38:30] kormat: :(, maybe we can move it to next week? [13:45:35] marostegui: updated the ticket with comms, removed from calendar, and Deployments. that's hopefully everything. [13:47:55] ok! [14:05:00] Amir1: hi. the grant for 'root'@'10.%' on db1128 has the same password as the usual root@cumin2001 grant ... [14:05:04] is this a knokwn issue? [14:05:08] (db1128 == m1) [14:08:19] cannot say, but the person that may know may be afk ATM [14:09:06] although I remember hearing misc hosts were not reviewed yet [14:41:20] kormat: I honestly don't know what is the situation with root grants are [14:41:48] ok :) [15:03:23] I broke replication to s2 on codfw, fixing it [15:52:49] fixed fully btw [16:51:17] would you be ok if we put the mailman recovery on the SRE meeting doc? [16:52:01] it is not super-important, and I would not consider it an incident, but I would like to highlight the importance or recovery and regular backups [16:55:55] i'm seeing a lot of icinga checks in UNKOWN: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=^\(db\|es\|pc\)[12]&style=detail&servicestatustypes=29 [16:56:11] most with the message: [16:56:11] > http://prometheus.svc.eqiad.wmnet/ops/api/v1/query error while decoding json: Expecting value: line 1 column 1 (char 0) [16:56:43] prometheus issue? [16:58:36] See operations- filippo is doing something, maintenance? [16:59:15] yeah, seems like it. they're going away, too. [16:59:58] sobanski: ^for my :52 question? [17:00:30] Sure thing [17:00:41] (with highlight I don't mean bold, I don't think it is important enogh to discuss it) [17:02:07] the s6 switch isn't happening so we can also remove it [17:02:41] Done [17:04:37] thanks