[00:22:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on aqs1021:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[00:37:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on aqs1021:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[04:49:38] <marostegui>	 I think the check_data_report for checking new wikis isn't working, we've not received any emails about the new wiki at https://phabricator.wikimedia.org/T390710
[05:01:59] <marostegui>	 It seems to be working, looks like I started way too early to work today and the cron didn't start yet XD
[05:40:24] <marostegui>	 switched es1 master in eqiad
[07:38:18] * Emperor still looking for reviews of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1138830 and its 2 related changes, please (though I won't deploy 'til Monday now)
[07:38:42] <marostegui>	 done
[07:46:45] <Emperor>	 thanks
[08:24:52] <marostegui>	 switched es1 master in codfw
[12:55:27] <marostegui>	 federico3: https://phabricator.wikimedia.org/T390710#10767298 if we don't get to this today it will be alerting the whole weekend
[12:56:21] <federico3>	 uh?
[12:56:45] <marostegui>	 federico3: Didn't you see today's emails?
[12:56:48] <marostegui>	 The private data ones
[12:57:45] <federico3>	 no, checking now
[12:58:20] <marostegui>	 federico3: Please create a proper filter for those, they are important
[13:01:43] <federico3>	 filter on what parameter?
[13:03:53] <marostegui>	 federico3: Check the email and whatever parameters works for you, but please, address the task and later the filters
[13:10:58] <federico3>	 https://phabricator.wikimedia.org/T390710#10767298 here are you referring to running cookbook sre.mysql.sanitize-wiki --wiki nupwiki --check-only --task T390710 ?  (and later on without --check-only ? )
[13:10:58] <stashbot>	 T390710: Prepare and check storage layer for nupwiki - https://phabricator.wikimedia.org/T390710
[13:11:30] <marostegui>	 federico3: Yes, the redaction cookbook you merged
[13:12:05] <marostegui>	 federico3: https://phabricator.wikimedia.org/T390710#10699405
[13:12:41] <federico3>	 hm , it fails to start 
[13:17:24] <federico3>	 mysql.get_dbs is detecting 2 hosts but the mysql module only handles one
[13:18:18] <marostegui>	 There are 5 hosts to be sanitized, 2 sanitariums (1 per dc) and 3 hosts in eqiad, 2 cloudb* and an-redactteddb1001
[13:18:20] <federico3>	 before I tweak the cookbook: is it expected to have db2186.codfw.wmnet and db1154.eqiad.wmnet here?
[13:19:08] <marostegui>	 Of course
[13:19:41] <marostegui>	 The whole process is at https://phabricator.wikimedia.org/T366146
[13:19:45] <federico3>	 we are filtering by A:db-sanitarium and A:db-section-s5   here
[13:27:21] <federico3>	 ok, it's moving forward with a little tweak
[13:27:30] <marostegui>	 good
[13:34:23] <marostegui>	 federico3: Once done, can you update https://wikitech.wikimedia.org/wiki/MariaDB/PII with how to run the cookbook and all that?
[13:34:34] <federico3>	 ok
[13:35:11] <federico3>	 I'm seeing an empty set of values in Non-public databases that are present , Non-public tables that are present, Unfiltered columns that are present
[13:35:37] <marostegui>	 What?
[13:41:16] <federico3>	 I'm referring to https://phabricator.wikimedia.org/P75464 - plus I'm seeing a glitch similar to the one in the related PR 
[13:42:27] <marostegui>	 s8 shouldn't have any private data
[13:42:30] <marostegui>	 only s5
[13:42:44] <federico3>	 (evidently it was tested on 1-host sets before)
[13:52:25] <federico3>	 now it failed to run the GRANT SELECT... @Amir1 perhaps you have more context about this?
[13:53:01] <marostegui>	 federico3: which command are you running? 
[13:53:26] <federico3>	 marostegui: you mean how I'm calling the runbook or what query is the runbook running?
[13:53:28] <marostegui>	 The exact command is the one I pasted above
[13:53:33] <marostegui>	 the GRANT SELECT
[13:53:39] <federico3>	 spicerack.mysql.MysqlError: Failed to run 'GRANT SELECT, SHOW VIEW ON `nupwiki_p`.* TO `labsdbuser`;' on an-redacteddb1001.eqiad.wmnet
[13:53:59] <marostegui>	 Maybe some escaping or something? what is the more detailed error?
[13:54:27] <federico3>	 sadly it's not logging the error :-/
[13:57:32] <federico3>	 I'm a bit puzzled by this syntax Instance(host, name=self.section)
[13:57:57] <federico3>	 or the original Instance(self.clouddb_hosts.remote_hosts, name=self.section)
[14:05:30] <Emperor>	 Can I get a +1 for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1139046 please, to fix an earlier error? ms-be2089 isn't able to rsync to other nodes ATM
[14:07:29] <Emperor>	 (this does need applying today)
[14:09:00] <Emperor>	 TY :)
[14:12:49] <urandom>	 whoops, A.mir1 beat me to it
[16:49:48] <federico3>	 marostegui: the cookbook requires more work, I suggest we make the create table/grant select by hand for now
[16:55:27] <marostegui>	 I am gone for the day , can you do it?
[16:57:11] <federico3>	 I'm trying and not able to connect to an-redacteddb1001.eqiad.wmnet