[05:42:20] <Amir1>	 optimize table not replicating bit me again.
[05:42:35] <Amir1>	 *put significant amount of swear words* 
[05:59:00] <_joe_>	 there is a good reason for it not replicating heh
[05:59:26] <marostegui>	 Amir1: what?
[05:59:55] <Amir1>	 optimize table not replicating to replicas, the mariadb bug you filed
[06:00:03] <marostegui>	 Amir1: https://phabricator.wikimedia.org/T296143#7519268
[06:00:06] <marostegui>	 I told you XD
[06:00:18] <Amir1>	 I know but I forgot :P
[06:00:25] <marostegui>	 that bug is so stupid
[06:00:48] <Amir1>	 like I did it in codfw but with the one to the cloud, I forgot
[06:04:30] <Amir1>	 marostegui: thanks to this, I need to run it on sanitarium master, just down time it and its replicas. Correct?
[06:05:09] <marostegui>	 Amir1: yep, that should be enough
[06:05:16] <Amir1>	 Thanks
[06:23:44] <_joe_>	 marostegui: I kinda remember optimize statement didn't replicate on myisam because of table-level locking
[06:24:10] <_joe_>	 but am I misremembering? maybe we had to explicitly skip replication
[06:24:16] <_joe_>	 my dba skills are so rusty
[06:24:23] <marostegui>	 _joe_: I think it does unless you tell it otherwise
[06:24:40] <marostegui>	 in this case, they've decided to change the behaviour of something without giving a heads up and apparently without even internal discussion
[06:25:18] <marostegui>	 _joe_: this is the bug I filed, and this is the comment where it looks like Monty decided to change it without asking anyone https://jira.mariadb.org/browse/MDEV-26618?focusedCommentId=199597&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-199597
[06:25:46] <_joe_>	 marostegui: rotfl
[06:25:50] <_joe_>	 and in a minor?
[06:26:42] <marostegui>	 ¯\_(ツ)_/¯
[06:27:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db1139:13311)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org
[06:27:40] <Amir1>	 I wonder when are we moving to mysql :P
[06:27:56] <marostegui>	 that prometheous alert is weird....
[06:28:05] <marostegui>	 Ah, I think I know why
[06:35:47] <_joe_>	 I hate the format of those IRC announcements
[06:42:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db1139:13311)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org
[08:26:44] <Emperor>	 I presume the formatting could be adjusted
[08:27:20] <Emperor>	 (the resolved format does contain the word "failed" quite a bit :-/ )
[08:28:12] <_joe_>	 Emperor: I mostly hate the (PrometheusMysqldExporterFailed) prefix
[08:47:26] <Emperor>	 it does seem a bit superfluous
[08:47:52] <Amir1>	 there is no natural rhythm in the message. For example, as a rule of thumb you should be able to remove anything inside () and it should still meaningful. Well in here, it's opposite
[08:48:40] <Amir1>	 marostegui: btw, I'm turning on actor write both on every wiki now. Buckle up
[08:56:33] <Emperor>	 the alert is defined here https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/refs/heads/master/team-data-persistence/prometheus-mysqld-exporter.yaml 
[09:00:33] <marostegui>	 Amir1: nice!
[09:00:54] <Amir1>	 so far everything looks good, I want to start the script on s3 later today
[09:01:30] <Amir1>	 the reason I want to do it per section is that later I want to parallelize them per section (otherwise it would take a year to finish)
[09:02:11] <Amir1>	 the fastest I can get, enwiki would take two months
[09:05:48] <Amir1>	 (keep in mind, we need to do this all over again with comment)
[09:29:28] <jynus>	 why did db1139 alerted? It is being imported, but in theory, prometheus should be working fine
[09:29:48] <marostegui>	 jynus: it had an empty password
[09:32:40] <jynus>	 I am confused- how did it work before? I setup prometheus account yesterday and it worked
[09:33:06] <marostegui>	 jynus: yeah, it was configured rightly (using the unix socket) but it also had the password as empty, so no idea
[09:33:11] <marostegui>	 i recreated it and it is all fine now
[09:33:18] <marostegui>	 no alerts, and graphing things correctly
[09:33:41] <marostegui>	 also ran the empty password script and it wasn't reported anymore
[09:54:13] <jynus>	 some backups failed due to wrong permissions on some files due to reimage changing uids
[09:54:20] <jynus>	 I am fixing that
[10:06:55] <jynus>	 and in the case of m1 it was a problem with bad grants for 2 out of 3 dbprov servers
[12:55:49] <Amir1>	 marostegui: btw, I'm not using the scripts I wrote to do schema changes but It prints commands it would run, I'm running them. that actually makes me avoid typos (depooling wrong db :D) or see if the code has any bugs 
[12:56:04] <Amir1>	 and working well so far \o/
[12:56:29] <marostegui>	 That's good!
[12:56:40] <marostegui>	 What is important are the safety checks to run before and after the deployment
[12:56:47] <marostegui>	 ie: Does this host have replicas? if so, then....
[12:56:52] <marostegui>	 that sort of thing
[12:57:09] <Amir1>	 yeah
[19:13:27] <razzi>	 Hey DBAs, going to create new view for this patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732740
[19:13:53] <razzi>	 It's a new view so it shouldn't cause trouble