[08:15:26] Amir1: I am finally updating the documentation on wikitech XD [08:15:55] haha, nice [08:15:57] thanks! [10:02:15] marostegui: db1156 (s2) has these directories: [10:02:19] root@db1156:/srv# ls [10:02:19] sqldata sqldata.bak tmp [10:02:34] shall I delete the second and the third? [10:05:14] only .bak [10:05:21] tmp is needed [10:18:22] yeah, okay, gonna clean up the bak ones [10:20:31] marostegui: next one, db2102 (s1) has sqldata.percona [10:20:44] last modifies Jan 20 2021 [10:20:50] yeah, can be deleted [10:21:06] done [10:21:51] now, db2096 (x1): sqldata.old last modified Oct 15 2018 [10:21:57] yes [10:22:10] awesome [10:22:51] so far all seems to be okay now, the misc cluster is quite weird though [10:23:34] https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=db1156&var-datasource=thanos&var-cluster=wmcs&from=now-1h&to=now&viewPanel=28 [10:23:44] 🙈 [10:40:02] there were some binlogs in db2102, I cleaned them too [11:40:06] db2100 seems to be leaking memory [11:40:12] I am going to restart it [11:41:23] not so sure it is leaking, maybe just overcommited [11:44:01] https://gerrit.wikimedia.org/r/c/operations/puppet/+/867548 [11:45:09] ok for me to restart db2100? [11:45:57] it's a backup source isn't it? [11:46:45] yes, s7 + s8 [11:46:59] I just +1ed [11:47:18] ok, doing [13:39:52] >Using index condition; Using where; Rowid-ordered scan; Using filesort [17:47:16] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (an-coord1001:9104) - TODO - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [17:51:16] Reedy: is there anything it's not using? :P [21:47:31] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (an-coord1001:9104) - TODO - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed