[07:41:36] x1 has been slowly but consistently reducing its dump size since around May 10, probably some purging that was enabled by some team? [07:45:09] seems to be related to echo [07:59:07] seems to be related to echo [08:02:10] :) [08:32:16] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db2175:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [08:37:10] ^ me [09:12:16] (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db2175:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [09:17:56] jynus: https://phabricator.wikimedia.org/T310070 reminder that b1, B2, B4 and B5 are going on maintenance today [09:18:19] i have stopped the db hosts, but there are some dbprov* there [09:18:28] in case you need something [09:20:06] Emperor: I have https://gerrit.wikimedia.org/r/c/operations/puppet/+/819503 out to fix a couple of things in grow_ssd_part, though after that I'm ready to actually --doit, I was thinking of adding 100G [09:20:10] on thanos-be that is [09:27:13] thanks, I wasn't aware of that [09:29:21] godog: thanks, sounds reasonable to me [09:29:47] marostegui: also A7, apparently? T309957 [09:29:47] T309957: (Need By:TBD) rack/setup/install row A new PDUs - https://phabricator.wikimedia.org/T309957 [09:30:27] Emperor: ok! thanks for the review [09:30:47] jynus: ys but we don't have anything in a7 [09:38:39] I can now shutdown dbprovs as m2 backup was still running (6h 12m) [09:40:51] no (so far) ongoing downtime issues after icinga restart, right? [09:41:00] yep all fine [09:52:50] question, is auto_schema only used for mediawiki replicas? How do you handle non-mw hosts? [11:00:27] Emperor: ok https://gerrit.wikimedia.org/r/c/operations/puppet/+/819522 should be the final one, tested on sda on thanos-be2004 and works as expected [11:00:49] if that looks good I'll roll it out to the rest of the thanos-be hosts this afternoon