[00:34:09] <jinxer-wm>	 FIRING: [10x] DiskSpace: Disk space backup1004:9100:/srv/objectstorage 0.6979% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[04:34:09] <jinxer-wm>	 FIRING: [10x] DiskSpace: Disk space backup1004:9100:/srv/objectstorage 0.6973% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[08:34:09] <jinxer-wm>	 FIRING: [10x] DiskSpace: Disk space backup1004:9100:/srv/objectstorage 0.6972% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[08:36:04] <Emperor>	 jynus: is it worth silencing those alerts for a bit? I assume you're more-than-aware of the disk shortage issues :)
[12:34:09] <jinxer-wm>	 FIRING: [10x] DiskSpace: Disk space backup1004:9100:/srv/objectstorage 0.6967% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[13:26:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db2231:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[15:33:37] <Amir1>	 marostegui: https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=db2147&var-datasource=000000026&var-cluster=mysql&viewPanel=panel-28&from=now-12h&to=now&timezone=utc 50GB reduction in size of s4 courtesy of zabe <3
[15:34:04] <marostegui>	 Nice, what was the change?
[15:34:12] <marostegui>	 I am also running the revision changes on s4 now
[15:34:23] <marostegui>	 it will take a few weeks
[15:35:47] <Amir1>	 rev_sha1 drop
[15:36:13] <Amir1>	 in s8 it'll be around the double given that it has around twice as many rows
[15:38:53] <marostegui>	 Amir1: ah sorry, I thought that drop came from something different :)
[16:34:09] <jinxer-wm>	 FIRING: [10x] DiskSpace: Disk space backup1004:9100:/srv/objectstorage 0.696% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[17:26:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db2231:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:32:38] <jynus>	 I acked the disk alerts and restarted this ^
[17:33:19] <jynus>	 I am not touching the same on sretest2*, unsure what's that, but probably some reimaging test, not gonna touch it
[17:36:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db2231:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed