[02:30:50] <jinxer-wm>	 FIRING: [7x] DiskSpace: Disk space backup1004:9100:/srv/objectstorage 3.807% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[06:30:50] <jinxer-wm>	 FIRING: [7x] DiskSpace: Disk space backup1004:9100:/srv/objectstorage 3.798% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[08:24:27] <Emperor>	 Anyone know about backups running out of disk? I see a note in the weekly minutes from jy.nus that he was working on it back in November, and then the essential work doc on 8th Dec says "...media backups, which will run out of space soon".
[08:25:18] <Emperor>	 kwakuofori: do you happen to know what jy.nus' thoughts were on media backups running short of space?
[08:26:11] <Emperor>	 (if the answer is "they're going to run out, nothing to be done about it 'til January", then I'll silence the alert until 6th Jan)
[08:29:51] <marostegui>	 Emperor: He brified me on that, that it will be expected for them to fail during the break and nothing can be done. I don't know anything else
[08:30:09] <marostegui>	 briefed
[08:31:36] <Emperor>	 marostegui: OK, thanks, I'll silence those alerts 'til he's back then.
[08:31:44] <marostegui>	 thank you
[08:33:40] <Emperor>	 {{done}}
[09:33:55] <kwakuofori>	 Emperor, marostegui: that's correct. we've asked to expand h/w for that. thanks for silencing
[13:30:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be2069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:55:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: swift_rclone_sync.service on ms-be2069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[22:29:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es2028:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed