[04:20:52] <jinxer-wm>	 FIRING: [2x] ThanosRuleSenderIsFailingAlerts: Thanos Rule is failing to send alerts to alertmanager. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/35da848f5f92b2dc612e0c3a0577b8a1/thanos-rule - https://alerts.wikimedia.org/?q=alertname%3DThanosRuleSenderIsFailingAlerts
[04:23:17] <denisse>	 ^ That's me, I was working on the alert hosts, should resolve soon.
[04:23:25] <jinxer-wm>	 FIRING: [4x] SystemdUnitFailed: check_icinga-alert1002.service on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[04:24:35] <denisse>	 ^ Also me, keyholder needed to be armed, done.
[04:25:52] <jinxer-wm>	 RESOLVED: [2x] ThanosRuleSenderIsFailingAlerts: Thanos Rule is failing to send alerts to alertmanager. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/35da848f5f92b2dc612e0c3a0577b8a1/thanos-rule - https://alerts.wikimedia.org/?q=alertname%3DThanosRuleSenderIsFailingAlerts
[04:28:25] <jinxer-wm>	 RESOLVED: [4x] SystemdUnitFailed: check_icinga-alert1002.service on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed