[00:02:42] <jinxer-wm>	 (SystemdUnitFailed) firing: statograph_post.service Failed on alert2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[00:07:42] <jinxer-wm>	 (SystemdUnitFailed) resolved: statograph_post.service Failed on alert2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[05:36:26] <jinxer-wm-test>	 (ThanosQueryIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosQueryIsDown
[05:36:27] <jinxer-wm-test>	 (ThanosRuleIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosRuleIsDown
[05:36:27] <jinxer-wm-test>	 (ThanosCompactIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosCompactIsDown
[05:36:31] <jinxer-wm-test>	 (ThanosStoreIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosStoreIsDown
[05:36:35] <jinxer-wm-test>	 (ThanosSidecarIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosSidecarIsDown
[07:38:29] <jinxer-wm>	 (RSyslogDeliveryFailure) firing: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog  - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure
[07:43:29] <jinxer-wm>	 (RSyslogDeliveryFailure) resolved: (2) rsyslog is failing to deliver messages - https://wikitech.wikimedia.org/wiki/Rsyslog  - https://alerts.wikimedia.org/?q=alertname%3DRSyslogDeliveryFailure
[08:06:23] <jinxer-wm-test>	 (ThanosSidecarIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosSidecarIsDown
[08:06:23] <jinxer-wm-test>	 (ThanosQueryIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosQueryIsDown
[08:06:23] <jinxer-wm-test>	 (ThanosStoreIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosStoreIsDown
[08:06:27] <jinxer-wm-test>	 (ThanosRuleIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosRuleIsDown
[08:06:32] <jinxer-wm-test>	 (ThanosCompactIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosCompactIsDown
[09:41:08] <jinxer-wm-test>	 (ThanosCompactIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosCompactIsDown
[09:41:08] <jinxer-wm-test>	 (ThanosStoreIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosStoreIsDown
[09:41:08] <jinxer-wm-test>	 (ThanosRuleIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosRuleIsDown
[09:41:13] <jinxer-wm-test>	 (ThanosSidecarIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosSidecarIsDown
[09:41:17] <jinxer-wm-test>	 (ThanosQueryIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosQueryIsDown
[12:11:03] <jinxer-wm-test>	 (ThanosQueryIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosQueryIsDown
[12:11:03] <jinxer-wm-test>	 (ThanosSidecarIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosSidecarIsDown
[12:11:03] <jinxer-wm-test>	 (ThanosStoreIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosStoreIsDown
[12:11:07] <jinxer-wm-test>	 (ThanosRuleIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosRuleIsDown
[12:11:12] <jinxer-wm-test>	 (ThanosCompactIsDown) firing: Thanos component has disappeared. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/0cb8830a6e957978796729870f560cda/thanos-overview - https://alerts.monitoring.wmflabs.org/?q=alertname%3DThanosCompactIsDown
[15:21:25] <godog>	 I've silenced the thanos test alerts, the nail in the coffin being having an actual blackhole for testing purposes
[16:55:03] <Guest1692>	 godog: I may have re enabled that after pushing my changes to pontoon. Apologies for that.
[17:04:23] <godog>	 Guest1692: sure np