[00:49:11] <jinxer-wm>	 FIRING: PrometheusSeriesCreationRateAnomalyHigh: Anomalous increase in series creation rate on ops (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks - https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=ops&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DPrometheusSeriesCreationRateAnomalyHigh
[03:01:05] <jinxer-wm>	 FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.688% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[03:21:05] <jinxer-wm>	 FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.99% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[04:49:11] <jinxer-wm>	 FIRING: PrometheusSeriesCreationRateAnomalyHigh: Anomalous increase in series creation rate on ops (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks - https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=ops&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DPrometheusSeriesCreationRateAnomalyHigh
[05:01:05] <jinxer-wm>	 FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.753% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[05:06:05] <jinxer-wm>	 RESOLVED: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.753% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[05:42:35] <jinxer-wm>	 FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-aux 3.99% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[06:02:35] <jinxer-wm>	 FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.979% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[07:02:35] <jinxer-wm>	 FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.829% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[07:12:35] <jinxer-wm>	 FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.99% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[07:43:56] <jinxer-wm>	 RESOLVED: PrometheusSeriesCreationRateAnomalyHigh: Anomalous increase in series creation rate on ops (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks - https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=ops&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DPrometheusSeriesCreationRateAnomalyHigh
[07:52:35] <jinxer-wm>	 RESOLVED: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.918% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[08:50:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: grafana-image-renderer.service on grafana2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:55:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: grafana-image-renderer.service on grafana2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:00:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: grafana-image-renderer.service on grafana2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:10:25] <jinxer-wm>	 RESOLVED: [2x] SystemdUnitFailed: grafana-image-renderer.service on grafana2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:24:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: report2drive-quarter_detail.service on grafana1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:26:52] <tappof>	 ^^ it's me
[10:29:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: report2drive-quarter_detail.service on grafana1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed