[08:56:34] FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-aux 3.99% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:01:34] RESOLVED: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-aux 3.984% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:48:56] FIRING: PrometheusSeriesCreationRateAnomalyHigh: Anomalous increase in series creation rate on ops (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks - https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=ops&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DPrometheusSeriesCreationRateAnomalyHigh [16:34:35] FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-aux 3.99% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:49:11] FIRING: PrometheusSeriesCreationRateAnomalyHigh: Anomalous increase in series creation rate on ops (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks - https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=ops&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DPrometheusSeriesCreationRateAnomalyHigh [16:54:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.99% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:04:34] RESOLVED: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.977% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [18:29:35] FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-aux 3.991% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [18:49:35] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.99% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:04:35] RESOLVED: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.969% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:06:34] FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-aux 3.942% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:11:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.967% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:49:11] FIRING: PrometheusSeriesCreationRateAnomalyHigh: Anomalous increase in series creation rate on ops (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks - https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=ops&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DPrometheusSeriesCreationRateAnomalyHigh [21:01:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.877% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:06:34] RESOLVED: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.878% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:19:35] FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-aux 3.99% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:39:35] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.99% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:04:35] RESOLVED: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.786% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:16:05] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.787% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:36:04] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-aux 3.989% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace