[02:43:25] FIRING: SystemdUnitFailed: sync-icinga-state.service on alert2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:59:34] FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.992% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:04:34] RESOLVED: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.99% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:38:25] RESOLVED: SystemdUnitFailed: sync-icinga-state.service on alert2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:07:34] FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.966% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:07:34] RESOLVED: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.848% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:07:34] FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.971% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:02:34] RESOLVED: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.895% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:08:34] FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.992% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:23:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.989% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:03:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.866% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:23:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.987% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:03:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.821% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:23:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.993% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:03:35] RESOLVED: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.973% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:21:04] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.657% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:36:04] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.989% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:06:04] RESOLVED: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.693% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [18:11:34] FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.992% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:06:34] RESOLVED: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.922% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:26:34] FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.993% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:41:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.987% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:06:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.894% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [22:41:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.992% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:06:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.827% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:56:34] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.992% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace