[00:53:43] <jinxer-wm>	 FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.992% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[01:03:34] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.981% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[02:07:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.96% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[03:02:34] <jinxer-wm>	 FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.894% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[06:07:34] <jinxer-wm>	 FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.975% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[06:45:05] <godog>	 hi folks, I got this for your eyes https://gerrit.wikimedia.org/r/c/operations/puppet/+/1342971?usp=dashboard
[07:02:34] <jinxer-wm>	 FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.804% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[07:12:34] <jinxer-wm>	 FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.99% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[09:02:34] <jinxer-wm>	 RESOLVED: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.703% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[09:03:05] <jinxer-wm>	 FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.955% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[09:07:49] <jinxer-wm>	 RESOLVED: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.704% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[13:09:32] <godog>	 hi folks, I got this for your eyes https://gerrit.wikimedia.org/r/c/operations/puppet/+/1342971
[13:09:52] <godog>	 hah, nevermind I just saw the +1, thank you tappof 
[15:03:46] <denisse>	 godog: Nice to see you around here!! :D We miss you!! <3
[15:05:31] <godog>	 denisse: awww thank you <3 <3 nice to be here
[19:50:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[19:54:29] <jinxer-wm>	 FIRING: SystemdUnitFailed: kafka.service on kafka-logging1006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[19:55:34] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[19:59:29] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: kafka.service on kafka-logging1006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[20:09:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 0.1262% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[20:14:29] <jinxer-wm>	 FIRING: SystemdUnitFailed: kafka.service on kafka-logging1006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[20:49:34] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 1.18e-07% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[20:59:29] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: kafka.service on kafka-logging1006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[21:37:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.678% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[21:47:34] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 2.255% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[21:54:03] <denisse>	 ^ Removed unused packages, if it keeps firing I'll see if we can increase the disk.
[21:54:29] <jinxer-wm>	 FIRING: SystemdUnitFailed: kafka.service on kafka-logging1006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed