[02:04:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[02:19:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[10:23:13] <topranks>	 hey folks if anyone had time for a quick review:
[10:23:14] <topranks>	 https://gerrit.wikimedia.org/r/c/operations/alerts/+/1198926
[10:23:40] <topranks>	 I merged a patch for some ospf alerts but they aren't working, seems I used the wrong label name though so hopefully a simple fix 
[10:57:53] <tappof>	 hi Cathal, reviewed
[10:58:27] <topranks>	 tappof: thanks! 
[15:40:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: thanos-rule-reload@pilot.service on titan1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[15:55:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: thanos-rule-reload@pilot.service on titan1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[16:47:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on titan1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[16:57:48] <jinxer-wm>	 FIRING: [2x] PuppetFailure: Puppet has failed on titan1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[17:02:48] <jinxer-wm>	 FIRING: [3x] PuppetFailure: Puppet has failed on titan1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[17:12:48] <jinxer-wm>	 FIRING: [4x] PuppetFailure: Puppet has failed on titan1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[17:37:48] <jinxer-wm>	 FIRING: [2x] PuppetFailure: Puppet has failed on titan1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[17:52:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on titan2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure