[01:30:35] <jinxer-wm>	 FIRING: DiskSpace: Disk space logging-sd1001:9100:/srv 2.806% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=logging-sd1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[01:40:35] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space logging-sd1001:9100:/srv 2.511% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=logging-sd1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[02:37:03] <jinxer-wm>	 FIRING: [2x] PuppetFailure: Puppet has failed on alert1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[06:37:03] <jinxer-wm>	 FIRING: [2x] PuppetFailure: Puppet has failed on alert1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[06:57:19] <denisse>	 ^ looking...
[07:06:46] <denisse>	 It was me, I broke it with this change, reverting: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1335233
[07:21:48] <jinxer-wm>	 FIRING: [2x] PuppetFailure: Puppet has failed on alert1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[07:23:21] <denisse>	 Should resolve now, I'm not sure why it failed tbh, I followed the instructions from git::clone https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/git/manifests/clone.pp#61
[07:26:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on alert2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[07:31:42] <denisse>	 Found the culprit, it was a small typo... 🤦‍♀️
[10:34:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[10:44:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[13:18:07] <inflatador>	 denisse apologies about the puppet merge yesterday, I guess I forgot to deploy it ;( https://gerrit.wikimedia.org/r/c/operations/puppet/+/1338042
[19:28:07] <denisse>	 inflatador: Hey, no worries!! It's all good. 🤗