[12:34:39] inflatador: I think the network policy allowing traffic from the Prometheus hosts is missing.. [13:18:44] tappof thanks for following up, will take a look [19:27:50] FIRING: SLOMetricAbsent: logstash-availability eqiad - https://slo.wikimedia.org/?search=logstash-availability - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [19:31:00] FIRING: ThanosRecordingRuleGaps: Thanos recording rule has stopped working correctly, check thanos-rule service. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/35da848f5f92b2dc612e0c3a0577b8a1/thanos-rule - https://alerts.wikimedia.org/?q=alertname%3DThanosRecordingRuleGaps [19:32:50] FIRING: [2x] SLOMetricAbsent: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [19:36:00] RESOLVED: ThanosRecordingRuleGaps: Thanos recording rule has stopped working correctly, check thanos-rule service. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/35da848f5f92b2dc612e0c3a0577b8a1/thanos-rule - https://alerts.wikimedia.org/?q=alertname%3DThanosRecordingRuleGaps [19:37:50] RESOLVED: [2x] SLOMetricAbsent: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [21:25:34] FIRING: DiskSpace: Disk space centrallog2002:9100:/srv 3.97% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog2002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace