[00:07:25] FIRING: SystemdUnitFailed: logrotate.service on centrallog1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:09:52] FIRING: [2x] DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.167% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:52:25] FIRING: [2x] SystemdUnitFailed: logrotate.service on centrallog1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:02:25] FIRING: [2x] SystemdUnitFailed: logrotate.service on centrallog1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:02:53] FIRING: [2x] DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.305% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [01:03:48] FIRING: PuppetFailure: Puppet has failed on prometheus3004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:13:57] FIRING: [4x] PuppetFailure: Puppet has failed on prometheus1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:23:49] FIRING: [6x] PuppetFailure: Puppet has failed on prometheus1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:28:49] FIRING: [7x] PuppetFailure: Puppet has failed on prometheus1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:29:43] FIRING: [2x] DiskSpace: Disk space centrallog1002:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [01:37:35] ^ looking. [01:37:53] FIRING: [2x] DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.193% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [01:43:39] All those alerts (except for disk space) are related to the PKI hosts... Checking if they're down. [02:36:42] !log Extended volume on prometheus2007 tor the disk space alert as per https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_host_running_out_of_space [02:36:42] denisse: Not expecting to hear !log here [02:42:21] !log Extended volume on prometheus2008 for the disk space alert as per https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_host_running_out_of_space [02:42:21] denisse: Not expecting to hear !log here [03:38:49] FIRING: [7x] PuppetFailure: Puppet has failed on prometheus1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:43:48] FIRING: [7x] PuppetFailure: Puppet has failed on prometheus1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:48:49] FIRING: [7x] PuppetFailure: Puppet has failed on prometheus1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:53:49] FIRING: [7x] PuppetFailure: Puppet has failed on prometheus1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [04:28:48] FIRING: [2x] PuppetFailure: Puppet has failed on prometheus1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [04:33:48] RESOLVED: [2x] PuppetFailure: Puppet has failed on prometheus1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [05:02:40] FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:29:43] FIRING: DiskSpace: Disk space centrallog1002:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:37:53] FIRING: [2x] DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.058% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:52:53] FIRING: [2x] DiskSpace: Disk space centrallog1002:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:02:40] FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:25:34] hello! in https://integration.wikimedia.org/ci/job/alerts-pipeline-test/3550/console what does "AssertionError: Alert TransportLinksInUsageNoRedundancy is not going to fire: external label reference in non-global alert (team-netops/interfaces.yaml)" mean? [09:25:34] And how to fix it? that's for https://gerrit.wikimedia.org/r/c/operations/alerts/+/1321906 thanks [09:37:53] FIRING: [2x] DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.022% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:54:43] FIRING: [2x] DiskSpace: Disk space centrallog1002:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:02:53] FIRING: [2x] DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.317% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:34:48] FIRING: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:42:54] FIRING: [2x] DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.189% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:44:48] RESOLVED: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:02:40] FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:54:43] FIRING: [2x] DiskSpace: Disk space centrallog1002:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:44:43] FIRING: [2x] DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 2.883% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:02:40] FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:17:25] RESOLVED: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:54:43] FIRING: [2x] DiskSpace: Disk space centrallog1002:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:44:43] FIRING: [2x] DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 2.747% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:04:30] XioNoX: So far, nothing stands out, just found a couple of typos that are not the cause of the issue. I'll continue digging... [22:54:43] FIRING: [2x] DiskSpace: Disk space centrallog1002:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:44:43] FIRING: [2x] DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 2.615% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace