[00:47:35] FIRING: DiskSpace: Disk space centrallog1002:9100:/srv 3.882% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:48:25] FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:19:34] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.984% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:04:34] RESOLVED: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.853% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [04:47:49] FIRING: DiskSpace: Disk space centrallog1002:9100:/srv 2.494% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [04:48:25] FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:53:34] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.984% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:03:34] RESOLVED: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.962% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:08:40] FIRING: LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=40&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures [07:13:40] RESOLVED: LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=40&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures [08:47:50] FIRING: DiskSpace: Disk space centrallog1002:9100:/srv 0.6851% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:48:40] FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:52:35] FIRING: [2x] DiskSpace: Disk space centrallog1002:9100:/srv 0.6537% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:15:34] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.983% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:05:34] RESOLVED: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.842% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:02:04] Heya, I'm trying to deploy a change to add more logs, and the logs are falling into dlq: https://logstash.wikimedia.org/app/discover#/doc/19d32430-85d8-11eb-8ab2-63c7f3b019fc/dlq-default-1-1.0.0-1-2026.08.05?id=J5HF0Z8BsYGHbYxISGCJ [12:02:06] Wondering if anyone knew why? I suspect maybe they event keyword? Which I added in an attempt to make moving to ecs later easier but maybe it's in the wrong spot and is getting dumped? [12:20:34] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.983% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:48:40] FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:52:50] FIRING: [2x] DiskSpace: Disk space centrallog1002:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:05:34] RESOLVED: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.859% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:56:34] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.984% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:57:40] Hi Mvolz! The reason they're going to the dlq is that some other application produces the `event` field as a string and citoid is producing it as an object. The legacy indexes rely on the first log encountered at a named field to automatically assign a type for the lifetime of the index. These conflicts are common and is one of the core reasons we started supporting ECS. The [13:57:42] legacy indexes are incompatible with the ECS schema, so they can't really coexist in the same index. [15:01:34] RESOLVED: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.71% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:08:35] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.984% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:48:25] FIRING: [2x] SystemdUnitFailed: rsync-var-lib-grafana.service on grafana2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:53:25] FIRING: [2x] SystemdUnitFailed: rsync-var-lib-grafana.service on grafana2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:52:50] FIRING: [2x] DiskSpace: Disk space centrallog1002:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:03:34] RESOLVED: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.559% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:05:06] heya! Is there some sort of publicly-available API I can use for monitoring alerts? It looks like I might be able to use grafana.wm.o but I'm not sure that the alerting component of grafana is configured to be showing alerts from icinga as well [17:13:34] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.984% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [18:06:20] brett: not really - all our alerting dashboards, tooling, and log streams are gated behind authentication [18:08:18] cwhite: Thanks for the confirmation. It appears I might be able to kludge something together through grafana, but the icinga-less alerts would make it a no-go. Am I correct in that grafana is not tracking icinga-sourced alerts? [18:10:50] brett: That's correct, but Grafana does have a copy of alert logs from icinga and prometheus in the Public Logs datasource. [18:11:25] ah, so loki [18:12:07] Thanks for the help! [18:16:28] yw! [19:53:25] FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:52:50] FIRING: [2x] DiskSpace: Disk space centrallog1002:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:53:25] RESOLVED: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:13:50] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.945% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:03:35] RESOLVED: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.535% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:59:34] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.984% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace