[00:24:34] RESOLVED: DiskSpace: Disk space arclamp2001:9100:/srv 3.14% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=arclamp2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:25:35] RESOLVED: DiskSpace: Disk space centrallog1002:9100:/srv 3.945% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:07:23] * gmodena waves [11:08:27] hey folks. I would like to onboard my team (wikidata platform) to alertmanager. Could someone maybe help me review this patch? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1218735 [11:11:29] Sure gmodena, we'll take a look [11:11:42] tappof thanks! [12:16:35] FIRING: DiskSpace: Disk space arclamp1001:9100:/srv 3.911% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=arclamp1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:21:35] RESOLVED: DiskSpace: Disk space arclamp1001:9100:/srv 3.894% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=arclamp1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:41:35] FIRING: DiskSpace: Disk space arclamp1001:9100:/srv 3.905% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=arclamp1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:41:58] Hey peeps, is there any way to have multiple dashboard links in an alertmanager alert? [12:42:09] like links to a grafana and logstash? [13:40:01] Well, claime, IIRC, on alerts.wikimedia.org additional labels other than runbook and dashboard are displayed (for example, logs). I'm not sure about IRC, but I think only runbook and dashboard are templated for each entry. [13:49:17] tappof: This would be for an alert posted to phab [13:50:25] I should check claime ... I'll let you know .. [13:50:32] tappof: tyvm <3 [14:01:35] FIRING: [2x] DiskSpace: Disk space arclamp1001:9100:/srv 2.968% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:15:35] FIRING: DiskSpace: Disk space centrallog1002:9100:/srv 3.97% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [18:00:35] RESOLVED: DiskSpace: Disk space centrallog1002:9100:/srv 3.367% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [18:01:50] FIRING: [2x] DiskSpace: Disk space arclamp1001:9100:/srv 0.484% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:02:25] FIRING: [2x] SystemdUnitFailed: excimer-log.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:07:25] FIRING: [4x] SystemdUnitFailed: arclamp_generate_metrics.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:17:25] FIRING: [4x] SystemdUnitFailed: arclamp_generate_metrics.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:22:25] FIRING: [5x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:47:25] FIRING: [4x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:52:25] FIRING: [5x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:07:25] FIRING: [6x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:17:25] FIRING: [6x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:22:25] FIRING: [9x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:27:25] FIRING: [10x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:28:18] looking at arclamp [20:32:25] FIRING: [10x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:37:25] FIRING: [10x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:39:11] FIRING: [2x] DiskSpace: Disk space arclamp1001:9100:/srv 3.266% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:47:25] FIRING: [10x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:07:25] FIRING: [6x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:12:25] FIRING: [6x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:17:25] FIRING: [6x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:19:11] RESOLVED: DiskSpace: Disk space arclamp2001:9100:/srv 3.511% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=arclamp2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:42:25] RESOLVED: [2x] SystemdUnitFailed: excimer-log.service on arclamp2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed