[00:52:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[04:52:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:46:50] <Lucas_WMDE>	 hi folks, I just saw the email about Grafana v11.6.1 and wanted to report an issue
[06:46:58] <Lucas_WMDE>	 Graphite data sources aren’t working due to CORS errors
[06:47:00] <Lucas_WMDE>	 > Reason: header ‘x-grafana-device-id’ is not allowed according to header ‘Access-Control-Allow-Headers’ from CORS preflight response
[06:47:18] <Lucas_WMDE>	 (that header name also sounds fishy from a privacy perspective anyway ^^)
[06:47:45] <Lucas_WMDE>	 I know Graphite is going away but we were also promised it would remain available read-only for a year, and I assume you want to upgrade Grafana before then :P
[06:48:20] <Lucas_WMDE>	 example affected dashboard: https://grafana-next.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts
[06:50:15] <Lucas_WMDE>	 (note that the prod version of that dashboard, https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts, was mostly ported to Prometheus in the meantime, but you can see the Graphite panels under “historical data” at the bottom)
[08:52:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:52:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[16:52:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:32:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_grafana-loki.service on grafana2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[21:32:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_grafana-loki.service on grafana2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed