[13:05:15] <XioNoX>	 Dashboard link doesn't support spaces https://usercontent.irccloud-cdn.com/file/fcSVswl5/Screenshot%20From%202025-05-13%2015-04-46.png
[13:15:31] <godog>	 XioNoX: interesting yeah, needs url quoting
[13:15:54] <godog>	 checking CI, it should complain
[13:16:27] <godog>	 ah yes of course because it comes from labels
[13:17:16] <XioNoX>	 yeah it should be fixed at the karma level I think
[13:24:03] <godog>	 XioNoX: I think sth like https://gerrit.wikimedia.org/r/c/operations/alerts/+/1145196
[13:24:10] <godog>	 to get properly escaped urls
[13:24:17] <godog>	 and karma will do the right thing
[13:24:47] <XioNoX>	 oh nice!
[13:24:57] <XioNoX>	 would be better for it to be fully automated, but that works
[13:25:18] <godog>	 indeed, good enough™
[15:55:41] <jinxer-wm>	 FIRING: [15x] PrometheusRuleEvaluationFailures: Prometheus rule evaluation failures (instance titan2001:17902) - https://wikitech.wikimedia.org/wiki/Prometheus - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fops - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRuleEvaluationFailures
[16:00:41] <jinxer-wm>	 RESOLVED: [15x] PrometheusRuleEvaluationFailures: Prometheus rule evaluation failures (instance titan2001:17902) - https://wikitech.wikimedia.org/wiki/Prometheus - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fops - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRuleEvaluationFailures
[16:24:41] <jinxer-wm>	 FIRING: [8x] PrometheusRuleEvaluationFailures: Prometheus rule evaluation failures (instance titan1001:17902) - https://wikitech.wikimedia.org/wiki/Prometheus - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fops - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRuleEvaluationFailures
[16:29:20] <denisse>	 ^ I tried to look at it but the alert no longer shows in Karma.
[16:30:28] <denisse>	 After refreshing Karma twice I can see the alert again.
[16:34:41] <jinxer-wm>	 FIRING: [13x] PrometheusRuleEvaluationFailures: Prometheus rule evaluation failures (instance titan1001:17902) - https://wikitech.wikimedia.org/wiki/Prometheus - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fops - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRuleEvaluationFailures
[16:39:41] <jinxer-wm>	 FIRING: [17x] PrometheusRuleEvaluationFailures: Prometheus rule evaluation failures (instance titan1001:17902) - https://wikitech.wikimedia.org/wiki/Prometheus - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fops - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRuleEvaluationFailures
[16:44:41] <jinxer-wm>	 FIRING: [12x] PrometheusRuleEvaluationFailures: Prometheus rule evaluation failures (instance titan1001:17902) - https://wikitech.wikimedia.org/wiki/Prometheus - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fops - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRuleEvaluationFailures
[16:49:41] <jinxer-wm>	 RESOLVED: PrometheusRuleEvaluationFailures: Prometheus rule evaluation failures (instance titan1001:17902) - https://wikitech.wikimedia.org/wiki/Prometheus - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fops - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRuleEvaluationFailures
[16:51:41] <jinxer-wm>	 FIRING: [12x] PrometheusRuleEvaluationFailures: Prometheus rule evaluation failures (instance titan1001:17902) - https://wikitech.wikimedia.org/wiki/Prometheus - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fops - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRuleEvaluationFailures
[16:56:41] <jinxer-wm>	 RESOLVED: [12x] PrometheusRuleEvaluationFailures: Prometheus rule evaluation failures (instance titan1001:17902) - https://wikitech.wikimedia.org/wiki/Prometheus - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fops - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRuleEvaluationFailures
[17:40:37] <denisse>	 I found the issue, I'm still investigating the root cause for it. I opened T394080 to track this.
[17:40:38] <stashbot>	 T394080: Duplicate SLOMetricAbsent rules generated by Pyrra for varnish-combined-eqsin-cache_upload.yaml - https://phabricator.wikimedia.org/T394080