[15:04:13] <jinxer-wm>	 (ThanosSidecarNoConnectionToStartedPrometheus) firing: Thanos Sidecar cannot access Prometheus, even though Prometheus seems healthy and has reloaded WAL. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarNoConnectionToStartedPrometheus
[15:09:13] <jinxer-wm>	 (ThanosSidecarNoConnectionToStartedPrometheus) resolved: Thanos Sidecar cannot access Prometheus, even though Prometheus seems healthy and has reloaded WAL. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarNoConnectionToStartedPrometheus
[15:17:51] <elukey>	 herron: o/ sorry total brain fail
[15:17:59] <elukey>	 I realized only now the horrible CR that I sent
[15:18:04] <elukey>	 hope the current one is better :D
[15:20:27] <herron>	 elukey: haha no worries, one more comment and then I think its good to go!
[15:23:24] <elukey>	 herron: I have a doubt though - when I check in Thanos istio_request_duration_milliseconds_bucket I see that multiple time series are reported, one for each le label.. This is why for total I used +Inf, since it should in theory be equal to the total amount of queries.. Is the concern of multiple time series not a problem? (Trying to understand sorry)
[15:28:57] <herron>	 elukey: yes you have a good point, ok lets go ahead and try with +Inf and see
[15:29:31] <elukey>	 super, ok to solve the comments?
[15:29:50] <herron>	 just did! 🚀
[15:30:06] <elukey>	 thanks!
[15:30:45] <elukey>	 if this turns out to be ok as well I'll create some basic puppet code to avoid copy/paste every time, since most of the SLOs will be similar
[15:31:16] <herron>	 will they be similar enough to use grouping, or different queries?
[15:33:00] <elukey>	 we could use grouping in theory but I'd need to remove restrictions on the number of time series returned, so thanos will not like it :(
[15:33:15] <elukey>	 basically now I restrict to only one service
[15:33:26] <elukey>	 but we have ~10 similar to it now
[15:40:11] <herron>	 gotya ok
[16:23:14] <jinxer-wm>	 (ThanosSidecarNoConnectionToStartedPrometheus) firing: Thanos Sidecar cannot access Prometheus, even though Prometheus seems healthy and has reloaded WAL. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarNoConnectionToStartedPrometheus
[16:24:29] <elukey>	 little follow up: https://gerrit.wikimedia.org/r/c/operations/puppet/+/979126
[16:24:37] <elukey>	 Pyrra suggests to use the histogram count
[16:24:58] <elukey>	 thanks :)
[16:25:25] <godog>	 the thanos alerts is me btw, I rolled out a change that restarted prometheus
[16:25:29] <herron>	 👍
[16:28:13] <jinxer-wm>	 (ThanosSidecarNoConnectionToStartedPrometheus) resolved: Thanos Sidecar cannot access Prometheus, even though Prometheus seems healthy and has reloaded WAL. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarNoConnectionToStartedPrometheus
[16:36:20] <elukey>	 all updated!
[16:36:24] <elukey>	 so far the new dashboard looks good
[16:36:38] <elukey>	 I'll check it again tomorrow 
[16:36:42] <elukey>	 with more data etc..