[08:35:07] FYI I'm getting 401s for jquery/popper and other stuff under /static/ on https://prometheus-eqiad.wikimedia.org/ops/classic/targets [09:24:54] claime: thank you for reaching out, known/expected ATM [09:24:56] https://phabricator.wikimedia.org/T326657#9095309 [09:25:46] ack [13:30:25] yeah, I'll go ahead and revert the change to use the prometheus-https load balancer which will pin the service to one host and avoid those 401s. fwiw https://phabricator.wikimedia.org/T331512 is also related (same issue there with sso sessions) [13:39:25] thank you [13:57:21] hi folks! [13:58:06] I took over https://gerrit.wikimedia.org/r/c/operations/grafana-grizzly/+/952226/4/slo_definitions.libsonnet from Tobias (holidays) and I am trying to follow up on what Keith suggested (thanks for the review!) [13:58:47] to test the new slo rule, I put a modified version of it (overriding the variables like $site etc..) in the thanos ui [13:59:01] but this recording rule "destsvc_rev_ns_rc_rf:istio_requests_total:rate5m" seems to lead to an error [13:59:10] is it supposed to happen? [13:59:23] (this is probably a pebcak I know) [14:14:01] as in, querying for destsvc_rev_ns_rc_rf:istio_requests_total:rate5m alone leads to an error? [14:26:40] herron: o/ or with anything else, I get an error (as if HTML was returned instead of json) [14:27:23] hmm, could I see a paste? [15:03:42] back sorry, I was in a meeting [15:03:45] lemme create one [15:06:01] herron: lol now it works [15:06:16] elukey: ahh classic [15:06:22] ok simply asking you made prometheus behave [15:06:29] good to know :D [15:06:34] haha 💪 [16:11:25] o/ I'm seeing a strange behaviour in grafana.w.org -- if I switch the time range I get an empty page, and I have to hard reload the page [16:11:34] dcaro can also reproduce it [16:11:42] example link where this happens: https://grafana.wikimedia.org/d/UUmLqqX4k/wmcs-openstack-api-latency?from=now-7d&orgId=1&refresh=5m&to=now&viewPanel=19 [16:11:50] (seems to happen only on maximized panels) [16:20:54] I can reproduce the issue dhinus mentions. [16:22:55] Reloading the dashboard doesn't seem to work. [16:32:56] Looks like it has something to do with dynamic panels. When cinter_api_backend is enabled and viewed, the time picker works fine, but if you view a "later" panel like designate_backend, the behavior manifests. It looks like grafana reassigns IDs to the panels on time picker change because they can be added/removed with the "backend" var. [16:34:23] You can see this in action if you check only the heat_backend, view the panel, then enable the cinder_api_backend then change the time. You'll see the heat_backend panel change to cinder_api_backend even though you were previously viewing the heat_backend panel.