[15:27:08] So I have a change to add a first LW SLO dashboard using Grizzly (https://gerrit.wikimedia.org/r/c/operations/grafana-grizzly/+/942448). But I get no data. Since the ro instance of Grafana doesn't let me mess with the queries to try and get them right, is there a way to make grr preview send me to the rw instance? [15:27:46] (also, are preview dashboards deleted automagically? since I clearly can't do so with the ro instance) [16:06:13] ack, thanks cwhite ! [16:07:46] klausman: the rw instance won't let you play with preview dashboard queries either. It's probably best to try in the explore feature, or a temporary dashboard. [16:08:15] ack. [16:08:51] Dashboards imported via grizzly cannot be changed via the ui. [16:09:55] Is there a way to make grizzly just give me the JSOn so I can make a temp DB to figure out what's going on? [16:11:50] Next to the title at the top there is a "share" icon. From there, Export->View JSON should give you a json rendering of that dashboard. [16:15:30] ah, right, should've thought of that. thx! [17:53:47] I kicked a new ZK cluster in https://phabricator.wikimedia.org/T341792 and it looks like prometheus is not trying to scrape it. Known-good cluster config at https://github.com/wikimedia/operations-puppet/blob/production/hieradata/role/common/analytics_cluster/zookeeper.yaml , new config at https://github.com/wikimedia/operations-puppet/blob/production/hieradata/role/common/zookeeper/flink.yaml . I confirmed the prom host can [17:53:47] reach the new zk node, so I don't think it's fw-related. Any suggestions? [18:04:42] inflatador: it's possible that Prometheus isn't aware of the new instances like it is for role::analytics_cluster::zookeeper and druid. https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/prometheus/analytics.pp#190 [19:37:28] cwhite thanks! We have https://gerrit.wikimedia.org/r/c/operations/puppet/+/942494 up now if you don't mind taking a look. No hurry though