[15:54:49] i all wondering if someone could helpe with some basic promethrous grafana questions. i create a scrap config here https://gerrit.wikimedia.org/r/c/operations/puppet/+/703727/1/modules/profile/manifests/prometheus/ops.pp and from testing can confirm that prom1003 can reach that endpoint (http://pki.discovery.wmnet/metrics) and im not trying to see the results in thaons [15:55:19] when i enter `requests_total` in the thanos ui i dont get any m,etricts back [15:55:47] requests_total is the last metric recived from http://pki.discovery.wmnet/metrics which is why i was searching for that but perhaps it gets decorated? [15:56:09] can some one point me to somewhere where i can debug this further or check that the scrapping is actully working as expected [15:58:50] also searching for "go_memstats_stack_inuse_bytes{cluster = 'pki'}" dosn;t show any results for the cfssl service which make me thing something is wrong with the scrapping config [16:40:59] hey jbond, at a quick glance I think this one still needs jobs defined (something like $pki_jobs or $cfssl_jobs) and added to the scrape_configs_extra param of prometheus::server. another definition in the same file e.g. $wikidough_jobs should have some good pointers to follow [16:46:46] thanks herror, dose this look right? https://gerrit.wikimedia.org/r/c/operations/puppet/+/715262 [16:46:55] jbond: ^^ and it appears pki1001:80 is reachable due to ferm `10_csr_and_ocsp_responder`. pki-root1001:80 is unreachable from the prometheus servers. [16:48:20] thanks cwhite in fact i thik i need to remove pki-root from the config. (pki1001 and pki2001 should be reachable) [16:53:57] * jbond updated the cr to uyse class_config instead of cluster_config which should remove pki-root [17:09:01] jbond: I think the jobs variables should be up above the call to prometheus::server [17:09:40] herron: yes thanks i just updated that, the latest pcc shows an update [17:10:34] this one: https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/30901/console [17:11:45] thanks, hadn't refreshed the page [17:24:00] huh, I don't see pki-root going away in the diff, but I also don't see it in the full change catalog. but anyway yeah looks good to me [17:25:29] thanks herron (and cwhite :)) the pki-root diffing is likley an issue with the pcc puppetdb not knowing about pki-root. ill check the diffes when applied and send a follow up ps if im wrong [17:26:04] ok sounds good [17:27:14] ack [17:28:36] * jbond just confirming `- - pki-root1001:80 [17:28:40] was removed [17:28:55] thanks again :) [17:37:35] np! seeing the metrics filtering in as well now, great [17:41:59] awesome thanks, also seeing stuff in thanos (will leave the grafana bored till monday :))