[16:35:43] <inflatador>	 Is anyone around to help me understand why a blackbox check is returning 500? Based on the access logs, it doesn't look like the check is polling the right URI . Check is here: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/query_service/monitor/ldf.pp
[16:36:12] <gehel>	 inflatador: is it deployed somewhere already?
[16:36:36] <inflatador>	 access logs show `"GET /bigdata/ldf HTTP/1.1" 500` , notice the lack of a query string. curling the exact query string as shown in the check from the prometheus hosts gives a 200
[16:36:58] <inflatador>	 gehel the check is delpoyed
[16:37:38] <gehel>	 where is it running? Where can I see it's configuration?
[16:38:07] <inflatador>	 gehel blackbox check is at prometheus1006:/etc/prometheus/blackbox.yml.d/query_wikidata_org_ldf.yml
[16:38:21] <inflatador>	 you can see the targeted host at  /srv/prometheus/ops/targets/probes-custom_puppet-http.yaml
[16:38:27] <inflatador>	 search for 'ldf'
[16:38:33] <gehel>	 looking
[16:39:03] <inflatador>	 access logs are at `wdqs1015:/var/log/nginn/access.log`, grep for 'prometheus-ldf-check'
[16:39:13] <inflatador>	 err...nginx not nginn
[16:40:36] <gehel>	 is the query expected to be in the body? sent with a GET?
[16:40:47] <gehel>	 Could you paste your test curl?
[16:41:08] <inflatador>	 gehel `curl -A "bking@prometheus"  -H 'host:query.wikidata.org' 'http://wdqs1015.eqiad.wmnet/bigdata/ldf?subject=wd%3AQ42&predicate=wdt%3AP31&object='`
[16:41:14] <inflatador>	 and yes, it should be send with a GET
[16:41:43] <gehel>	 blackbox exporter is configured to send the query in the body, it should be part of the path instead.
[16:43:16] <inflatador>	 gehel excellent, let me get a patch up
[16:43:48] <gehel>	 do you know how to manually run blackbox exporter on prometheus1006 so that you can test by manually editing the config? So you don't have to do one more back and forth with a patch before you know it is correct?
[16:44:01] <inflatador>	 Y
[16:46:08] <inflatador>	 gehel I'm a little weirded out by the fact that no other targets have a query string in their path, do you think that matters? 
[16:47:31] <gehel>	 In most cases, we probably check for more standard HTML pages, or REST APIs.
[16:48:00] <gehel>	 There might be some issues with escaping the query string properly through all the layers, but I don't see why it wouldn't work
[16:48:02] <inflatador>	 Your advice worked...at least, we are getting the query string access.log . Still a 500 though
[16:48:35] <inflatador>	 let me try quoting and a couple of other things
[16:49:21] <gehel>	 does the query look correct in the nginx log?
[16:51:04] * gehel needs to start the weekend by feeding the kids.
[16:51:12] <gehel>	 inflatador: I hope you find your way around!
[16:51:24] <inflatador>	 ;) me too, thanks for the advice
[20:41:50] <gehel>	 inflatador: https://github.com/wikimedia/operations-puppet/blob/production/hieradata/common/profile/trafficserver/backend.yaml#L201
[20:42:19] <gehel>	 is the check going through Varnish? In which case, there is a rewrite from /ldf to /bigdata/ldf