[17:50:29] <rzl>	 herron: hey, I passed you https://phabricator.wikimedia.org/T320749 optimistically, but if you don't have the cycles for it immediately, NBD :)
[17:50:44] <rzl>	 I can give it a try if you're too busy to pick it up, but I might still need a fair bit of handholding is all
[17:59:23] <rzl>	 oh brilliant thanks :D
[17:59:25] <herron>	 rzl: thx! just updated the task, yeah happy to take a stab at that
[18:17:07] <ryankemper>	 hey o11y, search team's working on developing a service level indicator to measure WDQS uptime. Our initial idea is to gather metrics and/or logs for nginx requests. I could use some help figuring out how to get prometheus metrics corresponding to, at a minimum, the http response code out of nginx. There's a bit of context in https://phabricator.wikimedia.org/T313751#8315187 if it helps
[19:12:13] <gehel>	 A bit more context about the question above. We don't know exactly what metric we want, but something around the percentage of successful requests (HTTP 200). This seems like a very generic problem, which should have a generic solution.
[19:12:45] <gehel>	 Do we ingest web requests from the traffic layer into prometheus? Could we query that, filtering on a hostname?
[20:18:13] * cwhite reads the task for context
[20:45:49] <cwhite>	 ryankemper, gehel: is it safe to assume that you'd like to scope this metric to requests against wdqs.discovery.wmnet?
[21:03:18] <ryankemper>	 cwhite: yes indeed
[21:05:39] <cwhite>	 ryankemper: dug around and found these metrics from trafficserver: job_method_status_backend_layer:trafficserver_backend_requests_seconds_count:rate2m{backend="wdqs.discovery.wmnet"} and job_method_status_backend_layer:trafficserver_backend_requests_seconds_count:rate5m{backend="wdqs.discovery.wmnet"}
[21:06:52] <cwhite>	 these are pre-aggregated metrics though
[21:09:55] <cwhite>	 trafficserver_backend_requests_seconds_count{backend="wdqs.discovery.wmnet"} looks like a counter that might get you more precise data
[21:45:56] <ryankemper>	 cwhite: thanks, this looks like it's on the right track
[23:12:19] <bblack>	 gehel: yeah it's just aggregate metrics, not really per-request data.  But it might have the status counters that matter to you.