[13:05:11] o/ we'd like to propagate the usual "request_id" in logs to help traceability but we produce ECS compatible logs, is there a field we could use for this? (thinking about trace.id) c.f. https://doc.wikimedia.org/ecs/#field-trace-id do you have opinions on this? [13:28:16] isn't that what http.request.id would be? https://doc.wikimedia.org/ecs/#field-http-request-id [13:31:07] brouberol: oh thanks, missed this one [13:36:08] np! [15:03:51] godog: Hey, what's up? I'm trying to use label_values (without a metric) to get the site from a prometheus datasource in grafana, and it won't let me. Thoughts? I have the variable query datasource set to ${dc}, which is one of the Promethus datasources filtered on /k8s/. My variable in grafana is of type query, I put in label_values(site) as the query, which should be equivalent to curl [15:03:52] localhost:9906/k8s/api/v1/label/site/values and... no cigar. Thoughts? [15:04:10] (the curl is on a prometheus instance) [15:04:55] claime: yo! I recommend using thanos as the datasource, then you can filter by "prometheus" and "site" [15:05:09] and also label_values(site) will do the right thing [15:05:41] No cigar? [15:05:52] That'd mean change the datasource in every graph too right? [15:06:03] denisse: From the expression "close but no cigar"? [15:06:24] Meaning "falling just short" iirc [15:06:51] claime: mmhh I take it the datasource is a variable too? in that case you can set thanos as the datasource variable [15:07:18] claime: interesting!! TIL. [15:07:24] if that's not the case and there's a whole lot of panels, what I usually do is download the json, bulk change, re-import the json [15:07:54] There are a few panels I think we can work it [15:08:04] Ah I thought kamila_ would be here but they aren't [15:08:06] whole lot I'd say is more than 15, otherwise changing 10 panels or so is just faster from the UI [15:08:24] in the order of 3 minutes of work [15:08:52] Yeah we're at 20+ [15:09:07] But datasource is a variable [15:09:20] So I'll just set thanos as the datasource variable :D [15:09:34] yeah that's it, then you can even hide the variable [15:10:06] the datasource variable that is, you still need the prometheus instance selector though, which will require changing the queries [15:10:16] but I'll need to go around and change the queries [15:10:18] yeah [15:10:43] to explain: I think the reason label_values(site) doesn't work with prometheus ds but does with thanos ds is that it is an external variable [15:11:01] so thanos picks it up, and a "straight" query to prometheus doesn't [15:11:49] going forward thanos is meant to be the only datasource to be used, and we don't have these discrepancies anymore [15:11:51] Doesn't grafana just do the equivalent of my curl above to the right datasource? [15:13:24] I'm actually not 100% sure what grafana does under the hood with label_values [15:13:32] ack [15:14:13] I'm gonna copy my dashboard to a WIP one and start hacking at it. We'll see what happens [15:14:45] claime: ok! let us know how it goes, I can poke at it too [15:22:15] lol create a new dash board, copy/paste the json over except title and uid [15:22:24] "A dashboard with the same uid already exists" [15:22:33] Yeah I know, it's the one I'm editing actually [15:23:41] claime: indeed, what I do to clone a dashboard is cog menu top right -> save as [15:23:49] Ah good shout thanks [15:24:08] sure np [15:24:30] kamila_: copied to https://grafana-rw.wikimedia.org/d/u-XHXoMSz/mw-on-k8s-wip-serviceops for shenanigans [15:24:55] thanks claime [15:30:00] godog: Just so I'm clear, that just means adding prometheus=k8s, site=$site to all my queries basically [15:36:33] claime: that's one way yes, or prometheus=$prometheus and have $prometheus be a variable, in case you need/want to query k8s-staging [15:36:54] No staging for mw-on-k8s, so that should be good [15:37:17] indeed [15:38:34] I wonder if we have a way to get namespaces from chart name via a metric or something [15:40:45] no idea tbh [15:45:27] claime: yes from my metrics [15:45:49] kamila_: Great, you'll be able to fix the hardcoded namespaces! :D [15:45:50] oh wait, wdym by from chart name? [15:46:05] Like mediawiki gives you mw-web, mw-api-ext, etc. [15:46:10] Helm chart name [15:48:28] so my metrics do have a namespace label but I'm not sure if it is useful for the other metrics... I'll have a look in a bit [16:51:52] claime: I think I don't understand the problem... what's wrong with just something like `label_values(apache_accesses_total, deployment)`? [16:52:15] I assume there is a reason why it's hardcoded instead of using something like that, but I don't know what it is [16:53:24] (these metrics were already available before the benthos thing, that's why I assume that that is somehow not good) [16:53:50] nah I just missed them I think [16:54:34] so I can just change to that instead of the hardcoded thing and it's all good? [16:56:12] I'd go with label_values(apache_accesses_total{app="mediawiki", deployment) and it should be all right [16:56:42] cool, will do, thanks