[12:11:12] Amir1: if you are around we can hop on a call and dig into LinkCache [12:11:22] Somethign isn't quite right [12:51:09] hey duesen, meet godog, superb grafana-fu colleague [12:51:22] hello [13:02:34] godog: hi! i think i got it, but let me fiddle a bit more. I'll share in a few minutes [13:02:59] for sure -- let us know [13:03:00] Amir1: i'm making a grafana dashboard for the LinkCache to confirm my assumptions about what's happening [13:11:08] godog: ok, this is as far as i got: https://grafana-rw.wikimedia.org/d/rG3oVe27k/link-cache [13:12:03] the two panels at the top are working. I want a third one ("volume") that is a lot like the "Good vs Missing" one, but showing total numbers (stacked) rather than percentage of rate [13:12:30] Somehow it's still showing things as percentage though, and at an insane scale... [13:12:37] godog: can you help me sort that out? [13:13:03] duesen: ok! I'm taking a look at the third panel now [13:15:21] godog: I have some unsaved changes (renaming stuff), can I save them now? [13:15:27] duesen: sure go ahead [13:15:43] ok done [13:17:13] duesen: in the third panel I see the metrics are different i.e. count vs rate, is taht expected? [13:20:01] duesen: try selecting 'percent' in the 'display' option under "stacking and null value" [13:20:23] duesen: also use 'none' or 'simple' as the y axis unit [13:21:33] hold on, looking now [13:22:57] I'm looking for count per unit time, which is still a rate... for the graph it doesn't really make a difference, but I suppose for a rate I'D expect an average for each bar, while for the count I'd expect the sum. [13:23:05] Summing rates doesn't make sense, does it? [13:24:10] if I select "percent", I seem to effectively get the same view as in the second panel. but I don't want percent... [13:24:39] it should be ok I think (summing rates) [13:25:06] ah I see, you want absolute numbers on the y scale but stacked as to form columns of the same height? [13:25:19] but it doesn't give me the actual total number for each unit time, which is what i'm looking for [13:26:01] ah, I found one thing to fix: y-max under Axes. [13:26:29] duesen: I see, the other thing to try might be passing through a graphite function to e.g. summarize per unit time? https://wikitech.wikimedia.org/wiki/Graphite#Functions [13:26:37] godog: stack, but not all the same hight. I want to see difference in volume over time [13:27:02] I think I got it now, actually. At least it looks plausible. Not sure the numbers are the ones I'm looking for [13:27:10] I'll save now so you can see what I'm doing [13:27:32] duesen: I'm around now [13:27:33] duesen: yeah the 'count' in graphite is misleading and iffy, but I think you already knew that [13:27:54] I vaguely remember that, but I don't remember how exactly... [13:28:08] re summing - isn't that what sumSeries does? [13:31:02] yes though across all timeseries you selected in the expresison, not sum over time for example [13:31:24] e.g. if you need to do buckets say average over 10 min or so [13:31:41] Amir1: https://grafana-rw.wikimedia.org/d/rG3oVe27k/pagestore [13:32:24] duesen: I'm 90% certain the hit rate is wrong [13:32:37] godog: ok, I think I have something that is good enough for investigating the issue at hand. thanks for your help! Can I poke you again if issues come up? [13:33:49] duesen: sure no worries, yeah if I'm free I'll take a look [13:34:15] thanks again! [15:45:39] duesen: https://wikitech.wikimedia.org/wiki/Graphite#Extended_properties [15:46:01] TLDR: you probably want '.rate'. Never count. [22:10:32] Krinkle: I'd like to see the actual cumulative volume, rather than rate... But I'm starting to realize that that doesn't make sense. Volume per what amount of time? I guess I'll settle for a throughput measure then. [22:11:23] it's a bit annoying that in the context of a cache, there is "hit rate" which is a ration, not a rate. And there is an access rate, which is actually a rate... [22:12:09] duesen: right. As long as your desired window is equal to or larger than the largest graphite/whisper aggregation window (currently 1 day) then summarize(thing.sum) should work. [22:13:10] similarly, you can use .sum in a pie chart to e.g. plot the total (or average, they're same thing when aggregated correctly) hits vs misses [22:13:55] but yeah, we usually do rate per second or minute, based on .rate or scale(.rate, 60) [22:14:17] and then maybe a chart plotting the total or average for some large window if there is an interest in that. [22:15:01] if anecdotally we want to know what happens per week or month we'd probalby eyeball the average rate and multiply it ad-hoc as part of whatever conversation we're having outside grafana [22:17:15] ok, thanks. i renamed the panels to make it more clear