[00:09:07] cwhite: interesting point; I for some reason assumed -- perhaps incorrectly -- that default is 1m [00:09:14] I will set it explicitly and we can try to test this again tomorrow [08:01:42] heh my understanding is (was) that without for the alert fires as soon as the expression evaluates to true, clearly not the case [14:45:04] we recently switched our read traffic from search.svc..wmnet to search.discovery.wmnet, but one downside is now our client side metrics all report against that same "discovery" cluster instead of a specific one. I can imagine a few possible solutions, but have others addressed this already? [14:48:32] ebernhardson: which kind of clients/metrics? do they have (or could) a site label? [14:50:37] volans: mostly latency and error rates, they come from mediawiki so they have a source cluster, but they don't know the destination cluster [14:51:11] the top few rows here mostly: https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1&from=now-7d&to=now&timezone=utc&var-cluster=elasticsearch&var-exported_cluster=production-search [14:52:58] ack [15:06:54] ebernhardson: interesting, I'm not aware of similar cases with client metrics, not a full solution though sort of a bandaid could be to display the discovery records pooled status for codfw/eqiad in the dashboard, we may have metrics already and if we don't then we should [15:12:15] that might work, i'm also going to check about adding a response header that says where the request was routed to [15:15:33] SGTM [16:39:15] ebernhardson: the metrics exported automatically by Envoy might also be helpful here