[11:39:46] <elukey>	 herron: o/ Leaving some questions related to pyrra/istio in the chan, feel free to answer anytime :)
[11:40:31] <elukey>	 1) I still see some Prometheus alerts created by Pyrra (the budget burn ones) that are firing after the clean up of the pilots, should I reload anything else on titan nodes to clean up?
[11:42:28] <elukey>	 2) I tried to follow up on your suggestion about dropping some labels in Istio metrics. Istio itself in theory can limit the labels, but due to some bugs etc.. our version doesn't support it (upgrading it a little tricky etc.). So I tried to follow up with some label_drop rules in Prometheus' config, but
[11:44:10] <elukey>	 the istio metrics are pulled by the 'k8s-pods' job (defined in k8s.pp), that is responsible for a broad range of metrics, not only istio ones. IIUC the label drop config wants a regex with the label names to drop, but there seems to be no way to restrict the metrics that it operates on.
[11:44:45] <elukey>	 So I'd be hesitant in adding label drops to k8s-pods, since if there is some overlapping between unrelated metrics we'd risk to drop data that we don't aim to.
[11:44:57] <elukey>	 Does it make sense? Is there anything that we can  try/do?
[23:46:57] <herron>	 elukey: yeah I see what you mean, maybe the thing to do is ask upstream about use of recording rules on the pyrra input
[23:56:04] <herron>	 re: the burn alert, I see those are firing in karma but the underlying config is no longer in thanos rule, looks stale.  godog have you ran into that before?