[08:25:35] <elukey>	 hi folks morning
[08:26:12] <elukey>	 I am reloading pyrra on titan[12]001 since I still see the old lift wing pilot that should have been removed yesterday
[08:26:30] <elukey>	 after that I'll also reload thanos-rules to hopefully the old rec-rules will be cleared up
[08:31:57] <elukey>	 ok found an issue, the old yamls are still there
[08:35:18] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/983352/
[14:41:44] <elukey>	 another one :) 
[14:41:44] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/983421
[14:41:55] <elukey>	 I still see the old pyra recording rules in thanos even after a reload
[14:41:59] <elukey>	 very weird
[14:55:08] <herron>	 looks like the output rules are still there too
[14:57:52] <herron>	 manually cleaned those up and reloaded thanos rule
[15:06:35] <herron>	 elukey: what are your thoughts in terms of next steps?  one idea that comes to mind is to drop any unneeded labels as a relabel config
[15:26:40] <elukey>	 herron: could be an option yes, I put some thoughts in the parent's task about it. Istio provides a way to reduce the labels but it is not compatible with our version (of course), so we'd need to do it from Prometheus.. the number of time series will still be big though