[02:38:59] <jinxer-wm>	 (LogstashKafkaConsumerLag) firing: Too many messages in kafka logging - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[06:43:41] <jinxer-wm>	 (LogstashKafkaConsumerLag) firing: Too many messages in kafka logging - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[10:43:41] <jinxer-wm>	 (LogstashKafkaConsumerLag) firing: Too many messages in kafka logging - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[11:13:09] <godog>	 I'm working on a tweak of that alert ^
[11:21:28] <godog>	 with https://gerrit.wikimedia.org/r/c/operations/alerts/+/991756/ we'll be able at least to silence consumer groups we know are currently lagged
[12:23:41] <jinxer-wm>	 (LogstashKafkaConsumerLag) firing: (2) Too many messages in kafka logging - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[15:19:25] <godog>	 FYI I've silenced the alert above only for group logstash7-codfw consuming from eqiad since that's know (i.e. mw k8s access log)
[16:11:29] <lmata>	 thanks!
[16:42:43] <inflatador>	 godog I'm working on an alerts review and had some questions: what is the default retention rate in Thanos? Is there any option to keep data longer than the default?
[16:47:18] <godog>	 inflatador: in thanos we keep 54 weeks of raw metric data, and 5 years for 5m and 1h resolution, that's for all metric data in a "all or nothing" fashion
[16:47:33] <godog>	 in terms of selecting which metrics are kept for longer/shorter
[16:57:34] <inflatador>	 godog excellent, thanks for the info
[16:58:15] <godog>	 sure np inflatador, happy to help
[17:00:12] <inflatador>	 not sure if this is the best place, but I updated https://wikitech.wikimedia.org/wiki/Thanos#Metrics_Retention w/your info
[17:02:10] <godog>	 thank you, yeah that's good