[08:29:10] <Tran>	 Is this still the best place to ask technical questions regarding observability? I have a question about if prometheus' pushgateway is still the preferred/best practice for job-based data or if I should just use statsd. Wikitech says yes (https://wikitech.wikimedia.org/wiki/Prometheus#Ephemeral_jobs_%28Pushgateway%29) but I just wanted to confirm.
[08:52:22] <jinxer-wm>	 (LogstashKafkaConsumerLag) firing: (2) Too many messages in logging-codfw for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-codfw&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[08:52:57] <jayme>	 that is me updating rsyslog
[08:53:03] <jayme>	 (I guess)
[08:53:28] <jinxer-wm>	 (LogstashKafkaConsumerLag) firing: (2) Too many messages in logging-codfw for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-codfw&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[09:02:22] <jinxer-wm>	 (LogstashKafkaConsumerLag) firing: (3) Too many messages in logging-codfw for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag  - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[09:03:28] <jinxer-wm>	 (LogstashKafkaConsumerLag) firing: (3) Too many messages in logging-codfw for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag  - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[09:08:28] <jinxer-wm>	 (LogstashKafkaConsumerLag) firing: (3) Too many messages in logging-codfw for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag  - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[09:17:22] <jinxer-wm>	 (LogstashKafkaConsumerLag) resolved: (2) Too many messages in logging-codfw for group logstash7-eqiad - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag  - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[09:31:35] <godog>	 jayme: likely yeah, thank you for that
[09:32:41] <godog>	 Tran: it is indeed the best place :) yes you should use pushgateway as opposed to statsd which is being deprecated. Also feel free to open a task to #observability to inquiry re: your use case and we can provide assistance/guidance
[10:12:24] <jayme>	 godog: I've rolled the rsyslog update to all of wikikube for now. Will do the other clusters in a bit
[10:12:34] <godog>	 jayme: \o/ nice
[10:13:10] <jayme>	 that we'll see :p
[10:22:22] <jinxer-wm>	 (LogstashKafkaConsumerLag) firing: Too many messages in logging-eqiad for group logstash7-eqiad - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[10:27:22] <jinxer-wm>	 (LogstashKafkaConsumerLag) resolved: Too many messages in logging-eqiad for group logstash7-eqiad - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag