[03:08:05] <jinxer-wm>	 (LogstashKafkaConsumerLag) firing: Too many messages in logging-eqiad for group benthos-eqiad - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[04:27:50] <jinxer-wm>	 (LogstashKafkaConsumerLag) resolved: Too many messages in logging-eqiad for group benthos-eqiad - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[09:23:49] <jinxer-wm>	 (PuppetDisabled) firing: Puppet disabled on mwlog1002:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled
[09:23:58] <godog>	 volans: icinga config is ok now, I'm guessing it self resolved, was that in production ?
[09:24:23] <volans>	 yes was production
[09:24:46] <volans>	 and I saw the recovery on IRC later, but not sure if anyone did anything
[09:25:49] <godog>	 yeah me neither, and I'm wondering how that cloud vps name got into the config in the first place
[09:26:14] <godog>	 I'll investigate
[09:28:38] <volans>	 yeah the cloud-related name was weird to see in the error
[09:28:57] <volans>	 I got that running icinga configuration check on alert1001
[09:29:02] <volans>	 at the time
[09:34:55] <godog>	 Jan 31 16:58:02 alert1001 puppet-agent[27797]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content) -#011parents                   
[09:34:58] <godog>	 toolsbeta-test-k8s-worker-nfs-1                                                                                                                           Jan 31 16:58:02 alert1001 puppet-agent[27797]: (/Stage[main]/Icinga::Naggen/File[/etc/icinga/objects/puppet_hosts.cfg]/content) +#011parents                   cloudsw1-d5-eqiad  
[09:35:20] <godog>	 I'm genuinely puzzled rn
[09:38:27] <taavi>	 this sounds like the LLDP parent fact was broken for a while and the fixed itself
[09:39:37] <godog>	 ah good point re: lldp fact, that makes sense
[13:23:49] <jinxer-wm>	 (PuppetDisabled) firing: Puppet disabled on mwlog1002:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled
[17:23:49] <jinxer-wm>	 (PuppetDisabled) firing: Puppet disabled on mwlog1002:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled
[21:24:04] <jinxer-wm>	 (PuppetDisabled) firing: Puppet disabled on mwlog1002:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled