[01:08:40] <jinxer-wm>	 FIRING: LogstashNoLogsIndexed: Logstash logs are not being indexed by Elasticsearch - https://wikitech.wikimedia.org/wiki/Logstash#No_logs_indexed - https://grafana.wikimedia.org/d/000000561/logstash?var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashNoLogsIndexed
[01:12:37] <jinxer-wm>	 FIRING: OpensearchClusterHealth: Opensearch cluster health reported as red - https://wikitech.wikimedia.org/wiki/Runbook - https://grafana.wikimedia.org/d/e7d7fa18-7bc3-4548-bb07-ef261a9d3b8b/opensearch-cluster-health?var-cluster=production-elk7-codfw - https://alerts.wikimedia.org/?q=alertname%3DOpensearchClusterHealth
[01:13:40] <jinxer-wm>	 FIRING: LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[01:14:34] <jinxer-wm>	 FIRING: ErrorBudgetBurn: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[01:18:40] <jinxer-wm>	 FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-codfw for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag  - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[01:29:34] <jinxer-wm>	 FIRING: [2x] ErrorBudgetBurn: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[01:37:40] <jinxer-wm>	 FIRING: LogstashClusterStatus: OpenSearch reports cluster status is red. - https://wikitech.wikimedia.org/wiki/Logstash#Unassigned_Shards_and_Cluster_Status - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=panel-49 - https://alerts.wikimedia.org/?q=alertname%3DLogstashClusterStatus
[02:07:37] <jinxer-wm>	 RESOLVED: OpensearchClusterHealth: Opensearch cluster health reported as red - https://wikitech.wikimedia.org/wiki/Runbook - https://grafana.wikimedia.org/d/e7d7fa18-7bc3-4548-bb07-ef261a9d3b8b/opensearch-cluster-health?var-cluster=production-elk7-codfw - https://alerts.wikimedia.org/?q=alertname%3DOpensearchClusterHealth
[02:07:40] <jinxer-wm>	 RESOLVED: LogstashClusterStatus: OpenSearch reports cluster status is red. - https://wikitech.wikimedia.org/wiki/Logstash#Unassigned_Shards_and_Cluster_Status - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=panel-49 - https://alerts.wikimedia.org/?q=alertname%3DLogstashClusterStatus
[02:43:40] <jinxer-wm>	 RESOLVED: LogstashNoLogsIndexed: Logstash logs are not being indexed by Elasticsearch - https://wikitech.wikimedia.org/wiki/Logstash#No_logs_indexed - https://grafana.wikimedia.org/d/000000561/logstash?var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashNoLogsIndexed
[02:48:40] <jinxer-wm>	 FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-codfw for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag  - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[02:49:34] <jinxer-wm>	 FIRING: [2x] ErrorBudgetBurn: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[02:52:37] <jinxer-wm>	 FIRING: OpensearchClusterHealth: Opensearch cluster health reported as red - https://wikitech.wikimedia.org/wiki/Runbook - https://grafana.wikimedia.org/d/e7d7fa18-7bc3-4548-bb07-ef261a9d3b8b/opensearch-cluster-health?var-cluster=production-elk7-codfw - https://alerts.wikimedia.org/?q=alertname%3DOpensearchClusterHealth
[03:14:34] <jinxer-wm>	 RESOLVED: ErrorBudgetBurn: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[03:17:40] <jinxer-wm>	 FIRING: LogstashClusterStatus: OpenSearch reports cluster status is red. - https://wikitech.wikimedia.org/wiki/Logstash#Unassigned_Shards_and_Cluster_Status - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=panel-49 - https://alerts.wikimedia.org/?q=alertname%3DLogstashClusterStatus
[03:33:40] <jinxer-wm>	 FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-codfw for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag  - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[03:35:40] <jinxer-wm>	 FIRING: LogstashNoLogsIndexed: Logstash logs are not being indexed by Elasticsearch - https://wikitech.wikimedia.org/wiki/Logstash#No_logs_indexed - https://grafana.wikimedia.org/d/000000561/logstash?var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashNoLogsIndexed
[03:39:34] <jinxer-wm>	 FIRING: ErrorBudgetBurn: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[03:54:34] <jinxer-wm>	 FIRING: [2x] ErrorBudgetBurn: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[04:50:40] <jinxer-wm>	 RESOLVED: LogstashNoLogsIndexed: Logstash logs are not being indexed by Elasticsearch - https://wikitech.wikimedia.org/wiki/Logstash#No_logs_indexed - https://grafana.wikimedia.org/d/000000561/logstash?var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashNoLogsIndexed
[04:53:40] <jinxer-wm>	 FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-codfw for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag  - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[04:54:34] <jinxer-wm>	 FIRING: [2x] ErrorBudgetBurn: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[04:57:37] <jinxer-wm>	 RESOLVED: OpensearchClusterHealth: Opensearch cluster health reported as red - https://wikitech.wikimedia.org/wiki/Runbook - https://grafana.wikimedia.org/d/e7d7fa18-7bc3-4548-bb07-ef261a9d3b8b/opensearch-cluster-health?var-cluster=production-elk7-codfw - https://alerts.wikimedia.org/?q=alertname%3DOpensearchClusterHealth
[04:57:40] <jinxer-wm>	 RESOLVED: LogstashClusterStatus: OpenSearch reports cluster status is red. - https://wikitech.wikimedia.org/wiki/Logstash#Unassigned_Shards_and_Cluster_Status - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=panel-49 - https://alerts.wikimedia.org/?q=alertname%3DLogstashClusterStatus
[05:19:34] <jinxer-wm>	 RESOLVED: ErrorBudgetBurn: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[06:14:40] <jinxer-wm>	 FIRING: LogstashNoLogsIndexed: Logstash logs are not being indexed by Elasticsearch - https://wikitech.wikimedia.org/wiki/Logstash#No_logs_indexed - https://grafana.wikimedia.org/d/000000561/logstash?var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashNoLogsIndexed
[06:18:34] <jinxer-wm>	 FIRING: ErrorBudgetBurn: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[06:23:40] <jinxer-wm>	 FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-codfw for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag  - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[06:24:40] <jinxer-wm>	 RESOLVED: LogstashNoLogsIndexed: Logstash logs are not being indexed by Elasticsearch - https://wikitech.wikimedia.org/wiki/Logstash#No_logs_indexed - https://grafana.wikimedia.org/d/000000561/logstash?var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashNoLogsIndexed
[06:28:34] <jinxer-wm>	 RESOLVED: ErrorBudgetBurn: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[06:28:40] <jinxer-wm>	 FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-codfw for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag  - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[06:34:34] <jinxer-wm>	 FIRING: [2x] ErrorBudgetBurn: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[06:54:34] <jinxer-wm>	 RESOLVED: ErrorBudgetBurn: logstash-availability codfw - https://slo.wikimedia.org/?search=logstash-availability   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[10:28:40] <jinxer-wm>	 FIRING: LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[10:58:40] <jinxer-wm>	 RESOLVED: LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[10:59:10] <jinxer-wm>	 FIRING: LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[11:03:55] <jinxer-wm>	 RESOLVED: LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag
[14:07:48] <jinxer-wm>	 FIRING: PuppetDisabled: Puppet disabled on titan2001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=titan&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled
[14:07:58] <tappof>	 ^^ it's me
[18:08:03] <jinxer-wm>	 FIRING: PuppetDisabled: Puppet disabled on titan2001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=titan&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled
[22:08:03] <jinxer-wm>	 FIRING: PuppetDisabled: Puppet disabled on titan2001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=titan&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled
[22:34:24] <tappof>	 ^^ It's always me... I acknowledged the alerts on Karma.