[20:08:30] (ORESFetchScoreJobKafkaLag) firing: Kafka consumer lag for ORESFetchScoreJob over threshold for past 1h. ... [20:08:35] - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#Kafka_Consumer_lag_-_ORESFetchScoreJobKafkaLag_alert - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&orgId=1&to=now&var-cluster=main-codfw&var-consumer_group=cpjobqueue-ORESFetchScoreJob - https://alerts.wikimedia.org/?q=alertname%3DORESFetchScoreJobKafkaLag [20:43:45] (ORESFetchScoreJobKafkaLag) resolved: Kafka consumer lag for ORESFetchScoreJob over threshold for past 1h. ... [20:43:45] - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#Kafka_Consumer_lag_-_ORESFetchScoreJobKafkaLag_alert - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&orgId=1&to=now&var-cluster=main-codfw&var-consumer_group=cpjobqueue-ORESFetchScoreJob - https://alerts.wikimedia.org/?q=alertname%3DORESFetchScoreJobKafkaLag [20:49:51] Alerts seem to work! Ofc the alert would fire on a Sunday :) [22:44:30] (ORESFetchScoreJobKafkaLag) firing: Kafka consumer lag for ORESFetchScoreJob over threshold for past 1h. ... [22:44:30] - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#Kafka_Consumer_lag_-_ORESFetchScoreJobKafkaLag_alert - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&orgId=1&to=now&var-cluster=main-codfw&var-consumer_group=cpjobqueue-ORESFetchScoreJob - https://alerts.wikimedia.org/?q=alertname%3DORESFetchScoreJobKafkaLag [23:04:30] (ORESFetchScoreJobKafkaLag) resolved: Kafka consumer lag for ORESFetchScoreJob over threshold for past 1h. ... [23:04:30] - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#Kafka_Consumer_lag_-_ORESFetchScoreJobKafkaLag_alert - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&orgId=1&to=now&var-cluster=main-codfw&var-consumer_group=cpjobqueue-ORESFetchScoreJob - https://alerts.wikimedia.org/?q=alertname%3DORESFetchScoreJobKafkaLag