[10:30:17] <wikibugs>	 (03PS4) 10Thiemo Kreuz (WMDE): build: Updating composer dependencies [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1246497 (owner: 10Libraryupgrader)
[11:06:26] <wikibugs>	 (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] "I think I found a more straightforward solution. Is this ok?" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1246497 (owner: 10Libraryupgrader)
[15:44:59] <wikibugs>	 06Machine-Learning-Team, 06Growth-Team, 10New-Engagement-Experiments, 06Research: [RFC] Personalized article recommendations for Newcomer Tasks using content-based filtering - https://phabricator.wikimedia.org/T418051#11711650 (10Aditya_Pola) Thanks for the context! Great to see @Lwilson-ctr and @Samwalton...
[21:19:44] <jinxer-wm>	 FIRING: LiftWingServiceErrorRate: ...
[21:19:44] <jinxer-wm>	 LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=codfw%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-damaging&var-backend=frwiki-damaging-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate
[21:51:50] <jinxer-wm>	 FIRING: ORESFetchScoreJobKafkaLag: Kafka consumer lag for ORESFetchScoreJob over threshold for past 1h. ...
[21:51:50] <jinxer-wm>	 - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#Kafka_Consumer_lag_-_ORESFetchScoreJobKafkaLag_alert - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&orgId=1&to=now&var-cluster=main-codfw&var-consumer_group=cpjobqueue-ORESFetchScoreJob&var-datasource=%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DORESFetchScoreJobKafkaLag
[23:11:50] <jinxer-wm>	 RESOLVED: ORESFetchScoreJobKafkaLag: Kafka consumer lag for ORESFetchScoreJob over threshold for past 1h. ...
[23:11:50] <jinxer-wm>	 - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#Kafka_Consumer_lag_-_ORESFetchScoreJobKafkaLag_alert - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&orgId=1&to=now&var-cluster=main-codfw&var-consumer_group=cpjobqueue-ORESFetchScoreJob&var-datasource=%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DORESFetchScoreJobKafkaLag
[23:14:44] <jinxer-wm>	 RESOLVED: LiftWingServiceErrorRate: ...
[23:14:44] <jinxer-wm>	 LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=codfw%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-damaging&var-backend=frwiki-damaging-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate