[02:11:44] <jinxer-wm>	 FIRING: LiftWingServiceErrorRate: ...
[02:11:44] <jinxer-wm>	 LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-damaging&var-backend=frwiki-damaging-predictor-default.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate
[02:36:44] <jinxer-wm>	 RESOLVED: LiftWingServiceErrorRate: ...
[02:36:44] <jinxer-wm>	 LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-damaging&var-backend=frwiki-damaging-predictor-default.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate
[07:25:48] <isaranto>	 Good morning !
[08:33:48] <aiko>	 morning!
[08:44:21] <isaranto>	 o/ Aiko!
[08:58:11] <elukey>	 早上好人人！ 
[08:58:41] <elukey>	 as FYI I have reimaged one of the Dragonfly supernodes in codfw, now runs Bookworm
[08:59:06] <isaranto>	 o/ grazie!
[08:59:08] <elukey>	 if you see anything weird during deployments lemme know (The host is part of the p2p network that we use to cache docker layers etc..)
[08:59:41] <wikibugs>	 06Machine-Learning-Team, 10ORES, 07dark-mode, 10MW-1.43-notes (1.43.0-wmf.22; 2024-09-10), and 3 others: Damaging edits are not readable in contributions page in dark mode due to ORES highlight - https://phabricator.wikimedia.org/T368680#10129055 (10Ebrahim) https://gerrit.wikimedia.org/r/1050501  {F57...
[09:49:23] <isaranto>	 I deployed articlequality to prod again, didnt face any issues!
[09:51:25] <aiko>	 elukey: were you trying to say "everyone"? we don't say 人人 (literally "people people"). we say 大家 (dà jiā, literally "big family") instead. I know it's confusing :D
[10:01:48] <elukey>	 aiko: I did yes! I thought 人人 was everyone! Of course 大家 
[10:01:52] <elukey>	 is better :)
[10:02:01] <elukey>	 I'll take notes :D
[10:28:35] <wikibugs>	 06Machine-Learning-Team, 06Content-Transform-Team, 06Research, 13Patch-For-Review: Add Article Quality Model to LiftWing - https://phabricator.wikimedia.org/T360455#10129300 (10isarantopoulos) All changes are deployed and the model is ready for use:  Simple request: ` % curl https://api.wikimedia.org/servi...
[11:38:15] <wikibugs>	 06Machine-Learning-Team, 10ORES, 07dark-mode, 10MW-1.43-notes (1.43.0-wmf.22; 2024-09-10), and 3 others: Damaging edits are not readable in contributions page in dark mode due to ORES highlight - https://phabricator.wikimedia.org/T368680#10129501 (10Ebrahim) Recent comments weren't related to this issu...
[11:39:02] * isaranto afk lunch!
[13:31:43] <isaranto>	 running an errand be back in 30'!
[14:12:55] <isaranto>	 back online!
[14:26:10] <wikibugs>	 06Machine-Learning-Team, 13Patch-For-Review: Migrate the ownership of ML-Owned Docker images in production-images repo to mailing lists - https://phabricator.wikimedia.org/T374233#10130138 (10elukey) I'd also add the knative images:  ` ls images/knative/serving/ activator  autoscaler  build  controller  domain...
[14:27:52] <wikibugs>	 06Machine-Learning-Team, 13Patch-For-Review: Migrate the ownership of ML-Owned Docker images in production-images repo to mailing lists - https://phabricator.wikimedia.org/T374233#10130128 (10elukey)
[14:37:07] <wikibugs>	 06Machine-Learning-Team, 13Patch-For-Review: Migrate the ownership of ML-Owned Docker images in production-images repo to mailing lists - https://phabricator.wikimedia.org/T374233#10130194 (10isarantopoulos)
[14:47:00] <wikibugs>	 06Machine-Learning-Team, 13Patch-For-Review: Migrate the ownership of ML-Owned Docker images in production-images repo to mailing lists - https://phabricator.wikimedia.org/T374233#10130267 (10isarantopoulos)
[15:22:46] <chrisalbon>	 Good morning all
[15:23:32] <isaranto>	 Mooorning Chris o/
[15:45:50] <aiko>	 o/ hi Chris
[17:14:59] <isaranto>	 going afk folks, have a nice evening/rest of day
[17:42:42] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team: Request to host the Reference Risk Model on LiftWing - https://phabricator.wikimedia.org/T372405#10131224 (10XiaoXiao-WMF) @AikoChou a side note - please make sure, when we integrate with Enterprise, we can record usage (User Agent String to the http Requests or something...
[18:40:59] <wikibugs>	 06Machine-Learning-Team, 06Content-Transform-Team, 06Research, 13Patch-For-Review: Add Article Quality Model to LiftWing - https://phabricator.wikimedia.org/T360455#10131485 (10Isaac) many thanks @isarantopoulos! I think from my end, we can mark this as resolved. If anything comes up when Enterprise starts...