[08:00:57] <isaranto>	 in the the alerts that fired yesterday I see there is a 29 min delay in ORESFetchScoreJob when MediaWiki is fetching scores (vs 20s normal time) https://grafana.wikimedia.org/d/LSeAShkGz/jobqueue?orgId=1&viewPanel=64
[08:04:03] <isaranto>	 this is caused by enwiki-damaging where we can see the same pattern visible in the p75 chart in preprocessing https://grafana.wikimedia.org/goto/jgvb9__SR?orgId=1
[08:27:00] <isaranto>	 and here are the logs with the revids taking too long. get_revscoring_extractor_cache is taking too long https://logstash.wikimedia.org/goto/82297a52c37ac94c58081b5471ce3075
[18:07:49] <jinxer-wm>	 FIRING: KubernetesDeploymentUnavailableReplicas: ...
[18:07:49] <jinxer-wm>	 Deployment wikidatawiki-itemquality-predictor-default-00018-deployment in revscoring-articlequality at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - ...
[18:07:49] <jinxer-wm>	 https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revscoring-articlequality&var-deployment=wikidatawiki-itemquality-predictor-default-00018-deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
[18:12:49] <jinxer-wm>	 RESOLVED: KubernetesDeploymentUnavailableReplicas: ...
[18:12:49] <jinxer-wm>	 Deployment wikidatawiki-itemquality-predictor-default-00018-deployment in revscoring-articlequality at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - ...
[18:12:49] <jinxer-wm>	 https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revscoring-articlequality&var-deployment=wikidatawiki-itemquality-predictor-default-00018-deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas