[07:00:18] hi folks! [07:05:06] Good morning [08:32:35] morning! o/ [08:33:44] FIRING: LiftWingServiceErrorRate: ... [08:33:44] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-damaging&var-backend=plwiki-damaging-predictor-default.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [08:38:44] RESOLVED: LiftWingServiceErrorRate: ... [08:38:44] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-damaging&var-backend=plwiki-damaging-predictor-default.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [10:15:55] 10Lift-Wing, 06Machine-Learning-Team, 10Wikimedia Enterprise - Content Integrity, 13Patch-For-Review: Load test the language agnostic article-quality model - https://phabricator.wikimedia.org/T388805#10732832 (10isarantopoulos) I ran 2 types of load tests for the existing service: A. simulating 2 concurre... [10:16:42] (03PS4) 10Ilias Sarantopoulos: articlequality: add async requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1135721 (https://phabricator.wikimedia.org/T388805) [10:29:38] * isaranto lunch [10:56:33] * klausman lunch as well [12:53:48] 06Machine-Learning-Team: [onboarding] Improving language agnostic articlequality model + service - https://phabricator.wikimedia.org/T391679 (10isarantopoulos) 03NEW [12:59:53] 10Lift-Wing, 06Machine-Learning-Team: [onboarding] Improving language agnostic articlequality model + service - https://phabricator.wikimedia.org/T391679#10733191 (10isarantopoulos) [13:00:31] 10Lift-Wing, 06Machine-Learning-Team: [onboarding] Improving language agnostic articlequality model + service - https://phabricator.wikimedia.org/T391679#10733197 (10isarantopoulos) [13:07:00] 10Lift-Wing, 06Machine-Learning-Team: [onboarding] Improving language agnostic articlequality model + service - https://phabricator.wikimedia.org/T391679#10733218 (10isarantopoulos) a:03OKarakaya-WMF [13:07:02] ozge_: aiko [13:07:32] I created the task with all the info we discussed. Please review it when you have time and add any information that is missing [13:26:32] 06Machine-Learning-Team, 06collaboration-services, 10Discovery-Search (2025.04.11 - 2025.05.02), 10Wikipedia-iOS-App-Backlog (iOS Release FY2024-25): [Spike] Fetch Topics for Articles in History on iOS app - https://phabricator.wikimedia.org/T379119#10733506 (10Gehel) [13:50:36] 10Lift-Wing, 06Machine-Learning-Team, 10Wikimedia Enterprise - Content Integrity, 13Patch-For-Review: Load test the language agnostic article-quality model - https://phabricator.wikimedia.org/T388805#10733663 (10isarantopoulos) a:03isarantopoulos [13:54:25] 10Lift-Wing, 06Machine-Learning-Team, 10EditCheck: Load test the peacock edit check service - https://phabricator.wikimedia.org/T388817#10733669 (10isarantopoulos) 05Open→03Resolved [14:04:19] 06Machine-Learning-Team, 07sre-alert-triage: Alert in need of triage: DiskSpace (instance ml-lab1001:9100) - https://phabricator.wikimedia.org/T391465#10733691 (10isarantopoulos) I've deleted 30GB from my home directory. @klausman are there any quick wins to clean up disk space for now? I think purging the h... [14:06:46] 06Machine-Learning-Team, 07sre-alert-triage: Alert in need of triage: DiskSpace (instance ml-lab1001:9100) - https://phabricator.wikimedia.org/T391465#10733710 (10klausman) >>! In T391465#10733690, @isarantopoulos wrote: > I've deleted 30GB from my home directory. > @klausman are there any quick wins to clean... [14:43:03] Great! Thank you @isaranto