[07:26:00] good morning [07:54:17] morning! :) [09:30:12] 06Machine-Learning-Team, 07Essential-Work: Upgrade ores-legacy from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400348#11056846 (10gkyziridis) 05Open→03Resolved [09:32:16] 06Machine-Learning-Team: Create a notebook for tone check Airflow pipeline - https://phabricator.wikimedia.org/T398937#11056851 (10achou) 05Open→03Resolved Here are the notebooks I created: - Training data generation: [[ https://gitlab.wikimedia.org/repos/machine-learning/exploratory-notebook/-/blob/main... [09:33:14] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.07.26 - 2025.08.15): Create an analytics service user for the ML team - https://phabricator.wikimedia.org/T400902#11056857 (10BTullis) Regarding this item: > Change ownership of /wmf/cache/artifacts/airflow/ml to the new analytics-ml service user. I think t... [10:24:13] 06Machine-Learning-Team, 06Data-Persistence, 06Growth-Team: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11057026 (10Michael) Some first thoughts: * it is //secondary data//, generated by the LiftWing Tone model based on parsed wikite... [10:47:46] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.07.26 - 2025.08.15): Create an analytics service user for the ML team - https://phabricator.wikimedia.org/T400902#11057106 (10BTullis) I have now created: {T401103} to deal with the file ownership issue of Airflow artifacts that are deployed by blunderbuss. [11:42:59] 06Machine-Learning-Team: Error in revscoring-editquality-damaging - itwiki-damaging-predictor-default - https://phabricator.wikimedia.org/T401109 (10OKarakaya-WMF) 03NEW [11:43:30] 06Machine-Learning-Team: Error in revscoring-editquality-damaging - itwiki-damaging-predictor-default - https://phabricator.wikimedia.org/T401109#11057204 (10OKarakaya-WMF) a:03OKarakaya-WMF [12:08:27] 06Machine-Learning-Team: Error in revscoring-editquality-damaging - itwiki-damaging-predictor-default - https://phabricator.wikimedia.org/T401109#11057268 (10OKarakaya-WMF) {F65710267} Although it's not new we get 500,502,503 codes: https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluste... [12:44:10] 06Machine-Learning-Team: Error in revscoring-editquality-damaging - itwiki-damaging-predictor-default - https://phabricator.wikimedia.org/T401109#11057370 (10OKarakaya-WMF) more logs from the pod: ` The above exception was the direct cause of the following exception: Traceback (most recent call last): File... [13:02:56] 06Machine-Learning-Team: Error in revscoring-editquality-damaging - itwiki-damaging-predictor-default - https://phabricator.wikimedia.org/T401109#11057412 (10OKarakaya-WMF) implementation where we get the error: https://gerrit.wikimedia.org/r/plugins/gitiles/machinelearning/liftwing/inference-services/+/refs/he... [13:39:33] 06Machine-Learning-Team: Error in revscoring-editquality-damaging - itwiki-damaging-predictor-default - https://phabricator.wikimedia.org/T401109#11057564 (10OKarakaya-WMF) I've started a branch here to have retry for this specific case: https://gerrit.wikimedia.org/r/plugins/gitiles/machinelearning/liftwing/... [13:54:17] 06Machine-Learning-Team: Error in revscoring-editquality-damaging - itwiki-damaging-predictor-default - https://phabricator.wikimedia.org/T401109#11057586 (10OKarakaya-WMF) We have logs in logstash since 01/06/2025 and I see this error is not new and we have it since the beginning of the logs. I'll try to find... [14:10:59] 06Machine-Learning-Team: Error in revscoring-editquality-damaging - itwiki-damaging-predictor-default - https://phabricator.wikimedia.org/T401109#11057648 (10OKarakaya-WMF) Checking with the related teams here: https://wikimedia.slack.com/archives/C01R06P8D1B/p1754316595382829 [14:14:48] 06Machine-Learning-Team, 06Data-Persistence, 06Growth-Team, 07OKR-Work: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11057667 (10ldelench_wmf) [14:59:54] 06Machine-Learning-Team, 06Data-Persistence, 06Growth-Team, 07OKR-Work: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11057900 (10Eevans) p:05Triage→03Medium [15:15:04] 06Machine-Learning-Team: Error in revscoring-editquality-damaging - itwiki-damaging-predictor-default - https://phabricator.wikimedia.org/T401109#11057954 (10OKarakaya-WMF) Yes, we have ~300 503, and 102 500 errors over the last week. I'll take a look to 500s further. [15:23:37] 06Machine-Learning-Team: Error in revscoring-editquality-damaging - itwiki-damaging-predictor-default - https://phabricator.wikimedia.org/T401109#11057979 (10Joe) I've taken a look from the side of the `api.log` mediawiki generates, and I was a bit surprised to find 16 identical calls over the span of an hour fo... [15:25:11] 06Machine-Learning-Team, 06Data-Persistence, 06Growth-Team, 07OKR-Work: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11057984 (10Eevans) >>! In T401021#11057026, @Michael wrote: > Some first thoughts: > > [ ... ] > * it is mostly st... [15:41:31] 06Machine-Learning-Team: Error in revscoring-editquality-damaging - itwiki-damaging-predictor-default - https://phabricator.wikimedia.org/T401109#11058037 (10Joe) I fear this is a well known we've already encountered. You can see here https://grafana.wikimedia.org/d/zsdYRV7Vk/istio-sidecar?orgId=1&from=now-7d&to... [16:40:23] 06Machine-Learning-Team, 06Data-Persistence, 06Growth-Team, 07OKR-Work: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11058265 (10Eevans)