[02:46:45] FIRING: ErrorBudgetBurn: - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [06:01:31] (03CR) 10Kevin Bazira: "Hi Eamedina, we are following these changes. When they are approved and merged on your end. We update the image and deployment config in L" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1072241 (owner: 10Nik Gkountas) [06:19:47] 06Machine-Learning-Team: Run load tests for the rec-api-ng and update production resources to meet expected load - https://phabricator.wikimedia.org/T365554#10151538 (10kevinbazira) 05Open→03Resolved [06:21:11] 06Machine-Learning-Team, 13Patch-For-Review: Host the recommendation-api container on LiftWing - https://phabricator.wikimedia.org/T339890#10151540 (10kevinbazira) 05Open→03Resolved [06:22:21] 06Machine-Learning-Team: Containerize Content Translation Recommendation API - https://phabricator.wikimedia.org/T338805#10151542 (10kevinbazira) 05Open→03Resolved [06:31:57] 06Machine-Learning-Team, 06Language-Team, 07Epic: Migrate Content Translation Recommendation API to Lift Wing - https://phabricator.wikimedia.org/T308164#10151547 (10kevinbazira) 05Open→03Resolved Closing this task as we have completed migrating the content translation recommendation API to LiftWing,... [06:32:27] 06Machine-Learning-Team, 10MW-1.43-notes (1.43.0-wmf.17; 2024-08-06), 07OKR-Work: Deploy Modernized Recommendation API to LiftWing - https://phabricator.wikimedia.org/T371465#10151552 (10kevinbazira) 05Open→03Resolved [06:46:45] FIRING: ErrorBudgetBurn: - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [07:32:10] Hello! [07:39:23] morning! [09:22:23] (03PS2) 10AikoChou: locust: fix formatting in README for reference_quality [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1073188 [09:28:50] (03CR) 10AikoChou: "I used a markdown preview and it looks okay, but "Response time percentiles" is in a separate line. Is it supposed to be in the table?" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1073188 (owner: 10AikoChou) [09:29:23] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1073404 [09:29:29] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1073400 [09:29:40] ---^ ref-quality [09:45:59] on it! [09:47:03] aiko: the Response time percentiles is also a table as far as I remember [09:48:09] I used an online preview tool and it seems good, my IDE wasn't rendering it properly [09:48:46] (03CR) 10Ilias Sarantopoulos: [C:03+1] locust: fix formatting in README for reference_quality [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1073188 (owner: 10AikoChou) [09:48:53] let's merge it and see [09:49:30] ok! [09:49:36] (03CR) 10AikoChou: [C:03+2] locust: fix formatting in README for reference_quality [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1073188 (owner: 10AikoChou) [09:49:43] (03CR) 10AikoChou: [V:03+2 C:03+2] locust: fix formatting in README for reference_quality [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1073188 (owner: 10AikoChou) [09:51:29] https://gerrit.wikimedia.org/r/plugins/gitiles/machinelearning/liftwing/inference-services/+/refs/heads/main/test/locust/models/reference_quality/ your IDE were right lol [09:52:56] aiko: Don't worry about it, I'll fix it! [09:53:51] thanks <3 [09:54:01] 06Machine-Learning-Team, 06Moderator-Tools-Team, 06Research, 10Temporary accounts, 06Trust and Safety Product Team: RevertRisk model readiness for temporary accounts - https://phabricator.wikimedia.org/T352839#10152242 (10kostajh) [10:00:12] (03PS1) 10Ilias Sarantopoulos: fix: markdown in ref-need locust README.md [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1073419 [10:10:34] klausman: o/ [10:10:34] Is this change enough in order to update the SLO dashboards? [10:10:59] Looking [10:11:34] Yes [10:12:59] Do you have merge permission on that repo? [10:13:06] nope! [10:13:13] ok, will merge and deploy [10:13:19] thank you! [10:18:45] And done [10:18:56] can you confirm the dashboards now show the current Q? [10:24:38] I checked 1-2 and they seem ok, they have the new dates, thanks again! [10:24:42] * isaranto afk - lunch [10:24:45] np! [10:24:48] * klausman also lunch [10:46:45] FIRING: ErrorBudgetBurn: - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [10:50:32] (03CR) 10AikoChou: [C:03+1] "Nice!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1073419 (owner: 10Ilias Sarantopoulos) [10:55:03] /me lunch [10:55:06] lol [11:31:34] (03CR) 10Ilias Sarantopoulos: [V:03+2 C:03+2] fix: markdown in ref-need locust README.md [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1073419 (owner: 10Ilias Sarantopoulos) [12:24:41] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1073444 [12:24:52] ---^ sth was missing [12:27:24] isaranto: o/ I found that the articlequality isvc is also missing that [12:28:19] I'll submit a patch to fix articlequality as well [12:28:52] good catch, thanks! probably some bad copy/pasting without checking [12:31:09] 10Lift-Wing, 06Machine-Learning-Team: Log and export preprocess size in inference services as a prometheus metric - https://phabricator.wikimedia.org/T374034#10152878 (10isarantopoulos) [13:08:16] sigh we still have gaps https://grafana.wikimedia.org/d/slo-Lift_Wing_Revert_Risk_LA/lift-wing-revert-risk-la-slo-s?orgId=1 [13:08:41] we probably need to ping observability for https://phabricator.wikimedia.org/T352756 [13:52:48] 06Machine-Learning-Team: [LLM] log input/output size per request - https://phabricator.wikimedia.org/T370775#10153211 (10isarantopoulos) This will be tackled after T374034 [13:56:05] 10Lift-Wing, 06Machine-Learning-Team, 13Patch-For-Review: [LLM] Use vllm for ROCm in huggingface image - https://phabricator.wikimedia.org/T370149#10153224 (10isarantopoulos) a:03isarantopoulos [14:36:08] 06Machine-Learning-Team, 05Goal: Goal 1: Non-technical users can make a request to a Hugging Face Large Language Model that is fast in production. - https://phabricator.wikimedia.org/T371395#10153511 (10isarantopoulos) - Slow progress on vllm. @isarantopoulos will discuss the issue with @klausman and @achou d... [14:38:09] 06Machine-Learning-Team, 05Goal: Goal 2: People outside the ML team can ssh into an ml-lab machine, run a Jupyter Notebook, and run PyTorch powered by a GPU. - https://phabricator.wikimedia.org/T371396#10153520 (10isarantopoulos) - Working on bundling things on the alb machines - We should be working with ROC... [14:42:13] 06Machine-Learning-Team, 05Goal: Goal 3: Operational Excellence - Improve base monitoring, alerting and logging of Lift Wing services. - https://phabricator.wikimedia.org/T371397#10153540 (10isarantopoulos) - SLO dashboards have been updated to show the dates for the current quarter. - Discussed adding some mi... [14:44:00] 06Machine-Learning-Team, 05Goal: Goal 4: Support product teams in deploying production models. - https://phabricator.wikimedia.org/T371398#10153565 (10isarantopoulos) - Reference need model is already deployed in staging and is going to be deployed to production this week. - Reference risk and article country... [14:46:45] FIRING: ErrorBudgetBurn: - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [14:58:17] 06Machine-Learning-Team, 06Structured-Data-Backlog, 07OKR-Work: Host a logo detection model for Commons images - https://phabricator.wikimedia.org/T358676#10153619 (10isarantopoulos) 05In progress→03Resolved [15:00:20] 10Lift-Wing, 06Machine-Learning-Team, 13Patch-For-Review: Request to update Readability model on Lift Wing - https://phabricator.wikimedia.org/T369712#10153628 (10achou) 05Open→03Resolved [15:13:01] 06Machine-Learning-Team, 10ORES, 06Moderator-Tools-Team, 07Spike: [SPIKE] Investigate how to install ORES in idwiki [8HRS] - https://phabricator.wikimedia.org/T374077#10153681 (10jsn.sherman) [16:07:25] 06Machine-Learning-Team, 10ORES, 10Moderator-Tools-Team (Kanban), 07Spike: [SPIKE] Investigate how to install ORES in idwiki [8HRS] - https://phabricator.wikimedia.org/T374077#10154000 (10DMburugu) [16:44:28] going afk folks, have a nice evening/rest of day [18:46:45] FIRING: ErrorBudgetBurn: - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [19:51:45] RESOLVED: ErrorBudgetBurn: - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [20:37:57] (03CR) 10Eamedina: [C:03+2] Remove parameter from WIKIMEDIA_API url [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1072241 (owner: 10Nik Gkountas) [20:39:28] (03Merged) 10jenkins-bot: Remove parameter from WIKIMEDIA_API url [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1072241 (owner: 10Nik Gkountas) [20:57:34] (03PS4) 10Eamedina: Fetch campaign metadata and return them with recommendations [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1070308 (https://phabricator.wikimedia.org/T373132) (owner: 10Nik Gkountas) [20:58:20] (03CR) 10CI reject: [V:04-1] Fetch campaign metadata and return them with recommendations [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1070308 (https://phabricator.wikimedia.org/T373132) (owner: 10Nik Gkountas) [21:01:36] (03CR) 10Eamedina: [C:04-1] Fetch campaign metadata and return them with recommendations (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1070308 (https://phabricator.wikimedia.org/T373132) (owner: 10Nik Gkountas) [21:03:14] (03CR) 10Eamedina: [C:03+2] "Thanks kevinbazira" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1072241 (owner: 10Nik Gkountas)