[06:17:07] (03PS3) 10Kosta Harlan: LiftWingService: Simplify revertRiskLiftWingRequest invocation [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980489 [06:17:13] (03PS2) 10Kosta Harlan: LiftWingService: Extract API endpoint as a config value [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980491 [06:17:19] (03PS1) 10Kosta Harlan: LiftWingService: Set Content-Type header for requests to RevertRisk [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980539 (https://phabricator.wikimedia.org/T348298) [06:18:03] (03CR) 10Kosta Harlan: [C: 03+2] Add revertrisk model to the list of models (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/971547 (https://phabricator.wikimedia.org/T348298) (owner: 10Ilias Sarantopoulos) [06:37:43] Good morning o/ [07:07:07] (03PS1) 10Kosta Harlan: LiftWingServiceTest: Fix property name in test [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980542 [07:07:20] (03PS3) 10Kosta Harlan: LiftWingService: Extract host header endpoint for RevertRisk as a config value [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980491 [07:28:14] 10Lift-Wing, 10Machine-Learning-Team: Enforce json payload in existing kserve model servers - https://phabricator.wikimedia.org/T352834 (10isarantopoulos) [07:29:15] 10Lift-Wing, 10Machine-Learning-Team: Enforce json payload in existing kserve model servers - https://phabricator.wikimedia.org/T352834 (10isarantopoulos) [07:35:18] (03CR) 10Ilias Sarantopoulos: [C: 03+1] LiftWingService: Set Content-Type header for requests to RevertRisk [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980539 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [07:35:50] (03CR) 10Ilias Sarantopoulos: [C: 03+1] LiftWingServiceTest: Fix property name in test [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980542 (owner: 10Kosta Harlan) [07:40:35] 10Lift-Wing, 10Machine-Learning-Team: Enforce json payload in existing kserve model servers - https://phabricator.wikimedia.org/T352834 (10isarantopoulos) [07:44:33] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "LGTM! Having the same format for all responses from the beginning seems way better. Nice1" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980489 (owner: 10Kosta Harlan) [07:57:07] (03CR) 10Ilias Sarantopoulos: [C: 03+2] LiftWingService: Set Content-Type header for requests to RevertRisk [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980539 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [08:00:17] (03Merged) 10jenkins-bot: LiftWingService: Set Content-Type header for requests to RevertRisk [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980539 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [08:02:26] Going afk for 30-60 minutes [08:44:23] morning folks! [08:52:29] 10Machine-Learning-Team, 10Moderator-Tools-Team, 10Temporary accounts, 10Trust and Safety Product Team: RevertRisk model readiness for temporary accounts - https://phabricator.wikimedia.org/T352839 (10kostajh) [09:01:40] o/ [09:01:43] Back [09:13:07] (03CR) 10AikoChou: [C: 03+2] revert-risk: add top level dir to PYTHONPATH (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980394 (https://phabricator.wikimedia.org/T352181) (owner: 10AikoChou) [09:20:29] hey, I'll need some help for https://phabricator.wikimedia.org/T352834 [09:21:12] I'm just planning to submit a patch for revertrisk for now in order to fix any issues that may have occurred but we should also update API docs etc [09:21:26] I'll ping u folks for reviews and we can coordinate here and on the task [09:22:42] I think it makes sense from now on if a service accepts only a specific payload (e.g. json) to validate that it is ok (and/or convert it if needed) otherwise return an error [09:23:52] (03Merged) 10jenkins-bot: revert-risk: add top level dir to PYTHONPATH [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980394 (https://phabricator.wikimedia.org/T352181) (owner: 10AikoChou) [09:39:28] isaranto: agree, that makes sense to me [09:55:02] (03PS1) 10Ilias Sarantopoulos: refactor: rename validate_input function to validate_json_input [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980820 (https://phabricator.wikimedia.org/T352834) [09:55:41] aiko: do you wanna w8 for the new patch for revertrisk before you deploy? [09:55:50] unless you're on it already [09:58:47] 10Machine-Learning-Team: Investigate prediction bug in article-descriptions model-server - https://phabricator.wikimedia.org/T352750 (10kevinbazira) After comparing the 2 requests to understand why the first one was working as expected and the second one was throwing an error, I noticed that although the second... [10:01:01] (03PS1) 10Ilias Sarantopoulos: revertrisk: validate json input [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980821 (https://phabricator.wikimedia.org/T352834) [10:07:41] isaranto: yep I'll wait for that [10:08:13] awesome, thanks! [10:08:20] I filed 2 patches --^ [10:08:52] (03CR) 10AikoChou: [C: 03+1] refactor: rename validate_input function to validate_json_input [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980820 (https://phabricator.wikimedia.org/T352834) (owner: 10Ilias Sarantopoulos) [10:10:38] (03CR) 10Ilias Sarantopoulos: [C: 03+2] refactor: rename validate_input function to validate_json_input [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980820 (https://phabricator.wikimedia.org/T352834) (owner: 10Ilias Sarantopoulos) [10:11:14] (03CR) 10AikoChou: "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980821 (https://phabricator.wikimedia.org/T352834) (owner: 10Ilias Sarantopoulos) [10:18:29] I'm going to w8 for ci to merge the first patch, I'll rebase and then merge the second one. Will ping you once things are ready [10:19:30] ack [10:20:01] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Enforce json payload in existing kserve model servers - https://phabricator.wikimedia.org/T352834 (10isarantopoulos) 05Open→03In progress p:05Triage→03Unbreak! a:03isarantopoulos [10:21:01] (03Merged) 10jenkins-bot: refactor: rename validate_input function to validate_json_input [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980820 (https://phabricator.wikimedia.org/T352834) (owner: 10Ilias Sarantopoulos) [10:24:18] (03CR) 10Ilias Sarantopoulos: [C: 03+2] "Tested locally!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980821 (https://phabricator.wikimedia.org/T352834) (owner: 10Ilias Sarantopoulos) [10:27:56] (03CR) 10Ilias Sarantopoulos: [C: 03+2] LiftWingService: Simplify revertRiskLiftWingRequest invocation [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980489 (owner: 10Kosta Harlan) [10:31:39] (03Merged) 10jenkins-bot: revertrisk: validate json input [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980821 (https://phabricator.wikimedia.org/T352834) (owner: 10Ilias Sarantopoulos) [10:34:09] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Enforce json payload in existing kserve model servers - https://phabricator.wikimedia.org/T352834 (10isarantopoulos) Updated revertrisk models to enforce json payload. I see that in all the pages in API GW documentation we instruct users to use the Co... [10:34:42] isaranto: o/ I ran into some issues when trying to run the llm refactor patch locally. maybe later today can we have a quick call to discuss it? [10:35:58] aiko: sure, if it is easy to describe let me know what the issue is so i can take a look before hand [10:36:14] going to run it one more time now to check [10:38:27] a, I'm also having an issue with the imports. let me test it and fix it. Sorry! [10:40:26] (03Merged) 10jenkins-bot: LiftWingService: Simplify revertRiskLiftWingRequest invocation [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980489 (owner: 10Kosta Harlan) [10:45:30] (03PS2) 10Ilias Sarantopoulos: llm: refactor directory structure to treat as python module. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980429 [10:58:06] (03PS3) 10Ilias Sarantopoulos: llm: refactor directory structure to treat as python module. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980429 [10:59:41] * klausman early lunch [11:03:33] aiko: let me know if it is ok now. I have tested it locally but building the image takes a year [11:04:27] (03PS7) 10AikoChou: revert-risk: add batch_model.py and USE_BATCHER env var [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/977135 (https://phabricator.wikimedia.org/T348536) [11:05:43] isaranto: okkk, I'll test it again [11:06:18] isaranto: and the batch patch is ready to be reviewed :) ---^ [11:06:25] ack! [11:07:05] I have a plan (in my head) to add unit tests for all these things that way we won't even have to test that things work. If y'all agree we can do this [11:14:40] that would be great! but it would probably require a considerable amount of work. we can approach it step by step, starting from one model server, etc [11:20:19] (03PS4) 10Ilias Sarantopoulos: llm: refactor directory structure to treat as python module. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980429 [11:22:30] 10Machine-Learning-Team, 10ORES: Add deprecation warnings to ORES-related repositories on Github - https://phabricator.wikimedia.org/T349632 (10klausman) Opened these pull requests: https://github.com/wikimedia/editquality/pull/240 https://github.com/wikimedia/draftquality/pull/45 https://github.com/wikimedia... [11:27:58] isaranto: AttributeError: module 'llm' has no attribute 'NLLB' .. did I miss something? [11:30:55] oh ruff removed everything I had in __init__.py [11:30:55] w8 [11:38:24] aiko: just fixed it. 🙏 [11:39:21] the following thing happened: I tested it, made the commit but didnt pay attention that ruff wiped out __init__.py. if there is no __all__ variable in the init py file ruff deletes everything [11:46:22] isaranto: i see, but you haven't updated the patch :D [11:46:35] (03PS5) 10Ilias Sarantopoulos: llm: refactor directory structure to treat as python module. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980429 [11:47:25] 🤦 I think I've been juggling too many things at once today [11:48:27] the funny thing is that I'm writing the Pull request guidelines we discussed and I'm just doing things wrong. in greek there is a saying that highlights this behavior :) [11:50:39] hahaha [11:51:08] teacher that teaches or sth like that would be the translation [11:52:23] * isaranto afk lunch [11:52:36] isaranto: ah one more thing https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/980819 if you have a moment :D [11:53:18] * aiko lunch as well [13:19:18] (03PS1) 10Ilias Sarantopoulos: article-descriptions: validate json input [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980862 (https://phabricator.wikimedia.org/T352834) [13:19:28] Morning [13:22:18] Buongiorno! [13:29:27] hello folks! [13:33:22] o/ Luca! [13:50:47] ¡Hola! [13:55:42] I'm currently working on an APIGW endpoint for the rec-api-ng, but am unsure what the best URL on the APIGW would be. Feel free to leave comments on the code review: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/980865/comments/feca9c38_738353e1 Or on the ticket: https://phabricator.wikimedia.org/T347263 [14:13:10] (03CR) 10Ilias Sarantopoulos: [C: 03+2] LiftWingService: Extract host header endpoint for RevertRisk as a config value [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980491 (owner: 10Kosta Harlan) [14:16:00] (03Merged) 10jenkins-bot: LiftWingService: Extract host header endpoint for RevertRisk as a config value [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980491 (owner: 10Kosta Harlan) [14:18:53] (03CR) 10Kevin Bazira: [C: 03+1] "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980862 (https://phabricator.wikimedia.org/T352834) (owner: 10Ilias Sarantopoulos) [14:25:22] (03CR) 10Ilias Sarantopoulos: [C: 03+2] article-descriptions: validate json input [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980862 (https://phabricator.wikimedia.org/T352834) (owner: 10Ilias Sarantopoulos) [14:26:08] (03Merged) 10jenkins-bot: article-descriptions: validate json input [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980862 (https://phabricator.wikimedia.org/T352834) (owner: 10Ilias Sarantopoulos) [14:54:42] still no sign of a new catboost release [14:54:43] sigh [15:07:38] 10Machine-Learning-Team, 10observability, 10SRE Observability (FY2023/2024-Q2): Gap in metrics rendered from Thanos Rules - https://phabricator.wikimedia.org/T352756 (10lmata) [15:07:52] 10Machine-Learning-Team, 10SRE Observability (FY2023/2024-Q2): Gap in metrics rendered from Thanos Rules - https://phabricator.wikimedia.org/T352756 (10lmata) [15:51:33] klausman: do you have a min for a puppet private code review? [15:51:45] sure [15:52:04] I am removing the cergen's cert for the istio egress gateway, we never really used it [15:52:12] all staged on puppetmaster1001 [15:52:38] looking [15:53:09] modules/secret/secrets/certificates/certificate.manifests.d/kube_services.certs.yaml is not staged [15:53:34] other than that, LGTM [15:53:54] yep yep, thanks, committing [15:55:39] 10Machine-Learning-Team, 10SRE Observability (FY2023/2024-Q2): Gap in metrics rendered from Thanos Rules - https://phabricator.wikimedia.org/T352756 (10herron) @elukey could the gaps possibly be attributed to changes in the queries/recording rules? I know we've been through a few iterations for these SLOs, is... [15:55:40] I see the issuer for kserve as group: cfssl-issuer.wikimedia.org [15:57:39] Is that not what is desired? [15:58:22] sorry wrong channel :) [15:58:24] yes yes :) [15:58:36] klausman: another one to check please [15:58:39] also if it makes sense [15:58:54] afaics kserve uses cfssl issuers, but please double check to be sure [16:00:58] LGTM [16:12:08] (03PS1) 10Amire80: Remove a space from the beginning of a message [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980897 [16:17:34] (03PS6) 10Ilias Sarantopoulos: llm: refactor directory structure to treat as python module. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980429 [16:24:38] (03CR) 10Ilias Sarantopoulos: [C: 03+2] Remove a space from the beginning of a message [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980897 (owner: 10Amire80) [16:27:46] isaranto: o/ yayy nllb is running locally [16:28:18] aiko: I'm also reviewin batcher at the moment! works nice! [16:30:05] isaranto: one thing, do we also want to validate json input for nllb model? [16:31:08] yes, it is better to have it for all model servers. I wrote the list here to track it down https://phabricator.wikimedia.org/T352834#9386042 [16:31:38] I actually created a local branch to do it for llm and nllb but didnt do anything yet [16:35:53] Okk got it :) [16:41:40] (03Merged) 10jenkins-bot: Remove a space from the beginning of a message [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980897 (owner: 10Amire80) [16:47:51] (03CR) 10DannyS712: [C: 03+2] LiftWingServiceTest: Fix property name in test [extensions/ORES] - 10https://gerrit.wikimedia.org/r/980542 (owner: 10Kosta Harlan) [16:49:03] (03CR) 10AikoChou: [C: 03+1] "LGTM! Only two nits in the README.md" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980429 (owner: 10Ilias Sarantopoulos) [17:26:42] (03CR) 10Ilias Sarantopoulos: "LGTM! I tested it locally with language-agnostic and it works great! Please add something in the README.md (how to run etc) especially sin" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/977135 (https://phabricator.wikimedia.org/T348536) (owner: 10AikoChou) [17:28:04] aiko: review done! [17:29:52] isaranto: thanks :) [17:31:48] I think it is a good first step. We now support this functionality and now we can build on top of it and make it faster [17:31:52] nice work! [17:35:52] (03PS7) 10Ilias Sarantopoulos: llm: refactor directory structure to treat as python module. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980429 [17:36:02] have a nice rest of the day folks! [17:37:39] (03PS8) 10Ilias Sarantopoulos: llm: refactor directory structure to treat as python module. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980429 [17:39:16] (03CR) 10Ilias Sarantopoulos: "Changes done! I stole stuff from the revertrisk README:)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980429 (owner: 10Ilias Sarantopoulos) [17:39:57] ciao Luca! I'm going afk as well. o/ [17:40:18] have a nice day/evening every1 [17:41:26] ciao Ilias and Luca o/ [17:47:13] (03CR) 10CI reject: [V: 04-1] llm: refactor directory structure to treat as python module. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980429 (owner: 10Ilias Sarantopoulos) [19:05:07] (03CR) 10Ilias Sarantopoulos: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/980429 (owner: 10Ilias Sarantopoulos) [19:49:00] 10Machine-Learning-Team, 10ORES, 10Growth-Team-Filtering, 10PageTriage: Add ORES topic prediction to the NewPagesFeed - https://phabricator.wikimedia.org/T218132 (10Novem_Linguae) [19:53:30] 10Machine-Learning-Team, 10ORES, 10Growth-Team-Filtering, 10PageTriage: Add ORES topic prediction to the NewPagesFeed - https://phabricator.wikimedia.org/T218132 (10Novem_Linguae) [19:56:11] 10Machine-Learning-Team, 10ORES, 10Growth-Team-Filtering, 10PageTriage: Add ORES topic prediction to the NewPagesFeed - https://phabricator.wikimedia.org/T218132 (10Novem_Linguae)