[08:11:30] (03PS1) 10Kevin Bazira: article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) [08:13:02] (03CR) 10CI reject: [V: 04-1] article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [08:19:54] (03PS2) 10Kevin Bazira: article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) [08:20:36] (03CR) 10CI reject: [V: 04-1] article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [08:38:57] Hi folks! [08:51:09] (03PS3) 10Kevin Bazira: article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) [08:56:31] isaranto: hi o/ [08:57:06] hello folks :) [08:57:10] got back earlier [09:10:22] hey! I see you got an approval in https://github.com/catboost/catboost/pull/2519 🎉 [09:11:08] yeah finally! Now I am puzzled, because I am not sure why it wasn't merged [09:11:57] anyway, I tested it with docker etc.. limiting cpus, and it seems to work fine [09:12:08] sooo in theory it should work out-of-the-box in our use case [09:12:15] I didn't create an extra ticket for the kserve upgrades for catboost models as we have separate tasks for each one in blocked [09:13:10] 10Machine-Learning-Team, 10Patch-For-Review: Upgrade Revert Risk Multilingual docker images to KServe 0.11 - https://phabricator.wikimedia.org/T347551 (10isarantopoulos) We can proceed with this after https://github.com/catboost/catboost/pull/2519 has been included in a new catboost release (support for Cgroup... [09:13:21] 10Machine-Learning-Team, 10Patch-For-Review: Upgrade the readability model server to KServe 0.11.1 - https://phabricator.wikimedia.org/T348664 (10isarantopoulos) We can proceed with this after https://github.com/catboost/catboost/pull/2519 has been included in a new catboost release (support for CgroupsV2) [09:23:19] it may be due to https://github.com/catboost/catboost/issues/2525 [09:23:42] but I am not 100% sure how to fix it, the cmake config of the repo is very dense [09:43:25] ¯\_(ツ)_/¯ [10:25:14] kevinbazira: I'm trying to run article-descriptions locally. have you ran it with the above patch? I'm having difficulty understanding if the BERT_PATH would work [10:25:50] would it be ok if I submit a patch (once I have it ready) to modify the model server so that it can run from localhost without docker? [10:27:53] same, I think that BERT_PATH needs to be absolute, something like output_dir/name-of-the-file [10:28:22] it may be working on ml-sandbox with the old file downloaded [10:28:30] but not sure about that either [10:44:20] restarted all ml-cache cassandra nodes to pick up a ca-bundle puppet change, all good [11:13:27] (03Abandoned) 10Ilias Sarantopoulos: revscoring: add missing type in function [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/976270 (owner: 10Ilias Sarantopoulos) [11:16:14] (03PS1) 10Ilias Sarantopoulos: article-descriptions: enable local run [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/976670 [11:36:07] (03PS3) 10Ilias Sarantopoulos: Change default config values to support local/patchdemo deployments [extensions/ORES] - 10https://gerrit.wikimedia.org/r/976157 (https://phabricator.wikimedia.org/T351703) [11:36:52] (03PS4) 10Ilias Sarantopoulos: Change default config values to support local/patchdemo deployments [extensions/ORES] - 10https://gerrit.wikimedia.org/r/976157 (https://phabricator.wikimedia.org/T351703) [11:39:48] (03CR) 10Ilias Sarantopoulos: "Thanks for the review Kosta!" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/976157 (https://phabricator.wikimedia.org/T351703) (owner: 10Ilias Sarantopoulos) [11:49:20] * elukey lunch! [11:52:16] * isaranto lunch as well [12:10:02] isaranto: yes, I did run that patch locally. on the ml-sandbox you can see it in the container with ID: 027e48595afe [13:17:42] ack! [14:02:47] (03Restored) 10Ilias Sarantopoulos: revscoring: add missing type in function [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/976270 (owner: 10Ilias Sarantopoulos) [14:10:04] (03CR) 10Ilias Sarantopoulos: [V: 03+2 C: 03+2] "Merging this as it is a dummy commit to trigger build pipeline for revscoring image" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/976270 (owner: 10Ilias Sarantopoulos) [14:16:09] (03Merged) 10jenkins-bot: revscoring: add missing type in function [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/976270 (owner: 10Ilias Sarantopoulos) [14:37:44] updated the docker images https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/976748 [14:39:28] isaranto, elukey: sorry for the back and forth - huggingface noob here :). [14:39:28] turns out the local env I've been using to test had cached the model files in: [14:39:28] "~/.cache/huggingface/hub/models--bert-base-multilingual-uncased/snapshots" [14:39:28] will prepare and send an updated patch. [14:54:02] no prob! [15:10:47] 10Machine-Learning-Team, 10observability, 10Patch-For-Review, 10SRE Observability (FY2023/2024-Q2): Istio recording rules for Pyrra and Grizzly - https://phabricator.wikimedia.org/T351390 (10lmata) [17:13:17] * elukey afk! [17:13:20] o/ [17:13:26] \o [17:18:21] afk as well!