[05:45:38] (03PS4) 10Kevin Bazira: article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) [05:51:27] (03PS5) 10Kevin Bazira: article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) [08:24:53] o/ [08:25:24] (03CR) 10Elukey: article-descriptions: update model-server to use local files only (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [08:28:45] morning! [08:35:37] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Newsletter, and 3 others: Jenkins jobs for npm-test fail on project with deps on node-gyp which requires python2.7 - https://phabricator.wikimedia.org/T213944 (10hashar) [08:43:56] (03CR) 10Kosta Harlan: Change default config values to support local/patchdemo deployments (034 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/976157 (https://phabricator.wikimedia.org/T351703) (owner: 10Ilias Sarantopoulos) [08:44:06] (03CR) 10Kosta Harlan: Change default config values to support local/patchdemo deployments (032 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/976157 (https://phabricator.wikimedia.org/T351703) (owner: 10Ilias Sarantopoulos) [08:44:31] (03CR) 10Kosta Harlan: [C: 03+2] Change default config values to support local/patchdemo deployments [extensions/ORES] - 10https://gerrit.wikimedia.org/r/976157 (https://phabricator.wikimedia.org/T351703) (owner: 10Ilias Sarantopoulos) [08:53:53] if someone has the time, https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/976748 [08:54:04] I initially had added aiko but anyone can review [08:55:05] checking [08:56:14] thanks! [08:59:34] (03CR) 10Kosta Harlan: add revertrisk model to the list of models (032 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/971547 (https://phabricator.wikimedia.org/T348298) (owner: 10Ilias Sarantopoulos) [09:00:11] (03CR) 10Kosta Harlan: add revertrisk model to the list of models (034 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/971547 (https://phabricator.wikimedia.org/T348298) (owner: 10Ilias Sarantopoulos) [09:02:31] (03PS14) 10Kosta Harlan: add revertrisk model to the list of models [extensions/ORES] - 10https://gerrit.wikimedia.org/r/971547 (https://phabricator.wikimedia.org/T348298) (owner: 10Ilias Sarantopoulos) [09:18:55] kevinbazira: I am reviewing you patch trying to run it locally to verify it and will comment in a bit [09:19:07] just fyi to let you know I'm on it :) [09:21:54] thanks isaranto :) [09:48:34] oof, it is that time of the week where I just close the 1 million browser tabs that I have open [09:50:28] (03CR) 10Kosta Harlan: add revertrisk model to the list of models (0313 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/971547 (https://phabricator.wikimedia.org/T348298) (owner: 10Ilias Sarantopoulos) [10:03:03] * elukey bbiab [10:40:55] (03CR) 10Ilias Sarantopoulos: article-descriptions: update model-server to use local files only (033 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [10:41:54] kevinbazira: I managed to run the model server locally. I left some comments regarding the paths and I will submit my patch aftewards [10:42:44] okok let me fix the reviews [11:15:53] started deploying servers on ml-staging! [11:15:58] * isaranto afk lunch [12:04:58] (03PS6) 10Kevin Bazira: article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) [12:05:40] (03CR) 10CI reject: [V: 04-1] article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [12:22:00] (03PS7) 10Kevin Bazira: article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) [12:24:11] (03CR) 10Kevin Bazira: article-descriptions: update model-server to use local files only (033 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [12:50:17] kevinbazira: please don't resolve the comments - especially when changes are not trivial. it makes it difficult to track changes [12:52:47] isaranto: I have not resolved the comments. :) [12:53:50] kevinbazira: oh well, gerrit noob here :) sorry [12:54:06] it's all good :) [12:57:09] (03CR) 10Ilias Sarantopoulos: article-descriptions: update model-server to use local files only (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [12:58:26] I guess code doesnt appear in place when it is removed from an updated patchset and makes it hard to follow up compared to other code review software [12:59:19] I left another comment, we can also sync here if you want and I can comment on the patch afterwards [13:18:51] thanks. let me fix the comment [13:26:39] (03PS8) 10Kevin Bazira: article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) [13:28:13] (03CR) 10Kevin Bazira: article-descriptions: update model-server to use local files only (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [13:59:57] (03CR) 10Ilias Sarantopoulos: article-descriptions: update model-server to use local files only (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [14:00:08] kevinbazira: one last comment! nice work [14:15:46] (03PS9) 10Kevin Bazira: article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) [14:16:28] (03CR) 10CI reject: [V: 04-1] article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [14:25:23] (03PS10) 10Kevin Bazira: article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) [14:27:21] (03CR) 10Kevin Bazira: article-descriptions: update model-server to use local files only (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [14:38:41] (03CR) 10Ilias Sarantopoulos: "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [14:38:52] (03CR) 10Ilias Sarantopoulos: [C: 03+1] article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [14:52:01] (03CR) 10Kevin Bazira: [C: 03+2] "Thanks for the reviews :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [14:52:48] (03Merged) 10jenkins-bot: article-descriptions: update model-server to use local files only [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/975936 (https://phabricator.wikimedia.org/T343123) (owner: 10Kevin Bazira) [15:14:34] side comment - it is not really nice when there is a team meeting, people are explaining things and code reviews are happening :) [15:36:23] 10Machine-Learning-Team, 10Research, 10Epic: Develop a ML-based service to predict reverts on Wikipedia(s) - https://phabricator.wikimedia.org/T314384 (10diego) [16:01:08] (03Abandoned) 10Elukey: Update KServe model servers from 0.11.1 to 0.11.2 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/974944 (owner: 10Elukey) [16:17:37] (03CR) 10Ilias Sarantopoulos: "I totally missed this one, sorry!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/974944 (owner: 10Elukey) [16:19:17] elukey: apologies for the review, you're totally right [16:28:01] isaranto: I'd need to test a theory with SLO metrics, namely if after a revscoring deploy the Pyrra/prometheus graphs go awol showing weird bumps (due to how aggregation was done). Are you planning to deploy one of the revscoring model servers (IIRC for kserve 0.11.2) any time soon? [16:28:14] (I can also do it in case, I don't recall if we were ready or not) [16:30:08] elukey: I deployed all staging ones today. I was planning to run 1-2 load tests before deploying to production [16:30:26] yes yes no rush, I wanted to know if you were planning to [16:30:44] when you do it ping me (or I can help if needed!) so I can check if Pyrra dashboards explode [16:32:53] Let's do it tomorrow morning then . I'm working on changes for the extension but I'll be done [16:41:45] yes yes not in a hurry [16:44:07] 10Machine-Learning-Team, 10observability, 10Patch-For-Review, 10SRE Observability (FY2023/2024-Q2): Istio recording rules for Pyrra and Grizzly - https://phabricator.wikimedia.org/T351390 (10elukey) Very interesting link: https://aws.amazon.com/blogs/mt/how-to-reduce-istio-sidecar-metric-cardinality-with-a... [17:00:43] going afk for today folks! [17:00:49] have a nice rest of the day :) [17:03:34] Going afk as well! [17:04:21] \o