[07:08:57] 10Machine-Learning-Team, 10Patch-For-Review: Host open source LLM (bloom, etc.) on Lift Wing - https://phabricator.wikimedia.org/T333861 (10santhosh) @isarantopoulos Did you consider optimizing bloom or nllb for inference? Even if we have GPUs, inference optimization can save lot of compute resources. In htt... [07:20:24] 10Machine-Learning-Team, 10Patch-For-Review: Host open source LLM (bloom, etc.) on Lift Wing - https://phabricator.wikimedia.org/T333861 (10elukey) @santhosh Hi! One side note about how we run NLLB - this is the [[ https://gerrit.wikimedia.org/r/plugins/gitiles/machinelearning/liftwing/inference-services/+/ref... [07:43:29] 10Machine-Learning-Team, 10Patch-For-Review: Host open source LLM (bloom, etc.) on Lift Wing - https://phabricator.wikimedia.org/T333861 (10isarantopoulos) @santosh Hi! at the moment we are basically doing a POCs with GPUs and LLMs on Lift Wing in order to procure GPUs in the upcoming months. Next quarter we p... [09:08:00] 10Machine-Learning-Team, 10Patch-For-Review: Host open source LLM (bloom, etc.) on Lift Wing - https://phabricator.wikimedia.org/T333861 (10santhosh) > One thing that puzzled me in these months is that MinT could have easily been added to Lift Wing, but for a lot of reasons we decided not to We had to do this... [09:18:35] 10Machine-Learning-Team, 10Patch-For-Review: Host open source LLM (bloom, etc.) on Lift Wing - https://phabricator.wikimedia.org/T333861 (10elukey) >>! In T333861#8963349, @santhosh wrote: > But looking back I think that was the right call. These days we update the codebase very frequently and we have full con... [09:26:25] klausman: o/ [09:26:33] \o [09:27:05] I've read https://wikitech.wikimedia.org/w/index.php?title=Machine_Learning%2FLiftWing%2FUsage&diff=2086791&oldid=2082465, it is good but I think that people without any knowledge about tokens could still get lost :( [09:27:22] for example, we say that "100000 requests/hour for every authenticated OAuth user." [09:27:34] and in theory that needs an elevation as well right? [09:27:43] Yes [09:27:51] I should clairfy that a bit more. [09:27:57] Will do so today [09:28:17] super, and also do you think that there is a way to have a quick python script to extract the audience bit? [09:28:28] yes, I already have it :) [09:28:32] or similar, it would be nice that people wouldn't need to ping us, unless really needed [09:28:35] ack :) [09:29:37] (03PS1) 10AikoChou: outlink: add logging of source event for get_outlinks function [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932826 (https://phabricator.wikimedia.org/T328899) [09:30:36] elukey: https://phabricator.wikimedia.org/P49478 [09:32:01] (03PS2) 10AikoChou: readability: add readability model server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) [09:32:06] If that looks a lot like Go-but-in-Python, well that's how my mind works these days [09:32:09] klausman: perfect, let's add it to the wikipage [09:32:49] Should we inline the script or link to the paste? [09:33:28] let's add it to the page so people can copy-paste [09:33:37] we should also explain why they need to do it [09:33:52] (briefly, maybe with a link to the api gateway pages) [09:34:03] Ack. [09:34:07] my aim is to let this be as self-service as possible [09:35:22] Yeah. Though the question of token revocation still remains open [09:35:40] (and if it's "too-self-service", token revocation might be moot) [09:36:01] But I don't know who really has access to mwmaint [09:39:12] (03CR) 10CI reject: [V: 04-1] readability: add readability model server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [09:40:10] klausman: token revocation is on our side though right? [09:40:46] mwmaint is accessible by deployers, so is should be broadly available, but good point.. Let's add a note about it, in case they can ping us [09:40:52] (like WME probably) [09:40:52] It's unclear to me, I must poke the api platform people about it [09:41:17] klausman: as far as I got, there is no way of doing it now [09:41:51] :-/ [09:41:52] so we should probably build some automation around it, like simulation how to remove a test token from live traffic etc.. [09:42:12] for example, a way could be to manually set maxint to the token in redis, the one that api-gateway uses [09:42:26] it is brutal but maybe with a python script could work, as initial fence [09:42:48] yeah, sounds good. [09:45:34] (03PS1) 10Santhosh: Add language identification service [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932828 (https://phabricator.wikimedia.org/T99666) [10:08:21] elukey: updated the page, Lmk what you think. [10:16:38] klausman: "Anonymous users via the API gateway are limited to 500 requests per hour" - isn't it 10k? [10:17:00] (03CR) 10AikoChou: "I don't know why the llm pipeline failed in CI test. Also I filed a patch for integration/config in https://gerrit.wikimedia.org/r/c/integ" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [10:17:06] https://wikitech.wikimedia.org/wiki/API_Gateway#Rate_limiting says 500 [10:17:35] klausman: sure but we configure 10k [10:17:47] and in theory the limit should be ok since there is not jwt check [10:17:56] Good point, fixing [10:18:34] also adding wording re: per-IP [10:20:39] (03PS2) 10Santhosh: Add language identification service [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932828 (https://phabricator.wikimedia.org/T99666) [10:25:28] (03CR) 10AikoChou: "To investigate the errors we saw in https://phabricator.wikimedia.org/P49459, we should actually add logging in get_outlinks() instead of " [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932826 (https://phabricator.wikimedia.org/T328899) (owner: 10AikoChou) [10:26:19] klausman: reworded it a little, added more references etc.. [10:27:12] :+1: [10:27:34] (03CR) 10CI reject: [V: 04-1] Add language identification service [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932828 (https://phabricator.wikimedia.org/T99666) (owner: 10Santhosh) [10:37:07] (03CR) 10Klausman: [C: 03+1] outlink: add logging of source event for get_outlinks function [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932826 (https://phabricator.wikimedia.org/T328899) (owner: 10AikoChou) [10:41:17] (03CR) 10Klausman: readability: add readability model server (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [10:41:49] aiko_: CI ran out of disk space when installing packages on this one ^^^ [10:42:34] elukey: The cassandra probles on the ml-cache machines are failing (https://alerts.wikimedia.org/?q=%40state%3Dactive&q=instance%3D~%28%5Eml%7C%5Eores%29) [10:42:54] I presume you're elbows-deep in that machinery atm :) [10:43:04] (03CR) 10Elukey: outlink: add logging of source event for get_outlinks function (033 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932826 (https://phabricator.wikimedia.org/T328899) (owner: 10AikoChou) [10:43:53] klausman: yep exactly! The usual "it takes 5 min" :D [10:43:58] https://gerrit.wikimedia.org/r/c/operations/puppet/+/932801 and next should solve [10:44:07] the prometheus blackbox exporter is different than I thought [10:44:30] It's a strange beast. I have had some experience with it. If you ever want help with it, lmk [10:48:58] the main issue is that it doesn't support different SNIs as it is built now [10:49:04] so we need the CN etc.. [10:53:32] Yeah, it's a bug people run into regularly. I have a task to take a look what would be needed in the BBE to be able to handle it on my todo pile, but you know that is :) [10:54:50] I presume our version already supports __param_hostname? [10:55:41] you can check in puppet, prometheus blackbox exporter tcp [10:57:02] ack [10:58:33] I'll have a look. It could be that what we need is actually possible with some Puppet changes (IF our version of BBE is new enough) [11:02:23] I am rolling out the new settings to see if the work [11:02:34] sadly cassandra didn't pick up the new keystore, as it was supposed to do [11:03:20] or not, it takes 10 mins, will see [11:07:45] Yep, our BBE is too old to support SNI via __params_hostname [11:08:07] you can open a task to observability if it is worth pursuing [11:08:21] Surely that will change once the Prometheus hosts are switched to Bookworm :D [11:08:25] * elukey lunch! [11:12:25] (03PS2) 10AikoChou: outlink: add logging of source event for get_outlinks function [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932826 (https://phabricator.wikimedia.org/T328899) [11:12:56] * klausman lunch as well. [11:15:36] (03CR) 10Elukey: [C: 03+1] outlink: add logging of source event for get_outlinks function [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932826 (https://phabricator.wikimedia.org/T328899) (owner: 10AikoChou) [11:20:26] (03CR) 10AikoChou: [C: 03+2] "Thanks for the review :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932826 (https://phabricator.wikimedia.org/T328899) (owner: 10AikoChou) [11:25:11] (03Merged) 10jenkins-bot: outlink: add logging of source event for get_outlinks function [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932826 (https://phabricator.wikimedia.org/T328899) (owner: 10AikoChou) [11:44:56] 10Machine-Learning-Team, 10Research, 10Section-Level-Image-Suggestions, 10Section-Topics, 10Structured-Data-Backlog: Let the model that learns section alignments consume section topics output - https://phabricator.wikimedia.org/T331968 (10AUgolnikova-WMF) [11:50:20] 10Machine-Learning-Team, 10Research-Backlog, 10Section-Level-Image-Suggestions, 10Structured-Data-Backlog: [XL] Productionize section alignment model training - https://phabricator.wikimedia.org/T325316 (10AUgolnikova-WMF) [11:51:14] 10Machine-Learning-Team, 10Research, 10Section-Level-Image-Suggestions, 10Section-Topics, 10Structured-Data-Backlog: Let the model that learns section alignments consume section topics output - https://phabricator.wikimedia.org/T331968 (10AUgolnikova-WMF) [12:25:37] (03CR) 10Ilias Sarantopoulos: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932828 (https://phabricator.wikimedia.org/T99666) (owner: 10Santhosh) [12:33:33] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "Just a note regarding logging and error handling of input params. Other than that it works like a charm and it is also blazing fast" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932828 (https://phabricator.wikimedia.org/T99666) (owner: 10Santhosh) [12:35:04] 10Machine-Learning-Team, 10WMF-General-or-Unknown, 10I18n, 10NewFunctionality-Worktype, 10Patch-For-Review: Provide a service to detect which language the user is writing on - https://phabricator.wikimedia.org/T99666 (10isarantopoulos) [12:51:55] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932828 (https://phabricator.wikimedia.org/T99666) (owner: 10Santhosh) [13:01:59] (03CR) 10Ilias Sarantopoulos: readability: add readability model server (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [13:08:16] (03PS3) 10Ilias Sarantopoulos: Add language identification service [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932828 (https://phabricator.wikimedia.org/T99666) (owner: 10Santhosh) [13:13:50] (03CR) 10Ilias Sarantopoulos: "Great work!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [13:14:04] (03PS3) 10AikoChou: readability: add readability model server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) [13:14:36] (03CR) 10CI reject: [V: 04-1] Add language identification service [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932828 (https://phabricator.wikimedia.org/T99666) (owner: 10Santhosh) [13:18:55] (03CR) 10Elukey: [C: 04-1] "Folks I love the idea of a new service but as I wrote in the task, let's follow https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftW" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932828 (https://phabricator.wikimedia.org/T99666) (owner: 10Santhosh) [13:23:25] (03CR) 10CI reject: [V: 04-1] readability: add readability model server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [13:49:42] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MW-1.41-notes (1.41.0-wmf.15; 2023-06-27), 10Patch-For-Review: Move backend of ORES MediaWiki extension to Lift Wing - https://phabricator.wikimedia.org/T319170 (10PatchDemoBot) Test wiki **created** on [[ https://patchdemo.wmflabs.org | Patch demo ]]... [13:56:28] (03CR) 10Ilias Sarantopoulos: "The ORES filters seem to be disabled in beta cluster after our recent deployment and at the moment I can't seem to figure out why." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/932289 (https://phabricator.wikimedia.org/T319170) (owner: 10Ilias Sarantopoulos) [13:57:09] Amir1: [13:58:04] I need some help with debugging the above as I've hit a wall at the moment. Let me know whenever you have some time, here or DM . thaanks [14:03:36] Hi. Sure I'm sick today. Ping me tomorrow if I forget to ping you [14:04:13] hope u feel better soon <3 . I'll ping you if I don't manage to figure it out [14:04:29] Thanks! [14:05:02] (03PS4) 10AikoChou: readability: add readability model server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) [14:11:38] (03CR) 10CI reject: [V: 04-1] readability: add readability model server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [14:12:47] (03CR) 10AikoChou: "Thanks for the review!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [14:36:13] (03PS2) 10Ilias Sarantopoulos: llm: test bitsandbytes-rocm package [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932227 (https://phabricator.wikimedia.org/T334583) [14:49:12] elukey: seems like the bitsandbytes-rocm patch passed CI [14:50:14] nice! [14:50:41] I have no idea how reliable the repo is [14:50:46] but we can surely try to test [14:51:14] (03CR) 10Elukey: [C: 03+1] "No idea how reliable https://github.com/Titaniumtown/bitsandbytes-rocm is, but let's try!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932227 (https://phabricator.wikimedia.org/T334583) (owner: 10Ilias Sarantopoulos) [14:53:20] me neither! def this is not be used in production. A simple transformers upgrade could break it. however we can learn a bit more about it and maybe even end up helping maintain it [14:53:33] (03CR) 10Ilias Sarantopoulos: [C: 03+2] llm: test bitsandbytes-rocm package [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932227 (https://phabricator.wikimedia.org/T334583) (owner: 10Ilias Sarantopoulos) [14:54:35] (03CR) 10Ilias Sarantopoulos: readability: add readability model server (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [14:55:07] (03Merged) 10jenkins-bot: llm: test bitsandbytes-rocm package [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/932227 (https://phabricator.wikimedia.org/T334583) (owner: 10Ilias Sarantopoulos) [14:58:29] 10Machine-Learning-Team, 10API Platform, 10Anti-Harassment, 10Cloud-Services, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10dancy) [15:19:44] (03PS5) 10AikoChou: readability: add readability model server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) [15:26:48] (03CR) 10CI reject: [V: 04-1] readability: add readability model server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [15:47:20] (03CR) 10AikoChou: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/931987 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [15:56:59] bitsandbytes-rocm failed again as it couldn't initialize cuda (cuda is translated to rocm in all these packages ) Dont know if it is because of our old gpu or sth else. I'll update the task with the msg etc [15:57:09] logging off folks, more tomorrow! [15:57:12] <3 [15:58:00] 10Machine-Learning-Team, 10Foundational Technology Requests: Content Translation Recommendations API - https://phabricator.wikimedia.org/T293648 (10elukey) @Pginer-WMF @santhosh - one extra question - do we have any plan/agreement about who will maintain the "new" Python API? Will it be a collaboration between... [16:16:59] klausman: added some info to https://api.wikimedia.org/wiki/Lift_Wing_API/Rate_limits as well [16:17:11] ty! [16:31:45] 10Machine-Learning-Team, 10Research-Backlog, 10Section-Level-Image-Suggestions, 10Structured-Data-Backlog (Current Work): [XL] Productionize section alignment model training - https://phabricator.wikimedia.org/T325316 (10AUgolnikova-WMF) [17:11:08] going afk for today folks, have a good one