[06:37:32] <wikibugs>	 (03CR) 10Elukey: revert-risk: handle unsupported edit types for wikidata model (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/924912 (https://phabricator.wikimedia.org/T333125) (owner: 10AikoChou)
[07:10:40] <elukey>	 hello folks
[08:03:55] <moritzm>	 FYI, installing containerd security updates on ml-serve* (already running on 2001)
[08:07:55] <klausman>	 moritzm: Will the install trigger a restart? Not asking because of disruption, but rather if any action is needed from us afterward.
[08:07:59] <klausman>	 Also: g'mornin'
[08:09:08] <moritzm>	 not needed, this doesn't affect running containers, this only affects new ones being started
[08:09:18] <klausman>	 ack, thx!
[08:23:55] <elukey>	 need to run some errands, back i na bit!
[08:25:07] <elukey>	 klausman: o/ can you open today the task for the api-gateway rate limits?
[08:25:16] <elukey>	 so we can start thinking about options etc.
[08:25:35] <klausman>	 Yep!
[08:25:38] <elukey>	 super
[08:36:55] <wikibugs>	 10Machine-Learning-Team: Investigate ad-hoc traffic class for API GW rate limits applied to Inference services as used by WME - https://phabricator.wikimedia.org/T338121 (10klausman)
[08:38:34] <wikibugs>	 10Machine-Learning-Team: Investigate ad-hoc traffic class for API GW rate limits applied to Inference services as used by WME - https://phabricator.wikimedia.org/T338121 (10klausman)
[08:43:15] <wikibugs>	 (03CR) 10Kevin Bazira: feat: reduce llm memory footprint (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/926507 (https://phabricator.wikimedia.org/T333861) (owner: 10Ilias Sarantopoulos)
[10:07:11] <klausman>	 <- lunch and groceries
[12:40:16] <wikibugs>	 10Machine-Learning-Team, 10API Platform: Investigate ad-hoc traffic class for API GW rate limits applied to Inference services as used by WME - https://phabricator.wikimedia.org/T338121 (10elukey)
[12:45:23] <wikibugs>	 10Machine-Learning-Team, 10API Platform: Investigate ad-hoc traffic class for API GW rate limits applied to Inference services as used by WME - https://phabricator.wikimedia.org/T338121 (10elukey) I think that we should try to figure out, form the API Platform perspective, what is be best road to follow, since...
[13:07:31] <elukey>	 klausman: what next steps do you have in mind for --^ ?
[13:07:50] <klausman>	 I am poking Hugh to find out what's involved in the ad-hoc class approach
[13:07:52] <elukey>	 I think that it is critical to find a solution, otherwise people can't migrate
[13:08:49] <elukey>	 let's try to find on phabricator if there are previous tasks in which people asked for different tiers/classes
[13:10:13] <elukey>	 also let's ask in the api-platform slack channel, not only Hugh
[13:10:25] <klausman>	 He's on PTO 'til the 12th
[13:10:29] <elukey>	 Bill Pirkle may be able to give us some guidance
[13:10:42] <klausman>	 (Hugh, that is)
[13:10:51] <elukey>	 sure, do you have time to follow up on slack?
[13:10:56] <klausman>	 Yep
[13:17:15] <elukey>	 so the tiers are defined in wmf-config/CommonSettings.php
[13:17:22] <elukey>	 maybe we could just add a class in there
[13:18:22] <elukey>	 as defined in https://phabricator.wikimedia.org/T246271
[13:19:26] <wikibugs>	 10Machine-Learning-Team, 10API Platform: Investigate ad-hoc traffic class for API GW rate limits applied to Inference services as used by WME - https://phabricator.wikimedia.org/T338121 (10elukey) I see that rate limits tiers were defined in T246271, and they live in `wmf-config/CommonSettings.php`. Should we...
[13:28:35] <klausman>	 Question is how those tiers are tied to services
[13:29:48] <elukey>	 judging from https://phabricator.wikimedia.org/T246271 it seems that they were created for the MVP 3 years ago
[13:30:44] <klausman>	 THm. It seems those classes are applied unconditionally per-service, so Preferred==25k is independent of the service being called.
[13:31:07] <klausman>	 The class of a non-anon client is determined by their token (and can be changed on mwmaint).
[13:31:40] <klausman>	 SO I am not quite sure how to make a new class that is "Like preferred, but the limit is X"
[13:31:48] <klausman>	 But _only_ on LW services
[13:36:01] <elukey>	 probably we could have another class for 200k req/hour, calling it in another way ("internal-high"?) and use it
[13:37:51] <klausman>	 Do we atm use the existing classes at all?
[13:37:55] <klausman>	 I mean, by name
[13:38:21] <elukey>	 I didn't get the question
[13:39:13] <klausman>	 So in helmfile.d/services/api-gateway/values.yaml we refer to the limit for anon users as 10k.
[13:39:20] <klausman>	 by using `anon_limit`
[13:39:44] <klausman>	 Hrm. Let me turn this over in my head.
[13:39:55] <elukey>	 this is the confusing bit, those are API-Gateway's limits
[13:39:59] <elukey>	 not MW ones
[13:40:08] <elukey>	 separate buckets
[13:40:11] <klausman>	 So your proposal is to have WME generate a new token, tell us it's audience ID, and then move it to this new 200k tier?
[13:40:34] <klausman>	 (audience ID is what you use on mwmaint)
[13:41:19] <elukey>	 yes exactly
[13:42:24] <klausman>	 Yeah, that should work. Who is "custodian" of `wmf-config/CommonSettings.php`?
[13:44:36] <elukey>	 a lot of folks, I think that api-platform should probably give us the green light first
[13:44:54] <klausman>	 ack. I'll explain this plan to Bill, see what he has to say
[13:46:29] <elukey>	 let's do it on slack so it is visible
[13:46:45] <klausman>	 I have been doing so in a PM, oops
[13:47:16] <klausman>	 I'll c7P
[13:47:20] <klausman>	 er c&p
[13:49:16] <elukey>	 super
[13:55:20] <elukey>	 dcops is moving two gpus to ml-serve1001
[13:55:28] <elukey>	 so we'll be able to test LLMs with AMD gpus soon
[13:55:29] <klausman>	 What time window?
[13:55:36] <elukey>	 now
[13:55:57] <klausman>	 Neato. Do we have to add silences/downtimes?
[13:56:10] <elukey>	 already drained, downtimed and turned off
[13:56:27] <klausman>	 Excellent
[14:06:25] <elukey>	 klausman: ok to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/927197 ?
[14:07:17] <klausman>	 Wait, so DSE would have 0 GPUs?
[14:07:45] <klausman>	 aj, you moved it to per-host files
[14:36:50] <elukey>	 aiko: nice work on the changeprop change! Hugh is out of the office, so I added Kamila (the other SRE in api-platform).. After their review we can merge and deploy :)
[14:37:17] <klausman>	 elukey: 200k, right? Or do we want to add more margin?
[14:37:39] <elukey>	 klausman: I'd say 250k, and then we set 200 at the api gateway level (so we have some margin)
[14:37:47] <klausman>	 ack, will do
[14:38:00] <elukey>	 back in a bit
[14:53:10] <wikibugs>	 (03PS3) 10AikoChou: revert-risk: handle unsupported edit types for wikidata model [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/924912 (https://phabricator.wikimedia.org/T333125)
[15:15:22] <wikibugs>	 (03CR) 10AikoChou: revert-risk: handle unsupported edit types for wikidata model (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/924912 (https://phabricator.wikimedia.org/T333125) (owner: 10AikoChou)
[15:16:40] <aiko>	 elukey: yay! thanks for the help :D
[15:44:56] <wikibugs>	 10Machine-Learning-Team, 10Epic: Experiment with GPUs in the Machine Learning infrastructure - https://phabricator.wikimedia.org/T333462 (10elukey)
[15:46:28] <elukey>	 ok gpus on ml-serve1001 are working!
[15:48:35] <elukey>	 but of course we have LLMs only on staging..
[15:49:29] <elukey>	 should have thought about it
[15:49:34] <elukey>	 uffff
[15:49:52] <elukey>	 (but ml-staging is in codfw so no opportunity to move gpus in there, right)
[15:52:07] <wikibugs>	 10Machine-Learning-Team, 10Spike: [Spike] Run models and frameworks on AMD GPU and identify challenges - https://phabricator.wikimedia.org/T334583 (10elukey) @isarantopoulos we have two GPUs on ml-serve1001, ready to be tested :)  We can probably think about rolling out again the experimental namespace in prod...
[15:52:32] <elukey>	 at this point we may need to have exprimental in production (again), so that we'll be able to experiment in there too
[15:52:35] <elukey>	 thoughts?
[15:56:27] <klausman>	 Fine by me
[15:56:49] <klausman>	 as long as we only use it for things that can't be tested in staging-exp
[15:58:47] <elukey>	 yep yep
[15:58:49] <elukey>	 filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/927235
[15:59:41] <elukey>	 in theory the diff should look ok
[15:59:59] <klausman>	 LGTM
[16:00:12] <klausman>	 Do we need private repo stuff for this as well?
[16:03:57] <elukey>	 yep, already there, we didn't remove it from before
[16:05:28] <klausman>	 ah, excellent :)
[16:08:11] <elukey>	 deploying bloom models to ml-serve-eqiad
[16:13:41] <klausman>	 ack
[16:23:42] <elukey>	 both bloom models are now running on ml-serve-eqiad
[16:24:05] <elukey>	 tomorrow I'll talk to Ilias to run them on a GPU, and see performance differences (hopefully)
[16:24:49] <klausman>	 sgtm
[16:25:05] <elukey>	 going afk, have a nice rest of the day folks
[16:26:19] <wikibugs>	 10Machine-Learning-Team, 10Spike: [Spike] Run models and frameworks on AMD GPU and identify challenges - https://phabricator.wikimedia.org/T334583 (10elukey) After a chat with Tobias we thought to restore the experimental namespace in prod, but limited to bloom/llm models for now (we can do it with the new tem...
[16:34:50] <klausman>	 \o same here