[07:13:31] <isaranto>	 hello!
[07:15:17] <isaranto>	 kevinbazira: o/ have a great start of the week!
[07:15:17] <isaranto>	 could you keep an eye for any issues related to this while deploying the new recapi image?
[07:17:16] <kevinbazira>	 isaranto: o/
[07:18:10] <kevinbazira>	 thanks! have  a great week too!
[07:18:44] <kevinbazira>	 I've started digging into the recent changes in rec-api so I can deploy them.
[07:19:34] <isaranto>	 ok, let me know if you need any assistance there
[07:19:50] <kevinbazira>	 I sure will
[09:41:00] <klausman>	 Morning!
[09:42:31] <isaranto>	 o/ Tobias!
[09:48:42] <klausman>	 ¡Hoal, Ilias! ¿Cómo estás?
[09:53:31] <klausman>	 That should be "Hola" of course %-)
[09:55:42] <isaranto>	 Hola!
[09:56:01] <isaranto>	 I redid a bit the model cards page moving some models to Production -> https://meta.wikimedia.org/wiki/Machine_learning_models
[09:57:02] <isaranto>	 I'd like to put the models in categories cause now the newer models are "lost" in the sea of the language specific revscoring models
[09:58:56] <klausman>	 ack. I think with the revscoring models in the table above (folded away by default), we can drop them from the prod section, maybe with a quick note that the folded-away section exists
[10:01:43] <isaranto>	 I think that would do. I was thinking to also have their full names under a foldable section but perhaps that's not needed
[10:02:54] <isaranto>	 I'll have to move all the pages though and add redirects as the Production models section just lists all models under `Machine_learning_models/Production/`
[10:03:02] <isaranto>	 will share once I have sth new!
[10:03:17] <klausman>	 ack, ty!
[10:04:05] <isaranto>	 thanks for the useful feedback!
[10:20:38] <elukey>	 hey folks!
[10:20:52] <elukey>	 I am doing some follow ups for https://phabricator.wikimedia.org/T373977
[10:21:18] <elukey>	 the py3.11 upgrade is not critical, but it would be good to eventually see all images converting to the new version
[10:21:48] <elukey>	 you can check https://debmonitor.wikimedia.org/packages/python3.11 and select "show only upgradable packages"
[10:22:11] <elukey>	 mostly pytorch etc..
[10:22:28] <elukey>	 when you have a min, could you please take a look?
[10:22:34] <klausman>	 ack.will do
[10:23:44] <klausman>	 elukey: btw, I am looking at packaging/importing rocm61 for the Labs machines. It's the latest version supported by Pytorch (I have been told all the cool kids use that now instead of Tensorflow)
[10:24:10] <elukey>	 okok
[10:24:15] <klausman>	 Of couNaturally, there will be a lot of dep checking first, making sure it even works on Bookworm
[10:24:36] <elukey>	 when you do it also keep an eye for licenses, especially if new packages are involved etc..
[10:24:40] <klausman>	 Fortunatly, I can mess up thelabs machines and just reimage them :)
[10:24:44] <elukey>	 just to avoid accidental closed source deployments
[10:24:47] <klausman>	 Yep, of course
[10:25:08] <klausman>	 The old closed-source package that we had to stub out is gone, fortunately.
[10:25:34] <klausman>	 I'll see what updates to https://wikitech.wikimedia.org/wiki/Machine_Learning/AMD_GPU#Upgrade_the_Debian_packages might be useful
[10:30:31] <isaranto>	 elukey: I don't have view access for https://phabricator.wikimedia.org/T373977
[10:30:31] <isaranto>	 We'll move all isvcs to bookworm and py3.11 and we're working on a deprecation plan for the revscoring ones which my not be upgraded to 3.11
[10:33:05] * isaranto lunch o clock
[10:37:21] <klausman>	 same!
[10:41:12] <elukey>	 isaranto: you should have access now
[10:41:41] <elukey>	 the upgrades are needed for the current isvcs using bookworm basically
[10:42:14] <elukey>	 the most notable one is the storage initializer, I think it should be set everywhere to use the newer version
[10:42:34] <isaranto>	 I have access now, thanks!
[11:43:11] <wikibugs>	 (03PS1) 10AikoChou: locust: fix formatting in README for reference_quality [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1073188
[12:26:31] <kevinbazira>	 thanks for the review Ilias.
[12:26:43] <kevinbazira>	 the rec-api is running in staging: https://phabricator.wikimedia.org/P69132
[12:26:48] <kevinbazira>	 going to deploy in prod
[12:30:16] <isaranto>	 ack!
[12:46:05] <kevinbazira>	 both the rec-api translation and section recommendation endpoints are up and running in prod:
[12:46:06] <kevinbazira>	 https://phabricator.wikimedia.org/P69133
[12:46:06] <kevinbazira>	 going to follow up on Stephane's question on slack in #talk-to-machine-learning
[13:05:23] <isaranto>	 nice! I tested the endpoints as well and they seem fine, although these connectivity failures to cxserver didn't happen all the time.
[13:05:34] <isaranto>	 thanks for following up on this Kevin
[13:33:45] <wikibugs>	 06Machine-Learning-Team, 10Temporary accounts: Allow calling revertrisk language agnostic and revert risk multilingual APIs in a pre-save context - https://phabricator.wikimedia.org/T356102#10148877 (10Strainu) >>! In T356102#10144392, @diego wrote: > We will need a different model for new article evaluati...
[14:46:45] <jinxer-wm>	 FIRING: ErrorBudgetBurn: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[14:59:25] <chrisalbon>	 Good morning all
[14:59:36] <chrisalbon>	 Welcome back kevinbazira!
[15:00:14] <isaranto>	 good morning o/
[15:03:38] <kevinbazira>	 thanks chrisalbon! good morning
[15:07:21] * isaranto afk - be back in an hour
[16:22:53] <isaranto>	 I updated the time window on the SLO dashboards https://gerrit.wikimedia.org/r/c/operations/grafana-grizzly/+/1073257
[17:46:20] * isaranto afk
[18:46:45] <jinxer-wm>	 FIRING: ErrorBudgetBurn: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn
[22:46:45] <jinxer-wm>	 FIRING: ErrorBudgetBurn: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn