[07:38:35] <isaranto>	 Good morning!
[08:28:07] <isaranto>	 I resent the invitations for this week's team meetings 
[08:41:47] <klausman>	 Morning :)
[08:42:12] <klausman>	 I'll be out for ~2h after ten, for an appointment, but back around lunchtime
[08:46:06] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: Makefile: add support for article-descriptions (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993481 (https://phabricator.wikimedia.org/T356176) (owner: 10Kevin Bazira)
[08:50:29] <isaranto>	 Morning Tobias!
[08:50:34] <aiko>	 morning o/
[08:50:42] <isaranto>	 Hey Aiko!
[08:51:09] <aiko>	 hi Ilias :)
[08:51:32] <isaranto>	 I wanted to sync about multilingual and kserve. I'm going afk for an errand but I'll be back in like 30'
[08:57:24] <wikibugs>	 (03CR) 10Kevin Bazira: Makefile: add support for article-descriptions (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993481 (https://phabricator.wikimedia.org/T356176) (owner: 10Kevin Bazira)
[09:32:09] <aiko>	 isaranto: ok o/
[10:08:53] <isaranto>	 o/
[10:09:14] <isaranto>	 aiko: do you want to have a call now instead of later?
[10:17:11] <aiko>	 isaranto: yeah we can do that :)
[10:20:09] <aiko>	 isaranto: in 10min?
[10:21:03] <isaranto>	 aiko: yep!
[11:50:31] <klausman>	 heads up: I am going to drain the staging workers in turn to update everything to use the newest version of runc (as discussed with Moritz yesterday). 
[11:52:04] <isaranto>	 ack!
[11:55:10] * isaranto afk lunch!
[12:05:08] <klausman>	 Ok, all done, and pods are running again
[12:05:52] <klausman>	 moritzm: all pods in staging have been restarted. I recommed we let this soak 'til tomorrow and do the serving cluster in eqiad then
[12:06:55] <moritzm>	 sounds good
[12:07:28] <klausman>	 while we're at it, are ther kernelupdates/rebooting-would-help things pending? Might as well doa reboot too for that if we're draining hosts
[12:13:54] <wikibugs>	 (03CR) 10AikoChou: "I agree to proceed with kserve 0.11.2" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/995198 (https://phabricator.wikimedia.org/T356501) (owner: 10AikoChou)
[12:28:03] <wikibugs>	 (03PS5) 10AikoChou: Makefile: add support for revertrisk-multilingual [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/995198 (https://phabricator.wikimedia.org/T356501)
[12:55:26] <wikibugs>	 (03CR) 10Kevin Bazira: Makefile: add support for revertrisk-multilingual (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/995198 (https://phabricator.wikimedia.org/T356501) (owner: 10AikoChou)
[15:33:05] <wikibugs>	 10Machine-Learning-Team, 10Patch-For-Review: Support running revertrisk-multilingual model-server via Makefile - https://phabricator.wikimedia.org/T356501 (10isarantopoulos)
[15:58:55] <wikibugs>	 10Machine-Learning-Team, 10SRE, 10Patch-For-Review: Requesting write access to ml-staging-codfw for ML team - https://phabricator.wikimedia.org/T354516 (10isarantopoulos) I tried to delete a revision and an inferenceservice on experimental namespace and it seems that I don't have access:  ` kubectl delete re...
[16:02:09] <isaranto>	 klausman: --^ I update the task. I also tried to patch an inference service with the same results
[16:02:34] <klausman>	 ty!
[16:06:16] <isaranto>	 for now I removed the gpu manually with a patch , cause I want to run a load test with the cpu only https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/997901
[16:07:39] <klausman>	 Yeah, I saw it (re) scheudle when doing the staging drains and got excited for a moment :D
[16:10:07] <isaranto>	 kevinbazira: manually downloading the model worked \o/
[16:11:29] <wikibugs>	 (03PS6) 10Kevin Bazira: Makefile: add support for article-descriptions [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993481 (https://phabricator.wikimedia.org/T356176)
[16:11:39] <isaranto>	 so if you can add the `--continue` option to wget and update the commit msg we'll be ready to go
[16:12:18] <isaranto>	 nice :D already done
[16:12:56] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [C: 03+1] Makefile: add support for article-descriptions [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993481 (https://phabricator.wikimedia.org/T356176) (owner: 10Kevin Bazira)
[16:13:27] <kevinbazira>	 isaranto: good to know it has finally worked :D
[16:13:37] <isaranto>	 nice work!
[16:14:20] <kevinbazira>	 thanks to all of you for the invaluable input in the meeting :)
[16:14:29] <isaranto>	 <3
[16:26:34] <wikibugs>	 (03CR) 10AikoChou: [C: 03+1] Makefile: add support for article-descriptions [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993481 (https://phabricator.wikimedia.org/T356176) (owner: 10Kevin Bazira)
[16:31:56] <wikibugs>	 (03CR) 10Kevin Bazira: [V: 03+2 C: 03+2] "Thanks for the reviews :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/993481 (https://phabricator.wikimedia.org/T356176) (owner: 10Kevin Bazira)
[17:43:05] <klausman>	 isaranto: one thing we could try tomorrow is wothout the v1 suffix. Maybe Alex was wrong about those.
[17:43:58] <isaranto>	 that's the only thing I could come up with, especially since all the other roles doent have version suffixes specified
[17:44:37] <klausman>	 My gut tells me that maybe versions are either not part of the schema at that point at all, or have a different syntax
[17:44:54] <klausman>	 anyway, heading out for now. Seeya tomorrow!
[17:45:24] <klausman>	 Oh and re: Analytics downloads, I'm coordinating with Ben over in #wikimedia-analytics
[17:51:04] <isaranto>	 ciao Tobias, I'm heading out as well o/