[04:07:10] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [C:03+2] revscoring: add flag to log JSON inputs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1020898 (https://phabricator.wikimedia.org/T362503) (owner: 10Elukey)
[04:13:22] <wikibugs>	 (03Merged) 10jenkins-bot: revscoring: add flag to log JSON inputs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1020898 (https://phabricator.wikimedia.org/T362503) (owner: 10Elukey)
[04:22:07] <isaranto>	 Good morning!
[04:26:16] <isaranto>	 things seem to be going much better over the last 9h - exactly after the replica increase
[05:21:59] <isaranto>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1021056
[05:55:48] <wikibugs>	 (03CR) 10Kevin Bazira: [C:03+1] "I've tested batch_model locally with both batch and non-batch requests and it works as advertised: https://phabricator.wikimedia.org/P6083" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1020835 (https://phabricator.wikimedia.org/T358744) (owner: 10AikoChou)
[07:28:47] <isaranto>	 I'm going to deploy the new change to revscoring-damaging. This means that all the model servers in the namespace will be redeployed
[08:05:46] <elukey>	 hello folks, passing by to check the latency, seems fine!
[08:05:59] <elukey>	 at this point it seems not revscoring ending up into a weird state
[08:06:13] <elukey>	 but rev-ids causing high preprocess cpu time, stalling everthing etc..
[08:06:19] <elukey>	 or maybe both
[08:06:21] <elukey>	 sigh
[08:12:43] <isaranto>	 hey Luca, I'm currently deploying the new changes so that we can see the revids at least
[08:14:26] <isaranto>	 oups, models are crashing on staging. I'm investigating and will report back. Note related with latest changes but there has been no new deployment since november, so looking at what has changed
[08:15:09] <isaranto>	 error from logs -> https://phabricator.wikimedia.org/P60868
[08:33:10] <klausman>	 Morning!
[08:33:47] <klausman>	 The circular import problem is odd in that I'd expect it to happen in off-LW testing as well. But then again, my familiarity with how the revscoring models work internally is limited.
[08:35:39] <isaranto>	 morning Tobias!
[08:36:06] <isaranto>	 I found the root cause! https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/981718
[08:36:26] <isaranto>	 that's what you get when you introduce a lot of changes
[08:37:05] <klausman>	 ah, I see. Is it just import reordering, or a new import?
[08:39:08] <isaranto>	 import ordering. But I'm trying to fix it so that order doesnt affect us
[08:42:22] <wikibugs>	 06Machine-Learning-Team: Fix revscoring model servers - https://phabricator.wikimedia.org/T362853 (10isarantopoulos) 03NEW
[09:17:52] <wikibugs>	 (03PS1) 10Ilias Sarantopoulos: revscoring: fix circular import [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1021384 (https://phabricator.wikimedia.org/T362853)
[09:24:05] <wikibugs>	 06Machine-Learning-Team, 13Patch-For-Review: Fix revscoring model servers - https://phabricator.wikimedia.org/T362853#9725578 (10isarantopoulos) This was caused by a change in the order of the imports. RevscoringModelMP depends on RevscoringModel and RevscoringModelType.  There are 2 solutions:     - import Re...
[09:38:36] <wikibugs>	 (03CR) 10Klausman: [C:03+1] revscoring: fix circular import [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1021384 (https://phabricator.wikimedia.org/T362853) (owner: 10Ilias Sarantopoulos)
[10:02:16] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [C:03+2] revscoring: fix circular import [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1021384 (https://phabricator.wikimedia.org/T362853) (owner: 10Ilias Sarantopoulos)
[10:03:02] <wikibugs>	 (03Merged) 10jenkins-bot: revscoring: fix circular import [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1021384 (https://phabricator.wikimedia.org/T362853) (owner: 10Ilias Sarantopoulos)
[10:03:53] <wikibugs>	 06Machine-Learning-Team, 10ORES: Create basic alerts for isvcs to catch outages - https://phabricator.wikimedia.org/T362661#9725696 (10klausman) a:03klausman
[10:19:06] * klausman lunch
[11:02:38] * isaranto lunch
[12:01:04] <isaranto>	 ok revscoring-damaging in staging is fine now with the new changes
[12:01:14] <isaranto>	 deploying them to codfw first and then to eqiad
[12:01:58] <klausman>	 :+1:
[12:19:18] <isaranto>	 done both in eqiad and codfw
[12:24:51] <elukey>	 hello folks!
[12:28:37] <isaranto>	 Bongiorno!
[12:31:11] <isaranto>	 it took some time to deploy the logging change as I found an older bug
[12:32:12] <isaranto>	 and I missed the typo in the logging patch.  paylod instead of payload but we've got bigger problems so will fix it later
[12:35:12] <elukey>	 ouch sorry :(
[12:35:26] <elukey>	 sending a patch
[12:35:40] <isaranto>	 it was my fault!
[12:36:13] <elukey>	 ah it is a typo in the error msg!
[12:36:22] <elukey>	 so everything works but it is mispelled right?
[12:36:52] <elukey>	 what bigger problems do we have?
[12:39:20] <isaranto>	 yes yes everything works lets leave it. by bigger problems I mean just the increased latencies. not sth new!
[12:40:48] <elukey>	 yes yes ok I thought I missed some other horror :)
[12:41:26] <elukey>	 we also opened a task a while ago to figure out if some model server needed multiprocess, maybe some revscoring pods are a good candidate
[12:46:46] <isaranto>	 yes we could try deploy ruwiki in ml-staging with mp
[12:48:42] <isaranto>	 I noticed some increased latencies 
[12:48:42] <isaranto>	 ```
[12:48:42] <isaranto>	 INFO:root:JSON paylod for the request: {'rev_id': 137320697}
[12:48:42] <isaranto>	 INFO:root:Function get_revscoring_extractor_cache took 26.5574 seconds to execute.
[12:48:42] <isaranto>	 ```
[12:49:34] <wikibugs>	 06Machine-Learning-Team, 13Patch-For-Review: Fix revscoring model servers - https://phabricator.wikimedia.org/T362853#9726201 (10isarantopoulos) 05Open→03Resolved a:03isarantopoulos
[12:49:37] <elukey>	 niceee
[12:50:23] <elukey>	 26s is also "fast" compared to yesterday :D
[13:18:00] <elukey>	 klausman: thanks for the reviews!
[13:21:03] <klausman>	 elukey: np! also have a first go at an alert for LW services having lots of 500s (and 0s)
[13:33:55] <elukey>	 the knative-serving's chart change is basically a no-op, only few comments moved
[13:34:00] <elukey>	 but the config now seems cleaner
[13:36:37] <klausman>	 Agreed, it took me a bit to grok the model-value changes until I realized how much of it was just indentation
[13:37:20] <klausman>	 I'm also ignoring the mw-api-into-ro changes from c.laime for now; if you want me to have a look at them, lmk
[13:40:21] <elukey>	 nono I'll have to send more :D
[13:51:15] <chrisalbon>	 good morning all
[13:52:14] <isaranto>	 good morning!
[14:09:44] <elukey>	 o/
[14:47:14] <klausman>	 Heyo Chris
[15:03:58] * isaranto afk! bbl to wrap things up
[15:18:24] <elukey>	 ok so the final Istio config should be https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1021490
[15:25:58] <elukey>	 and also filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1021506 to fix the liftwing_staging's httpbb tests
[15:55:52] <wikibugs>	 (03CR) 10AikoChou: "Thank you for testing it and providing feedback! I added a limit on batch size. That's a really good point. After these changes are made i" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1020835 (https://phabricator.wikimedia.org/T358744) (owner: 10AikoChou)
[16:21:28] <elukey>	 going afk folks! Have a nice rest of the day!
[16:22:18] <klausman>	 have a nice evening, Luca
[16:23:54] <isaranto>	 I'm back for a bit! 
[16:23:59] <isaranto>	 ciao Luca o/
[16:41:12] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [C:03+1] "Nice! I like the idea of having 1 endpoint/deployment that does both." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1020835 (https://phabricator.wikimedia.org/T358744) (owner: 10AikoChou)
[16:43:39] <isaranto>	 Llama3! https://llama.meta.com/llama3/
[16:44:28] <isaranto>	 I went afk for an hour and so many things happened already! 
[16:52:30] <isaranto>	 eventually logging off for the evening folks. cu tomorrow o/
[17:07:43] <aiko>	 bye Ilias! 
[17:15:47] <wikibugs>	 06Machine-Learning-Team, 13Patch-For-Review: Deploy logo-detection model-server to LiftWing staging - https://phabricator.wikimedia.org/T362749#9727434 (10kevinbazira) I have deployed the logo-detection model-server in the experimental namespace on LiftWing staging. On checking the pod, I noticed it was not st...
[17:19:31] <wikibugs>	 06Machine-Learning-Team, 13Patch-For-Review: Deploy logo-detection model-server to LiftWing staging - https://phabricator.wikimedia.org/T362749#9727438 (10kevinbazira) I have reviewed the logs and found that the `CrashLoopBackOff` error is occurring because the model-server lacks the necessary permissions to l...
[17:27:25] <kevinbazira>	 klausman: o/ whenever you get a minute tomorrow, please help resolve this issue where the logo-detection model-server deployed in the experimental namespace on staging doesn't have permission to load the model: https://phabricator.wikimedia.org/T362749#9727438 thanks in advance!
[17:27:59] <klausman>	 will do. I suspect /mnt/models doesn't have the right perms
[17:29:01] <kevinbazira>	 great. thanks!
[17:38:30] <klausman>	 Currheading out now, g'night everyone
[19:35:30] <chrisalbon>	 night all!