[07:04:54] Good morning [07:19:53] hello! [07:39:17] ozge_: I see from https://catboost.ai/docs/en/concepts/python-installation that catboost requires numpy (>=1.16.0) so we might be ok indeed with using the same image. [07:43:25] morning folks \o [07:47:53] isaranto: Ok cool, I think I’ll serve both models in the same endpoint where each instance in the request has an additional input model_version which is by default v1. So we will expect an input similar to this: curl -s localhost:8080/v1/models/articlequality:predict -X POST -d '{"instances": [{"rev_id": 12345, "lang": "en", model_version="v1"}, {"rev_id": 1285650302, "lang": "en", model_version="v2"}]}' -i -H "Content-type: [07:47:53] application/json" would you agree? Or would it be better to introduce a new endpoint for the new model? [07:52:37] I do like this approach but I think i would prefer to use multi model serving from the same service instead which would use different endpoints. e.g articlequality:predict and articlequality_v2:predict. The only reason why i would prefer the latter is because of monitoring. It would be easier to monitor traffic through istio [07:53:42] hmm on the other hand we could also introduce a dashboard in logstash and check by version but I'm not exactly sure about this [08:16:08] Ok cool! I’ll check. Do we already have some examples? any docs would be appreciated. [09:01:22] (03CR) 10Ilias Sarantopoulos: inference-services: edit-check locust tests (033 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138678 (https://phabricator.wikimedia.org/T392460) (owner: 10Gkyziridis) [09:02:25] Something like this worked: ``` server = ModelServer() [09:02:25] server.register_model(model) [09:02:25] server.register_model(model2) [09:02:25] server.start([model, model2])``` [09:02:31] o/ [09:02:36] exactly! [09:02:47] I am available to chat if you want [09:03:24] https://meet.google.com/sgg-rrms-zye [09:03:44] if anyone else wants to join feel free! [10:17:19] ozge_: looking at the code it seems that we don't need to manually register the model by calling register_model. https://github.com/kserve/kserve/blob/998a95c84d0c6100f80555e875daeaff1937cfb1/python/kserve/kserve/model_server.py#L400 [10:18:35] so server.start([model1, model2]) would do. Unless there is some edge case where this would be helpful but I can't think of sth at the moment [10:45:06] Nice, it worked without register_model as well. [11:53:22] (03CR) 10Ilias Sarantopoulos: [C:03+1] "LGTM, just a comment regarding the commit message" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) (owner: 10Gkyziridis) [11:54:31] kevinbazira: o/ how is the docker image horror going ? :D do you need any help? I am here if you want to pick my brain or just need a debugger duck [11:55:49] isaranto: I am chipping on it slowly by slowly, I'll share more once I hit a milestone ... [11:56:47] ok! just want to make sure you're not hitting a wall there, so ping me/us if there is sth we can help with [11:59:26] okok ... [12:24:35] (03CR) 10AikoChou: [C:03+1] inference-services: edit-check. (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) (owner: 10Gkyziridis) [12:53:26] (03PS10) 10Ozge: feat: adds articlequality_v2 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138337 [12:56:16] (03PS11) 10Ozge: feat: adds articlequality_v2 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138337 [13:00:35] Hey, patch above is ready to review again :) I’ve uploaded the model as well so makefile should work locally if you’d like to test. [13:08:05] (03CR) 10AikoChou: inference-services: edit-check locust tests (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138678 (https://phabricator.wikimedia.org/T392460) (owner: 10Gkyziridis) [13:24:54] (03PS3) 10Gkyziridis: edit-check: Update readme and add pydantic tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) [13:25:15] (03CR) 10Gkyziridis: edit-check: Update readme and add pydantic tests. (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) (owner: 10Gkyziridis) [13:53:28] (03PS7) 10Gkyziridis: inference-services: edit-check locust tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138678 (https://phabricator.wikimedia.org/T392460) [13:55:47] (03PS8) 10Gkyziridis: edit-check locust tests: detect and report failed instances in batch responses [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138678 (https://phabricator.wikimedia.org/T392460) [13:55:52] (03CR) 10Gkyziridis: edit-check locust tests: detect and report failed instances in batch responses (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138678 (https://phabricator.wikimedia.org/T392460) (owner: 10Gkyziridis) [13:56:38] (03CR) 10Gkyziridis: edit-check locust tests: detect and report failed instances in batch responses (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138678 (https://phabricator.wikimedia.org/T392460) (owner: 10Gkyziridis) [14:01:49] (03CR) 10Ilias Sarantopoulos: [C:03+1] "LGTM, nice work George, thanks for tackling this!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138678 (https://phabricator.wikimedia.org/T392460) (owner: 10Gkyziridis) [14:03:05] Folks can I merge this one: https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1136981 ?? [14:03:18] Thnx for your time for reviewing both [14:06:57] (03PS12) 10Ozge: feat: adds articlequality_v2 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138337 [14:07:31] (03CR) 10Gkyziridis: [C:03+2] edit-check locust tests: detect and report failed instances in batch responses [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138678 (https://phabricator.wikimedia.org/T392460) (owner: 10Gkyziridis) [14:08:30] (03CR) 10Gkyziridis: [V:03+2 C:03+2] edit-check locust tests: detect and report failed instances in batch responses [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138678 (https://phabricator.wikimedia.org/T392460) (owner: 10Gkyziridis) [14:41:47] georgekyz: I'm re-reviwing cause you mentioned you're still getting the error [14:41:59] referring to https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1136981 [15:00:06] yeap [15:00:30] isaranto: We can merge it as it is... I am getting the error when I use the code in the comment. [15:03:23] (03CR) 10Ilias Sarantopoulos: "It works great as is! It is up2u if you want to include the test you tried in the comment" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) (owner: 10Gkyziridis) [15:04:01] georgekyz: I think you missed Aiko's comment. If you install kserve (or just mock InvalidInput) it would work [15:04:49] regerring to this comment https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1136981/comment/cd9a96e3_3dae181d/ [15:05:35] ozge_ I'll review your patch. Can someone else also review/test it please? Monday morning is fine :) [15:06:48] * isaranto afk bbl [15:08:05] (03CR) 10Ilias Sarantopoulos: feat: adds articlequality_v2 (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1138337 (owner: 10Ozge) [15:08:24] just a question/comment for now [17:03:03] (03PS4) 10Gkyziridis: edit-check: Update readme and add pydantic tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) [17:03:53] (03CR) 10CI reject: [V:04-1] edit-check: Update readme and add pydantic tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) (owner: 10Gkyziridis) [17:04:25] (03PS5) 10Gkyziridis: edit-check: Update readme and add pydantic tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) [17:05:15] (03CR) 10CI reject: [V:04-1] edit-check: Update readme and add pydantic tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) (owner: 10Gkyziridis) [17:07:38] (03PS6) 10Gkyziridis: edit-check: Update readme and add pydantic tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) [17:10:35] (03PS7) 10Gkyziridis: edit-check: Update readme and add pydantic tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) [21:35:06] (03PS8) 10Gkyziridis: edit-check: Update readme and add pydantic tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) [21:36:00] (03CR) 10CI reject: [V:04-1] edit-check: Update readme and add pydantic tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100) (owner: 10Gkyziridis) [21:36:24] (03PS9) 10Gkyziridis: edit-check: Update readme and add pydantic tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1136981 (https://phabricator.wikimedia.org/T386100)