[06:36:20] <elukey>	 good morning :)
[06:36:48] <elukey>	 I tried to delete a pod in the revscoring-editquality ns and it didn't lead to any issue, the storage initializer wasn't blocked by firewall etc..
[06:37:02] <elukey>	 so the new network rules shouldn't be the issue
[07:14:07] <elukey>	 I just realized that the STORAGE_URI used for articlequality is different from the others, filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/734559
[07:14:15] <elukey>	 let's see if this is the issue
[07:22:00] <elukey>	 lol it was the right fix
[07:22:05] <elukey>	 now article quality works!
[07:23:44] <elukey>	 I mean sort of, the pods are up but the HTTP api returns 500 :D
[07:24:25] <elukey>	       File "model-server/model.py", line 20, in predict
[07:24:25] <elukey>	         inputs = request["article_text"]
[07:24:26] <elukey>	     KeyError: 'article_text'
[07:24:39] <elukey>	 so something to follow up
[07:39:26] <elukey>	 running errand for a bit, ttl!
[10:05:25] <elukey>	 back
[10:05:37] <elukey>	 now I get the 500, I am not passing the correct input.json 
[10:06:45] <elukey>	 yeah it works perfectly :)
[10:07:04] <elukey>	 as a follow up, maybe we could be a little more defensive in the model.py code
[10:07:33] <elukey>	 maybe returning a 400 ?
[10:27:13] <elukey>	 mmm I think that I have an idea why the storage-intializer behaved in that way
[10:27:16] <elukey>	 https://github.com/kserve/kserve/blob/master/python/kserve/kserve/storage.py#L114
[10:28:03] <elukey>	 so it expects to list objects in a bucket_path
[10:28:12] <elukey>	 meanwhile we gave the full model.bin path
[10:28:37] <elukey>	 so it returned without logging and then ended up in the "successfully copied etc.."
[10:29:07] <elukey>	 that is https://github.com/kserve/kserve/blob/master/python/kserve/kserve/storage.py#L85
[10:30:34] <elukey>	 okok the logs seem to point to this direction, I am inclined to open a github issue to upstream
[11:45:17] <elukey>	 created https://github.com/kserve/kserve/issues/1882
[11:51:04] <elukey>	 mmmm wait a minute, after creating the github issue I realized that in theory the code should work if we specify /model.bin
[14:39:17] <elukey>	 created https://github.com/kserve/kserve/pull/1883
[14:49:15] <elukey>	 taking a little break 
[16:02:24] <accraze>	 o/
[16:02:55] <accraze>	 elukey: omgg i can't believe the uri issue was because of appending `/model.bin`
[16:21:22] <accraze>	 either way,  the isvc should give a better warning/error other than no file/directory found
[16:22:08] <elukey>	 accraze: o/ https://github.com/kserve/kserve/pull/1883
[16:22:18] <accraze>	 nice one elukey!
[16:42:06] <elukey>	 thanks :)
[16:42:17] <elukey>	 need to run now, but I'll read later in case there are questions!
[16:42:23] <elukey>	 the article quality model works fine afaics
[16:45:51] <accraze>	 see ya later!
[18:54:42] <accraze>	 been reading through the updated kserve docs, looks like they have some cool stuff on the horizon, new dataplane v2 api looks cool
[18:55:34] <accraze>	 also impressed by the example of using feast (feature store) as a transformer in front of a model inference service