[08:02:33] <elukey>	 good morning :)
[08:07:06] <isaranto>	 o/
[08:26:37] <klausman>	 \o
[10:13:24] <elukey>	 isaranto: o/ something is moving, from stat1004:
[10:13:25] <elukey>	 curl "https://ores-legacy.k8s-ml-staging.discovery.wmnet:31443/v3/scores" -i --http1.1 --resolve ores-legacy.k8s-ml-staging.discovery.wmnet:31443:10.192.0.201  -k
[10:13:36] <elukey>	 (the -k is still needed since I have to fix a TLS cert problem yet)
[10:13:44] <elukey>	 the main issue is that sometimes I get 404
[10:14:57] <isaranto>	 🙌
[10:15:15] <isaranto>	 nice! I got 404 every second call I made 
[10:16:05] <elukey>	 in the uvicorn logs I only see 200s, so I guess that the istio gateway is doing something weird
[10:18:24] <isaranto>	 note to self: once we get this up and running I also need to improve error handling
[10:18:48] <isaranto>	 e.g. for unknown revision ids I got a client error in the response instead of the standard ores message
[10:19:38] <elukey>	 atm contacting lift wing is not possible since we have to allow POSTs in the TLS proxy
[10:22:43] <isaranto>	 ack
[10:53:43] <wikibugs>	 (03PS1) 10Ilias Sarantopoulos: feat: hardcode threshold calls to switch to Lift Wing [extensions/ORES] - 10https://gerrit.wikimedia.org/r/915541 (https://phabricator.wikimedia.org/T332953)
[10:54:38] <wikibugs>	 (03PS2) 10Ilias Sarantopoulos: feat: hardcode threshold calls to switch to Lift Wing [extensions/ORES] - 10https://gerrit.wikimedia.org/r/915541 (https://phabricator.wikimedia.org/T332953)
[10:56:03] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] feat: hardcode threshold calls to switch to Lift Wing [extensions/ORES] - 10https://gerrit.wikimedia.org/r/915541 (https://phabricator.wikimedia.org/T332953) (owner: 10Ilias Sarantopoulos)
[11:13:43] * elukey lunch!
[11:39:04] * isaranto lunch
[13:42:48] <elukey>	 ok I found the issue of the random 404
[13:43:20] <elukey>	 the K8s svc for the new ingress gateway targets all the istio gateway pods, including the kserve ones
[13:43:42] <elukey>	 so when the load balancing heuristic decides to use a "kserve" ingress it returns 404
[13:43:47] <elukey>	 or other errors
[13:44:28] <isaranto>	 ack
[13:58:40] <klausman>	 Ah, the rare "too many backends" problem :)
[14:02:10] <elukey>	 it is very weird since it may be a limitation of istio
[14:03:31] <elukey>	 https://github.com/istio/istio/blob/release-1.15.7-patch/manifests/charts/gateway/templates/service.yaml#L49
[14:03:47] <elukey>	 ok so I need to add that bit, let's see
[14:23:26] <elukey>	 finally, now it works :)
[14:23:26] <elukey>	 curl "https://ores-legacy.k8s-ml-staging.discovery.wmnet:31443/v3/scores" -i --http1.1 --resolve ores-legacy.k8s-ml-staging.discovery.wmnet:31443:10.192.0.201
[14:24:54] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/915685
[14:25:13] <elukey>	 ok so now the next problem is to allow envoy to make posts
[14:28:02] <wikibugs>	 10Machine-Learning-Team, 10Patch-For-Review: Create a staging ingress configuration for ml-staging-codfw - https://phabricator.wikimedia.org/T335756 (10elukey) First result! from any node, like stat100x, one could use:  ` curl "https://ores-legacy.k8s-ml-staging.discovery.wmnet:31443/v3/scores" -i --http1.1 --...
[15:09:20] <isaranto>	 magic... :)
[15:14:01] <elukey>	 s/magic/horror
[15:35:47] <elukey>	 isaranto: o/ so I found the source of the problem, in ores-legacy we define only "app.post"
[15:35:54] <elukey>	 err sorry app.gets
[15:36:26] <elukey>	 mmm wait, am I missing something? Is ORES only working with GETs?
[15:37:13] <elukey>	 ahhhh of course I was trying to POST to the ores-legacy Ui
[15:37:35] <isaranto>	 ORES works with GET..
[15:38:10] <isaranto>	 so we issue a GET request to ores-legacy which then POST(s) to Lift Wing
[15:38:27] <elukey>	 curl "https://ores-legacy.k8s-ml-staging.discovery.wmnet:31443/v3/scores/enwiki/1234331345/damaging" -i --http1.1 --resolve ores-legacy.k8s-ml-staging.discovery.wmnet:31443:10.192.0.201 works!
[15:38:42] <elukey>	 isaranto: yes yes I am stupid, too much confusion with Lift Wing!
[15:39:15] <elukey>	 I mean sort of works, we still need to apply the correct URL for lift wing
[15:39:24] <elukey>	 okok but the rest works fine
[15:41:27] <isaranto>	 🎉
[15:41:59] <isaranto>	 on the other hand I lost some changes to a script I had in a jupyter notebook 
[15:42:30] <isaranto>	 not checked in VCS and restarted the laptop
[15:42:42] <isaranto>	 oh well, I love notebooks when this happens :)
[15:43:30] <isaranto>	 ok, false alarm minor changes were not saved
[15:50:02] <elukey>	 isaranto: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/915727 should also fix the lift wing URL
[15:53:44] <isaranto>	 should/is it be the same for all deployments? staging/prod
[15:56:07] <elukey>	 good point, we can definitely override for staging
[15:56:17] <elukey>	 also, I firewalled prod from staging, so we have to
[15:56:19] <elukey>	 sigh
[15:56:22] <elukey>	 lemme check
[15:56:49] <elukey>	 ahhh no wait
[15:57:20] <elukey>	 on the mesh settings (the envoy proxy config that serviceops manages as sidecar) we have only the "inference" settings, that is prod
[15:57:43] <elukey>	 so we'll need to add one for staging as well probably
[16:00:53] <elukey>	 mmmm no in theory the settings should allow to call prod from staging (need to be fixed)
[16:01:41] <elukey>	 isaranto: not sure why but I see a 200 contacting ores from the sidecar, but in the response I get "ClientError"
[16:03:08] <isaranto>	 elukey: don't mind the responses for now I need to figure out what they will be
[16:03:38] <isaranto>	 I mean it returns a 200 because the call succeeds (although the underlying LW calls fail)
[16:04:06] <elukey>	 isaranto: sure sure but it seems that an error response is generated, and from the create_error_response() code it seems that it is due to lift wing
[16:04:54] <elukey>	 I don't see logs in the pod except the access log, maybe also the python logging config for uvicorn needs to be set (or similar)
[16:05:40] <elukey>	 I think we can postpone the tests for when I am back next week (when you have time), but feel free to do any tests and report in the task!
[16:05:43] <isaranto>	 hmm I will check 
[16:05:48] <elukey>	 going afk for today folks, o/
[16:05:52] <elukey>	 talk with you next week
[16:05:56] <isaranto>	 ok I will for sure 
[16:05:59] <elukey>	 have a nice rest of the day!
[16:06:08] <isaranto>	 ciao!
[16:51:04] <isaranto>	 logging off, more threshold debugging tomorrow!
[17:02:06] <aiko>	 bye Ilias!
[17:05:07] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 17th round of wikis - https://phabricator.wikimedia.org/T308143 (10kevinbazira)
[17:05:31] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 17th round of wikis - https://phabricator.wikimedia.org/T308143 (10kevinbazira) The conclusion on the backtesting results is that most of the languages look fine besides: - tiwiki (0.54), tkwiki (0.74), urwiki (0.62)...