[00:50:37] <wikibugs>	 10Machine-Learning-Team, 10ORES: ORES possibly blocking PetScan from loading? - https://phabricator.wikimedia.org/T347367 (10Quiddity)
[03:16:28] <wikibugs>	 10Machine-Learning-Team, 10ORES: ORES possibly blocking PetScan from loading? - https://phabricator.wikimedia.org/T347367 (10calbon) This is definitely ORES/Lift Wing related. We are working on it.
[05:02:59] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: "This is ready for review now. I fixed the issues but didn't add a unit test." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/960587 (https://phabricator.wikimedia.org/T347193) (owner: 10Ilias Sarantopoulos)
[05:12:49] <isaranto>	 o/ Good morning
[05:29:45] <wikibugs>	 (03PS1) 10Ilias Sarantopoulos: ores-legacy: fix CORS headers [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/960763 (https://phabricator.wikimedia.org/T347367)
[05:30:46] <wikibugs>	 (03PS2) 10Ilias Sarantopoulos: ores-legacy: fix CORS headers [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/960763 (https://phabricator.wikimedia.org/T347367)
[06:20:18] <wikibugs>	 (03PS3) 10Ilias Sarantopoulos: ores-legacy: fix CORS headers and avoid redirects [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/960763 (https://phabricator.wikimedia.org/T347367)
[06:20:43] <wikibugs>	 (03CR) 10Kevin Bazira: [C: 03+1] "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/960763 (https://phabricator.wikimedia.org/T347367) (owner: 10Ilias Sarantopoulos)
[06:24:21] <isaranto>	 The above should work but I'm not sure about the allow all policies I've set
[06:24:21] <isaranto>	 ```
[06:24:22] <isaranto>	 allow_origins=["*"],
[06:24:22] <isaranto>	     allow_methods=["*"],
[06:24:22] <isaranto>	     allow_headers=["*"],
[06:24:22] <isaranto>	 ```
[06:24:43] <isaranto>	 I mean in terms of security
[06:47:31] <wikibugs>	 (03PS4) 10Ilias Sarantopoulos: ores-legacy: fix CORS headers and avoid redirects [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/960763 (https://phabricator.wikimedia.org/T347367)
[06:49:17] <isaranto>	 Deploying the above --^
[06:50:25] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [C: 03+2] ores-legacy: fix CORS headers and avoid redirects [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/960763 (https://phabricator.wikimedia.org/T347367) (owner: 10Ilias Sarantopoulos)
[06:51:17] <wikibugs>	 (03Merged) 10jenkins-bot: ores-legacy: fix CORS headers and avoid redirects [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/960763 (https://phabricator.wikimedia.org/T347367) (owner: 10Ilias Sarantopoulos)
[06:55:33] <wikibugs>	 10Machine-Learning-Team, 10ORES, 10Patch-For-Review: User-scripts running on Wikipedia can no longer use ORES (CORS issue) - https://phabricator.wikimedia.org/T347344 (10Aklapper)
[06:55:40] <wikibugs>	 10Machine-Learning-Team, 10ORES, 10Wikimedia-Site-requests: ORES article quality is gone from euwiki in Mozilla Firefox 117.0.1 - https://phabricator.wikimedia.org/T347243 (10Aklapper)
[07:25:44] <isaranto>	 The above doesn't seem to work with the specified origins :(
[07:26:00] * isaranto commuting, back in a bit
[07:41:59] <aiko>	 morning o/
[07:43:57] <kevinbazira>	 aiko: morning o/
[07:44:00] <kevinbazira>	 isaranto: o/
[07:44:00] <kevinbazira>	 The origins specified use `allow_origins=["https://*.wikipedia.org", "https://*.wmflabs.org"]` we could use regex `allow_origin_regex=['https://.*\.wikipedia\.org', 'https://.*\.wmflabs\.org']` as shown in the FastAPI CORS docs: https://fastapi.tiangolo.com/tutorial/cors/?h=%20cors#use-corsmiddleware
[08:06:51] <aiko>	 hi Kevin :)
[08:11:48] <isaranto>	 o/
[08:12:36] <isaranto>	 nice kevinbazira: Thaanks!
[08:12:48] <isaranto>	 seems like I was in a hurry!
[08:15:26] <isaranto>	 I'm filing a new patch ok?
[08:26:09] <kevinbazira>	 sure, no problem :)
[08:33:08] <elukey>	 hello folks
[08:43:57] <wikibugs>	 (03CR) 10AikoChou: [C: 03+1] "Ahh I didn't see that it is ready for review. LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/960587 (https://phabricator.wikimedia.org/T347193) (owner: 10Ilias Sarantopoulos)
[08:48:14] <wikibugs>	 (03PS1) 10Ilias Sarantopoulos: ores-legacy: fix allowed cross origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/961044 (https://phabricator.wikimedia.org/T347367)
[08:48:29] <isaranto>	 created the new patch
[08:51:01] <wikibugs>	 (03PS4) 10AikoChou: ores-legacy: support empty boolean parameters [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/960587 (https://phabricator.wikimedia.org/T347193) (owner: 10Ilias Sarantopoulos)
[08:51:51] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] ores-legacy: support empty boolean parameters [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/960587 (https://phabricator.wikimedia.org/T347193) (owner: 10Ilias Sarantopoulos)
[08:58:14] <isaranto>	 when you have time please review https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/961044
[08:58:14] <isaranto>	 hope that the origins work correctly now. aiko: let me merge this one first to see if it fixes the issue. thanks!
[08:59:51] <aiko>	 isaranto: ack!
[09:02:18] <wikibugs>	 (03CR) 10Elukey: ores-legacy: fix allowed cross origins (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/961044 (https://phabricator.wikimedia.org/T347367) (owner: 10Ilias Sarantopoulos)
[09:04:19] <wikibugs>	 (03CR) 10AikoChou: ores-legacy: fix allowed cross origins (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/961044 (https://phabricator.wikimedia.org/T347367) (owner: 10Ilias Sarantopoulos)
[09:05:37] <wikibugs>	 (03PS2) 10Ilias Sarantopoulos: ores-legacy: fix allowed cross origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/961044 (https://phabricator.wikimedia.org/T347367)
[09:06:43] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: ores-legacy: fix allowed cross origins (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/961044 (https://phabricator.wikimedia.org/T347367) (owner: 10Ilias Sarantopoulos)
[09:07:56] <wikibugs>	 (03CR) 10Elukey: [C: 03+1] ores-legacy: fix allowed cross origins (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/961044 (https://phabricator.wikimedia.org/T347367) (owner: 10Ilias Sarantopoulos)
[09:08:06] <wikibugs>	 (03CR) 10Kevin Bazira: [C: 03+1] "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/961044 (https://phabricator.wikimedia.org/T347367) (owner: 10Ilias Sarantopoulos)
[09:08:15] <wikibugs>	 (03CR) 10AikoChou: [C: 03+1] ores-legacy: fix allowed cross origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/961044 (https://phabricator.wikimedia.org/T347367) (owner: 10Ilias Sarantopoulos)
[09:12:19] <isaranto>	 I see that now most of the requests are 200, so at least the redirection thingy worked
[09:12:58] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [C: 03+2] "Thanks for the reviews and spotting the missing \ 😊" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/961044 (https://phabricator.wikimedia.org/T347367) (owner: 10Ilias Sarantopoulos)
[09:13:51] <wikibugs>	 (03Merged) 10jenkins-bot: ores-legacy: fix allowed cross origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/961044 (https://phabricator.wikimedia.org/T347367) (owner: 10Ilias Sarantopoulos)
[09:15:30] <wikibugs>	 10Machine-Learning-Team, 10ORES, 10Patch-For-Review: ORES possibly blocking PetScan from loading? - https://phabricator.wikimedia.org/T347367 (10taavi) PetScan has also a separate issue: {T347317}
[09:17:27] <wikibugs>	 (03PS5) 10AikoChou: ores-legacy: support empty boolean parameters [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/960587 (https://phabricator.wikimedia.org/T347193) (owner: 10Ilias Sarantopoulos)
[09:17:31] <wikibugs>	 10Machine-Learning-Team: petscan expects javascript function callback from ORES - https://phabricator.wikimedia.org/T347317 (10isarantopoulos) We acknowledge the issue and are working on this to fix it.
[09:19:55] <isaranto>	 redeploying now!
[09:21:04] <elukey>	 ack thanks!
[09:21:09] <elukey>	 nice work :)
[09:21:26] <elukey>	 I'd like to follow up on the two * in the CORS, if we can limit them
[09:21:43] <elukey>	 but I am fairly ignorant as well
[09:23:06] <elukey>	 should be fine from what I am reading
[09:24:16] <elukey>	 we also need to add CORS to lift wing right? But not urgently
[09:24:28] <isaranto>	 yes I wanted your input on that. i think we can limit them (the * I mean)
[09:24:56] <isaranto>	 will also have to check if we need to allow it from other hosts. I just deployed waiting to check it soon
[09:32:26] <elukey>	 since we don't authenticate etc.. it seems that we should be fine
[09:32:44] <elukey>	 maybe we can limit the methods to GET/POST, but it is probably a nit
[09:32:51] <elukey>	 restriction on the origins is more important
[09:33:35] <elukey>	 I am a little skeptical about https://phabricator.wikimedia.org/T347317
[09:34:20] <elukey>	 I am probably ignorant about the JS use case, but I am not sure why ores-legacy needs to answer with a jquery snippet
[09:34:33] <elukey>	 it seems an extra feature that we could avoid
[09:34:44] <elukey>	 maybe following up with whoever maintains petscan
[09:34:44] <isaranto>	 me neither but is seems there has been such usage. we need to check
[09:35:31] <elukey>	 I think the usage is really minimal, and probably folks can bypass the functionality simply writing some js code on the client side
[09:37:43] <isaranto>	 the above patch for CORS doesn't seem to work yet...
[09:38:46] <elukey>	 how are you testing it?
[09:39:36] <isaranto>	 Opening a browser window, go to petscan.wmflabs.org go to developer tools , open console and type:
[09:39:36] <isaranto>	 ```
[09:39:36] <isaranto>	 $.ajax({
[09:39:36] <isaranto>	     url: "https://ores.wikimedia.org/v3/scores/"
[09:39:36] <isaranto>	 }).done(function(response) {
[09:39:37] <isaranto>	     console.log(response);
[09:39:37] <isaranto>	 });
[09:39:38] <isaranto>	 ```
[09:40:17] <isaranto>	 When I did the same on localhost (but with wildcard for origins in the FastAPI CORS settings) it worked
[09:41:04] <isaranto>	 So  I am assuming it is misconfigured origins
[09:45:16] <elukey>	 isaranto: when does the response headers are added? because with curl I don't see them 
[09:48:22] <elukey>	 maybe fastapi returns Access-Control-Allow-Origin etc.. only for some clients
[09:49:48] <isaranto>	 I don't really know. but I assume that it does for the origins allowed in the middleware
[09:50:25] <isaranto>	 from what I read in the Steps section: https://fastapi.tiangolo.com/tutorial/cors/
[09:50:25] <isaranto>	 we should allow the OPTIONS method
[09:50:56] <klausman>	 aiko: fyi: 
[09:50:58] <klausman>	 $ curl https://staging.svc.eqiad.wmnet:8087/service/lw/inference/v1/models/readability:predict -X POST -d '{"rev_id": 123456, "lang": "en"}'; echo
[09:51:00] <klausman>	 {"model_name":"readability","model_version":"2","wiki_db":"enwiki","revision_id":123456,"output":{"prediction":false,"probabilities":{"true":0.4793056845664978,"false":0.5206943154335022},"fk_score":8.277095534538086}}
[09:51:14] <klausman>	 So readability works on the API GW's staging instance, will push to prod in a minute
[09:53:28] <elukey>	 isaranto: yeah I was checking as well
[09:55:23] <elukey>	 isaranto: so from the response headers when I try OPTIONS I see that Varnish itself responds with HTTP 405
[09:55:47] <elukey>	 so it seems the wrong road
[09:56:07] <klausman>	 All done. Readability isvc is now reachable via API GW
[09:56:08] <wikibugs>	 10Machine-Learning-Team: petscan expects javascript function callback from ORES - https://phabricator.wikimedia.org/T347317 (10isarantopoulos) Is there a codebase somewhere for petscan?
[09:56:49] <elukey>	 klausman: nice!
[09:57:40] <isaranto>	 elukey: you mean that as things are at the moment it won't be allowed by varnish whatever we do, right?
[10:02:33] <elukey>	 isaranto: yeah.. there are some rules in the VCL, but I just checked on ores1001 and I see
[10:02:36] <elukey>	 access-control-allow-origin: *
[10:02:37] <elukey>	 even for a GET
[10:02:57] <elukey>	 it is added by nginx, that does the TLS termination/proxy in front of uwsgi
[10:03:15] <elukey>	 see https://phabricator.wikimedia.org/rOPUP5288692a3226547d2591cbc400b0dd4db0bb08de
[10:03:49] <isaranto>	 ok, I will check. on my local deployment the regex we added doesnt seem to work
[10:04:03] <elukey>	 but do you see the response header if you do a GET?
[10:04:28] <isaranto>	 nope
[10:04:36] <elukey>	 I think it should be there
[10:04:45] <isaranto>	 going for a quick lunch and will come back soon
[10:13:49] <elukey>	 ok so with curl we need to add something like "-H "Origin: en.wikipedia.org""
[10:13:58] <elukey>	 but I still don't see the response header
[10:14:32] <elukey>	 in https://fastapi.tiangolo.com/tutorial/cors/ I see that there is a "expose_headers" field
[10:19:07] <wikibugs>	 10Machine-Learning-Team: petscan expects javascript function callback from ORES - https://phabricator.wikimedia.org/T347317 (10Aklapper) >>! In T347317#9198582, @isarantopoulos wrote: > Is there a codebase somewhere for petscan?  https://github.com/magnusmanske/petscan_rs/
[10:20:00] <wikibugs>	 10Machine-Learning-Team: petscan expects javascript function callback from ORES - https://phabricator.wikimedia.org/T347317 (10Asartea) >>! In T347317#9198582, @isarantopoulos wrote: > Is there a codebase somewhere for petscan? Perhaps if it is the only use case then we could avoid returning Javascript from the...
[10:23:44] <elukey>	 ahh! So with the Origin request header I still see a 405, but this time from istio-envoy
[10:23:50] <elukey>	 for OPTIONS I mean
[10:24:05] <elukey>	 so that is probably where it is blocked
[10:24:48] <elukey>	 https://istio.io/latest/docs/reference/config/networking/virtual-service/#CorsPolicy
[10:30:41] <elukey>	 trying some values live on ml-serve-codfw
[10:30:53] <elukey>	 (precisely the ores-legacy's istio virtual server)
[10:50:30] <isaranto>	 oh ok!
[10:53:23] <isaranto>	 so in my setup it seems indeed that there is an issue with the regexes we added.
[10:53:23] <isaranto>	 I have the following setup: ores-legacy running on statbox and I am accessing it via ssh tunneling. If I explicitly specify petscan.wmflabs.org in remote origin the script will run succesfully
[10:54:13] * klausman lunch
[10:55:38] <elukey>	 isaranto: ack, still trying to make istio work, I think that without it we can set anything on our python code but it will not work :(
[10:56:00] <isaranto>	 ack. yes just reporting on my end
[10:56:04] <wikibugs>	 10Machine-Learning-Team: Add JavaScript examples to LiftWing API gateway docs - https://phabricator.wikimedia.org/T347387 (10kevinbazira)
[10:57:15] <wikibugs>	 10Machine-Learning-Team: Add JavaScript examples to LiftWing API gateway docs - https://phabricator.wikimedia.org/T347387 (10kevinbazira) JavaScript examples have been added to the LiftWing API gateway docs as shown below: 1. revscoring goodfaith prediction: https://api.wikimedia.org/wiki/Lift_Wing_API/Reference...
[10:57:52] <wikibugs>	 10Machine-Learning-Team: Add JavaScript examples to LiftWing API gateway docs - https://phabricator.wikimedia.org/T347387 (10kevinbazira) Also added the user-agent header in the the LiftWing usage docs for all examples (curl, python, and JS): 1. internal endpoints: https://wikitech.wikimedia.org/w/index.php?titl...
[10:58:38] <elukey>	 isaranto: I think I have it :)
[11:00:16] <elukey>	 mmm petscan still not working 
[11:01:24] <wikibugs>	 10Machine-Learning-Team, 10Documentation: Add JavaScript examples to LiftWing API gateway docs - https://phabricator.wikimedia.org/T347387 (10Aklapper)
[11:01:25] <chrisalbon>	 Good morning!
[11:01:32] <elukey>	 curl -s https://ores-legacy.wikimedia.org/v3/scores/ -i -H "Origin: https://petscan.wmflabs.org" | head
[11:01:35] <elukey>	 shows
[11:01:40] <elukey>	 access-control-allow-origin: https://petscan.wmflabs.org
[11:01:42] <elukey>	 chrisalbon: o/
[11:02:31] <elukey>	 ah no the snipped doesn't work
[11:02:45] <elukey>	 but if I open browser tools -> console I don't see any error
[11:02:50] <elukey>	 only the warning for the mime type
[11:04:20] <isaranto>	 Hey Chris o/
[11:04:40] <taavi>	 elukey: did you try refreshing the page after opening the console?
[11:05:06] <elukey>	 taavi: I did yes, we didn't fix the mime type thing issue though
[11:05:21] <elukey>	 so the general CORS headers block should be lifted
[11:05:40] <taavi>	 I see `Uncaught SyntaxError: unexpected token: ':'` in the console. it's trying to interpret the JSON output as javascript code
[11:08:29] <elukey>	 I am trying $.ajax({url: "https://ores.wikimedia.org/v3/scores/"}).done(function(response){console.log(response)}) but still CORS isuses
[11:08:32] <elukey>	 uff
[11:08:38] <elukey>	 going to lunch folks, will get back in abit :)
[11:11:42] <isaranto>	 these issues are because it doesn't allow CORS requests from petscan
[11:12:52] <taavi>	 oddly enough that ajax thing you pasted works for me
[11:13:03] <taavi>	 this smells like a caching issue
[11:22:02] <isaranto>	 the ajax script works for ores-legacy but not for ores for me
[12:55:24] <elukey>	 back!
[12:55:32] <elukey>	 taavi: ah nice point!
[12:55:39] <elukey>	 lemme try one thing
[12:58:21] <elukey>	 taavi: where did you try the ajax line if I may ask?
[12:58:43] <taavi>	 petscan
[12:58:54] <elukey>	 it doesn't work for me
[12:58:56] <elukey>	 but if I try
[12:58:58] <elukey>	 curl -s https://ores-legacy.wikimedia.org/v3/scores/ -i -H "Origin: https://petscan.wmflabs.orgss" | head
[12:59:05] <taavi>	 mmmm
[12:59:14] <elukey>	 I can see: access-control-allow-origin: https://petscan.wmflabs.org
[12:59:18] <taavi>	 are you setting the Vary header correctly?
[12:59:22] <elukey>	 (sorry extra ss in the above link)
[12:59:51] <elukey>	 taavi: what I did was open the dev tool tab -> console basically
[12:59:55] <elukey>	 then copy paste
[13:02:58] <isaranto>	 elukey: it is similar for me. The script runs for ores-legacy but not ores 
[13:04:44] <elukey>	 isaranto: for ores-legacy I get VM20:1 Uncaught TypeError: $.ajax is not a function
[13:04:55] <elukey>	 I know I am a realy n00b in js
[13:05:53] <elukey>	 ah wait maybe you are doing it from petscan?
[13:06:18] <elukey>	 ok I tried from en.wikimedia.org and indeed ores-legacy works
[13:06:24] <elukey>	 but not ores.w
[13:06:58] <isaranto>	 yes u need to try it from a website that has jquery loaded
[13:07:21] <isaranto>	 I'm n00b in js as well :)
[13:07:32] <chrisalbon>	 same
[13:10:39] <elukey>	 ok so there is a problem with caching
[13:10:48] <elukey>	 namely, it seems that varnish caches /v3/scores
[13:11:32] <isaranto>	 (now it works)
[13:11:37] <taavi>	 elukey: you need to emit a `Vary: Origin` header from the responses if you use origin to generate the CORS header
[13:11:55] <isaranto>	 did u clear the cache manually?
[13:11:59] <elukey>	 I did ye
[13:12:00] <elukey>	 *yes
[13:12:11] <elukey>	 taavi: ack yes that too
[13:12:16] <elukey>	 but this is another interesting difference:
[13:12:22] <elukey>	   ores.wikimedia.org:
[13:12:22] <elukey>	     caching: 'normal'
[13:12:22] <elukey>	   ores-legacy.wikimedia.org:
[13:12:24] <elukey>	     caching: 'pass'
[13:12:40] <elukey>	 these are the cache settings, we pass ores-legacy but we cache ores
[13:14:01] <elukey>	 yeah now ores.w works with the ajax script
[13:14:55] <elukey>	 to be on the safe side, I'd add 'pass' for ores.w too
[13:15:15] <elukey>	 then we can implement a decent caching mechanism, including Vary headers like taavi mentioned
[13:18:50] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/961106/
[13:19:45] <elukey>	 after this we should be ok in theory
[13:22:55] <elukey>	 taavi: thanks as always for the brainbounce!
[13:44:35] <elukey>	 isaranto: so far it seems working, I didn't apply the istio/cors setting to deployment-charts because I think I need to add support for it first (following up with serviceops) but I have applied it manually to ml-serve-codfw
[13:45:08] <elukey>	 in theory we could avoid setting it on our pods
[13:45:31] <elukey>	 and rely only on the istio ingress config
[13:45:36] <isaranto>	 ok, thank you!
[13:50:37] <wikibugs>	 10Machine-Learning-Team, 10Commons: Utilize ChatGPT for categorizing and extracting metadata from files on Commons - https://phabricator.wikimedia.org/T345898 (10calbon) 05Open→03Resolved
[14:28:02] <wikibugs>	 10Machine-Learning-Team: Refactor inference services repo to allow local runs - https://phabricator.wikimedia.org/T347404 (10isarantopoulos)
[14:30:17] <wikibugs>	 10Machine-Learning-Team, 10Goal: Zero traffic on bare metal ORES servers - COMPLETE - https://phabricator.wikimedia.org/T341696 (10calbon)
[14:30:40] <wikibugs>	 10Machine-Learning-Team, 10Goal: Defined and measured SLO for every production service - COMPLETE - https://phabricator.wikimedia.org/T341693 (10calbon)
[14:31:07] <wikibugs>	 10Machine-Learning-Team, 10Goal: Content Recommendation API migration completed - https://phabricator.wikimedia.org/T341704 (10kevinbazira) We finally got rec-api deployment settings that could run on LiftWing: 1. a 4th test was run and this dropped the memory usage by 5x  2. we disabled the related_articles s...
[14:31:18] <wikibugs>	 10Machine-Learning-Team, 10Goal: Content Recommendation API migration  - https://phabricator.wikimedia.org/T341704 (10calbon)
[14:32:07] <wikibugs>	 10Machine-Learning-Team, 10Goal: Support WME migration to Lift Wing - COMPLETE - https://phabricator.wikimedia.org/T341698 (10calbon)
[14:34:31] <wikibugs>	 10Machine-Learning-Team, 10Goal: Order 1 GPU for Lift Wing - https://phabricator.wikimedia.org/T341699 (10calbon)
[14:36:46] <wikibugs>	 10Machine-Learning-Team, 10Goal: Order 1 GPU for Lift Wing - https://phabricator.wikimedia.org/T341699 (10calbon) Maybe: big order decision jan/feb
[14:42:12] <wikibugs>	 10Machine-Learning-Team, 10ORES: User-scripts running on Wikipedia can no longer use ORES (CORS issue) - https://phabricator.wikimedia.org/T347344 (10calbon) a:03elukey
[14:42:31] <wikibugs>	 10Machine-Learning-Team: petscan expects javascript function callback from ORES - https://phabricator.wikimedia.org/T347317 (10calbon) a:03isarantopoulos
[14:43:10] <wikibugs>	 10Machine-Learning-Team, 10ORES: ORES possibly blocking PetScan from loading? - https://phabricator.wikimedia.org/T347367 (10calbon) a:03isarantopoulos
[14:44:16] <wikibugs>	 10Machine-Learning-Team: Create external endpoint for recommendation-api-ng hosted on LiftWing - https://phabricator.wikimedia.org/T347263 (10calbon) a:03kevinbazira
[14:45:07] <wikibugs>	 10Machine-Learning-Team: Create external endpoint for recommendation-api-ng hosted on LiftWing - https://phabricator.wikimedia.org/T347263 (10calbon) a:05kevinbazira→03klausman
[14:45:49] <wikibugs>	 10Machine-Learning-Team: Set SLO for the recommendation-api-ng service hosted on LiftWing - https://phabricator.wikimedia.org/T347262 (10calbon) a:03klausman
[14:46:50] <wikibugs>	 10Machine-Learning-Team, 10ORES: Cannot set Api-User-Agent header when making requests to ORES from a user script - CORS - https://phabricator.wikimedia.org/T347214 (10calbon) a:03elukey
[14:50:02] <wikibugs>	 10Machine-Learning-Team, 10ORES: Feature injection does not appear to work in ores-legacy - https://phabricator.wikimedia.org/T347194 (10calbon) a:03calbon
[14:53:43] <wikibugs>	 10Machine-Learning-Team, 10ORES, 10Patch-For-Review: Support for basic boolean flags in ores-legacy - https://phabricator.wikimedia.org/T347193 (10calbon) a:03isarantopoulos
[15:29:57] <wikibugs>	 10Machine-Learning-Team: Create external endpoint for recommendation-api-ng hosted on LiftWing - https://phabricator.wikimedia.org/T347263 (10Isaac) Thanks @kevinbazira ! API spec worked great when I first tested. Maybe not turned on yet but I tried to test the actual predictions (following [[https://wikitech.wi...
[15:39:24] <elukey>	 isaranto: the api-user-agent issue seems fixed
[15:39:33] <isaranto>	 ack, nice!
[15:39:52] <elukey>	 should we try to rollback the ores-legacy fast api changes to see if all works?
[15:40:34] <wikibugs>	 10Machine-Learning-Team, 10ORES: Cannot set Api-User-Agent header when making requests to ORES from a user script - CORS - https://phabricator.wikimedia.org/T347214 (10elukey) Applied a manual fix on ml-serve-codfw, it should now work. I need to add the proper config to deployment-charts to make it permanent :)
[15:43:29] <isaranto>	 yes we could
[15:45:20] <wikibugs>	 10Machine-Learning-Team: Decommission ORES configurations and servers - https://phabricator.wikimedia.org/T347278 (10elukey)
[15:46:18] <isaranto>	 I have to go afk now. Feel free to do it otherwise I can revert the deployment a bit later or in the morning 
[15:54:39] <elukey>	 isaranto: we can do it tomorrow, have a nice evening!
[15:55:33] <wikibugs>	 10Machine-Learning-Team: petscan expects javascript function callback from ORES - https://phabricator.wikimedia.org/T347317 (10isarantopoulos) Thanks for pointing that out! I have added to the existing issue https://github.com/magnusmanske/petscan_rs/issues/141 The issue is that the new ores-legacy service that...
[15:58:33] <wikibugs>	 10Machine-Learning-Team, 10ORES: User-scripts running on Wikipedia can no longer use ORES (CORS issue) - https://phabricator.wikimedia.org/T347344 (10elukey) I applied the following config to both ml-serve-eqiad and codfw:  ` # kubectl edit vs -n ores-legacy [..]   - corsPolicy:       allowCredentials: false...
[15:58:36] <elukey>	 klausman: --^
[15:58:50] <elukey>	 lemme know if there is anything not clear - a revert is very simple in case it is needed
[16:27:42] <wikibugs>	 10Machine-Learning-Team, 10ORES: User-scripts running on Wikipedia can no longer use ORES (CORS issue) - https://phabricator.wikimedia.org/T347344 (10Nux) Petscan also depends on this. Or to be more exact it assumes calback parameter is available to avoid CORS issues.  Petscan is dead because of this (white pa...
[16:35:16] <klausman>	 elukey: which of the things were you pointing to?
[16:35:40] <elukey>	 the task above the arrow
[16:35:55] <klausman>	 ah, thx
[16:56:16] * elukey afk!
[17:05:46] <chrisalbon>	 have a good night all!
[21:23:42] <wikibugs>	 10Machine-Learning-Team, 10ORES: ORES possibly blocking PetScan from loading? - https://phabricator.wikimedia.org/T347367 (10Novem_Linguae)
[21:27:16] <wikibugs>	 10Machine-Learning-Team, 10ORES: ORES possibly blocking PetScan from loading? - https://phabricator.wikimedia.org/T347367 (10Novem_Linguae) >>! In T347367#9198439, @taavi wrote: > PetScan has also a separate issue: {T347317}  Hey @taavi. I think this ticket and your ticket {T347317} are the same bug. Note how...
[21:29:05] <wikibugs>	 10Machine-Learning-Team, 10ORES: User-scripts running on Wikipedia can no longer use ORES (CORS issue) - https://phabricator.wikimedia.org/T347344 (10Novem_Linguae) I am no longer able to reproduce this. Thank you for the quick patches. Can probably mark this as resolved, but will let the machine learning team...