[03:26:13] (03CR) 10Abijeet Patro: "recheck" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/981302 (owner: 10L10n-bot) [03:27:52] (03CR) 10Abijeet Patro: [V: 03+2] Localisation updates from https://translatewiki.net. [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/981302 (owner: 10L10n-bot) [07:09:23] o/ good morning! [09:47:37] morning! [09:51:25] 10Machine-Learning-Team, 10Patch-For-Review: Reduce default API response fields for article-descriptions model-server - https://phabricator.wikimedia.org/T352959 (10kevinbazira) The default API response fields for article-descriptions model-server have been reduced to 5/8: ` kevinbazira@deploy2002:~$ time curl... [09:55:43] 10Machine-Learning-Team: Create external endpoint for recommendation-api-ng hosted on LiftWing - https://phabricator.wikimedia.org/T347263 (10klausman) From my workstation at home, both the API endpoint and the spec query work, but there are still things that could improve (more on that below): `$ time curl -s... [09:56:47] $ curl -s "https://api.wikimedia.org/service/lw/recommendation/v1/api/?s=en&t=fr&n=3&article=France" [09:56:49] [{"pageviews": 1831, "title": "Turks_in_Europe", "wikidata_id": "Q4821600", "rank": 498.0}, {"pageviews": 3271, "title": "European_emigration", "wikidata_id": "Q3819571", "rank": 496.0}, {"pageviews": 790, "title": "History_of_France_(1900\u2013present)", "wikidata_id": "Q290313", "rank": 476.0}] [09:57:00] kevinbazira: ^^^ The rec-api-ng is now reachable via API GW [09:57:12] Also, good morning :) [09:57:33] klausman: o/ [09:57:40] super :) [09:57:45] let me check ... [09:59:20] Note the caveats at the end of my Phab update, especially .../api/ vs .../api [10:10:20] nice! [10:10:54] I have tried with the request below and the rec-api-ng external endpoint works like a charm: [10:11:13] https://www.irccloud.com/pastebin/K9BWgHRU/ [10:11:31] klausman: thank you for working on this :) [10:11:31] next week I'll add the rec-api-ng to the APIGW docs [10:38:24] kevinbazira: thanks! I wonder if the api/ vs api thing could be solved in the service itself (handling both api/ and api the same way). [10:40:59] looks like it does handle it, in the browser when I visit: http://recommend.wmflabs.org/api?s=en&t=fr&n=3&article=Apple [10:40:59] it resolves to: https://recommend.wmflabs.org/api/?s=en&t=fr&n=3&article=Apple [10:52:41] It may be that wmflabs does some magic there? [10:53:25] if you hit the url with e.g. curl and don't have a slash after api you get a HTML page with a redirect to the wment internal endpoint [10:54:00] (the api gw one, that is) [10:54:19] I agree with Tobias her. I think we should handle that in the flask app (if it is not already handled) [10:58:34] *here [11:16:07] * klausman lunch [11:16:19] 10Lift-Wing, 10Machine-Learning-Team: Investigate increase p99 latencies in ml-serve-eqiad - https://phabricator.wikimedia.org/T352958 (10achou) I ran the model server on a stat machine to use the discovery endpoint for an internal call to debug the issue. I had the following error message when trying with the... [11:24:33] * aiko brunch! [11:24:41] fancy :) [11:27:33] hahaha :D [11:28:32] isaranto: o/ I added some thoughts regarding the zh-yue issue ---^ [11:31:44] klausman: interesting, the container in my local env is able to resolve `api` to `api/` when I add the `-L` flag curl: [11:32:00] https://www.irccloud.com/pastebin/1eFGXlcF/ [11:32:12] same appies to: `curl -sL "http://recommend.wmflabs.org/api?s=en&t=fr&n=3&article=Apple"` [11:32:34] I'll investigate why the LiftWing instance of the rec-api-ng doesn't resolve URLs. [11:52:47] aiko: thanks for checking this! I suggest to add a configuration file for host rewrites. I can file a patch if you agree [11:53:27] also: brunch sounds fancy lol [11:53:44] although dunch is the new thing 😜 [12:12:57] TIL : there is also linner (late lunch). whateever [12:13:01] * isaranto lunch break [12:51:09] isaranto: yes adding a config file sounds good :) [12:56:25] dunch and linner.. haha interesting πŸ˜† [13:58:17] (03PS1) 10Kevin Bazira: article-descriptions: update descartes tag to fix prediction bug [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/981427 (https://phabricator.wikimedia.org/T352750) [14:06:09] Good morning [14:09:43] Heyo Chris. [14:10:01] rec-api-ng is now usable via apigw [14:10:02] o/ Chris! [14:29:32] (03PS1) 10Ilias Sarantopoulos: revertrisk: add config for mw host rewrites [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/981541 (https://phabricator.wikimedia.org/T352958) [14:32:25] I really like the commit message description format (Why/What/Tested) as I find it helpful. what do you folks think about this? [14:32:36] or even something similar/better [14:37:28] What is an example of that? Purely for my own learning. [14:37:37] --^ [14:37:46] https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/981541/ [14:38:17] I saw folks use something similar in mediawiki code (the why /what) [14:40:54] I think the most important thing is to focus on "Why a change is made" in a commit message. For all the rest (what changes and how) the code can be considered self explanatory but we want to decribe that as well out of respect for the ppl that review our code [14:41:04] anyway, just some Friday thoughts! [14:42:10] aiko: The above patch fixes the issue on my local setup. want to try it as well with api-ro though [14:43:17] git review -R [14:43:26] lool , wrong window [14:43:32] (03PS2) 10Ilias Sarantopoulos: revertrisk: add config for mw host rewrites [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/981541 (https://phabricator.wikimedia.org/T352958) [14:48:16] * klausman out for an errand, bbiab [15:02:40] Isaranto let’s talk about it next meeting. I like it but my use case is me reading all the messages every morning [15:07:08] πŸ‘ [15:12:00] 10Machine-Learning-Team: Investigate how to improve model card integration with existing user flows - https://phabricator.wikimedia.org/T353025 (10Isaac) Example of robot icon for Android's machine-assisted article description recommendations (though in that case, clicking it opens the dialog box with the actual... [15:17:26] (03CR) 10Ilias Sarantopoulos: [C: 03+1] article-descriptions: update descartes tag to fix prediction bug [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/981427 (https://phabricator.wikimedia.org/T352750) (owner: 10Kevin Bazira) [15:19:18] (03CR) 10Kevin Bazira: [C: 03+2] "Thanks for the review :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/981427 (https://phabricator.wikimedia.org/T352750) (owner: 10Kevin Bazira) [15:20:05] (03Merged) 10jenkins-bot: article-descriptions: update descartes tag to fix prediction bug [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/981427 (https://phabricator.wikimedia.org/T352750) (owner: 10Kevin Bazira) [15:53:26] isaranto: I tested the patch with api-ro. it works with no issue! [15:55:32] isaranto: and I checked the existing kserve logs, I found three other cases like zh-yue [15:55:59] gsw -> als [15:56:09] sgs -> bat-smg [15:56:17] nan -> zh-min-nan [15:56:37] but the first two don't redirect to the new domain, instead they redirect to an incubator page [15:56:41] https://incubator.wikimedia.org/wiki/Wp/gsw?goto=mainpage [15:56:46] https://incubator.wikimedia.org/wiki/Wp/sgs?goto=mainpage [15:57:50] interesting,seems like we have to go through them all and find if we can do sth [15:57:57] nice work aiko ! [16:00:23] yeah maybe there are more [16:01:06] :D [16:04:32] for above 3 cases, adding them to the config file should solve the issue [16:16:11] on it! I was confused with nan! as in not-a-number [16:19:56] (03PS3) 10Ilias Sarantopoulos: revertrisk: add config for mw host rewrites [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/981541 (https://phabricator.wikimedia.org/T352958) [16:20:29] done! I'll open a follow up task (or we can do it in the context of this task) to check all the codes [16:22:29] I can get some revids like I did for the revscoring manual tests and run a revertrisk request for each one [16:26:36] What is api-ro? [16:27:40] 10Machine-Learning-Team: Investigate prediction bug in article-descriptions model-server - https://phabricator.wikimedia.org/T352750 (10kevinbazira) Thanks to @Isaac who provided a fix to this bug [[ https://github.com/epfl-dlab/descartes/commit/593c7c467115820dfb564826430b62d13301c0e9 | upstream ]], which we ha... [16:32:48] chrisalbon: api-ro.discovery.wmnet, the internal discovery endpoint we use for mediawiki api within WMF network [16:35:34] 10Machine-Learning-Team: Investigate prediction bug in article-descriptions model-server - https://phabricator.wikimedia.org/T352750 (10Isaac) thanks for narrowing down the cause to something that was easy to diagnose/fix and deploying the patch! [16:40:18] isaranto: haha, btw zh-min-nan is basically taiwanese [16:43:46] (03CR) 10AikoChou: [C: 03+1] "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/981541 (https://phabricator.wikimedia.org/T352958) (owner: 10Ilias Sarantopoulos) [16:46:52] (03PS4) 10Ilias Sarantopoulos: revertrisk: add config for mw host rewrites [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/981541 (https://phabricator.wikimedia.org/T352958) [16:54:08] hello folks :) [16:54:44] just passing by, qq - Instead of https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/981541, shouldn't be sufficient to follow redirects in the code when they happen? [16:56:53] (03CR) 10Elukey: "Quick question - is the problem the fact that we receive a HTTP 30X and that causes the call to https://etc.. ? If so did we check if ther" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/981541 (https://phabricator.wikimedia.org/T352958) (owner: 10Ilias Sarantopoulos) [17:01:30] o/ elukey: I think that would be the best option [17:01:43] probably it would work, but we have to change mwapi code etc first [17:02:26] isaranto: o/ let's see if we can do it, in this way we'd avoid a specific map etc.. for these use cases [17:02:50] it shouldn't change much, but every time we need to add a use case (find the current ones, and add them in the future as we discover troubles) [17:03:14] yeah you're right, anyway aint gonna push the above [17:03:34] yep yep sorry didn't mean to intrude, feel free to go if the others are ok :) [17:03:47] I was just curious [17:03:57] no no thanks for the input, always valuable <3 [17:04:58] <3 [17:05:31] I was checking https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html [17:05:34] 192GB [17:05:34] lol [17:06:12] also TIL PCIe 5.0 [17:07:52] ahhaha wow 304 Compute Units [17:08:35] there is another card with 128GB, they will cost a fortune [17:09:22] wow [17:09:51] ran a quick check and it seems that redirects are allowed by default in aiohttp https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession.request [17:10:16] also explicitly setting it in session.request didnt change anything https://github.com/mediawiki-utilities/python-mwapi/blame/master/mwapi/async_session.py#L85 [17:11:59] anyway I ran a quick test to try it we can follow up and tackle this in a better fashion. an issue could be that the redirect changes to https (not 100% I need to properly debug) [17:12:35] I'm stepping away for the weekend folks. cu on Monday! [17:13:43] yeah it may be aiohttp tries the redirect with the new endpoint [17:13:53] curious, let's check it on monday :) [17:13:58] have a good weekend! [17:14:57] ok so I see - in the aiohttp's session we can set allow_redirects=False, but then we'll need to check ourselves if it is a redirect [17:15:02] and call the right endpoint [17:15:15] (may be a good candidate for a shared function) [17:17:55] that is a good idea at least to start with debugging (manually following redirects)! [17:26:22] heading out as well. have a great weekend, everybody \o [17:48:09] elukey: I didn't think of that, ty!! I think it's worth a try. I would look into that