[05:29:18] morning! [05:29:42] Thanks for the deployment Luca! I also deployed it now on ml-serve (both codfw and eqiad) [06:07:40] deployed api gateway changes as well. Langid model is reachable via [06:07:40] ``` [06:07:40] curl https://api.wikimedia.org/service/lw/inference/v1/models/langid:predict -X POST -d '{"text": "Some sample text in any language that we want to identify"}' -H "Content-type: application/json" [06:07:40] ``` [06:21:26] 10Lift-Wing, 10Machine-Learning-Team, 10I18n, 10NewFunctionality-Worktype, 10Patch-For-Review: Create a language detection service in LiftWing - https://phabricator.wikimedia.org/T340507 (10isarantopoulos) New model is deployed on Lift Wing and can be accessed through the API Gateway Example request: ` c... [07:08:43] kalimera! [07:09:43] isaranto: for the api-gw changes, please coordinate first with Hugh or ServiceOps before proceeding [07:09:55] usually it is fine to do it self service, but better safe than sorry [07:10:06] and in case something is off, they can check [07:10:34] Ok, noted! Will do next time [07:10:35] also, check https://grafana.wikimedia.org/d/UOH-5IDMz/api-and-rest-gateway?orgId=1&refresh=30s afterwards for 20/30 mins [07:10:43] eqiad and codfw, just to be sure [07:11:03] I mean you did it right and perfectly, it is just me being over cautious [07:11:14] those are services not 100% self service etc.. [07:15:31] isaranto: I am going to file a change for the SLO dashboard [07:16:03] yeah , I understand, you're right [07:35:30] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Data-Engineering-Icebox, 10Event-Platform, 10Platform Team Initiatives (Modern Event Platform (TEC2)): ORES hook integration with EventBus - https://phabricator.wikimedia.org/T201869 (10Aklapper) [07:52:50] afk - commuting! [08:08:01] SLO for langid: https://gerrit.wikimedia.org/r/c/operations/grafana-grizzly/+/966486 [08:17:33] (thanks for the review Tobias) [08:17:37] going to merge and publish [08:20:12] ta daan https://grafana.wikimedia.org/d/slo-Lift_Wing_LangId/lift-wing-langid-slo-s?orgId=1 [08:20:37] 10Lift-Wing, 10Machine-Learning-Team, 10I18n, 10NewFunctionality-Worktype, 10Patch-For-Review: Create a language detection service in LiftWing - https://phabricator.wikimedia.org/T340507 (10elukey) And https://grafana.wikimedia.org/d/slo-Lift_Wing_LangId/lift-wing-langid-slo-s?orgId=1 :) [08:39:23] Nice! [08:41:28] I am seeing holes in the metris though, in various of our SLO dashboards [08:41:49] I think it may be the recording rule, going to open a task to observability [08:49:50] 10Machine-Learning-Team, 10observability: Some Istio recording rules may be missing data - https://phabricator.wikimedia.org/T349072 (10elukey) [08:49:54] created --^ [08:57:10] interesting --^ [09:03:13] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: kserve CORS error - https://phabricator.wikimedia.org/T348511 (10isarantopoulos) I've verified that the above works. I tested the request mentioned on the PR through a browser console and returns a proper response: ` $.ajax({ url: 'https://api.wikim... [09:26:18] 10Lift-Wing, 10Machine-Learning-Team, 10I18n, 10NewFunctionality-Worktype, 10Patch-For-Review: Create a language detection service in LiftWing - https://phabricator.wikimedia.org/T340507 (10santhosh) Thank you @isarantopoulos and @elukey ! [09:48:40] isaranto: did we set some autoscaling for langid? I am wondering if we should allow say 3/4 pods, to scale up in case people try the service [09:50:10] no, we haven't added any autoscaling to start with, since it requires 2Gi of ram [09:50:43] but yes I think we should add, but I want to run a load test and [09:50:47] *first [09:50:49] +1 okok [10:01:57] 10Machine-Learning-Team: Upgrade the readability model server to KServe 0.11.1 - https://phabricator.wikimedia.org/T348664 (10elukey) a:03achou [10:04:41] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Documentation: Update docs for ORES Extension - https://phabricator.wikimedia.org/T346761 (10elukey) This task was in in-progress status without an owner, moving it back to unsorted to re-triage. [10:52:01] * elukey lunch! [10:59:43] (03CR) 10Ilias Sarantopoulos: "Nice work! I left some suggestions to clean things up a bit." [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/965142 (https://phabricator.wikimedia.org/T348607) (owner: 10Kevin Bazira) [11:00:28] * isaranto lunch as well! [11:01:22] and same :) [12:11:34] (03PS7) 10Kevin Bazira: Use envoy proxy to access endpoints external to k8s/LiftWing [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/965142 (https://phabricator.wikimedia.org/T348607) [12:14:48] (03CR) 10Kevin Bazira: Use envoy proxy to access endpoints external to k8s/LiftWing (036 comments) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/965142 (https://phabricator.wikimedia.org/T348607) (owner: 10Kevin Bazira) [12:48:15] (03CR) 10Ilias Sarantopoulos: "Left 2 last comments and we're good to go" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/965142 (https://phabricator.wikimedia.org/T348607) (owner: 10Kevin Bazira) [13:11:30] (03PS8) 10Kevin Bazira: Use envoy proxy to access endpoints external to k8s/LiftWing [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/965142 (https://phabricator.wikimedia.org/T348607) [13:12:02] (03CR) 10CI reject: [V: 04-1] Use envoy proxy to access endpoints external to k8s/LiftWing [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/965142 (https://phabricator.wikimedia.org/T348607) (owner: 10Kevin Bazira) [13:14:15] (03PS9) 10Kevin Bazira: Use envoy proxy to access endpoints external to k8s/LiftWing [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/965142 (https://phabricator.wikimedia.org/T348607) [13:15:23] (03CR) 10Kevin Bazira: Use envoy proxy to access endpoints external to k8s/LiftWing (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/965142 (https://phabricator.wikimedia.org/T348607) (owner: 10Kevin Bazira) [13:23:34] Hey all. I’ll be out sick today. But just checking in. All good? [13:23:42] all good :) [13:33:22] hope you feel better soon Chris! [13:35:08] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "LGTM!" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/965142 (https://phabricator.wikimedia.org/T348607) (owner: 10Kevin Bazira) [13:37:07] (03CR) 10Kevin Bazira: [C: 03+2] "Thank you all for reviews :)" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/965142 (https://phabricator.wikimedia.org/T348607) (owner: 10Kevin Bazira) [13:37:39] (03Merged) 10jenkins-bot: Use envoy proxy to access endpoints external to k8s/LiftWing [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/965142 (https://phabricator.wikimedia.org/T348607) (owner: 10Kevin Bazira) [13:48:46] (03PS1) 10Elukey: Add precommit support [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542 [13:48:48] (03PS1) 10Elukey: Fix pre-commit errors and bump version [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543 [13:50:47] (03CR) 10CI reject: [V: 04-1] Fix pre-commit errors and bump version [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543 (owner: 10Elukey) [13:54:10] isaranto: o/ I'd need some of your time to fix --^ [13:54:28] I don't recall exactly how to run tests with our tox setup, and what CI calls [13:54:38] I'll take a look after our meeting! [13:58:43] Will be 2-3' late for the meeting! [14:01:41] me too sorry [14:01:47] just finished another meeting [14:19:19] 10Machine-Learning-Team, 10Section-Level-Image-Suggestions, 10Patch-For-Review, 10Structured-Data-Backlog (Current Work): [XL] Productionize section alignment model training - https://phabricator.wikimedia.org/T325316 (10Ottomata) > I think so? It seems like there was hesitation about the line that failed.... [14:25:35] 10Machine-Learning-Team, 10observability: Some Istio recording rules may be missing data - https://phabricator.wikimedia.org/T349072 (10elukey) a:03elukey [14:43:21] 10Lift-Wing, 10Machine-Learning-Team, 10I18n, 10NewFunctionality-Worktype, 10Patch-For-Review: Create a language detection service in LiftWing - https://phabricator.wikimedia.org/T340507 (10elukey) a:05santhosh→03isarantopoulos [15:05:05] * elukey bbiab [15:09:13] 10Machine-Learning-Team, 10Add-Link, 10Chinese-Sites, 10Growth-Team (Sprint 1 (Growth Team)), 10User-notice: Deploy "add a link" to 14th round of wikis - https://phabricator.wikimedia.org/T308139 (10KStoller-WMF) [15:09:15] I'm checking the above patch related to ci [15:11:11] 10Machine-Learning-Team, 10Section-Level-Image-Suggestions, 10Patch-For-Review, 10Structured-Data-Backlog (Current Work): [XL] Productionize section alignment model training - https://phabricator.wikimedia.org/T325316 (10xcollazo) >>! In T325316#9258075, @Ottomata wrote: >> I think so? It seems like there... [15:24:59] isaranto: I'd need some info about CI config, I don't really recall how it is setup [15:26:02] for example, for rec-api-ng, what is calling CI when running? Like, what specs do I need to check to see what command is executed? Is it in integration/config? [15:30:34] ack ok [15:30:48] I was checkin the errors to start with [15:31:34] yeah it is my bad, I haven't run the tests locally [15:31:49] not sure what is best, ./run_tests.sh seems ok but I'd love something tox-driven [15:31:50] what CI runs is defined in .pipeline/blubber.yaml in the test image. In this case it is the run_tests.sh script [15:32:17] ahhhh right blubber [15:32:23] ok my brain is fried [15:34:23] rec api is a monorepo so we can add the tests as a last step in the pre-commit [15:40:37] makes sense, working on it [15:43:27] I have a suggestion I tried. Will comment on the patch [15:48:31] (03CR) 10Ilias Sarantopoulos: "I suggest we run tests every time we make a commit by removing the manual step" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542 (owner: 10Elukey) [15:50:02] if you want to run the pre-commit steps via tox in that patch run `tox -e ci` otherwise outside of tox you need to do: [15:50:02] `pip install pre-commit && pre-commit install` [15:52:36] ack! [16:16:16] there is an issue caused by the conftest.py files. They include imports from other conftest files (the structure of the repo is not good at all) so they are removed as unused imports [16:17:11] I'm logging off, I can try to help tomorrow - I was checking ruffs extend-exclude option [16:17:29] TIL new ruff release v0.1.0 19h ago! [16:18:00] with py 3.11 there is also [16:18:01] ImportError: cannot import name 'MutableMapping' from 'collections' [16:18:28] seems to need an extra import [16:18:34] anyway, tomorrow I'll keep working on it, thanks! [16:18:43] going afk as well, have a nice rest of the day folks [18:41:52] Night Luca!