[05:59:54] Good morning! [06:05:10] Hi musikanimal: ! Unfortunately it doesn't! the design has been kept simple cause features for revisions are fetched from mwapi. [06:16:21] Hey team I'm working on https://phabricator.wikimedia.org/T346175 trying to figure out how the UI works :) [07:00:14] o/ [07:00:28] interesting, so the highlight is gone? [07:02:47] it seems so, although I need to verify when exactly it disappeared (probably after the enablement of LW ) [07:29:57] 10Machine-Learning-Team, 10Patch-For-Review, 10Research (FY2023-24-Research-July-September): Deploy multilingual readability model to LiftWing - https://phabricator.wikimedia.org/T334182 (10MGerlach) > @MGerlach before proceeding, do you have any plan for the model? I mean, are there any known consumers/clie... [07:59:20] 10Machine-Learning-Team, 10Patch-For-Review, 10Research (FY2023-24-Research-July-September): Deploy multilingual readability model to LiftWing - https://phabricator.wikimedia.org/T334182 (10elukey) Thanks for the info @MGerlach! In my opinion we are ok to proceed. @klausman @achou (if you agree to proceed a... [08:00:33] Running an errand. Bbl [08:02:11] * elukey errand as well [08:46:55] Morning! [08:47:17] aiko: I saw you just uploaded a patch to change 951461, was that just a rebase? [08:47:44] ah, and there's the comment :) [08:48:40] yes just added it :) [08:56:05] aiko: as far as I am concerned, the change is ready to be merged. Did my reply address your question re: max/minreplicas? [08:59:21] aiko: o/ one thing - is the autoscaling target (4) related to rps? [08:59:30] if so we need to set the annotation for it [08:59:52] otherwise the default is "concurrency" [09:02:46] good catch, I missed that [09:12:36] elukey: ahh yes, it's rps! [09:16:39] Will +2 once my browser is willing to start again [09:20:04] klausman: could you help to deploy it after the merge? [09:20:10] of course [09:20:28] thanks!! :) [09:24:36] 10Machine-Learning-Team, 10Patch-For-Review: Host the recommendation-api container on LiftWing - https://phabricator.wikimedia.org/T339890 (10kevinbazira) A 4th memory usage test was run using a combination of the 2nd (float downcasted to np.float32) and 3rd (preprocessed numpy arrays). Below are the steps tak... [09:41:25] elukey: halp. I seem to have lost my deployment skills. [09:41:44] klausman: blame good holidays! [09:41:45] :D [09:41:47] what's up? [09:42:07] I am trying to deploy the change I just merged, but helm is yelling errors at me. sec while I pastebin the output [09:43:15] same/similar output for serve-eq and staging as well [09:44:02] Config not found: /etc/kubernetes/readability-deploy-ml-serve-codfw.config [09:44:18] there is the puppet patch etc.. to merge beforehand for the new ns [09:44:35] I thought I did that a few mins ago [09:44:55] i.e. 951460 [09:45:04] did puppet run on the host? [09:45:13] running it now, I had suspected that [09:46:21] yeah, quite a bit of updates, inclduing certs. [09:46:52] which certs? [09:47:07] Info: /Stage[main]/Main/Cfssl::Cert[mlserve_staging__readability]/Cfssl::Csr[/etc/cfssl/csr/mlserve_staging__readability.csr]/File[/etc/cfssl/csr/mlserve_staging__readability.csr]: Scheduling refresh of Exec[Generate cert mlserve_staging__readability refresh] [09:47:33] also, helmfile ... diff now looks more credible :) [09:47:58] will apply/deploy in staging first [09:48:33] okok [09:48:47] did you apply the changes to admin_ng ? [09:48:51] for the new ns I mean [09:48:53] yes [09:48:56] okok [09:48:58] that went fine. [09:49:19] NAME READY STATUS RESTARTS AGE [09:49:20] readability-predictor-default-00001-deployment-6b86cb8bc6-8jm84 3/3 Running 0 42s [09:49:22] yaaay [09:49:54] Note that the service in experimental is still running. [09:50:46] it should go away when you deploy in there [09:51:29] yep, on that rn [09:51:58] and it's terminating [09:56:21] aiko: do you have acurl cmdline handy for testing if the move to non-experimental in staging worked? [09:58:30] I am unsure what the right Hots: header is [10:01:19] 10Machine-Learning-Team: Adapt the recommendation-api to use float32 preprocessed numpy arrays from swift - https://phabricator.wikimedia.org/T346218 (10kevinbazira) [10:01:30] klausman: IIRC in the inference-service's repo there is a test dir, in which Aiko is adding all wrks scripts to test the endpoints [10:01:53] ah, goody, looking [10:05:17] klausman: host header in this case should be readability.readability.wikimedia.org [10:06:04] curl -v https://inference-staging.svc.codfw.wmnet:30443/v1/models/readability:predict -H readability.readability.wikimedia.org is still giving me a 404 [10:09:16] curl -s https://inference-staging.svc.codfw.wmnet:30443/v1/models/readability:predict -H "Host: readability.readability.wikimedia.org" works :D [10:09:17] I tried: curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/readability:predict" -d '{"rev_id": 123456, "lang": "en"}' -H "Host: readability.readability.wikimedia.org" --http1.1 -k [10:09:21] it works [10:09:35] klausman: niceee [10:09:54] ok, deploying to serve-codfw [10:13:13] seems to also work there. aiko, do you want to load test it there before I deploy to eqiad or should I go ahead? [10:15:00] I think we can just go ahead! :) [10:15:07] Alright, deploying [10:17:43] elukey: have you ever gotten helm errors about gorup-readable files in /etc/kubernetes on deploy1002? [10:18:04] It's erroring out on /etc/kubernetes/readability-deploy-ml-serve-eqiad.config, but the files for codfw have the exac tsame permissions [10:21:05] do you have an example of those errors? [10:21:11] https://phabricator.wikimedia.org/P52494 [10:21:46] I think those are fine [10:22:33] But apply errors out the same way, it didn't on codfw. [10:24:30] I am an idiot :) [10:24:36] forgot to kube_env [10:24:57] ah no, that wasn't it (I _was_ surprised) [10:28:00] ah, I forgot to push the admin-ng stuff to eqiad [10:29:17] now it works.! [10:31:46] and eqiad readability is serving \o/ [10:34:43] 10Machine-Learning-Team, 10Research (FY2023-24-Research-July-September): Deploy multilingual readability model to LiftWing - https://phabricator.wikimedia.org/T334182 (10klausman) The service has been moved from the `experimental` namespace to `readability` in staging-codfw, and newly deployed to the same name... [10:34:49] and with that: lunch! [10:36:24] 10Machine-Learning-Team, 10Patch-For-Review: Host the recommendation-api container on LiftWing - https://phabricator.wikimedia.org/T339890 (10isarantopoulos) Nice work @kevinbazira! This is much better and we should proceed with this (I mean the combination of 32bit floats and preprocessed numpy arrays). I th... [11:11:54] * elukey lunch [11:13:51] Good morning all [11:17:31] Morning o/ [11:20:17] * isaranto lunch! [11:29:44] * aiko lunch as well [11:31:15] 10Machine-Learning-Team, 10Patch-For-Review: Host the recommendation-api container on LiftWing - https://phabricator.wikimedia.org/T339890 (10calbon) oh wow! Great job @kevinbazira [12:56:48] 10Machine-Learning-Team, 10Research (FY2023-24-Research-July-September): Deploy multilingual readability model to LiftWing - https://phabricator.wikimedia.org/T334182 (10leila) >>! In T334182#9122433, @elukey wrote: >>>! In T334182#9070479, @leila wrote: >> @elukey Research accepts accountability for the reada... [14:00:52] folks I am helping SRE with an issue in #operations, will join a little later sorry [14:01:40] ack [14:25:57] 10Machine-Learning-Team, 10Wikipedia-Android-App-Backlog (Android Release - FY2023-24): Migrate Machine-generated Article Descriptions from toolforge to liftwing. - https://phabricator.wikimedia.org/T343123 (10Isaac) Putting the above in context: I think we're now ready from a technical perspective to port the... [15:08:48] (03PS6) 10Kevin Bazira: Load preprocessed numpy arrays from swift [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/956846 (https://phabricator.wikimedia.org/T346218) [16:00:48] Leaving for a doc appt.cu tomorrow folks. [16:00:57] seeya, Ilias [16:03:02] Haven't made real progress fixing the bug.. if I don't have a breakthrough early morning I'll ask for help [16:34:06] o/ [16:50:04] * klausman heading out now as well, talk to 'yall tomorrow [19:38:38] anyone know offhand what the articlequality categories mean for Swedish Wikipedia? https://meta.wikimedia.org/wiki/Machine_learning_models/Production/Swedish_Wikipedia_article_quality [19:39:12] the categories are "u b r s" but I know know what those things mean or stand for [19:40:04] and the model page doesn't provide any hints or links to how sv.wiki does these things [19:42:15] I am guessing that U is Utmärkta (excellent), the equivalent of featured, so probably U B R S go in decreasing order of quality, but I'm not sure where to confirm that [19:42:44] B = Bra (good) [19:43:53] R = Rekommenderade (recommended), which is a lower tier that B, and S is all the rest... or so I infer