[06:55:59] morning folks! [06:56:04] isaranto: merged, thanks! [06:56:45] Good morning! [06:57:09] just ran httpbb for ml-serve-eqiad, worked nicely [07:21:37] (03CR) 10Elukey: [C: 03+2] nsfw,revert-risk: update method signature for Kserve 0.10 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/894695 (https://phabricator.wikimedia.org/T329032) (owner: 10Elukey) [07:27:58] (03Merged) 10jenkins-bot: nsfw,revert-risk: update method signature for Kserve 0.10 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/894695 (https://phabricator.wikimedia.org/T329032) (owner: 10Elukey) [07:44:59] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [07:45:12] (03CR) 10Elukey: "Great work Ilias! I added some questions to better understand the logic, first quick pass :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [07:50:15] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [07:51:41] created https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/895065 to update the docker images [07:52:19] will test them in staging, that should also verify the behavior of the control plane [08:12:11] thanks for the reviews :) [08:19:46] * isaranto commuting to coworking [08:23:24] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [08:26:46] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [08:28:54] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [08:31:16] ok interesting, I wasn't able to make the nsfw model to run on localhost (curl hanged indefinitely), tried in staging and same thing [08:31:25] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [08:47:59] * isaranto is back online [08:51:12] (03PS1) 10Elukey: python: remove FIXME and refactor some exceptions [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/895132 (https://phabricator.wikimedia.org/T329032) [08:52:33] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10MoritzMuehlenhoff) [08:53:53] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10elukey) [08:59:44] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) [09:02:48] elukey: I'm looking at your review comments. will check nsfw model as soon as I'm finished [09:07:22] isaranto: np! I have an errand to do in a bit, no hurry.. it seems hanging in tensorflow, I am trying to debug it via gdb [09:10:52] https://phabricator.wikimedia.org/P45183 [09:10:57] this is the stacktrace [09:11:01] * elukey afk for some errand [09:18:41] lol [09:22:05] wow, that stack trace rivals Java BTs [09:33:23] (03CR) 10Ilias Sarantopoulos: feat: Create a migration endpoint between LiftWing/ORES (034 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [09:38:22] u left out the exception thrown though :) [09:38:44] I'm taking a look to see but not much I can do since I can't run it locally (sucks) [09:51:07] btw kserve 0.10.1 has minor updates (famous last words :D ) but also some relaxed requirements constraints (e.g. no upper bound for numpy) [10:04:18] 10Machine-Learning-Team, 10Patch-For-Review: Create ORES migration endpoint (ORES/Liftwing translation) - https://phabricator.wikimedia.org/T330414 (10isarantopoulos) [Updating the ticket with some information from the related patches] First version of this service is ready and currently under review. This in... [10:14:23] (03CR) 10Klausman: [C: 03+1] python: remove FIXME and refactor some exceptions [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/895132 (https://phabricator.wikimedia.org/T329032) (owner: 10Elukey) [10:18:56] (03CR) 10Klausman: "One minor nit: I am not super happy about the "ORES migration" naming, since it sorta implies this _isn't_ a temporary thing. Including so" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [10:27:30] (03CR) 10Ilias Sarantopoulos: feat: Create a migration endpoint between LiftWing/ORES (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [10:28:35] klausman: valid point raised! I'd like to discuss it further. [10:28:46] re: naming? [10:29:36] if we are going to make any improvements to ORES's behavior [10:29:46] (03CR) 10Klausman: feat: Create a migration endpoint between LiftWing/ORES (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [10:29:50] in the context of the discussion that partial fault=overall fault [10:30:09] Addressed: I think mimicking ORES to a reasonable degree is the best course [10:30:30] me2. but still it got me thinking [10:30:45] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10achou) Thank you for sharing your thoughts on this. They all make sense! I agree that doing Option 1 for now and having a docum... [10:32:26] (03CR) 10Ilias Sarantopoulos: feat: Create a migration endpoint between LiftWing/ORES (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [10:32:33] totally agree about naming as well [10:42:34] Unfortunately I don't have a good, succinct name handy :) [10:43:36] * elukey back [10:43:52] (03PS15) 10Ilias Sarantopoulos: feat: Create a migration endpoint between LiftWing/ORES [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) [10:46:45] (03CR) 10Ilias Sarantopoulos: feat: Create a migration endpoint between LiftWing/ORES (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [10:47:14] klausman: I used the name I had in the code for now `ORES/LiftWing calls legacy service` [10:48:18] Sounds good! [10:48:31] * elukey back [10:48:42] isaranto: there is not exception, nsfw hangs :( [10:48:56] aa ok [10:48:57] it seems inside tensorflow, not sure why (yet) [10:49:11] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10achou) [10:50:01] I tried to run it locally by upgrading tensorflow but caught up in some uvicorn errors 💫 [10:55:34] mmm I noticed /opt/lib/python/site-packages/starlette/routing.py in the back trace [10:55:50] I removed the startlette constraint in requirements.txt since it was causing issues [10:55:57] I am wondering if that is the problem [10:59:03] (03CR) 10Elukey: feat: Create a migration endpoint between LiftWing/ORES (033 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [11:00:29] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Create new mediawiki.page_links_change schema based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399 (10Ottomata) [11:01:04] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Create new mediawiki.page_links_change schema based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399 (10Ottomata) [11:03:19] 10Machine-Learning-Team: Upgrade ML clusters to Kubernetes 1.23 - https://phabricator.wikimedia.org/T324542 (10JMeybohm) [11:03:24] 10Machine-Learning-Team: Upgrade the ml-serve-eqiad cluster to k8s 1.23 - https://phabricator.wikimedia.org/T330758 (10JMeybohm) 05Open→03Resolved This is all done [11:03:49] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10Ottomata) > having a documented plan to switch to Option 3 in the future Just created {T331399} [11:04:43] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Create new mediawiki.page_links_change schema based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399 (10Ottomata) We may want to emit this change directly from EventBus, not as from a strea... [11:13:33] 10Machine-Learning-Team, 10Patch-For-Review: Implement new mediawiki.revision-score streams with Lift Wing - https://phabricator.wikimedia.org/T328576 (10Ottomata) > Research it seems that they don't really need/care about any of the above streams Research might not, but what about the community at large? [... [11:15:29] 10Machine-Learning-Team, 10Patch-For-Review: Implement new mediawiki.revision-score streams with Lift Wing - https://phabricator.wikimedia.org/T328576 (10elukey) >>! In T328576#8672003, @Ottomata wrote: >> Research it seems that they don't really need/care about any of the above streams > > Research might no... [11:19:09] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream, 10Research: Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10Ottomata) [11:21:40] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10Ottomata) > - predicted_topics Let's bikeshed and implement the event schema for this in {T331401} [11:21:48] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10Ottomata) [11:21:52] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream, 10Research: Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10Ottomata) [11:24:54] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10aborrero) [11:25:34] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10aborrero) [11:25:48] 10Machine-Learning-Team, 10Patch-For-Review: Implement new mediawiki.revision-score streams with Lift Wing - https://phabricator.wikimedia.org/T328576 (10Ottomata) +1 ty [11:25:59] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10aborrero) Sent a ping to @Marostegui regarding clouddb[1013-1014,1021] Also @Andrew regarding cloudservices host, but I think the host... [11:32:41] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) @aborrero regarding clouddb* hosts, it is up to your team but I think it would be nice if you could depool them. Better use... [11:54:30] (03PS16) 10Ilias Sarantopoulos: feat: Create a migration endpoint between LiftWing/ORES [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) [12:08:08] (03PS17) 10Ilias Sarantopoulos: feat: Create a migration endpoint between LiftWing/ORES [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) [12:09:03] (03CR) 10Ilias Sarantopoulos: feat: Create a migration endpoint between LiftWing/ORES (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [12:17:18] 10Machine-Learning-Team, 10ORES, 10Wikimedia Enterprise: Investigate tools that use ORES - https://phabricator.wikimedia.org/T330854 (10isarantopoulos) a:03isarantopoulos [12:18:54] * isaranto is going for lunch [12:47:29] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [12:55:09] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10fnegri) @Marostegui @aborrero the patch above should depool clouddb1013 and clouddb1014. I don't think clouddb1021 can be depooled eas... [12:57:35] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [13:07:27] ok I reverted the patches for kserve 0.10, and nsfw is fine [13:14:01] so this is probably something related to http request routing [13:42:48] mmmm no if I comment the self.model.predict it works [13:44:08] what do u mean? line 59? `output = self.model.predict(preprocessed_image).tolist()` [13:44:21] yes exactly [13:44:35] it makes zero sense, in https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/894006/3/nsfw-model/model-server/requirements.txt it seems that only routing-related stuff was changed [13:44:41] then the model doesn't make a prediction 🙃 [13:45:49] yes, uvicorn and grpcio. I had (other) problems running it with updated tensorflow [13:46:25] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10cmooney) [13:46:29] also https://phabricator.wikimedia.org/P45183$12 pointed to predict() [13:46:56] no idea, do we know who built the model? [13:47:01] we could open a task to debug it [13:47:38] https://phabricator.wikimedia.org/T313526 [13:49:29] I have no idea who built it, but I agree not to spend more time in the context of kserve upgrade [13:51:22] 10Machine-Learning-Team: The nsfw model hangs in predict() after moving to Kserve 0.10 - https://phabricator.wikimedia.org/T331416 (10elukey) [13:51:28] opened a task [13:56:46] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [13:56:58] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream, 10Research: Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10achou) It seems we can remove the map field since we are now building a stream per model. Additional... [13:58:30] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10jbond) [13:58:56] nice, I continued to try but still couldnt make it run locally, I ran into this exact issue related to uvicorn https://github.com/encode/uvicorn/issues/478 [13:59:05] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10jbond) [13:59:14] nothing to check here, just pasting it in case anyone else saw/sees it [13:59:41] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10MatthewVernon) [14:00:21] https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/895132 is also pending, lemme know folks what do you prefer for the remaining exceptions [14:01:16] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 6th round of wikis - https://phabricator.wikimedia.org/T304550 (10Trizek-WMF) @Sgs, how possible is it to fix these languages? The idea is to deploy front-end next week. If we can't fix some languag... [14:02:00] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 6th round of wikis - https://phabricator.wikimedia.org/T304550 (10Trizek-WMF) a:05Trizek-WMF→03Sgs [14:08:51] I started reviewing but forgot , going at it again [14:09:54] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=f4ffc353-a529-4620-994f-ae7b737f3c7a) set by cmooney@cumin1001 for 2:0... [14:14:37] 10Machine-Learning-Team, 10Patch-For-Review: Upgrade the inference-services repo codebase to kserve 0.10 (fastapi) - https://phabricator.wikimedia.org/T329032 (10elukey) I opened T331416 for nsfw, since the model's predict function seems to hang with the new version of Kserve. Since it is an experimental model... [14:17:05] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0a07bba2-0f50-4eec-9718-0c768add34f3) set by cmooney@cumin1001 for 2:0... [14:41:58] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10jbond) [14:50:14] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10cmooney) Happy to say the upgrade went as expected, no issues encountered. All devices now back online running 21.4R3-S1.5. [14:52:26] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Andrew) the following hosts paged during this maintenance: ` NodeDown wmcs cloudvirt1023:9100 (node eqiad) NodeDown wmcs cloudvirt102... [14:54:33] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10MoritzMuehlenhoff) [14:54:57] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "Nice cleanup! 👌" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/895132 (https://phabricator.wikimedia.org/T329032) (owner: 10Elukey) [14:58:52] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [15:10:40] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10cmooney) >>! In T329073#8672931, @Andrew wrote: > the following hosts paged during this maintenance: > > > ` > NodeDown wmcs cloudvir... [15:13:18] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10MoritzMuehlenhoff) [15:27:22] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream, 10Research: Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10calbon) a:03achou [15:31:11] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [15:37:18] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream, 10Research: Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10Ottomata) Nice! Quick naming bikeshed: is `score` the best name for this field? Is that a generall... [15:39:56] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream, 10Research: Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10Ottomata) [15:39:59] 10Machine-Learning-Team, 10Patch-For-Review: Implement new mediawiki.revision-score streams with Lift Wing - https://phabricator.wikimedia.org/T328576 (10Ottomata) [15:44:38] 10Machine-Learning-Team, 10Growth-Team, 10PageTriage: Detection and flagging of articles that are AI/LLM-generated - https://phabricator.wikimedia.org/T330346 (10calbon) a:03calbon [15:47:04] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream, 10Research: Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10Isaac) > Do we like this scores field? We have the opportunity to do whatever we want here, so let's... [15:50:37] 10Lift-Wing, 10Machine-Learning-Team: Investigate Explainer for Revert-Risk model - https://phabricator.wikimedia.org/T330131 (10calbon) a:03achou [16:01:13] I completed https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing#Usage today with internal/external endpoints config [16:01:19] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream, 10Research: Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10Isaac) > is score the best name for this field? Is that a generally used term for ML predictions? I... [16:01:24] (basically what we were discussing in the meeting) [16:01:39] if you want to proof read it I'd be happy (whenever you want, not urgent) [16:06:32] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10colewhite) [16:09:22] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES: EnWiki Recent Changes Page no longer displays damaging filters - https://phabricator.wikimedia.org/T331045 (10elukey) I am totally ignorant about how the extension works, but I'll try to come up with some links. The first one is: https://github.com/wikimed... [16:11:34] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES: EnWiki Recent Changes Page no longer displays damaging filters - https://phabricator.wikimedia.org/T331045 (10calbon) update: @isarantopoulos and @elukey are investigating [16:11:48] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES: EnWiki Recent Changes Page no longer displays damaging filters - https://phabricator.wikimedia.org/T331045 (10calbon) a:03elukey [16:13:53] wait a min [16:13:54] https://en.wikipedia.org/wiki/Special:RecentChanges?hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=7&urlversion=2 [16:13:58] I see the filters now [16:18:49] (03PS18) 10Ilias Sarantopoulos: feat: Create a migration endpoint between LiftWing/ORES [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/892998 (https://phabricator.wikimedia.org/T330414) [16:18:53] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES: EnWiki Recent Changes Page no longer displays damaging filters - https://phabricator.wikimedia.org/T331045 (10elukey) So from the config: ` 'damaging' => [ 'likelygood' => [ 'min' => 0, 'max' => 'maximum recall @ precision >= 0.99' ], 'maybebad' =>... [16:19:06] isaranto: --^ It seems working now, but I am very confused :D [16:19:17] when you have time lemme know if I am crazy or not [16:19:32] but I am 99% sure it wasn't working before [16:20:43] going afk for a bit [16:20:46] tbh I don't understand a lot so lemme take a look on my own. if u want we can set up sth tomorrow morning [16:21:04] yes yes anytime! [16:21:07] The recent changes page is workng now? [16:21:17] chrisalbon: I checked and I see the filters [16:21:24] * isaranto goes afk - commuting back home [16:21:37] chrisalbon: see the screenshot in https://phabricator.wikimedia.org/T331045#8673400 [16:23:34] what a weird bug [16:29:17] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream, 10Research: Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10Ottomata) > maybe it just makes sense to have three separate schema (one for classification models,... [16:53:59] aiko: o/ [16:54:40] when you have time (tomorrow etc..) lemme know if you are ok with https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/895132 [16:54:50] or if you prefer different exceptions etc.. [17:20:01] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES: EnWiki Recent Changes Page no longer displays damaging filters - https://phabricator.wikimedia.org/T331045 (10achou) After investigating some past RecentChanges filter tasks (T144451, T149734, T149853), it appears that the damaging filter is actually the "C... [17:23:15] elukey: ahh ok I will have a look tomorrow morning first thing! [17:32:52] 10Machine-Learning-Team, 10ORES, 10Wikimedia Enterprise: Investigate tools that use ORES - https://phabricator.wikimedia.org/T330854 (10elukey) @HShaikh Hi! I had a chat with Chris Albon and he pointed me to you :) We'd like to get some info about how WME uses the ORES endpoint, so that we could set up a smo... [17:35:17] aiko: <3 [17:36:20] * elukey afk! [18:09:57] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream, 10Research: Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10diego) > And for the revert-risk model: > ` > score: > model_name: revertrisk > model_versi... [21:32:38] https://www.youtube.com/watch?v=WO2X3oZEJOA is wonderful