[06:46:32] good morning :) [07:31:04] (03CR) 10Elukey: "There are some dependency upgrades that looks strange, it seems that we are adding:" [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/761015 (https://phabricator.wikimedia.org/T300195) (owner: 10Halfak) [07:33:50] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10elukey) >>! In T300195#7694465, @Halfak wrot... [07:34:20] left some comments for Aaron, I see new dependencies being added and celery's version changed, it looks a little strange [10:27:08] \o [10:27:19] o/ [10:27:24] https://twitter.com/JanelleCShane/status/1227262111769751554 <- We should have a service like this. For, uh... reasons. [10:28:19] mmmm [11:09:57] * elukey lunch break [15:55:14] o/ [15:59:22] elukey: thanks for looking at the wheels repo, some of the reverts do seem strange [16:04:15] o/ [16:04:19] very weird indeed [16:06:00] Got a link to one? [16:06:09] Maybe an ignorant viewpoint can help :) [16:08:46] klausman: https://gerrit.wikimedia.org/r/c/research/ores/wheels/+/761015/ [16:09:00] also i love the neural candy hearts! [16:09:40] They are delightfully bizarre. The person who tweeted them also wrote a book called "You are an object and I love you" about the weird things voming out of ML models. [16:10:01] Sorry, "You look like a thing and I love you" [16:10:11] https://www.janelleshane.com/book-you-look-like-a-thing [16:14:52] elukey: I think the celery 4.4.1->4.4.7 bump is either a) accidental, and a consequence of "I want 4.4.x, with x=latest" or b) using the same logic, but intentionally. 5.x.y has a quite different API. [16:15:22] (anf 4.4.7 is well over a year old, so it's unlikely that it has (many) breakages) [16:18:24] klausman: sure but we are changing too may things at once, the scope of the deployment is getting bigger and bigger every time [16:18:32] ^^^ [16:18:40] I agree that it shouldn't be folded into that rollback [16:18:45] klausman: also, this is new from today :( https://phabricator.wikimedia.org/T300180#7697143 [16:18:59] Arguably, it isn't even one unless we exactly undo the update that is being rolled back [16:19:13] there are also new wheels added [16:19:29] Yeah, no, that is not a good idea. [16:19:32] the current state is a mess, I asked for a clean rollback but Aaron preferred another way [16:20:00] I'm with you on the clean rollback [16:20:14] agreed, there no reason to do anything else at this point [16:20:18] the updates/additions should be done separately, with some testing and soaking, if possible [16:22:28] (03CR) 10Elukey: "See also this comment for celery's version: https://phabricator.wikimedia.org/T300180#7697810 :(" [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/761015 (https://phabricator.wikimedia.org/T300195) (owner: 10Halfak) [16:25:25] I moved https://phabricator.wikimedia.org/T300270 to active tasks [16:25:28] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Return meaningful HTTP responses in Lift Wing's revscoring backends - https://phabricator.wikimedia.org/T300270 (10elukey) [16:25:30] cc kevinbazira, accraze --^ [16:25:41] I think it is needed for the MVP, what do you all think? [16:25:46] (so we can groom it) [16:26:07] agreed, it should be fairly straight-forward to implement [16:26:12] I agree that we should throw 400s or 500s instead of Python backtraces [16:26:55] 500 is currently a catch-all, and it doesn't provide much info [16:27:23] Agreed, I meant generally speaking. This specific one should probably be a literal 400 (Bad Request) [16:28:43] _maybe_ 418 ;) [16:29:40] 🫖 [16:33:28] Thinking about it, an invariant like that ("These fields need to be in the request JSON") probably should be a generic library call that generates a 400 if anything is missing. [16:37:42] yeah it really depends how granular we want to get with the status codes [16:38:15] also wondering how these will work with the api-gateway [16:38:33] I tend to favor useful errors (as in messages, not necessarily codes) for client errors and be a bit more opaque for server-side errors [16:38:53] there is also the question mark about input sanitization [16:38:59] I would hope that the GW just forwards them as-is, with maybe a few fields added, but none overwrittent [16:39:12] Sanitization by the GW? [16:40:31] sanitization by gw would be hard because all the models will take in different raw data [16:40:51] the transformers should have some defensive code in my opinion [16:40:58] i suppose we can do sanitization at the transformer and/or predictor level [16:41:03] yes exactly [16:41:26] in the preprocess method [16:41:43] Agreed, I think defensive code should be in LW, not the GW. Well, for request-level stuff. Defending against HTTP(S)-level stuff can be generic and in the GW [17:14:55] 10Lift-Wing, 10Machine-Learning-Team: Custom enwiki-goodfaith Explainer - https://phabricator.wikimedia.org/T301378 (10ACraze) [17:15:42] (03PS1) 10Accraze: [WIP]editquality: adding explainer aixserver [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/761401 (https://phabricator.wikimedia.org/T301378) [17:16:28] "aixserver" gives me IBM AIX flashbacks. And not good ones :) [17:19:53] lolol [17:21:12] ibm may be trying to rehab the "aix" term, their research team put together an AI eXplainability toolkit called aix360: http://aix360.mybluemix.net/ [17:23:52] ^ i think Hal (privacy engineer) might have actually talked to these folks recently about model cards [17:24:16] I guess most of the old farts that care about AIX-the-OS are not the target market, so confusion is unlikely [17:45:00] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Custom enwiki-goodfaith Explainer - https://phabricator.wikimedia.org/T301378 (10ACraze) [17:53:56] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Custom enwiki-goodfaith Explainer - https://phabricator.wikimedia.org/T301378 (10ACraze) I managed to get an explainer attached to the enwiki-goodfaith Inference Service on ml-sandbox and was able to retrieve an explanation from the `:explain` endpoin... [18:04:03] (03PS3) 10Halfak: Reverts back to python 3.5 wheels and includes mwparserfromhell 0.6.3 [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/761015 (https://phabricator.wikimedia.org/T300195) [18:09:29] (03PS4) 10Halfak: Reverts back to python 3.5 wheels and includes mwparserfromhell 0.6.3 [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/761015 (https://phabricator.wikimedia.org/T300195) [18:09:59] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) OK updates made. I'm not sure why... [18:42:33] hello [18:42:56] I think someone from ML should be on the RACI for MediaWiki Event Carried State Transfer [18:43:02] who could it be [18:43:06] accraze: you perhaps?! [18:43:22] https://docs.google.com/document/d/1uGDTExRXFw-w51F8WkmQNZwP_pCZxiu6jZv3ZPKlvdM/edit# [18:43:46] we are working on the Decision Record (^) and I need to consult with folks who might be interested in the solution [18:43:55] i think ML is, especially since we put ORES scores into streams now [18:44:07] and scores partically are calculated from content grabbed from the MW API, [18:44:13] and we are probably going to put content into streams [18:47:02] ottomata: my understanding is that the scores on streams now are the ones that mediawiki trigger via async jobqueue right? [18:48:27] yes [18:48:36] not jobqueue, just change-prop IIRC [18:48:41] but it has always been very hacky [18:48:52] a special piece of code in change-prop ittself does it, iirc [18:49:20] ottomata: mmm wait, that piece of code afaics does only cache warm up [18:49:33] do we have scores for all edits of all wikis? [18:49:41] (well or a subset of wikis) [18:50:35] yeah, elukey all that code does is call the ORES precache API url and get the scores for the revision, and then produce the new event to kafka [18:50:54] it would be better if ORES or something produced the event :p [18:51:14] ottomata: what is the topic though? The only one that I found was the mediawiki-job related [18:51:31] mediawiki.revision-score [18:51:38] (eqiad|codfw). [18:51:46] its in eventstreams too [18:51:57] https://stream.wikimedia.org/v2/ui/#/?streams=mediawiki.revision-score [18:54:45] but how many people consume those topics? [18:54:51] this is definitely a discussion to make [19:01:37] Andy is going to comment in a bit, we were discussing this over the team meeting (perfect timing) [19:01:52] too much tribal knowledge about ORES :D [19:02:01] need to go now, thanks for the ping! [19:18:58] ottomata: i wasn't aware of the scores being in streams now but happy to discuss more on monday [19:19:23] gr8 stuff [19:19:37] but yes as elukey, said we'll probably want to figure out how many people are consuming those topics [19:22:30] ottomata: I confused it with the mediawiki.job.ORESFetchScoreJob (that is from the job queue) [19:24:31] aye [19:39:46] o/ is there anyone who knows whether this file (cached english wikipedia edits + vandalism labels + ores features) is available somewhere on the WMF infrastructure or can be sent to me (https://github.com/wikimedia/editquality/blob/master/Makefile#L872)? We've been talking a bit about vandalism models in the research team and I'm curious to see the data [19:44:58] isaacj: i can send it to you, i recently had to generate it again to use while exploring some explainability stuff [19:45:23] oh awesome -- much appreciated accraze !! [19:50:42] isaacj: sent to yr email, lemme know if you have any issues [19:55:32] great, received and downloaded. if I run into any issues, I'll let you know. many thanks again and if we learn anything, we'll share back! [20:02:03] 10Machine-Learning-Team, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install ml-serve100[5-8] - https://phabricator.wikimedia.org/T294949 (10Jclark-ctr) [20:56:05] 10Lift-Wing, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Migrate editquality models - https://phabricator.wikimedia.org/T301409 (10ACraze) [20:56:42] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10revscoring, 10Machine-Learning-Team (Active Tasks): Create migration plan for editquality models from ORES to Lift Wing - https://phabricator.wikimedia.org/T284689 (10ACraze) [20:56:44] 10Lift-Wing, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Migrate editquality models - https://phabricator.wikimedia.org/T301409 (10ACraze) [21:07:19] 10Lift-Wing, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Revert editquality isvc architecture to predictor-only - https://phabricator.wikimedia.org/T301412 (10ACraze) [21:23:44] 10Lift-Wing, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Upload model binaries to storage - https://phabricator.wikimedia.org/T301413 (10ACraze) [21:38:32] 10Lift-Wing, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Add editquality isvc configurations to ml-services helmfile - https://phabricator.wikimedia.org/T301415 (10ACraze) [21:46:58] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Migrate editquality models - https://phabricator.wikimedia.org/T301409 (10ACraze) p:05Triage→03High [22:44:53] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Factor out feature retrieve functionality to a transformer - https://phabricator.wikimedia.org/T294419 (10ACraze) To recap from discussion at the team technical meeting today: * revscoring transformers for editquality/draftquality/topics are very h... [23:15:53] 10Machine-Learning-Team, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install ml-cache100[1-3] - https://phabricator.wikimedia.org/T299435 (10Jclark-ctr)