[06:32:19] 10Machine-Learning-Team: Store and fetch the recommendation-api embedding from Swift - https://phabricator.wikimedia.org/T343576 (10kevinbazira) a:03kevinbazira [06:36:14] 10Machine-Learning-Team: Store and fetch the recommendation-api embedding from Swift - https://phabricator.wikimedia.org/T343576 (10kevinbazira) The recommendation-api embedding was uploaded successfully to [[ https://wikitech.wikimedia.org/wiki/Thanos | Thanos Swift ]]. Here is its storage URI: `s3://wmf-ml-mo... [07:49:51] hello folks [07:49:59] it seems that we have a problem in cswiki too [07:58:28] created https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/946510/ [08:09:48] deploying to mediawiki now [08:25:32] Morning [08:25:39] Same problem as fiwiki? [08:26:07] ahyes [08:27:34] yep [08:31:53] ok done, it will take a bit for cache to catch up [08:52:03] I have not a clear idea what's happening though [08:52:47] I thought a bit about it over the weekend, but in my mental model I can't find a spot for this to go wrong. I am clearly not understanding how MW extensions work (or not well enough). [08:53:21] what are your doubts? [08:53:38] So we know that LW gives the right answers, and that they arrive in the right place. [08:53:49] So it becomes an UI issue, of sorts. [08:54:03] (or maybe I'm already off track at this point?) [08:54:34] IIRC, we also ruled out that the thresholds, as the UI uses them, are correct. [08:54:44] er we ruled out that they are _wrong_ [08:55:57] it should be a UI issue since the ores_classification table in the mariadb instances seems good (at least afaics) [08:56:32] yes, that matches my understanding as well [08:58:57] one really important thing that Aiko noticed though, on fiwiki, is that when we had lift wing turned on, the UI showed a "very likely bad intention" filter [08:59:07] that was basically tagging everything as bad faith [08:59:12] but now it is not there [08:59:35] we have only "May be malicious" and "probably malicious" [09:00:57] That sounds like the change not only touched where the score is coming from, but its interpretation as well, or we added add'l queries. Which is kinda weird. Maybe the config inheritance (or the defaults) broke somehow. If I didn't know better, I'd say it's YAML breakage [09:01:45] the config that Ilias added is stored in mediawiki-config: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/922512/8/wmf-config/ext-ORES.php [09:02:23] wgOresModelVersions and wgOresModelThresholds are fetched from the ORES API when lift wing is not enabled [09:03:39] Have we made 1000% sure that the config added in that CL is identical to what ORES emits? [09:10:30] not 100% sure, I think that Ilias fetched it from the ores api and added it to the wmf-config repo [09:10:38] Aiko checked and it seems the same [09:10:45] at least for fiwiki [09:12:22] https://phabricator.wikimedia.org/T343308#9062850 [09:13:33] and the leading theory is https://phabricator.wikimedia.org/T343308#9064198 [09:15:23] While that hints at a fix, I wonder why it didn't break with ORES (vs LW), unless the threshold-as-used is somehow different [09:15:34] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Patch-For-Review, 10User-notice: Deployment of Lift Wing usage to all wikis that use ores extension - https://phabricator.wikimedia.org/T342115 (10elukey) >>! In T342115#9070976, @Quiddity wrote: > @elukey Thanks! I've added it to https://meta.wikimed... [09:22:11] one thing that I don't get is the mediawiki-config's setting wgOresFiltersThresholds [09:22:23] for example, cswiki has [09:22:23] 'goodfaith' => [ [09:22:23] // likelygood, maybebad, likelybad use defaults [09:22:24] 'verylikelybad' => [ 'min' => 0, 'max' => 'maximum recall @ precision >= 0.98' ], [09:22:26] ], [09:22:40] so there is a verylikelybad [09:22:54] same thing for fiwiki [09:24:25] Ilias didn't change it, nor he added code for it [09:24:34] so it should be common to both ORES and Lift Wing [09:25:00] it is parsed by ThresholdParser.php in the ores extension [09:25:07] and the code wasn't changed [09:27:10] I wonder if we could make the extension dump to the logs what thesholds it actually uses [09:31:24] At least that way we'd have some floating-point numbers to grep for [09:36:15] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) >>! In T343308#9071507, @matej_suchanek wrote: > On [[ https://cs.wikipedia.org/wiki/Speci%C3%A1ln%C3%AD:Posledn%C3%AD_zm%C4%9Bny | cswiki ]], I'm also observing an unus... [09:38:58] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) >>! In T343308#9063805, @achou wrote: > The RC filters now only show "maybe malicious" and "probably malicious" which is correct. Don't see "very likely malicious" anym... [09:45:46] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) From the ORES access logs, this is the exact URI used to fetch thresholds: * [[ https://ores.wikimedia.org/v3/scores/fiwiki/?models=goodfaith&model_info=statistics.thre... [09:46:06] klausman: I fetched from the ores access logs the exact URIs used for cswiki and fiwiki [09:52:11] klausman: the only big difference that I see is that in the ores API "null" is the first element of the array [09:52:32] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10matej_suchanek) >>! In T343308#9073088, @elukey wrote: >>>! In T343308#9071507, @matej_suchanek wrote: >> On [[ https://cs.wikipedia.org/wiki/Speci%C3%A1ln%C3%AD:Posledn%C3%AD_z... [09:54:26] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) cswiki: ORES API: ` {'cswiki': {'models': {'goodfaith': {'statistics': {'thresholds': {'false': [{'!f1': 0.989,... [09:55:03] same thing for cswiki, the false array is not in the same order [09:56:49] elukey: wow good catch! [09:57:05] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) @achou maybe it is nothing but the "false" array elements order is not the same in fiwiki and cswiki: * in fiwiki, `null` is the first in the ORES API, meanwhile it is... [09:57:05] aiko: hello! [09:57:12] I posted a question for you in the task :) [10:00:27] now if this is the issue, we need to check all settings in the wmf-config repo [10:01:26] elukey: morning! [10:01:48] elukey: I wonder how NULL is interpreted there. [10:02:02] order does matter. I think that's the cause of the issue, so it also affects other wikis. [10:02:03] \o also, morning aiko :) [10:04:25] elukey: is it possible to know the URIs for all other wikis from the ORES access logs? [10:05:09] klausman: morning! :) [10:06:43] aiko: in theory yes, but we should look back in time since those calls are not made anymore for most of the wikis (since we have enabled lift wing) [10:07:15] I was able to check cswiki and fiwiki since they are currently making requests to ores (I used the logstash dashboard) [10:08:00] I see [10:08:32] there is some logic in the extension about how to calculate the URI for ores though, if we replicate it in python it should be fine [10:08:51] then we parse its result with what in ext-ORES.php's thresholds [10:08:59] finding mismatches [10:09:07] IF this is the issue of course [10:09:52] Ilias must have used a script to generate the config [10:13:34] elukey: this https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ORES/+/refs/heads/master/includes/Storage/ThresholdLookup.php#170, right? [10:14:04] aiko: nono I mean to generate the config in ext-ORES.php [10:14:16] that one reads it IIUC [10:16:28] so I checked itwiki's config, and the order is fine [10:16:34] fr wiki is not for example [10:17:36] yeah in fr wiki I see a ton of revs in RC [10:17:37] https://fr.wikipedia.org/w/index.php?goodfaith=verylikelybad&hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=7&title=Sp%C3%A9cial:Modifications_r%C3%A9centes&urlversion=2 [10:19:08] Sao do I understand correctly: for the config to work as it should, its first value should be null, and then an ordered list of threshold configs? [10:19:34] klausman: nono IIUC the values in the array matters, and we should preserve it [10:19:40] in the lift wing hardcoded config I mean [10:19:57] my suspicion is that changing their order leads to the wrong thresholds in the wrong categories [10:20:10] That's what I meant ;) [10:20:29] But where should the null value be? at the end? beginning? entirely absent? [10:20:32] but the null should be a placeholder [10:20:49] in cswiki they don't use t for example [10:20:55] see https://phabricator.wikimedia.org/T343308#9073102 [10:21:05] but the order is wrong [10:21:14] for goodfaith -> false I mena [10:21:16] *mean [10:21:25] aiko: am I totally crazy or some of this makes sense? [10:23:03] I wonder why the code doesn't use (the equivalent to a Python dictionary) but seems to rely on indexing into an array. Seems a lot less robust. [10:26:13] elukey: in the config in ext-ORES.php, the order is all following 0: maybe bad faith -> 1: likely bad faith -> 2: very likely bad faith, and in Ilias's code for parsing the config, it assumes to get the order. [10:26:42] but the urls you saw from ORES API don't follow that. That was unexpected for me. [10:28:33] I still have a lot of doubts, like why we don't see "very likely bad faith" in fiwiki now that it uses the ores thresholds [10:29:20] aiko: the order above is for the "false" use case? [10:30:53] elukey: we don't see "very likely bad faith" because the threshold is null [10:32:43] elukey: yeah for the goodfaith false use case [10:37:39] aiko: ok but "null" is the first slot from the ORES API [10:37:52] or better, the first element of the array [10:38:27] you mentioned 0 -> maybe bad faith [10:39:16] (I am talking about the fiwiki use case) [10:43:56] the order I mentioned is what ext-ORES.php uses [10:44:23] I'm thinking in ORES, the order does not matter because it has a variable for mapping categories to the corresponding thresholds. I see a $levelName => $config in the code. But in lift wing ver, we don't use that [10:46:07] aiko: I am checking fetchThresholdsFromApi in ThresholdLookup.php, in the ores extension [10:46:21] this is that we use if we don't turn on lift wing, afaics [10:46:46] the code makes a request to ores (the one that we see in the access logs) with this->oresService->request etc.. [10:47:22] then it must use the result in some order to create all the categories (very likely etc..) [10:49:10] ahhh wait, that levelName must be what we define in wgOresFiltersThresholds no? [10:49:31] like // likelygood, maybebad, likelybad use defaults [10:49:32] 'verylikelybad' => [ 'min' => 0, 'max' => 'maximum recall @ precision >= 0.9' ], [10:51:15] then at line 196 or ThresholdLookup.php there is an array_merge op [10:51:36] that probably merges model-thresholds and filter-thresholds [10:52:36] ending up in what RC filters expose [10:53:06] ThresholdLookupConfig.php uses the same array_merge code [10:53:25] (since it overrides fetchThresholdsFromApi to fetch data from the config rather than from the ores api) [10:55:43] (you can tell me that I am totally off, this is a brainbounce) [10:59:46] the result of the lookup seems to be handled by ChangeListHooksHandler.php [11:00:01] that probably run when we ask for the RC filters [11:00:15] at line 211 there is a function called handleGoodFaith [11:00:41] that registers the new filters [11:00:44] at least, IIUC [11:02:02] that function uses $thresholdLookup->getThresholds( 'goodfaith' ), and if I am right it is where the result of the lookup is loaded [11:02:08] and then later on misintepreted [11:11:57] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) My theory, that is probably really wrong since I am not great in PHP: * In the ORES extension, the `ThresholdLookup` class is used to fetch model thresholds from the O... [11:12:02] summarized all in --^ [11:13:39] Amir1: o/ [11:14:00] when you have a moment do you mind to read https://phabricator.wikimedia.org/T343308#9073434 and let me know if I am totally crazy/off or not? [11:15:47] I suspect that more wikis are broken, we didn't get reports yet [11:16:35] (for example https://fr.wikipedia.org/w/index.php?goodfaith=verylikelybad&hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=7&title=Sp%C3%A9cial:Modifications_r%C3%A9centes&urlversion=2, it looks wrong to me) [11:16:44] going afk for lunch, ttl! [11:18:32] elukey: o/ in ThresholdLookup.php, the code makes a request to ores with this->oresService->request etc.. and the result is saved to $data. But in ThresholdLookupConfig.php, data is gotten from ext-ORES with this->mainConfig->get( 'OresModelThresholds' ), which should be equivalent to $data, but we see they do not match. that's the problem, right? [11:22:37] aiko: exactly yes, this is what I suspect [11:22:49] because the code for the array_merge aftewards is the same [11:24:30] please double check if it makes sense, it is just a hunch, I get lost in the PHP code [11:29:36] * klausman lunch as well [11:32:29] elukey: sorry I was in a meeting, let me read [11:35:02] yeah my guess is that the data is not matching [11:42:24] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10Ladsgroup) the json representation of the cswiki's config is this: `lang=json {"goodfaith":{"statistics":{"thresholds":{"false":[{"!f1":0.966,"!precision":0.998,"!recall":0.935,... [13:11:49] (03PS1) 10Kevin Bazira: Fetch embedding from Swift [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) [13:18:02] Amir1: ack thanks for the check, at this point I'd be in favor of reverting all wikis to use ORES [13:18:15] I noticed that fr wiki seems affected, and many more probably [13:18:57] elukey: I can try to write a script to update it based on ores api responses [13:19:07] would that help? :P [13:19:18] Good morning all [13:19:21] Amir! [13:19:26] morning! [13:19:36] (I actually woke up like two hours ago) [13:20:22] Ha [13:20:25] Amir1: ahahah no don't worry, Ilias will probably check when he is back from holidays, I don't like the idea of having a half broken status now [13:20:42] filing the change to mw-config [13:20:43] why are you so polite [13:20:50] I'd do it later today [13:21:26] polite is just how Luca rolls :) [13:21:50] (about to enter a four-hour meeting marathon, wish me luck) [13:21:56] otherwise I would have done it sooner [13:22:06] I took a quick look at the two json blobs on the latest comment, it seems like the three blocks in the "false" key are in reverse order [13:22:33] well, not reverse, but definitely wrong [13:23:10] The !f1 attribute in the api is, in order, 0.989, 0.966, 0.991. In the config it's 0.966, 0.991, 0.989 [13:23:33] yes what we were discussing earlier on [13:23:37] The other sub-attrs look the same, but I only spot-checked them [13:23:38] the order in the array matters [13:23:53] How they got jumbled, I don't know. [13:23:56] Amir1: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/946546/ ready to go :) [13:24:27] chrisalbon_: o/ we need to revert the changes to the ores extension [13:24:42] there is a minor issue but we are impacting some wikis now [13:24:46] Yeah [13:24:57] I am just catching up [13:25:55] gotta run a quick errand, bbiab [13:27:17] Amir1: I disagree with the -1 :) we need to restore the previous behavior, and then we can take a couple of days to make sure we fix all the thresholds correctly. If you file another change somebody (like me) will need to review it, and check thresholds etc.. [13:27:22] so it will take a bit [13:27:39] let's restore the prev behavior [13:28:41] sure [13:29:27] Amir1: and I say this with all the wikilove possible for your work, I just don't want you to start coding like crazy to fix OUR bugs :) [13:30:03] yeah I know [13:30:05] no worries [13:30:48] so the config looks good right? [13:30:53] if so I'll proceed [13:36:57] thanks for the review :) [13:37:13] I'll wait for folks to complete mw deployments and I'll roll out the revert [13:48:09] (03CR) 10Elukey: "Left some comments :)" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [13:56:15] Thanks all [14:00:14] rollback completed [14:00:42] 10Machine-Learning-Team, 10Patch-For-Review: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10achou) @Ladsgroup What I don't understand is how the order for the ORES API response is defined. It seems that for fiwiki and cswiki, the order is ["very l... [14:00:56] elukey: thanks Luca! [14:01:43] :) [14:01:44] 10Machine-Learning-Team, 10Patch-For-Review: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) For all users: we rolledback the extension to its previous behavior, so we can investigate what is the bug that caused this to various wikis. Sorry... [14:01:49] I'll also post to wikitech-l [14:06:39] 10Machine-Learning-Team, 10Patch-For-Review: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10Ladsgroup) >>! In T343308#9073999, @achou wrote: > @Ladsgroup What I don't understand is how the order for the ORES API response is defined. It seems that... [14:07:37] aiko: did I answer your question? [14:09:43] Amir1: the cswiki example that you added should be like https://phabricator.wikimedia.org/T343308#9073102 IIUC - the data overall is the same, but the order of the false array is different [14:10:21] order shouldn't matter here at all [14:10:36] unless there is a bug making it pick the first one instead of max [14:12:02] Amir1: see https://phabricator.wikimedia.org/P50161 [14:12:23] the !f1 values are the same, but in different orders [14:12:49] (and the rest afaics) [14:14:43] my theory is that it gets combined with wgOresFiltersThresholds in a different way [14:14:55] if the "false" array is not the same [14:16:06] I'll debug this [14:16:27] it is fine if you don't have time, we can take care of it [14:16:38] at some point you'll need to get freed by the ORES curse [14:16:40] :D [14:17:39] very very likely there is a bug in ores ext [14:17:44] I can take a look [14:47:35] (03PS2) 10Kevin Bazira: Fetch embedding from Swift [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) [14:52:01] (03CR) 10Kevin Bazira: Fetch embedding from Swift (035 comments) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [14:57:57] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Patch-For-Review, 10User-notice: Deployment of Lift Wing usage to all wikis that use ores extension - https://phabricator.wikimedia.org/T342115 (10JJMC89) I've pulled the Tech News entry per T343308#9074000. [15:23:25] klausman: I filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/946569 [15:23:44] it seems a simplification to the current proxy path setup [15:23:46] checking [15:24:10] I checked with "kubectl get sks etc.." and the activators are always in Proxy mode [15:24:19] this is probably why they are throttled [15:24:31] in theory they should Proxy only when there are surges of traffic [15:24:43] keeping the values up to date with our traffic may become a nightmare [15:24:47] right. but the passthorough still incurred resource usage [15:25:01] LGTM [15:25:21] thanks :) [15:25:50] did you check https://knative.dev/docs/serving/load-balancing/target-burst-capacity/ etc..? [15:25:56] just to verify what I said :) [15:26:27] Not right now, but I looked at it last week when we were discussing throttling [15:26:35] But I can mnake 101% sure now :) [15:27:19] it is an invasive change, we should check carefully etc. now that we have real traffic [15:27:23] From what you describe it sounds than even with positive values, it was in the serving path? That's kinda contradictory to the docs. But we should try it [15:27:56] Ah, it's ambiguous [15:28:06] there are all the use cases in https://knative.dev/docs/serving/load-balancing/target-burst-capacity/#setting-the-target-burst-capacity [15:28:13] would cap=2 mean that _at 2,_ the proxy is in the path? [15:28:34] Either way, worth a try to see if things improve/ [15:28:49] no there is a formula for the target burst capacity [15:28:56] I think it is 211 or similar now [15:30:30] *nod* I stand by my LGTM :) [15:33:03] (03CR) 10Elukey: Fetch embedding from Swift (032 comments) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [15:36:38] ok I see in staging and codfw that the revision move to "Serve" [15:42:22] 10Machine-Learning-Team, 10Item Quality Evaluator, 10Wikidata, 10Wikidata Dev Team, 10Wikidata.org: Update API calls from ORES to Lift Wing - https://phabricator.wikimedia.org/T343731 (10Arian_Bozorg) [15:48:17] all updated [15:51:20] so results are great https://grafana-rw.wikimedia.org/d/Q1HD5X3Vk/elukey-k8s-throttling?forceLogin&from=now-3h&orgId=1&to=now&var-container=activator&var-dc=thanos&var-ignore_container_regex=&var-prometheus=k8s-mlserve&var-service=knative-serving&var-site=eqiad&var-sum_by=container&var-sum_by=pod [15:51:28] but the decrease happened BEFORE the deployment [15:51:41] and I think it is due to my previous rollback of the mw traffic [15:51:48] (so the activators do way less now) [15:52:03] so, we'll see when traffic ramps up again :D [15:52:17] but now ingress -> kserve pods should happen without activator in the middle [16:11:53] (03PS3) 10Kevin Bazira: Fetch embedding from Swift [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) [16:19:47] (03CR) 10Kevin Bazira: Fetch embedding from Swift (032 comments) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [16:35:21] 10Machine-Learning-Team: Review HTTP 500 reported by articletopic-outlink's transformer for wikisource.org - https://phabricator.wikimedia.org/T343740 (10elukey) [16:40:06] aiko: o/ around by any chance? [16:40:13] rough day, thanks all for the hard work [16:40:59] <3 [16:42:54] 10Machine-Learning-Team: Review HTTP 500 reported by articletopic-outlink's transformer for wikisource.org/wikibase.org - https://phabricator.wikimedia.org/T343740 (10elukey) [16:46:41] 10Machine-Learning-Team: Review HTTP 500 reported by articletopic-outlink's transformer for wikisource.org - https://phabricator.wikimedia.org/T343740 (10elukey) [16:48:03] (03PS1) 10Daimona Eaytoy: Avoid DB access in non-Database tests [extensions/ORES] - 10https://gerrit.wikimedia.org/r/946592 [16:52:27] I filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/946593 since we have some 500s for outlink, but low prioritu [16:52:31] *priority [16:52:35] we can merge tomorrow [16:52:58] Is anything literally on fire right now? [16:57:10] nono few 500s to change prop [16:57:15] very low priority [16:57:26] but I check logstash every now and then to see what's wrong :D [16:57:54] Then might I suggest a nice beer, maybe a workout, hanging with the family, dinner, some amazing trashy netflix movie. It will all be here tomorrow. [16:59:35] oh yes I was about to say "have a nice rest of the day" :) [16:59:41] o/ [17:07:00] Night Luca! [17:19:36] (03CR) 10Umherirrender: [C: 03+2] Avoid DB access in non-Database tests [extensions/ORES] - 10https://gerrit.wikimedia.org/r/946592 (owner: 10Daimona Eaytoy) [17:41:55] (03Merged) 10jenkins-bot: Avoid DB access in non-Database tests [extensions/ORES] - 10https://gerrit.wikimedia.org/r/946592 (owner: 10Daimona Eaytoy) [19:52:15] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10Ladsgroup) The thresholds are not wrong but what represent is wrong: ` ladsgroup@mwmaint1002:~$ mwscript shell.php --wiki=fiwiki Psy Shell v0.11.10 (PHP 7.4.33 — cli) by Justin... [20:08:41] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10Ladsgroup) I think I found out what's wrong. Now trying to see how can we fix it. It doesn't look easy to be honest.