[06:07:24] hi folks! [06:07:36] I am deploying new limits for knative again, still seeing throttling [06:12:25] will do the same for istio, there is some throttling too [06:43:20] still seeing some throttling in knative - https://grafana-rw.wikimedia.org/d/Q1HD5X3Vk/elukey-k8s-throttling?forceLogin&orgId=1&var-dc=thanos&var-site=codfw&var-prometheus=k8s-mlserve&var-sum_by=container&var-sum_by=pod&var-service=knative-serving&var-ignore_container_regex=&from=now-1h&to=now [06:43:31] but 0ms of throttling in istio gateways, that is good [06:44:07] ah no zero in codfw, few ms in eqiad [07:52:55] running some errands, bbiab! [08:25:10] back :) [09:16:21] https://grafana.wikimedia.org/d/Q1HD5X3Vk/elukey-k8s-throttling?forceLogin&orgId=1&var-dc=thanos&var-ignore_container_regex=&var-prometheus=k8s-mlserve&var-service=revscoring-editquality-goodfaith&var-site=eqiad&var-sum_by=container&var-sum_by=pod&from=now-12h&to=now [09:16:31] very nice set of throttling happening to goodfaith model servers [09:16:46] not sure if it is pods starting up after autoscale [09:16:49] or something other [09:19:20] aiko: o/ +2ed your change :) [09:19:48] for https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate/Administration#eventgate-wikimedia_schema_repository_change we'll also need to rollout a new version of eventgate-main's docker images (sigh) [09:26:08] elukey: thanks! I'll prepare a patch for eventgate-wikimedia [09:33:40] here https://gerrit.wikimedia.org/r/c/eventgate-wikimedia/+/944872 [09:38:58] aiko: there is also another commit going out, seems not a big issue but I added more people to the change [09:39:07] when we get +1s we'll deploy [09:52:35] okee [10:28:34] hey folks we have a issue [10:29:11] fiwiki's folks opened a thread in Village Pump saying that RC shows only "likely bad edit" flags for all the reviews [10:29:20] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10taavi) [10:29:27] this means that something is off with our dear extension [10:29:38] task is the above one [10:31:55] yeah: https://fi.wikipedia.org/wiki/Toiminnot:Tuoreet_muutokset?goodfaith=verylikelybad&hidebots=1&hidecategorization=1&hideWikibase=1&limit=500&days=30&goodfaith__verylikelybad_color=c5&urlversion=2 [10:33:41] so in itwiki this doesn't happen https://it.wikipedia.org/w/index.php?title=Speciale:UltimeModifiche&damaging=all&days=7&goodfaith=all&hideWikibase=1&hidebots=1&hidecategorization=1&limit=50&urlversion=2 [10:33:49] it shouldn't be widespread [10:35:50] If only fiwiki has this problem, maybe it's about thresholds..? will look into this later [10:36:28] aiko: I think it may also be itwiki, see [10:36:28] https://it.wikipedia.org/w/index.php?damaging=maybebad%3Blikelybad%3Bverylikelybad&goodfaith=maybebad%3Blikelybad%3Bverylikelybad&hidebots=1&hidecategorization=1&hideWikibase=1&limit=500&days=7&damaging__maybebad_color=c5&damaging__likelybad_color=c5&damaging__verylikelybad_color=c5&goodfaith__maybebad_color=c5&goodfaith__likelybad_color=c5&goodfaith__verylikelybad_color=c5&title=Speciale:UltimeModifi [10:36:34] che&urlversion=2 [10:36:37] sorry long link [10:36:51] I added the red highlight [10:38:41] ahh no so we should check how many wiki got affected [10:38:55] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) If I got it correctly this is happening to `itwiki` as well: https://it.wikipedia.org/w/index.php?damaging=maybebad%3Bverylikelybad&hidebots=1&hidecategorization=1&hide... [10:40:57] I am using https://www.mediawiki.org/wiki/Extension:ORES/Debugging to check on dbs (via quarry) [10:43:00] kevinbazira, klausman - o/ [10:43:18] we have a bit of an issue in here, do you have time to check/help? [10:43:23] elukey o/ [10:44:38] what's the issue? [10:45:05] kevinbazira: see the task above [10:48:09] I am a bit ignorant about the RC filters, so we should figure out if the issue is real and if it is also present in other wikis [10:58:14] elukey: I got confused about the highlights as well [10:58:24] checking zhwiki [11:01:17] it looks fine https://zh.wikipedia.org/zh-tw/Special:%E6%9C%80%E8%BF%91%E6%9B%B4%E6%94%B9?damaging=all&goodfaith=all&hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=7&damaging__likelybad_color=c3&goodfaith__verylikelybad_color=c5&urlversion=2 [11:04:16] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) Tried to analyze: https://fi.wikipedia.org/w/index.php?title=Andranik_Ozanjan&curid=1761901&diff=21724224&oldid=21724211 Picked `diff=21724224` as `rev-id` and ran the... [11:04:46] aiko: how did you manage to add the highlights without seeing only bad edits ? [11:05:43] anyway, I added an entry in the task [11:05:53] a good edit is marked as bad for some reason [11:05:56] maybe the thresholds? [11:06:15] elukey: I followed the conversation on https://fi.wikipedia.org/wiki/Wikipedia:Kahvihuone_(tekniikka)#Tuoreissa_muutoksissa_vain_pahantahtoisia_muokkauksia [11:06:25] and it looks like there are fiwiki bots that are yet to be migrated. These bots are https://phabricator.wikimedia.org/T343010 and https://phabricator.wikimedia.org/T343011 [11:07:33] kevinbazira: how does it impact on the extension's results though? [11:12:05] Amir1: o/ [11:12:44] so IIUC, to recap: [11:12:54] what's up? [11:13:33] hello :) If you have time, we have some fiwiki community reports about RCfilters going all red https://phabricator.wikimedia.org/T343308 [11:14:07] so far I didn't see anything weird with the scores themselves, but I admit my ignorance in the extension.. could it be threshold-related? [11:16:01] yeah, that looks like a threshold issue [11:16:30] can you double check what thresholds was set for fiwiki in ores legacy and what is the config? [11:17:13] Amir1: in ores-legacy? [11:17:17] I'm confused about this patch https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ORES/+/932289 adding default config for thresholds.. [11:17:36] ah ok you meant "old/current ores" [11:17:45] I got confused, we also have "ores-legacy" on k8s :D [11:17:52] ha, sorry yeah [11:17:58] aiko: that's not the value in production [11:18:47] ahh ok where I can find the value in prod [11:19:12] one sec [11:19:45] https://noc.wikimedia.org/wiki.php?wiki=fiwiki#wgOresModelThresholds [11:19:56] h/t to Timo for building this awesome tool [11:20:34] I see https://ores.wikimedia.org/v3/scores/fiwiki/?models=damaging&model_info=statistics.thresholds.true.%22maximum+recall+@+precision+%3E=+0.9%22 [11:21:19] so IIUC we use a single set of thresholds for all wikis, and this is biting us right? [11:23:16] Are we? [11:23:19] I think we use different thresholds for different wikis, isn't it? [11:23:32] I think they are all different [11:24:02] mmm and where do we store the in the config though? [11:24:14] I see only a single set (ignorant question) [11:24:17] I mean in Lift WIng [11:24:46] ah no wait sorry, nevermind, I checked the noc link [11:24:49] pebcak [11:30:50] elukey: sorry, was out for lunch, looking now. Is the issue still present? [11:32:23] yep [11:34:05] so far it seems limited to fiwiki [11:39:27] just to understand, what are we currently doing? [11:39:33] let's sync in here folks [11:39:47] I'm trying to figure out if there were recent changes to fiwiki's config [11:40:50] ack, aiko / kevinbazira ? [11:41:16] looking at https://noc.wikimedia.org/wiki.php?wiki=fiwiki#wgOresModelThresholds [11:43:58] I feel something is weird. goodfaith-> statistics-> thresholds-> true is null and under #wgOresFiltersThresholds, we only set rules for goodfaith-> verylikelybad [11:45:10] if we see itwiki https://noc.wikimedia.org/wiki.php?wiki=itwiki#wgOresModelThresholds there are more rules and no null [11:46:26] aiko: wow good catch! [11:46:36] let's check what we have in ores, probably we need some update [11:46:48] isaranto: you added most of the wmf-config/ext-ORES.php config in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/922512 --- where did the funbers (e.g. recall etc) come from? [11:47:03] klausman: Ilias is on holiday :) [11:47:07] dang [11:48:20] but https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/922512/8/wmf-config/ext-ORES.php#2331 is interesting [11:48:40] for goodfaith, at line 2414 we set "0" [11:48:49] and at 2446 we reset 0 to null [11:48:53] could it be the issue? [11:49:25] aiko: --^ [11:50:11] It does sound internally inconsistent, yes. [11:50:43] Gerrit's choic of indent formatting doesn't make it easy to read. I'll make a git clone and see if I can compare fiwiki with sewiki [11:51:30] no I misread, we only have 0 => null for fiwiki [11:52:06] what do we have in ores now? [11:54:08] https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/58c8718a9221b856493bfc1aadfac9d837a43d8e/wmf-config/ext-ORES.php#230 ok so here it says other rules use default [11:55:06] What is the meaning of the 0/1/2 subsections in the config? [11:57:03] aiko: I think that we should find the thresholds that ores uses now [11:57:16] E.g. in enwiki, 'goodfaith' => [ 'statistics' => [ 'thresholds' => [ 'false' has sections 0, 1, 2. fiwiki onluy had 0 and 1, with 2 being assigned null (does that mean use defaults?) [11:57:50] elukey: https://ores.wikimedia.org/v3/scores/fiwiki/?models=goodfaith&model_info=statistics.thresholds.true.%22maximum+recall+@+precision+%3E=+0.9%22 [11:58:03] https://ores.wikimedia.org/v3/scores/fiwiki/?models=goodfaith&model_info=statistics.thresholds.true [11:58:04] the 'true' subsection of enwiki has 0, and some config values in it, where as the true subsection of fiwiki has 0 assigned to null [11:58:57] elukey: depends on a rule given like "maximum+recall+@+precision+>=+0.9", it returns thresholds [11:59:34] ah, you think maybe for fiwiki, the extension should use the tuning functionality Ilias mentioned? (Not sure I understood his explanation at the time) [12:00:48] another problem: default is empty? https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/refs/changes/12/922512/8/wmf-config/ext-ORES.php#115 [12:01:52] klausman: 0/1/2 should be the level of precision you want when filtering rev-ids in RC (very likely bad, likely bad, etc..) [12:03:10] (maybe not sure) [12:03:17] ah, makes sense [12:03:21] aiko: interesting, we have other wikis with "damaging uses defaults for everything" [12:03:32] ca/es/fi [12:03:36] Did we maybe lose the default settings in the migration? [12:03:49] if we see the same issue in there maybe it is the culprit [12:04:01] i.e. they were there "somehow" for the old setup but disappeared and now wikis that rely on the defaults are misconfigured? [12:06:23] elukey: yeah very confusing now what defaults they are using [12:07:10] Amir1: qq - we see in https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/refs/changes/12/922512/8/wmf-config/ext-ORES.php#115 that "default" is null, and some configs later on refers to "xxx uses the defaults" [12:07:11] look like defaults has always been empty [12:08:29] hmm, that's weird [12:08:54] those are the "filters" threshold though [12:08:57] ah, it's the extension default I think [12:09:06] good point! [12:10:40] they seem in extension.json [12:11:23] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ORES/+/refs/heads/master/extension.json#196 [12:11:26] aiko: --^ [12:12:20] elukey: thanks! [12:12:51] folks I have to go in a bit for family errands, will be back in ~2/3 hours (not in time for meetings) [12:12:59] please keep the task updated :) [12:13:01] ttl! [12:15:05] worst case, if we don't find a solution, is to revert fiwiki's settings to use ores and not lift wing [12:19:39] ack [12:27:09] checked the fiwiki thresholds for goodfaith, seems the same that ores uses [13:16:42] RC filters docs, first time seeing it https://www.mediawiki.org/wiki/Help:New_filters_for_edit_review/Quality_and_Intent_Filters [14:04:09] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10achou) The thresholds used in ores-extension: **wgOresModelThresholds** (https://noc.wikimedia.org/wiki.php?wiki=fiwiki#wgOresModelThresholds) ` "goodfaith": {... [14:04:12] will be a few mins late for the meeting [14:12:16] and I'm theonly one? [14:12:32] ah, Slack. [14:23:25] back! [14:25:41] aiko: nice anaylisis :) [14:30:01] created https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/944916/ [14:30:08] so worst case scenario we revert [14:32:30] so, to recap: [14:33:01] 1) the rev-ids are scored correctly (namely the values saved on the db are consistent with what we expect) [14:33:16] 2) the threshold are ok (namely equals to the ones in ores) [14:34:01] 3) we don't see the same issue on other wikis (this is not 100% clear to me yet) [14:38:21] what could be wrong then? [14:40:25] One thing I don't understand is that the threshold for "very likely bad" is null, so iiuc the filter shouldn't be shown in the RC filters, just like "likely good faith". but instead we see "maybe bad" and "very likely bad" filters, and all edits are classified as "very likely bad" [14:44:31] ok so let's revert [14:44:47] +1 [14:44:56] Amir1: o/ if you have a moment https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/944916 [14:45:07] so we can revert and check how it goes [14:45:14] that community is being impacted [14:45:45] elukey: I +1'ed, wanna deploy it yourself? [14:46:05] do "scap backport 944916" on deploy1002 [14:46:10] new stuff, fancy [14:46:12] Amir1: I haven't deployed mw config in a while, if you give me some infos more than happy [14:46:15] ahhh [14:46:18] wow okok [14:46:28] thanks :) [14:47:56] Amir1: and not git pull etc.. needed right? [14:48:09] nope, not even +2 is needed [14:48:14] ah snap [14:48:32] there you go, first mistake, I +2ed [14:48:49] haha, it's fine, I do it as well [14:48:53] it's noop [14:49:02] deploying :) [14:52:09] Amir1: live on mwdebug, but I guess I can proceed (can't think a way to check) [14:52:49] (went forward) [14:53:21] yeah [14:57:53] I think that we should send an email to wikitech-l explaining that most of wikis use Lift Wing [14:58:00] so people know to contact us etc.. [14:58:57] backport done, thanks Amir1 :) [14:59:23] that's a good point [15:01:10] so I checked https://fi.wikipedia.org/wiki/Toiminnot:Tuoreet_muutokset?goodfaith=verylikelybad&hidebots=1&hidecategorization=1&hideWikibase=1&limit=500&days=30&goodfaith__verylikelybad_color=c5&urlversion=2 [15:01:19] and I still see a lot of red [15:03:05] yeah looks like the same [15:03:50] maybe new edits (i.e. the scoring is messed up), or some caching [15:04:04] elukey: maybe also tech news? [15:05:03] Amir1: ignorant about it, is it a mailing list? [15:05:16] https://meta.wikimedia.org/wiki/Tech/News [15:05:27] just add user-notice tag to the ticket [15:07:11] super TIL [15:09:19] wow [15:11:02] aiko: the fi page above still looks red sigh [15:11:41] it is weird that even new edits are red [15:11:42] mmmm [15:11:50] maybe cached content as A*mir said [15:13:58] yep with curl I see x-cache-status: hit-front [15:18:35] purging the URL doesn't really help [15:19:30] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) We reverted the change for `fiwiki` but the page in the description is still showing red flags, even for new changes. Tried to purge the URL from the CDN but didn't work... [15:21:01] aiko: do you mind to test https://fi.wikipedia.org/wiki/Toiminnot:Tuoreet_muutokset?goodfaith=verylikelybad&hidebots=1&hidecategorization=1&hideWikibase=1&limit=500&days=30&goodfaith__verylikelybad_color=c5&urlversion=2 ? [15:21:09] do you see all red as well? Even for new rev-ids? [15:21:24] let me see [15:21:43] kevinbazira, klausman - we are in a little bit of a urgent situation, could you please help? [15:22:34] yes all red, new rev-ids as well [15:23:58] wondering what it could be then [15:25:56] elukey: I've trying to find a way to understand what is fundamentally worng, but I honestly am a bit stuck. is the config as seen on NOC wrong? AIUI, Aiko established that the raw numbers for fiwiki are the same as before, but still, a score that used to be "passing" now isn't? [15:26:57] klausman: sure but we need to report in here, we already rolled back to ORES for fiwiki and we still see the same issue [15:27:10] If that is the case, there must be additional interpretation of the returned verdict, but I don't know the extension's inner workings well enough [15:27:24] same thing for Aiko and me :) [15:27:32] but three brains work better than two [15:29:51] Let's count mine at 0.5 ;) [15:31:08] the translated filter is "Very likely malicious" [15:31:16] is it goodfaith or damaging? [15:31:42] Have we tried moving fi to an older version of damaging? Many wikis use 0.5.0 (though some like en and hu use 0.5.1) [15:31:52] same for goodfaith [15:32:21] (I also don't know much work that change+rollout would be, so might not be worth that shot in the dark) [15:32:37] nono it was working before, let's not change ORES [15:34:12] ok so from the translation is "user intent" [15:34:12] elukey: it is goodfaith [15:34:15] so it must be goodfaith [15:34:19] aiko: :) [15:34:28] ack [15:34:38] elukey: I think now we have problem in goodfaith, damaging is fine [15:34:43] ok so the good news is that damaging filters work fine [15:34:56] aiko: I didn't check damaging before, was it working fine?? [15:35:00] I mean with Lift Wing [15:35:33] it was working fine [15:37:42] so https://fi.wikipedia.org/w/index.php?title=HK_Almaty&curid=773378&diff=21725284&oldid=21648025 is marked by the filter, but I checked with quarry and in the ores_classification table we have the correct value for goodfaith [15:37:49] and this is definitely a new revision that uses ores [15:38:13] so the DB values are fine [15:39:38] but the filter/extension flags it as malicious [15:40:14] I'm currently trying to understand how the extension works internally. Man, my PHP is rusty [15:41:23] klausman: the extension's biggest work is to enqueue an event when a new edit happens, that in turn calls ores and populates the ores_classification table (mariadb) [15:41:29] all asyncronously [15:41:42] then the RC Filter is just pulling from the DB [15:41:48] where do the extensions logs end up? logstash? [15:41:58] in theory yes [15:42:06] there are mediawiki ones [15:43:55] it looks like no matter the goodfaith score is, they got all highlighted as malicious [15:45:50] Which to me sounds very UI-side rather than something Liftwing broke. But we ruled out Extension config, so it's quite puzzling how this broke [15:46:35] elukey: I am trying to find any kind of log line from the extension, no luck so far. [15:47:33] maybe something is still cached [15:47:59] you mentioned that the extension writes to the DB and the UI part only reads from there. [15:48:05] maybe those tables are messy? [15:48:27] It would not mark new revisions as bad, though [15:49:38] the values on the db are good [15:50:43] makes sense, since the API values we provide seem to be as well. [15:55:39] reading https://www.mediawiki.org/wiki/ORES/Thresholds and trying to understand [15:55:44] Can we think of anyone we haven't asked yet to provide some fresh insight? [15:56:10] not really [15:56:32] I hope that if it is caching it eventually expire and auto-resolve [15:56:47] I'd prefer to know what that caching thing is. [15:57:50] this is the same feeling that we all have [16:08:18] aiko, klausman seems working now [16:08:33] yep [16:08:53] so it was definitely caching [16:08:58] Well, wait-out-the-cache seems to have worked. How frustrating. [16:09:14] I finished to deploy one hour ago [16:09:47] Naturally, we still don't know what broke in the first place. Do you think that maybe if we _hand't_ rolled back it would have "fixed" itself in a similar way? SOunds unlikely to me, but I wanted to mention it. [16:09:56] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) It seems to be working now, we'll keep this task open to figure out what went wrong. [16:10:43] klausman: no I don't think so since users reported it broken yesterday [16:11:06] yeah, then it definitely wasn't a one-hour cache issue. [16:11:23] I just hope we can find an off-mainline repro. [16:11:23] finally 0.0 [16:12:37] I rechecked itwiki and it seems working fine to me [16:13:39] so in theory it shouldn't be widespread, but we need to alert folks [16:14:12] Maybe via Tech/NEws again? [16:14:20] (not sure if we already worte there) [16:17:37] (03PS3) 10Ladsgroup: Add flag for host header [extensions/ORES] - 10https://gerrit.wikimedia.org/r/938267 (https://phabricator.wikimedia.org/T319170) (owner: 10Ilias Sarantopoulos) [16:17:42] (03CR) 10Ladsgroup: [C: 03+2] Add flag for host header [extensions/ORES] - 10https://gerrit.wikimedia.org/r/938267 (https://phabricator.wikimedia.org/T319170) (owner: 10Ilias Sarantopoulos) [16:21:50] aiko: elukey https://ores.wikimedia.org/v3/scores/fiwiki/?models=damaging&model_info= this is what ores gives, maybe you can compare goodfaith with the config in noc? [16:22:09] the RC filters now only show "maybe malicious" and "probably malicious" which is correct. don't see "very likely malicious" anymore [16:22:32] it's I think an issue in threshold config [16:23:18] good point aiko [16:23:25] can you write that into the task please? [16:23:37] Amir1: definitely yeah we'll follow up [16:23:44] I am sending an email to wikitech-l now [16:23:52] will do [16:25:00] done! [16:25:11] ok I think it is enough for today, have a good rest of the day folks! [16:25:15] thanks for the help Amir1 [16:25:42] <3 [16:25:50] \o And yes, thanks, Amir! [16:25:51] thanks Amir!!! [16:26:15] (03Merged) 10jenkins-bot: Add flag for host header [extensions/ORES] - 10https://gerrit.wikimedia.org/r/938267 (https://phabricator.wikimedia.org/T319170) (owner: 10Ilias Sarantopoulos) [16:27:14] bye luca :) [16:30:54] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10achou) The RC filters now only show "maybe malicious" and "probably malicious" which is correct. Don't see "very likely malicious" anymore. [16:48:18] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Patch-For-Review, 10User-notice: Deployment of Lift Wing usage to all wikis that use ores extension - https://phabricator.wikimedia.org/T342115 (10JJMC89) draft based on the wikitech-l post: ORES is now using a new backend service, Lift Wing, on all w... [18:08:13] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team: Create a single table with evaluation metrics from all trained add-a-link models - https://phabricator.wikimedia.org/T343374 (10kevinbazira) [18:10:31] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10achou) I have an assumption about what is causing this problem, but I need to verify it by examining the code further. My assumption is that the new setting has somehow messed u... [18:11:20] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team: Create a single table with evaluation metrics from all trained add-a-link models - https://phabricator.wikimedia.org/T343374 (10kevinbazira) Below is a disaggregated report that shows evaluation metrics for all models trained in 18 batches: |**Batch**|**Wik... [18:13:36] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team: Completion report on training 18 rounds of add-a-link models - https://phabricator.wikimedia.org/T336927 (10kevinbazira) The Research team requested for a single table that shows evaluation metrics for all trained wikis and we shared in T343374#9064194