[02:13:21] 10Machine-Learning-Team, 10Add-Link, 10Esperanto-Sites, 10Growth-Team (Current Sprint), and 2 others: Deploy "add a link" to 7th round of wikis - https://phabricator.wikimedia.org/T304551 (10FriedrickMILBarbarossa) [02:15:41] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10Gujarati-Sites, 10User-notice-archive: Deploy "add a link" to 8th round of wikis - https://phabricator.wikimedia.org/T308133 (10FriedrickMILBarbarossa) [02:17:33] 10Machine-Learning-Team, 10Add-Link, 10Croatian-Sites, 10Growth-Team (Current Sprint), 10User-notice-archive: Deploy "add a link" to 9th round of wikis - https://phabricator.wikimedia.org/T308134 (10FriedrickMILBarbarossa) [02:31:02] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10Malayalam-Sites, 10User-notice: Deploy "add a link" to 12th round of wikis - https://phabricator.wikimedia.org/T308137 (10FriedrickMILBarbarossa) [02:35:43] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10Turkish-Sites, 10User-notice: Deploy "add a link" to 17th round of wikis - https://phabricator.wikimedia.org/T308143 (10FriedrickMILBarbarossa) [02:36:55] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10Serbian-Sites, and 2 others: Deploy "add a link" to 16th round of wikis - https://phabricator.wikimedia.org/T308142 (10FriedrickMILBarbarossa) [02:38:40] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10Serbian-Sites, and 3 others: Deploy "add a link" to 16th round of wikis - https://phabricator.wikimedia.org/T308142 (10FriedrickMILBarbarossa) [02:55:07] 10Machine-Learning-Team, 10Add-Link, 10CommRel-Specialists-Support, 10Growth-Team, 10Chinese-Sites: Support languages whose add-a-link models were not published - https://phabricator.wikimedia.org/T309263 (10FriedrickMILBarbarossa) [03:10:29] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10Chinese-Sites: Investigate `UnicodeEncodeError` thrown by Add-A-Link training pipeline for fywiki model - https://phabricator.wikimedia.org/T325521 (10FriedrickMILBarbarossa) [07:13:17] 10Machine-Learning-Team: Review HTTP 500 reported by articletopic-outlink's transformer for wikisource.org - https://phabricator.wikimedia.org/T343740 (10elukey) Deployed a fix, pending verification :) (namely in logstash no more 500s related to this issue) [07:16:01] Hi folks! [07:16:06] need to run an errand, bbl! [08:33:07] elukey: \o [08:33:17] elukey: if you want, I can take care of 2004 [08:33:38] morning :) already powercycled, in getsel I noticed [08:33:40] Multi-bit memory errors detected on a memory device at location(s) DIMM_A1. [08:33:51] oh, ugh. [08:34:28] should we get DCops involved to have the memory reseated/replaced? [08:34:31] it may be a temporary thing, or we can open a task to dcops to check [08:34:33] not sure [08:34:59] I don't remember 2004 having problems before, so maybe we'll just cycle it and see if it happens again [08:37:28] yeah could be sufficient [08:38:05] If you want to take care of a host just tell me next time, even in #operations, I thought you were afk and rebooted :) [08:41:54] I was digging around in my old mail to see if 2004 had been problematic, but it wasn't [08:45:57] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) @Ladsgroup if you want to add here you ideas we'll pick up when Ilias is back from holidays! [08:46:15] 10Machine-Learning-Team: Review HTTP 500 reported by articletopic-outlink's transformer for wikisource.org - https://phabricator.wikimedia.org/T343740 (10elukey) a:03elukey [08:56:12] (03CR) 10Elukey: Fetch embedding from Swift (033 comments) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [09:04:49] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10Ladsgroup) I actually found a rather easy solution. Let me explain. Why this is happening in the first place? Your comment in T343308#9073434 is in the right direction and just... [09:05:57] elukey: the order was the issue all along https://phabricator.wikimedia.org/T343308#9076366 [09:07:29] nice. I always like bug fixes that also get rid of complex code (or make it clearly unused) [09:10:39] my net contributions to mw core is negative (actually growing, used to -3K lines, now it's like -7K :D) [09:10:59] At the Goo there was an actual reward for the people who deleted the most code [09:11:09] niice [09:11:36] I think the top-10 was O(50k lines) [09:11:45] All of it Java :D [09:23:18] Amir1: thanks all the time that you are putting into this :) [09:25:03] Amir1: looks really great, but please stop working on it, you already have a plate full :) Ilias will work on it after getting back from holidays [09:25:43] morning o/ [09:27:47] elukey: sorry yesterday I was not around [09:28:04] elukey: outlink issue? [09:28:28] aiko: np! it was a quick question, but then I self-answered, it was basically related to wikisource.org (the bare domain) [09:28:37] I updated the istio config and it should be ok now [09:28:50] not sure if we want to score those edits as well [09:28:57] this was the question :) [09:30:35] ok ok :) outlink should only score wikipedia domain [09:32:37] sounds good! [09:48:26] <3 [10:32:33] * elukey lunch! [11:04:19] 10Machine-Learning-Team, 10ORES: Help migrate WikiLoop - https://phabricator.wikimedia.org/T342959 (10achou) Wikiloop is using the `mediawiki.revision-score` stream in https://github.com/google/wikiloop-doublecheck/blob/master/server/ingest/ores-stream.ts. I have added a comment to inform them that the `media... [11:32:40] * klausman lunch [11:36:08] (03CR) 10Kevin Bazira: Fetch embedding from Swift (032 comments) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [12:14:12] (03CR) 10Elukey: Fetch embedding from Swift (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [12:17:09] (03CR) 10Elukey: [C: 03+1] "Looks good for a test!" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [12:21:51] (03CR) 10Elukey: "Forgot one question, then it is good to go" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [12:23:45] aiko: in https://github.com/google/wikiloop-doublecheck/issues/444 we could mention the fact that they can use revert risk instead of goodfaith and damaging [12:23:56] we still don't have the stream but it should be easy to set up [12:36:28] elukey: ah I thought revert risk is going to use flink, so it will take some time [12:37:10] aiko: yeah, maybe.. if we need to migrate people we can test it for other models [12:37:18] (flink I mean) [12:37:44] or maybe we do agnostic with changeprop and multi-lingual with flink [12:38:16] yeah we could do that [12:38:17] but if people don't migrate because of a stream, we'd need to come up with a solution [12:39:39] I see. I will add that in the issue [12:48:13] updated :) [12:50:33] thanks! [12:54:14] throttling for knative improved: https://grafana-rw.wikimedia.org/d/Q1HD5X3Vk/elukey-k8s-throttling?forceLogin&from=now-1h&orgId=1&to=now&var-dc=thanos&var-ignore_container_regex=&var-prometheus=k8s-mlserve&var-service=knative-serving&var-site=eqiad&var-sum_by=container&var-sum_by=pod [12:54:20] still not optimal but better [13:11:39] (03PS4) 10Kevin Bazira: Fetch embedding from Swift [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) [13:16:47] (03PS5) 10Kevin Bazira: Fetch embedding from Swift [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) [13:19:20] (03CR) 10Kevin Bazira: Fetch embedding from Swift (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [13:20:41] elukey: it looks like after warmup (1235~1303 in the graph), things kept getting even better. [13:21:18] ~30ms throttling rate for the webhook seems to be the max now [13:21:51] but we'll have to see how it looks over 24h, with traffic patterns (and then later with more load on the services) [13:30:24] 10Machine-Learning-Team, 10Data Engineering and Event Platform Team, 10Data-Engineering, 10Wikimedia Enterprise, 10Event-Platform: Implement PoC Event-Driven Data Pipeline for Revert Risk Model Scores using Event Platform Capabilities - https://phabricator.wikimedia.org/T338792 (10JArguello-WMF) [13:30:28] (03CR) 10Elukey: "Left a comment, also I just realized that candidate_finder.py is already there, you told me in the previous comments but for some reason I" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [13:42:43] klausman: https://medium.com/directeam/kubernetes-resources-under-the-hood-part-3-6ee7d6015965 is interesting, even if the author has a strong opinion [13:54:10] I'll give it a read [13:59:07] Ah, Tim Hockin. I used to work with him :D [13:59:52] If he supports the "No cpu limits" POV, it's very likely correct. [14:05:42] (03PS6) 10Kevin Bazira: Fetch embedding from Swift [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) [14:11:04] (03CR) 10Kevin Bazira: Fetch embedding from Swift (032 comments) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [14:17:57] 10Machine-Learning-Team: Future support for ores scores in RC API - https://phabricator.wikimedia.org/T343813 (10Strainu) [14:18:03] 10Machine-Learning-Team: Future support for ores scores in RC API - https://phabricator.wikimedia.org/T343813 (10Strainu) @elukey this follows up on your email from wikitech-l: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/EK65B7QCQHEG37C2ERPIUSP64OX3ZEUJ/ [14:44:52] 10Machine-Learning-Team: fiwiki RC filters classify all edits as 'very likely bad faith' - https://phabricator.wikimedia.org/T343308 (10elukey) a:03isarantopoulos [14:46:12] 10Machine-Learning-Team: Some outlink events are rejected by EventGate - https://phabricator.wikimedia.org/T343002 (10elukey) 05Open→03Resolved a:03achou [14:48:51] 10Machine-Learning-Team: Add JavaScript examples to LiftWing usage docs - https://phabricator.wikimedia.org/T343031 (10kevinbazira) 05Open→03Resolved [14:49:04] 10Machine-Learning-Team: Add deprecation message for ORES UI - https://phabricator.wikimedia.org/T342118 (10kevinbazira) 05Open→03Resolved [14:51:03] (03CR) 10Elukey: Fetch embedding from Swift (032 comments) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [15:05:43] (03CR) 10Kevin Bazira: Fetch embedding from Swift (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [15:09:38] 10Machine-Learning-Team: Add deprecation message for ORES UI - https://phabricator.wikimedia.org/T342118 (10Aklapper) >>! In T342118#9043450, @kevinbazira wrote: > 2. [[ https://github.com/wikimedia/mediawiki-services-ores/blob/13be375f9aef4f90a3f8b82bd68812ca69c8dad1/ores/wsgi/templates/scorer.html | mediawiki-... [15:14:47] (03CR) 10Elukey: Fetch embedding from Swift (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [15:33:01] kevinbazira: o/ we can discuss the rec-api change in here if you want [15:33:19] my point is that it is difficult to understand what changes happened in the module [15:34:18] sure, now working on the code review made on the patch [15:37:11] okok [16:20:25] (03PS7) 10Kevin Bazira: Fetch embedding from Swift [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) [16:22:32] (03CR) 10Kevin Bazira: Fetch embedding from Swift (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/945834 (https://phabricator.wikimedia.org/T343576) (owner: 10Kevin Bazira) [16:27:22] elukey: o/ the code review has been fixed [16:36:45] 10Machine-Learning-Team: Add deprecation message for ORES UI - https://phabricator.wikimedia.org/T342118 (10kevinbazira) @Aklapper, thank you for the ping. Based on the [[ https://wikitech.wikimedia.org/w/index.php?title=ORES&oldid=2097286 | current timeline ]] ORES will be decommissioned in January 2024 (about... [16:36:51] kevinbazira: ack thanks! I am going afk but I'll review it tomorrow first thing :) [16:37:26] sure, no problem. thanks! [16:37:53] have a nice rest of the day folks [16:38:08] enjoy your evening too! [16:49:06] bye Luca! [17:54:49] (03PS4) 10Jdlrobson: fix: set default lift wing url to null [extensions/ORES] - 10https://gerrit.wikimedia.org/r/937142 (https://phabricator.wikimedia.org/T319170) (owner: 10Ilias Sarantopoulos)