[07:36:55] (03CR) 10Elukey: [C: 03+2] revscoring: relax schema version checks when retrieving the rev-id [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/895825 (owner: 10Elukey) [07:43:39] (03Merged) 10jenkins-bot: revscoring: relax schema version checks when retrieving the rev-id [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/895825 (owner: 10Elukey) [09:23:31] 10Machine-Learning-Team, 10ORES, 10Wikimedia Enterprise: Investigate tools that use ORES - https://phabricator.wikimedia.org/T330854 (10elukey) @prabhat hi! Do you have some info about how Enterprise uses ORES? More specifically, I see two use cases in [[ https://github.com/wikimedia/OKAPI | the OKAPI repo ]... [09:32:45] hello folks [09:32:54] I am deleting the old docker images [09:44:14] go ahead! ๐Ÿงน [09:52:34] 10Machine-Learning-Team: Delete old ml-related docker images that are deprecated - https://phabricator.wikimedia.org/T331513 (10elukey) Cleaned up from build2001 following Wikitech's docs. Let's wait https://docker-registry.wikimedia.org/ to sync with the new changes before closing :) [09:52:43] 10Machine-Learning-Team: Delete old ml-related docker images that are deprecated - https://phabricator.wikimedia.org/T331513 (10elukey) a:03elukey [09:55:48] * isaranto commuting [10:08:46] folks yesterday I didn't really read https://api.wikimedia.org/wiki/Documentation/Getting_started/Authentication#2._Get_an_access_token correctly [10:09:07] there is already a way to retrive a bearer token from meta.w.o, without the API portal [10:09:13] this is what we were looking for in theory [10:18:14] 10Machine-Learning-Team, 10API Platform, 10Platform Team Initiatives (API Gateway): API-Gateway: lift auth restriction for POST requests - https://phabricator.wikimedia.org/T331547 (10elukey) I see that the blocker should be the following in `_api_gateway_ratelimit.tpl`: ` # For everything e... [10:20:57] 10Machine-Learning-Team: Delete old ml-related docker images that are deprecated - https://phabricator.wikimedia.org/T331513 (10elukey) Updated, all images that we don't use are gone :) [10:27:37] ok fix rolled out [10:33:47] nice! (for the bearer token) [10:34:00] and the fix too ๐Ÿ˜› [10:34:04] klausman: o/ I left a proposal in https://phabricator.wikimedia.org/T331547 for a possible next step to unblock POSTs without JWT token [10:37:11] checking [10:38:11] yeah, that sounds reasonable [10:40:44] the other thing that I am trying to understand is if we get some identification of auth-ed user in an HTTP header [10:41:09] I mean if at our Istio gateway we see some Authentication-related header with the client id [10:41:22] it would be useful to create ad hoc rate limit rules if needed [10:44:41] Yeah, potentially also for metrics: add a specific TS for "known/big" users and pile the rest into a summary TS [10:45:10] similar to how we separate TSes by UA in ORES [10:50:44] I am reading https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/809198/5/charts/api-gateway/templates/_ratelimit.yaml [10:51:10] what was the idea for specific rate limits? Like based on IPs/tokens/UAs/etc.. ? [10:53:24] I see only anon/default(that I guess is auth-ed) right? [10:54:42] in theory for rate limits we could leverage also requestctl at the Varnish level [10:54:47] There are other `key` values to match on, but I don't know if all the extractor logic (populating keys) is there yet [10:54:49] in case of emergency i mean [10:55:17] Yes, for emergencies, Varnish would be better _if_ it has access to the right info (like IPs) [10:55:32] what do you mean ? [10:56:01] As far out to the edge of the setup as is feasible is usually the best way to drop traffic that is damaging [10:56:27] no I mean the "right info part" [10:56:32] And AIUI, Varnish is the place of choice for IP-based filtering alredy [10:56:33] in theory https://wikitech.wikimedia.org/wiki/Requestctl can see anything [10:56:49] not only IP-based, we can match on http headers [10:57:03] it is way more powerful [10:57:53] Yeah, but it has one downside: if the traffic uses rapidly-switching JWTs, we can't match on those. Granted, a less likely scenario (and I would suspect a ratelimit on token generation or disabling the account to be another approach) [10:58:14] The old uniformtiy-of-traffic problem [10:58:43] sure but if the same set of IPs are switching JWTs we'll see them anyway, since we'll see the traffic volume and block them [10:58:55] Yeah, it's a corner case. [10:59:47] For the abuse-case/emergency, requestctl is probably the right way. For more "soft" ways of telling users to chill a bit, something on APIGW or LW would be nicer. Or at least my gut tells me that [11:05:04] probably yes, we'll see what plans they have to improve it.. [11:14:04] 10Machine-Learning-Team, 10artificial-intelligence, 10SRE, 10serviceops-radar: New Service Request 'open_nsfw' - https://phabricator.wikimedia.org/T250110 (10akosiaris) I am tentatively removing #service-deployment-requests as I don't see how #serviceops (the onwer of that tag) has anything to do with this... [11:25:50] 10Machine-Learning-Team, 10serviceops-radar, 10Language-Team (Language-2023-January-March): New Service Deployment Request: NNLB-200 for machine translation - https://phabricator.wikimedia.org/T329971 (10akosiaris) [11:27:39] 10Machine-Learning-Team, 10SRE, 10serviceops, 10Language-Team (Language-2023-January-March), 10Service-deployment-requests: New Service Deployment Request: NNLB-200 for machine translation - https://phabricator.wikimedia.org/T329971 (10akosiaris) I 've transformed (roughly) this to a #service-deployment-... [11:36:51] going afk in a bit, logging off for today folks! [11:44:56] ciao Luca! ๐Ÿ‘‹ [11:51:58] \o [11:52:03] * klausman lunch [12:31:05] o/ [13:06:25] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 6th round of wikis - https://phabricator.wikimedia.org/T304550 (10Trizek-WMF) >>! In T304550#8677050, @Sgs wrote: >>>! In T304550#8669991, @Trizek-WMF wrote: >> All models work fine except: >> * **cb... [13:10:29] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 6th round of wikis - https://phabricator.wikimedia.org/T304550 (10Trizek-WMF) [13:10:49] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 6th round of wikis - https://phabricator.wikimedia.org/T304550 (10Trizek-WMF) [13:11:30] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 6th round of wikis - https://phabricator.wikimedia.org/T304550 (10Trizek-WMF) Note: we can add the three excluded wikis to the list if we have a fix before Monday early afternoon UTC. [14:13:13] 10Machine-Learning-Team, 10API Platform, 10Platform Team Initiatives (API Gateway): API-Gateway: lift auth restriction for POST requests - https://phabricator.wikimedia.org/T331547 (10hnowlan) >>! In T331547#8679127, @elukey wrote: > @hnowlan Is the idea viable or completely off-track? In the former case I c... [14:42:00] o/ What is the preferred way to run MediaWIki locally? I want to try to run OREs extension. Have been trying with vagrant following this https://www.mediawiki.org/wiki/MediaWiki-Vagrant but no success yet (having a problem with a VM on virtualbox). WIll now try just with docker [16:03:56] ok. docker worked fine! [16:09:43] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) [16:24:05] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) [17:23:36] 10Machine-Learning-Team, 10ORES, 10Wikimedia Enterprise: Investigate tools that use ORES - https://phabricator.wikimedia.org/T330854 (10prabhat) @elukey We use ores to get damaging and goodfaith scores for each revision that shows up in the [[ https://stream.wikimedia.org/?doc#/streams | revision-create ]] e... [23:30:27] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 6th round of wikis - https://phabricator.wikimedia.org/T304550 (10kevinbazira) @Sgs, yes, the number of links in a wiki affects how the model performs.