[00:34:53] (03PS1) 10Jforrester: build: Update MediaWiki requirement to 1.39.0 [extensions/ORES] (REL1_39) - 10https://gerrit.wikimedia.org/r/898171 [03:39:07] (03PS1) 10Jforrester: build: Update MediaWiki requirement to 1.40.0 [extensions/ORES] (REL1_40) - 10https://gerrit.wikimedia.org/r/898353 [07:53:07] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 12th round of wikis - https://phabricator.wikimedia.org/T308137 (10kevinbazira) The conclusion on the backtesting results is that most of the languages look fine besides: - mnwwiki's precision and recall are bad. Both... [08:04:37] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 12th round of wikis - https://phabricator.wikimedia.org/T308137 (10kevinbazira) [08:18:33] in SRE there's a weekly rotation called SRE Clinic Duty, which assigns one person (among other things) to keep an eye on incoming Phab tasks with the intention that anything which gets filed with just SRE gets bubbled up to the workboards of the more specialised sub teams [08:18:51] https://phabricator.wikimedia.org/dashboard/view/527/ is the triage Phab page in question [08:19:40] what would be the appropriate tag to exclude for issues relating to SRE-related ML tasks? Machine-Learning-Team? [08:20:04] (so that the search can be amended to no longer present issues tagged with that project on the workboard) [08:32:47] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10fgiunchedi) [08:37:48] 10Machine-Learning-Team, 10ORES, 10Wikimedia Enterprise: Investigate tools that use ORES - https://phabricator.wikimedia.org/T330854 (10Ottomata) > since for some conditions we also query scores of certain related revisions of the revision @prabhat, for my own curiosity (not for Luca's project), do you nee... [08:49:54] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Vgutierrez) [09:06:43] o/ [09:08:18] I got an API key from API portal and successfully queried some model servers 🎉 [09:08:32] nice docs, easy process! [09:14:02] I also tried and got an temporary key for testing using the 3rd method (one that lasts 4 hours) [09:28:08] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10elukey) [09:32:55] isaranto: o/ nice! [09:33:12] I think that most of our users will just need a personal api token [09:33:40] one thing that I am still not clear is what to do when a client will need to be heavily rate limited, maybe for too much traffic etc.. [09:33:57] one nit on the docs there is a `$` in the `export $ACCESSTOKEN=` [09:33:59] like selectively apply a rate limit to a client-id (after auth) [09:34:08] ack [09:34:10] ah snap, didn't see it! Can you fix it! [09:34:11] ?? [09:34:15] sure! [09:34:20] thanks :) [09:34:45] I'm reading some docs about rate limiting etc [09:46:27] 10Machine-Learning-Team, 10Data-Engineering, 10Research, 10Event-Platform Value Stream (Sprint 10): Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10Ottomata) > it's like the mediawiki/revision/score schema can be used by many streams e.... [09:53:35] 10Machine-Learning-Team, 10Research, 10Section-Topics, 10Structured-Data-Backlog: Let the model that learns section alignments consume section topics output - https://phabricator.wikimedia.org/T331968 (10mfossati) [09:54:02] isaranto: IIUC we have three layers for rate-limit/block/etc.. [09:54:40] 1) at the varnish level with requestctl, it is a SRE-only way to add rate limits for specific combinations of UA+IP+etc.. very powerful and generic, not api-gateway specific [09:54:45] 1.api gateway, 2. istio on liftwing [09:54:48] ? [09:54:57] aa ok u covered me :) [09:55:55] 2) anon and default rate limits at the api-gateway level, that we can customize per-service. Here the granularity is coarse, we can set stuff like "anonymous requests allowed 500/hour for a single ip, 5000 if a client is oauth-authenticated, etc..) [09:56:32] 3) Lift Wing with istio/envoy custom filters, but we'll need to figure out how to do it (based on auth-header? IP+UA+etc..?) [09:56:44] so I am a little confused atm, not sure what's best [10:00:39] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10elukey) [10:12:58] 10Machine-Learning-Team, 10API-Portal: Add documentation about LiftWing to the API Portal - https://phabricator.wikimedia.org/T325759 (10elukey) a:05klausman→03elukey [10:17:39] 10Machine-Learning-Team, 10SRE: Migrate ml-cache to Bullseye - https://phabricator.wikimedia.org/T331712 (10akosiaris) [10:23:37] 10Machine-Learning-Team, 10SRE: Migrate ml-cache to Bullseye - https://phabricator.wikimedia.org/T331712 (10akosiaris) p:05Triage→03Medium [10:53:08] 10Machine-Learning-Team: Review ORES traffic to better understand Lift Wing's requirements - https://phabricator.wikimedia.org/T325763 (10elukey) Checked https://grafana-rw.wikimedia.org/d/HIRrxQ6mk/ores?forceLogin&from=now-7d&orgId=1&refresh=1m&to=now-1m&var-datasource=codfw%20prometheus%2Fops&var-model=All&vie... [10:54:26] 10Machine-Learning-Team, 10API Platform, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway), 10Platform Team Workboards (Platform Engineering Reliability): API-Gateway: lift auth restriction for POST requests - https://phabricator.wikimedia.org/T331547 (10hnowlan) [11:43:16] now the API gateway supports unauth requests! [11:43:17] curl https://api.wikimedia.org/service/lw/inference/v1/models/enwiki-articletopic:predict -X POST -d '{ [11:43:20] "rev_id": 123555 }' [11:43:24] this works without bearer token [11:43:38] I'll follow up with some ratelimits for anon requests [11:47:54] niiiiice [11:48:18] yes it is getting an MVP shape [11:49:14] (03PS5) 10Ilias Sarantopoulos: ores-legacy: add LW error messages and exceptions to response [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/897910 (https://phabricator.wikimedia.org/T330414) [11:49:53] I opened the patch with some error handling from liftwing side. --^ [11:50:32] lemme know your thoughts then we could add more. I want to gather some erroneous requests from ores so that I include them in unit tests [11:56:02] isaranto: one qs - IIRC the aiohttp's ClientSession needs a special flag to be true (raise_for_status) to raise exceptions for non HTTP20x/30x. We are not using it in this case right? We just check manually the return code etc.. [12:00:02] raise for status will raise an error if status code is above 400. In our case we don't want that because it is not the api we expose to the user [12:00:21] We want to catch the exception and manipulate a response [12:01:38] This is my interpretation of the whole thing because it refers to the liftwing calls. If it were on the top level (fastapi main) we would raise directly [12:01:57] super I wanted to double check, sounds good [12:09:06] * elukey lunch! [12:16:34] * isaranto lunch [13:36:23] 10Machine-Learning-Team, 10Research, 10Section-Topics, 10Structured-Data-Backlog: Let the model that learns section alignments consume section topics output - https://phabricator.wikimedia.org/T331968 (10mfossati) [13:36:31] 10Machine-Learning-Team, 10Research, 10Section-Topics, 10Structured-Data-Backlog: Let the model that learns section alignments consume section topics output - https://phabricator.wikimedia.org/T331968 (10mfossati) [13:36:49] 10Machine-Learning-Team, 10Research, 10Section-Topics, 10Structured-Data-Backlog: Let the model that learns section alignments consume section topics output - https://phabricator.wikimedia.org/T331968 (10mfossati) [13:45:48] 10Machine-Learning-Team, 10Research, 10Section-Topics, 10Structured-Data-Backlog: Let the model that learns section alignments consume section topics output - https://phabricator.wikimedia.org/T331968 (10CBogen) [13:59:50] (03PS1) 10Ilias Sarantopoulos: pretty print json [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898748 (https://phabricator.wikimedia.org/T330414) [14:19:44] 10Machine-Learning-Team, 10Research, 10Section-Topics, 10Structured-Data-Backlog: Let the model that learns section alignments consume section topics output - https://phabricator.wikimedia.org/T331968 (10elukey) Hi! Is there anything that the ML team needs to do? (just to organize the work etc..) [14:51:22] 10Machine-Learning-Team: [nsfw] Upgrade python and debian in docker image - https://phabricator.wikimedia.org/T329612 (10isarantopoulos) 05Open→03Resolved [14:51:32] 10Machine-Learning-Team: [revertrisk] Upgrade python from 3.7 to 3.9 in docker images - https://phabricator.wikimedia.org/T328439 (10isarantopoulos) 05Open→03Resolved [14:51:38] 10Machine-Learning-Team: [outlink] Upgrade python from 3.7 to 3.9 in docker images - https://phabricator.wikimedia.org/T328438 (10isarantopoulos) 05Open→03Resolved [14:56:26] 10Machine-Learning-Team, 10ORES, 10Item Quality Scoring Improvement, 10Wikidata, and 2 others: Wikidata Articlequality Ores/ML model needs updating after MUL - https://phabricator.wikimedia.org/T332021 (10Michael) [14:59:02] 10Machine-Learning-Team, 10API Platform, 10Platform Team Initiatives (API Gateway), 10Platform Team Workboards (Platform Engineering Reliability): API-Gateway: lift auth restriction for POST requests - https://phabricator.wikimedia.org/T331547 (10elukey) 05Open→03Resolved [14:59:13] 10Machine-Learning-Team: Upgrade ML clusters to Kubernetes 1.23 - https://phabricator.wikimedia.org/T324542 (10elukey) 05Open→03Resolved [14:59:23] 10Machine-Learning-Team: Upgrade Kserve's k8s control plane to 0.10 - https://phabricator.wikimedia.org/T331114 (10elukey) 05Open→03Resolved [14:59:29] 10Machine-Learning-Team: Upgrade the inference-services repo codebase to kserve 0.10 (fastapi) - https://phabricator.wikimedia.org/T329032 (10elukey) 05Open→03Resolved [14:59:36] 10Machine-Learning-Team: Delete old ml-related docker images that are deprecated - https://phabricator.wikimedia.org/T331513 (10elukey) 05Open→03Resolved [14:59:41] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES: EnWiki Recent Changes Page no longer displays damaging filters - https://phabricator.wikimedia.org/T331045 (10elukey) 05Open→03Resolved [14:59:51] 10Lift-Wing, 10Machine-Learning-Team: Deploy revert-risk multilingual model to production - https://phabricator.wikimedia.org/T325218 (10elukey) 05In progress→03Resolved [15:07:41] 10Machine-Learning-Team, 10ORES, 10Item Quality Scoring Improvement, 10Wikidata, and 2 others: Wikidata Articlequality ORES/ML model needs updating after MUL - https://phabricator.wikimedia.org/T332021 (10Michael) [15:08:38] * elukey afk for abit [15:11:19] (03PS1) 10Ilias Sarantopoulos: Add logging for FastAPI app [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) [15:23:18] (03PS2) 10Ilias Sarantopoulos: Add logging for FastAPI app [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) [15:24:28] (03PS3) 10Ilias Sarantopoulos: Add logging for FastAPI app [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) [15:26:23] I am issuing multiple patches for the ores legacy endpoint - idea is to keep changes as small as possible so that things dont get blocked and I can work in parallel on different stuff [15:26:44] hope u dont mind the extra review friction <3 [15:29:57] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10herron) [15:45:39] (03PS2) 10Ilias Sarantopoulos: ores-legacy: pretty print json responses [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898748 (https://phabricator.wikimedia.org/T330414) [15:58:00] 10Machine-Learning-Team, 10Sustainability (Incident Followup): How can we test all the wiki/page combinations that can be affected by ORES? - https://phabricator.wikimedia.org/T181830 (10akosiaris) 05Open→03Invalid 5+ years since last comment, 2+ years since last action (which was just moving around a task... [16:49:14] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10Item Quality Scoring Improvement, and 3 others: Wikidata Articlequality ORES/ML model needs updating after MUL - https://phabricator.wikimedia.org/T332021 (10Michael) [16:53:12] (03CR) 10Elukey: [C: 03+1] ores-legacy: pretty print json responses [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898748 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [16:55:05] (03CR) 10Elukey: Add logging for FastAPI app (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [16:57:22] (03CR) 10Elukey: ores-legacy: add LW error messages and exceptions to response (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/897910 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [17:21:24] (03CR) 10Ilias Sarantopoulos: Add logging for FastAPI app (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [17:33:43] (03CR) 10Elukey: Add logging for FastAPI app (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [17:52:01] 10Machine-Learning-Team, 10ORES, 10Advanced-Search, 10All-and-every-Wikisource, and 72 others: Remove unnecessary targets definitions - https://phabricator.wikimedia.org/T328497 (10Tchanders) Thanks - untagging IPInfo since the value is different from the default. [19:01:41] (03CR) 10Jforrester: [C: 03+2] build: Update MediaWiki requirement to 1.39.0 [extensions/ORES] (REL1_39) - 10https://gerrit.wikimedia.org/r/898171 (owner: 10Jforrester) [19:11:04] 10Machine-Learning-Team, 10Research, 10Section-Topics, 10Structured-Data-Backlog: Let the model that learns section alignments consume section topics output - https://phabricator.wikimedia.org/T331968 (10mfossati) Ciao @elukey , not really. I thought it would be useful to keep you folks in the loop, since... [19:18:16] (03Merged) 10jenkins-bot: build: Update MediaWiki requirement to 1.39.0 [extensions/ORES] (REL1_39) - 10https://gerrit.wikimedia.org/r/898171 (owner: 10Jforrester) [21:04:55] (03CR) 10Jforrester: [C: 03+2] build: Update MediaWiki requirement to 1.40.0 [extensions/ORES] (REL1_40) - 10https://gerrit.wikimedia.org/r/898353 (owner: 10Jforrester)