[05:11:15] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 12th round of wikis - https://phabricator.wikimedia.org/T308137 (10kevinbazira) @kostajh, we published datasets for all 21/23 models that passed the evaluation in this round. [05:11:46] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 12th round of wikis - https://phabricator.wikimedia.org/T308137 (10kevinbazira) [06:40:01] 10Machine-Learning-Team, 10ORES, 10Wikimedia Enterprise: Investigate tools that use ORES - https://phabricator.wikimedia.org/T330854 (10prabhat) @Ottomata By revision ids (as in the example above). Queries are typically for lastest **few revisions** of a page. [08:12:26] 10Machine-Learning-Team, 10ORES, 10Wikimedia Enterprise: Investigate tools that use ORES - https://phabricator.wikimedia.org/T330854 (10Ottomata) > latest few revisions of a page Okay, so you do need some 'historical' page data, meaning an API that only allowed you to get content / scores / whatever for the... [09:29:56] aiko: o/ [09:30:12] have you had any issues with proxy/etc.. on stat100x nodes when running aiohttp? [09:30:38] for some reason I'd need to contact an endpoint with some reqs, curl works but python doesn't [09:33:32] o/ I run the ores legacy app on stat4 and have no issue with aiohttp calls(although I access only liftwing) [09:34:13] * isaranto will be afk for the next 30-60' [09:50:05] isaranto: o/ thanks, really weird then, not sure why it hangs [10:00:53] ah interesting, requests hangs too [10:06:00] yeah ok it is something related to firewall or similar [10:06:01] tcp6 0 1 stat1004.eqiad.wm:53216 kubestage1003.eqia:8087 SYN_SENT 14981/python3 [10:06:12] but curl works [10:31:50] ok the mistery gets weirder [10:31:56] lift wing calls work [10:43:34] loool it was ipv6 [10:43:37] * elukey cries in a corner [11:35:57] the change is https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/898695/3/helmfile.d/services/api-gateway/values.yaml, so it should be 10k for goodfaith anon IIUC [11:38:26] (this was for hugh :D) [11:39:48] * elukey lunch! [11:43:06] (03CR) 10Ilias Sarantopoulos: Add logging for FastAPI app (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [11:52:02] 10Machine-Learning-Team, 10SRE, 10serviceops, 10Language-Team (Language-2023-January-March), 10Service-deployment-requests: New Service Deployment Request: NNLB-200 for machine translation - https://phabricator.wikimedia.org/T329971 (10Clement_Goubert) p:05Triage→03Medium [11:53:42] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10Clement_Goubert) [12:34:19] elukey: what stuff about the user would we like to log? by default the request has host ip and port [12:34:26] are these ok? [12:36:14] * isaranto goes for lunch [13:10:26] 10Machine-Learning-Team, 10ORES, 10Wikimedia Enterprise: Investigate tools that use ORES - https://phabricator.wikimedia.org/T330854 (10prabhat) Correct. [13:50:20] isaranto: User Agent would be nice too [13:52:09] kk, any preference if logging format? [13:56:54] I don't recall exactly if there is a format that logstash would ingest without issues [13:57:20] IIRC there was an ECS schema [14:00:08] TIL -> https://wikitech.wikimedia.org/wiki/Logstash/Common_Logging_Schema [14:00:12] will take a look [14:00:16] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10Marostegui) [14:23:17] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10ssingh) [14:26:00] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10herron) [14:26:34] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10ssingh) [14:58:23] tried the meta bearer token, it also works fine :) [15:00:34] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10bking) [15:01:54] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10ayounsi) [15:02:07] (03PS4) 10Ilias Sarantopoulos: Add logging for FastAPI app [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) [15:39:14] (03CR) 10Elukey: Add logging for FastAPI app (034 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [15:44:20] (03CR) 10Ilias Sarantopoulos: Add logging for FastAPI app (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [16:16:32] finally I was able to test the rate limit per endpoint feature, seems to work! [16:18:59] chrisalbon: I am deploying the new rate limits to the api-gateway (many thanks to Hugh for the help), the new functionality works [16:19:34] I'll fix the docs and deploy autoscaling changes tomorrow, but I can say that we hit MVP :) [16:20:04] it took ONLY two years :D [16:21:19] 10Machine-Learning-Team, 10Data-Engineering, 10Research, 10Event-Platform Value Stream (Sprint 10): Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10achou) > If a model had 1000 classes though, maybe doesn't make so much sense to include... [16:29:19] (03PS5) 10Ilias Sarantopoulos: Add logging for FastAPI app [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) [16:30:27] updated the docs in https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing#External_endpoints [16:30:49] going afk for today folks, have a nice rest of the day :) [16:30:56] (03CR) 10Ilias Sarantopoulos: Add logging for FastAPI app (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [16:32:38] (03PS6) 10Ilias Sarantopoulos: ores-legacy: add LW error messages and exceptions to response [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/897910 (https://phabricator.wikimedia.org/T330414) [16:34:05] (03CR) 10Ilias Sarantopoulos: ores-legacy: add LW error messages and exceptions to response (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/897910 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [16:34:22] (03CR) 10Ilias Sarantopoulos: [C: 03+2] ores-legacy: pretty print json responses [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898748 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [16:35:41] (03Merged) 10jenkins-bot: ores-legacy: pretty print json responses [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898748 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [16:38:15] (03PS7) 10Ilias Sarantopoulos: ores-legacy: add LW error messages and exceptions to response [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/897910 (https://phabricator.wikimedia.org/T330414) [17:00:19] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES: Migrate ORES MediaWiki Extension to LiftWing - https://phabricator.wikimedia.org/T332200 (10isarantopoulos) [17:08:32] (03PS6) 10Ilias Sarantopoulos: Add logging for FastAPI app [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) [17:09:26] rebase --^ [17:09:36] (03CR) 10CI reject: [V: 04-1] Add logging for FastAPI app [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) (owner: 10Ilias Sarantopoulos) [17:10:25] (03PS7) 10Ilias Sarantopoulos: Add logging for FastAPI app [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/898780 (https://phabricator.wikimedia.org/T330414) [17:11:08] just rebased both as I don't know which one may be merged first [17:12:06] I'll continue tomorrow with trying to tackle some basic caching in endpoint - not that easy to do with async 💫 [17:12:27] logging off folks, have a nice rest of day/afternoon [17:19:25] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-notice: Deploy "add a link" to 6th round of wikis - https://phabricator.wikimedia.org/T304550 (10Sgs) 05Open→03In progress [18:11:05] 10Machine-Learning-Team, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Eevans) [18:47:03] 10Machine-Learning-Team, 10ORES, 10Wikimedia Enterprise: Investigate tools that use ORES - https://phabricator.wikimedia.org/T330854 (10Ottomata) [18:47:05] 10Machine-Learning-Team: Migrate ORES clients to LiftWing - https://phabricator.wikimedia.org/T312518 (10Ottomata) [19:25:44] 10Machine-Learning-Team, 10ORES, 10Advanced-Search, 10All-and-every-Wikisource, and 72 others: Remove unnecessary targets definitions - https://phabricator.wikimedia.org/T328497 (10Jdlrobson) [20:25:41] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-notice: Deploy "add a link" to 7th round of wikis - https://phabricator.wikimedia.org/T304551 (10Sgs) a:05Sgs→03Trizek-WMF [20:27:04] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-notice: Deploy "add a link" to 8th round of wikis - https://phabricator.wikimedia.org/T308133 (10Sgs) a:05Sgs→03Trizek-WMF [20:27:33] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-notice: Deploy "add a link" to 9th round of wikis - https://phabricator.wikimedia.org/T308134 (10Sgs) a:05Sgs→03Trizek-WMF [20:41:35] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 6th round of wikis - https://phabricator.wikimedia.org/T304550 (10Sgs) 05In progress→03Open