[06:22:08] 10Machine-Learning-Team: Define SLI/SLO for Lift Wing - https://phabricator.wikimedia.org/T327620 (10elukey) Final status for all the dashboards: * https://grafana.wikimedia.org/d/slo-Lift_Wing_Article_Topic_Outlink/lift-wing-article-topic-outlink-slo-s?orgId=1 * https://grafana.wikimedia.org/d/slo-Lift_Wing_Re... [06:28:49] \o/ [06:30:34] 10Machine-Learning-Team, 10Patch-For-Review: Upgrade revscoring Docker images to KServe 0.11 - https://phabricator.wikimedia.org/T346446 (10isarantopoulos) kserve python package v0.11 has been upgraded for revscoring model servers and deployed to staging. The previous issue with the Content-type has been resol... [06:35:07] o/ [06:44:42] 10Machine-Learning-Team: Move the kserve custom helm chart to the upstream one - https://phabricator.wikimedia.org/T327241 (10elukey) The K8s SIG group added a new policy for upstream charts imported in our repo: https://wikitech.wikimedia.org/wiki/Kubernetes/Upstream_Helm_charts_policy [06:45:58] 10Machine-Learning-Team: Define SLI/SLO for Lift Wing - https://phabricator.wikimedia.org/T327620 (10elukey) 05Open→03Resolved [06:46:01] 10Machine-Learning-Team, 10Epic: Lift Wing improvements to get out of MVP state - https://phabricator.wikimedia.org/T333453 (10elukey) [06:46:12] 10Machine-Learning-Team: Deprecate mediawiki revision-score stream - https://phabricator.wikimedia.org/T342116 (10elukey) 05Open→03Resolved [06:48:13] 10Machine-Learning-Team, 10Patch-For-Review: Upgrade revscoring Docker images to KServe 0.11 - https://phabricator.wikimedia.org/T346446 (10elukey) >>! In T346446#9189875, @isarantopoulos wrote: > kserve python package v0.11 has been upgraded for revscoring model servers and deployed to staging. The previous i... [06:49:08] isaranto: o/ so https://gerrit.wikimedia.org/r/c/operations/puppet/+/959762 should be the traffic switch that we'll do on monday [06:49:22] it needs to be applied on all cp servers via puppet basically [06:49:33] so people hitting ores.wikimedia.org will be proxied to ores-legacy [06:56:15] Ok! Do you want to send the mail to wikitech-l or shall I do it? [06:56:52] In any case if anyone asks more technical question it's more likely you'll respond :) [06:57:09] Going to be afk for ~1h [06:57:19] definitely go ahead! [06:57:29] I wanted to brain bounce with you the solution, if you are ok etc.. [08:01:51] 10Machine-Learning-Team, 10Research: Review Revert Risk reports from WME - https://phabricator.wikimedia.org/T347136 (10elukey) [08:02:29] 10Machine-Learning-Team, 10Research: Review Revert Risk reports from WME - https://phabricator.wikimedia.org/T347136 (10elukey) [08:04:22] 10Machine-Learning-Team, 10Research: Review Revert Risk reports from WME - https://phabricator.wikimedia.org/T347136 (10elukey) For the first use case, I tried to check logs in Lift Wing for ruwiki:133170407, and I see this: ` 2023-09-22 07:34:41.247 71 root ERROR [preprocess():97] get_current_revision return... [08:04:40] * elukey bbiab [08:38:57] 10Machine-Learning-Team, 10Research: Review Revert Risk reports from WME - https://phabricator.wikimedia.org/T347136 (10achou) I quickly checked the two problematic revisions (ruwiki:133170407 and nlwiki:66027941) reported in the [doc](https://docs.google.com/document/d/19BV9rEyjOo9v_HFV6Dl4VO6PfNTgwJ5P53UGOv... [08:59:32] Amir1: o/ [08:59:40] guten tag [08:59:43] We are ready with https://gerrit.wikimedia.org/r/c/operations/puppet/+/959762 for Monday, what time do you prefer? [08:59:56] 10Machine-Learning-Team, 10Observability-Metrics, 10serviceops, 10Kubernetes: Don't scrape every containerPort for metrics - https://phabricator.wikimedia.org/T318707 (10JMeybohm) 05Open→03Resolved a:03JMeybohm [09:01:29] elukey: Monday is basically meeting day, so feel free to drop me a calendar invite in between those [09:03:54] Amir1: done! [09:04:01] Awesome! [09:28:59] little nit found while reviewing the ores traffic [09:29:18] https://ores-legacy.wikimedia.org/v3/scores/fakewiki/1695374769 replies with not found, and it is used by our check_ores nagios stuff [09:29:34] need to verify the alert, it may fire when we switch [09:30:36] the top talkers are using ancient python versions afaics, and they request 50 scores at the time [09:30:42] so we'll definitely see a lot of 400s [09:30:52] but they don't have a UA that we can use to follow up with them etc.. [09:30:59] so we can live with that [09:31:13] (traffic seems to come from AWS, so I don't think it is a community bot) [09:33:06] even the https://ores-legacy.wikimedia.org/v3/scores/fakewiki/1695374769 check should work, we return a HTTP 200 [09:35:36] so, to summarize, we are good afaics [09:38:41] ack [09:42:12] we could deploy fakewiki in LW or redirect the request in ores-legacy [09:43:35] actually I don't really know what fakewiki is. lemme check in ores [09:45:56] a ok it is just a fake extractor etc [09:46:01] just mocks [09:55:38] isaranto: yeah but the check is fine, it returns 200, it doesn't inspect anything afaics [09:56:38] Ack! [09:56:42] Then it is ok! [10:18:28] * aiko lunch [10:28:32] * elukey lunch! [10:57:34] * isaranto lunch! [12:37:29] Coffee! [12:46:14] morning o/ [12:46:36] same here, last coffee of the day ! [12:47:54] famous last words [12:59:23] coffee +1 :D [13:21:07] just sent the wikitech mail [13:30:43] isaranto: nice! +1 to the wording [13:31:20] Sorry, I reviewed it and then didn't say anything because I got distracted [13:31:25] retroactive +1 to the wording [13:31:29] I sent it as a follow up to the previous mails for more context [13:32:05] now we just need to do it :D [13:33:26] I am genuinely excited [13:36:10] \o/ [13:36:30] 10Machine-Learning-Team, 10Research: Review Revert Risk reports from WME - https://phabricator.wikimedia.org/T347136 (10prabhat) Thanks for looking into this. So, in summary: 1. We can only call revertrisk-language-agnostic API when parent revision exists. 2. Language support issue will be fixed. We can contin... [13:37:24] Thanks aiko for looking into the above issue [13:43:41] np! :) [14:06:50] While working on kserve logs I'm thinking of some required refactoring in order to be able to run model servers in a local env without packaging everything in a docker container (just for speed in local development/debugging) [14:07:25] I'll open a task and we can discuss it. What I have in mind would be ~1 day of work, but if other ppl dont find it useful we can skip it [14:15:57] I would be very interested in that, and probably so would research [14:22:23] I think it would be useful! [14:30:28] ok! sounds good then [14:31:06] cause I've been postponing that for a while doing some sorts of hacks locally [14:59:43] 10Machine-Learning-Team, 10Epic: Add meaningful access logs to KServe's pods - https://phabricator.wikimedia.org/T333804 (10isarantopoulos) 05Stalled→03Resolved Shall we proceed with the format mentioned above? ` '{"remote_address": "%(h)s", "user_name": "%(a)s", "date": "%(t)s", "status": "%(s)s", "method... [14:59:45] 10Machine-Learning-Team, 10Epic: Lift Wing improvements to get out of MVP state - https://phabricator.wikimedia.org/T333453 (10isarantopoulos) [15:00:03] 10Machine-Learning-Team, 10Epic: Add meaningful access logs to KServe's pods - https://phabricator.wikimedia.org/T333804 (10isarantopoulos) 05Resolved→03Open [15:00:10] 10Machine-Learning-Team, 10Epic: Lift Wing improvements to get out of MVP state - https://phabricator.wikimedia.org/T333453 (10isarantopoulos) [15:01:47] lol I had accidentally Resolved the above task earlier https://media.tenor.com/rMxNr07CxSMAAAAC/cat-crazy-cat.gif [15:03:23] I think we can close it! [15:06:43] 10Machine-Learning-Team, 10Add-Link, 10Chinese-Sites, 10Growth-Team (Current Sprint), and 2 others: Deploy "add a link" to 14th round of wikis - https://phabricator.wikimedia.org/T308139 (10Sgs) a:03Sgs [15:06:55] 10Machine-Learning-Team, 10Add-Link, 10Chinese-Sites, 10Growth-Team (Current Sprint), and 2 others: Deploy "add a link" to 14th round of wikis - https://phabricator.wikimedia.org/T308139 (10Sgs) 05Open→03In progress [15:25:44] I thought it would be better to have logging on a separate task for more context that's why I wrote on that one [15:39:41] isaranto: already answered to wikitech-l [15:42:26] ack [15:54:23] 10Machine-Learning-Team, 10ORES: Support for basic boolean flags in ores-legacy - https://phabricator.wikimedia.org/T347193 (10Halfak) [15:56:22] 10Machine-Learning-Team, 10ORES: Support for basic boolean flags in ores-legacy - https://phabricator.wikimedia.org/T347193 (10elukey) [15:56:42] 10Machine-Learning-Team, 10ORES: Feature injection does not appear to work in ores-legacy - https://phabricator.wikimedia.org/T347194 (10Halfak) [15:58:07] 10Machine-Learning-Team, 10ORES: Feature injection does not appear to work in ores-legacy - https://phabricator.wikimedia.org/T347194 (10elukey) We currently don't support feature injection, what is the use case for it? From our traffic analysis this is not a feature that is really used. Ores is being deprecat... [15:59:31] 10Machine-Learning-Team, 10ORES: Support for basic boolean flags in ores-legacy - https://phabricator.wikimedia.org/T347193 (10elukey) The following works nicely and it seems more precise: ` https://ores-legacy.wikimedia.org/v3/scores/enwiki/12312342/damaging?features=true ` Is there a use case for maintaini... [16:07:04] going afk folks! Have a nice weekend! [16:13:23] Going afk as well o/ [17:23:12] 10Machine-Learning-Team, 10Add-Link, 10Chinese-Sites, 10Growth-Team (Current Sprint), and 2 others: Deploy "add a link" to 14th round of wikis - https://phabricator.wikimedia.org/T308139 (10Sgs) I ran this script for adding the link-recommendation task type and populating the excluded sections entries: `la... [19:08:34] 10Machine-Learning-Team, 10ORES: Support for basic boolean flags in ores-legacy - https://phabricator.wikimedia.org/T347193 (10Halfak) It would not break currently working code. It's a compatibility issue. Also it's a common pattern in the MediaWiki API. E.g. see https://en.wikipedia.org/w/api.php?action=... [19:32:54] 10Machine-Learning-Team, 10ORES: Feature injection does not appear to work in ores-legacy - https://phabricator.wikimedia.org/T347194 (10Halfak) 1. People use the functionality to use ORES to make recommendations. E.g. the WikiEdu outreach dashboard uses feature injection to make predictions about hypothetica... [19:35:47] 10Machine-Learning-Team, 10ORES: Feature injection does not appear to work in ores-legacy - https://phabricator.wikimedia.org/T347194 (10Halfak) Relevant extract: > 5.2.1 Dependency injection. When we originally developed ORES, we designed our feature > engineering strategy based on a dependency injection f... [21:09:59] 10Machine-Learning-Team, 10ORES: Feature injection does not appear to work in ores-legacy - https://phabricator.wikimedia.org/T347194 (10Halfak) I checked in with @Sage_Wiki_Ed who said they are no longer using feature injection in their dashboard. [21:40:52] 10Machine-Learning-Team, 10ORES: Cannot set Api-User-Agent header when making requests to ORES from a user script - CORS - https://phabricator.wikimedia.org/T347214 (10Halfak)