[00:19:47] (03CR) 10DannyS712: [C:03+2] Migrate usage of Database::delete, insert, update and upsert to QueryBuilder [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1007862 (https://phabricator.wikimedia.org/T358831) (owner: 10MPGuy2824) [00:22:41] (03Merged) 10jenkins-bot: Migrate usage of Database::delete, insert, update and upsert to QueryBuilder [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1007862 (https://phabricator.wikimedia.org/T358831) (owner: 10MPGuy2824) [06:06:39] (03CR) 10Kevin Bazira: "Thank you for your comment, Ilias. This patch's task description (T363449) outlined two steps that were going to be implemented:" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1023542 (https://phabricator.wikimedia.org/T363449) (owner: 10Kevin Bazira) [07:21:02] Good morning folks! [07:33:40] 06Machine-Learning-Team, 10Foundational Technology Requests: Content Translation Recommendations API - https://phabricator.wikimedia.org/T293648#9751890 (10Aklapper) a:05DAbad→03None Removing inactive task assignee. (Please do so as part of offboarding - thanks.) [08:59:41] * isaranto brb running an errand [09:35:35] Guten tag! [09:38:10] Wünsche dir einen guten Start in die Woche :) [09:44:34] Guten tag Aiko! [11:05:48] (03CR) 10Ilias Sarantopoulos: [C:03+1] "LGTM!" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1014572 (https://phabricator.wikimedia.org/T356281) (owner: 10Jsn.sherman) [11:15:24] 06Machine-Learning-Team, 06Language-Team, 07Epic: Migrate Content Translation Recommendation API to Lift Wing - https://phabricator.wikimedia.org/T308164#9752643 (10KartikMistry) This is great news! CC @ngkountas I think we need to do some more changes apart from changing some parameters here though. [11:22:44] I'm going to deploy revertrisk to prod [11:33:23] ack [11:45:42] ah it also has the kserve upgrade [11:45:54] I'm going to run some load tests on ml-staging [12:03:20] put the revertrisk deployment on pause for a bit [12:10:50] deploying revscoring goodfaith to prod so that we have logging for revision ids (hasnt been deployed for a while). After following up with the alert fired over the weekend, I found out some spikes on ruwiki-goodfaith so want to check revisions that could have caused that [12:13:27] and I don't see any cpu spikes for this model server https://grafana-rw.wikimedia.org/d/-D2KNUEGk/kubernetes-pod-details?orgId=1&var-datasource=eqiad%20prometheus%2Fk8s-mlserve&var-namespace=revscoring-editquality-goodfaith&var-pod=All&var-container=All&from=now-2d&to=now&forceLogin [12:20:25] hello folks [12:20:58] isaranto: o/ [12:21:10] have you already deployed revscoring? [12:21:36] because we should also bump the docker image version, so we have the new/better logging with request-id [12:23:07] code review incoming [12:23:28] I thought I filed it on Friday but I forgot [12:23:35] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1025344 [12:25:55] merged thanks :) [12:26:08] hey! I have deployed revscoring-goodfaith sto prod. [12:26:39] a yes , I missed that this changes wan't eincluded [12:26:45] *included [12:27:03] we can wait a bit and then re-deploy, wdyt? [12:27:08] cool I'll redeploy it now. And do the same for the other ones in the following days [12:27:13] maybe to the editquality namespaces nly [12:27:24] yep, I'll do it and let you know! [12:27:30] okok! [12:27:55] Also please note that I have the WIKI_URL removal change lined up for hopefully this week, I'll need to deploy to all revscoring namespaces [12:30:41] ok, no prob. I'll just deploy goodfaith now and damaging then [12:31:38] yes yes even all of them if you prefer, it was just to let you know that I can complete what you leave out :) [12:32:41] ack! [12:37:15] i just redeployed goodfaith with the latest image [12:37:19] * isaranto lunch! [13:25:15] good morning all! [13:25:19] o/ [13:39:38] \o [13:48:09] ok, all well with the deployment, now we can see logs like this `INFO:root:JSON payload for the request-id...` [13:48:27] \o/ [13:48:40] way better, in the previous way it was a nightmare to correlate stuff [14:04:06] nice work \o/ [14:05:56] so folks I asked to serviceops about the implicit http -> https redirects that may happen with the MW API, since they use envoy as explicit proxy in their pods when calling the MW Api [14:06:27] so they may have code that runs http://localhost:65xx/w/api/etc.. returning HTTP 301 responses with a https location header [14:06:39] like we have for yue/zh-yue for example [14:06:50] but I've seen it also for wikidata.org -> www.wikidata.org [14:07:03] there is nothing in their envoy config that fixes the location header [14:07:10] it is offloaded to the application [14:07:44] if we want to be bullet proof we may need to set aiohttp to not follow redirects, and use some special python code to fix the location header in case [14:11:11] does it make sense? [14:14:53] ok. so we would actually follow the redirect manually right? [14:15:04] exactly yes [14:15:10] fixing the header if needed [14:15:22] sgtm! [14:15:34] in our case, after the next refactoring, it would be s/https/http [14:15:50] but it may be not super easy with mwapi's Async class [14:16:03] anyway, opening a task [14:17:46] 06Machine-Learning-Team, 10Structured-Data-Backlog (Current Work): Host a logo detection model for Commons images - https://phabricator.wikimedia.org/T358676#9753395 (10mfossati) [14:19:35] 06Machine-Learning-Team, 10Structured-Data-Backlog (Current Work): Host a logo detection model for Commons images - https://phabricator.wikimedia.org/T358676#9753403 (10mfossati) [14:20:31] 06Machine-Learning-Team, 06serviceops: Rename the envoy's uses_ingress option to sets_sni - https://phabricator.wikimedia.org/T346638#9753407 (10JMeybohm) [14:20:38] 06Machine-Learning-Team, 06Structured-Data-Backlog: Ignored exception in the logo detection prototype - https://phabricator.wikimedia.org/T363503#9753408 (10mfossati) [14:21:22] 06Machine-Learning-Team, 10Structured-Data-Backlog (Current Work): Host a logo detection model for Commons images - https://phabricator.wikimedia.org/T358676#9753410 (10mfossati) [14:21:43] 06Machine-Learning-Team, 06Structured-Data-Backlog: Pass the maximum number of uploads to the logo detection service - https://phabricator.wikimedia.org/T363505#9753411 (10mfossati) [14:21:58] 06Machine-Learning-Team, 10Structured-Data-Backlog (Current Work): Host a logo detection model for Commons images - https://phabricator.wikimedia.org/T358676#9753413 (10mfossati) [14:22:27] 06Machine-Learning-Team, 06Structured-Data-Backlog: Pass image objects to the logo detection service - https://phabricator.wikimedia.org/T363506#9753414 (10mfossati) [14:24:45] (03CR) 10Elukey: [C:03+1] "Very nice :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1024425 (https://phabricator.wikimedia.org/T362663) (owner: 10Ilias Sarantopoulos) [14:42:14] 10ORES, 10ChangeProp, 13Patch-For-Review: changeprop ORES tests failing - https://phabricator.wikimedia.org/T363046#9753493 (10elukey) [14:42:56] this --^ wasn't related to ORES or us, removed our tags :) [14:43:20] 10ORES, 10ChangeProp, 13Patch-For-Review: changeprop ORES tests failing - https://phabricator.wikimedia.org/T363046#9753497 (10elukey) Untagging ML since this is an issue with the nodejs code, not ORES etc.. Filed a patch to fix, lemme know :) [14:53:58] nice work! [14:54:51] I mean, nodejs... [14:54:57] always a pleasure [15:13:19] (03PS7) 10Ilias Sarantopoulos: utils: slow function execution wrapper [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1024425 (https://phabricator.wikimedia.org/T362663) [15:13:49] (03CR) 10Ilias Sarantopoulos: utils: slow function execution wrapper (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1024425 (https://phabricator.wikimedia.org/T362663) (owner: 10Ilias Sarantopoulos) [15:59:17] 06Machine-Learning-Team: Patch Location headers of HTTP redirects coming from the MW API in Lift Wing services - https://phabricator.wikimedia.org/T363725 (10elukey) 03NEW [16:00:25] 06Machine-Learning-Team: Improve Istio's mesh traffic transparent proxy capabilities for external domains accessed by Lift Wing - https://phabricator.wikimedia.org/T353622#9753959 (10elukey) Opened T363725 for the redirects, as it can be tackled separately. [16:00:39] aiko: remember the issue with the inconsistent revertrisk results in load testing? https://phabricator.wikimedia.org/T361881 [16:01:40] I ran some load tests today and indeed I can't seem to get the results that are documented. I'm starting to think if these results were taken with just one rev id as input (instead of the list of revids we have) [16:02:05] 06Machine-Learning-Team, 10MW-on-K8s, 06serviceops, 06SRE, 13Patch-For-Review: Migrate ml-services to mw-api-int - https://phabricator.wikimedia.org/T362316#9753974 (10elukey) After a lot of tests and config changes, we are almost ready to proceed with prod. Hopefully we'll get to it on April 2nd. [16:02:48] I'll run some more tomorrow, but I suggest if we don't figure anything out we replace the values and we start tracking with the current stats [16:03:49] going afk for the evening folks, cu tomorrow! [16:10:17] o/ [16:35:16] going afk for today folks! Have a nice rest of the day [16:36:59] o/ bye Luca and Ilias [16:38:22] isaranto: that's possible! I agree we shouldn't spend too much time on it [17:42:34] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MW-1.43-notes (1.43.0-wmf.3; 2024-04-30), 07Technical-Debt: Migrate usage of Database::delete, insert, update and upsert to QueryBuilder in ORES - https://phabricator.wikimedia.org/T358831#9754555 (10Umherirrender) 05Open→03Resolved a:03MPGuy... [17:46:05] (03PS1) 10Umherirrender: Migrate to IReadableDatabase::newSelectQueryBuilder [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1025429 (https://phabricator.wikimedia.org/T312454) [17:54:58] (03CR) 10DannyS712: [C:03+2] Migrate to IReadableDatabase::newSelectQueryBuilder [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1025429 (https://phabricator.wikimedia.org/T312454) (owner: 10Umherirrender) [18:00:38] (03Merged) 10jenkins-bot: Migrate to IReadableDatabase::newSelectQueryBuilder [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1025429 (https://phabricator.wikimedia.org/T312454) (owner: 10Umherirrender) [18:24:14] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 07Technical-Debt: Migrate usage of Database::select to SelectQueryBuilder in ORES - https://phabricator.wikimedia.org/T312454#9754839 (10Umherirrender) 05Open→03Resolved [19:27:49] 10ORES, 10ChangeProp, 13Patch-For-Review: changeprop ORES tests failing - https://phabricator.wikimedia.org/T363046#9755109 (10taavi) 05Open→03Resolved a:03elukey [20:14:36] 06Machine-Learning-Team, 06Language-Team, 07Epic: Migrate Content Translation Recommendation API to Lift Wing - https://phabricator.wikimedia.org/T308164#9755275 (10ngkountas) The response from the two endpoints Isaac listed above, are identical. If this is the case, it seems like a straightforward switch. [22:45:08] 07artificial-intelligence, 10Phabricator, 10research-ideas: Phab task classifier (bug, feature request, spike, epic, etc.) via AI - https://phabricator.wikimedia.org/T156415#9755784 (10Aklapper) 05Open→03Declined People are free to do that upstream. Closing for now as not actionable (though I'd reall... [23:01:35] night all!