[05:26:39] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10kevinbazira) [05:30:57] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10kevinbazira) @kostajh, we published datasets for all 17/19 models that passed the evaluation in this round. [08:03:42] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10achou) Luca has added support for mediawiki.page_change in Lift Wing, so now we can use page_change as the source event. (see T... [08:04:28] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 11th round of wikis - https://phabricator.wikimedia.org/T308136 (10kevinbazira) a:03kevinbazira [08:21:37] (03PS2) 10Ilias Sarantopoulos: nsfw: upgrade python to 3.9 and debian to bullseye [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/890473 (https://phabricator.wikimedia.org/T329612) [08:32:38] (03CR) 10Kevin Bazira: [C: 03+1] nsfw: upgrade python to 3.9 and debian to bullseye [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/890473 (https://phabricator.wikimedia.org/T329612) (owner: 10Ilias Sarantopoulos) [09:13:04] (03CR) 10Elukey: [C: 03+1] nsfw: upgrade python to 3.9 and debian to bullseye [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/890473 (https://phabricator.wikimedia.org/T329612) (owner: 10Ilias Sarantopoulos) [09:19:44] (03CR) 10Ilias Sarantopoulos: [C: 03+2] nsfw: upgrade python to 3.9 and debian to bullseye [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/890473 (https://phabricator.wikimedia.org/T329612) (owner: 10Ilias Sarantopoulos) [09:20:46] (03Merged) 10jenkins-bot: nsfw: upgrade python to 3.9 and debian to bullseye [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/890473 (https://phabricator.wikimedia.org/T329612) (owner: 10Ilias Sarantopoulos) [10:31:06] ok we also don't have any istio/knative metric anymore :D [10:31:14] same thing for wikikube codfw, not sure why [10:32:38] 10Machine-Learning-Team, 10Shared-Data-Infrastructure, 10Epic, 10Patch-For-Review: Upgrade DSE to k8s 1.23 - https://phabricator.wikimedia.org/T330261 (10elukey) New PKI intermediates added, puppet/deployment-charts changes are out for review as well :) [11:22:15] klausman: o/ so I am almost sure that the 10s issue lies in the Knative Activator pods [11:22:18] https://knative.dev/docs/serving/load-balancing/activator-capacity/ [11:22:28] I am trying to change some settings but so far not a lot of luck [11:22:38] the pods are inserted in the request path under certain conditions [11:22:48] and I think that the 10s are a sort of backpressure mechanism [11:30:28] ok very nice, now that we have the metrics in istio I see that it is definitely the knative-local-gateway adding the latency [11:34:17] * elukey lunch! [12:19:17] Fixed all my issues with the laptop upgrade \o/. Turns out my pip issue was caused by my ISP 🔥 🤦‍♀️ [12:28:54] elukey: very nice work! [12:37:04] * isaranto lunch o clock [13:41:11] 10Machine-Learning-Team: Create separate blubberfile and pipeline for revert-risk multilingual model - https://phabricator.wikimedia.org/T329936 (10achou) 05Open→03Resolved [13:53:58] 10Machine-Learning-Team, 10Research: Proposal: Create a stream end point for Revision Risk Model - https://phabricator.wikimedia.org/T326179 (10achou) [13:58:37] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10Ottomata) Please do! :) [14:26:48] klausman: o/ [14:26:59] we'd need to restart ORES for openssl upgrades, do you have time? [14:55:38] (I'll start with codfw) [15:01:54] 10Machine-Learning-Team: Investigate ModelMesh architecture - https://phabricator.wikimedia.org/T330408 (10isarantopoulos) [15:02:34] 10Machine-Learning-Team: Investigate ModelMesh architecture - https://phabricator.wikimedia.org/T330408 (10isarantopoulos) a:03isarantopoulos [15:04:28] 10Machine-Learning-Team: Investigate ModelMesh architecture - https://phabricator.wikimedia.org/T330408 (10isarantopoulos) [15:06:46] elukey: sure [15:07:08] elukey: by restart you mean only ORES components? Or full reboots to be 100% sure? [15:09:13] klausman: nono I mean running the cookbook [15:09:18] I am doing it for codfw [15:09:22] you can do eqiad if you have time [15:09:45] in parallel to you, or do you want me to wait? [15:15:11] 10Machine-Learning-Team, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10Isaac) > Is it ok if I arrange a meeting (maybe next week?), including the ML team, Isaac and Ottomata to discuss the source ev... [15:17:25] elukey: ^^ parallel yay/nay? [15:22:26] klausman: mine just finished, so all good [15:22:33] alright, starting now [15:35:03] 10Machine-Learning-Team, 10Edit-Review-Improvements-Integrated-Filters, 10Growth-Team, 10Research: Integration of Revert Risk Scores to Recent Changes as a filter - https://phabricator.wikimedia.org/T329071 (10elukey) @Ladsgroup Hi :) Sorry to drag you in this task but I think that you are definitely one o... [15:42:13] elukey: eqiad done, checking alerts etc now [15:43:09] super [15:59:04] 10Machine-Learning-Team: Upgrade the ml-staging-codfw cluster to k8s 1.23 - https://phabricator.wikimedia.org/T327767 (10elukey) This is what the istio ingress gateway sees: ` { "method":"POST", "authority":"enwiki-goodfaith-predictor-default.revscoring-editquality-goodfaith.svc.cluster.local", "downstream_l... [16:11:04] going to log off for today folks, have a nice evening! [16:18:00] \o [16:28:59] 10Machine-Learning-Team: Create ORES migration endpoint (ORES/Liftwing translation) - https://phabricator.wikimedia.org/T330414 (10isarantopoulos) [16:29:42] o/ [16:30:15] ☝️ I created the task we discussed in the team meeting and logging off as well [16:30:31] good evening/rest of day 👋 [16:47:07] 10Machine-Learning-Team: Migrate ORES clients to LiftWing - https://phabricator.wikimedia.org/T312518 (10Aklapper) [16:47:10] 10Lift-Wing, 10Machine-Learning-Team, 10Documentation: Create technical documentation for Lift Wing Infrastructure - https://phabricator.wikimedia.org/T276601 (10Aklapper)