[07:43:09] hi folks :) [07:47:15] 10Machine-Learning-Team, 10Epic: Lift Wing improvements to get out of MVP state - https://phabricator.wikimedia.org/T333453 (10elukey) [07:47:17] 10Machine-Learning-Team, 10serviceops, 10Platform Team Initiatives (API Gateway): Review LiftWing's usage of the API Gateway - https://phabricator.wikimedia.org/T340982 (10elukey) 05Open→03Resolved a:03elukey Change deployed to the API Gateway, thank's all for the feedback and the chats on IRC (Alexand... [08:19:24] FYI, ml-etcd1002 will briefly go down for a reboot [08:22:55] 10Machine-Learning-Team: Revert Risk multi-lingual model performance and reliability may need a review - https://phabricator.wikimedia.org/T340822 (10achou) > is it possible to get an overview of the model's response time (not only for the errors, but in general?) I reviewed kserve's logs from June 30 to July 3... [08:31:48] (03CR) 10AikoChou: [C: 03+2] "Thanks for the review :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/935068 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [08:35:57] moritzm: ack! [08:36:17] * elukey running some errands [08:37:36] (03Merged) 10jenkins-bot: readability: add nltk tokenizers download to blubber's builder [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/935068 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [08:41:41] hello! [08:45:08] o/ morning! [09:07:10] going afk for 30'-40' [09:31:05] ml-etcd1001 will also briefly go down [10:02:02] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10Sgs) >>! In T308135#8639664, @kevinbazira wrote: > @kostajh, we published datasets for all 17/19 models that passed the evaluation in... [10:27:54] (03CR) 10Elukey: "Looks very nice, I just added two comments related to the uwsgi.ini, that is very specific to the cloud vps instance and it will probably " [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) (owner: 10Kevin Bazira) [10:33:03] 10Machine-Learning-Team, 10API Platform, 10Anti-Harassment, 10Cloud-Services, and 18 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10kostajh) In Gerrit / PipelineLib workflow, the PipelineBot makes a comment in Gerrit with the newly published image tag names, [exa... [10:36:57] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10Sgs) I ran this script for adding the link-recommendation task type and populating the excluded sections entries: `lang=bash for WIKI... [10:37:36] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 11th round of wikis - https://phabricator.wikimedia.org/T308136 (10Sgs) a:03Sgs [10:39:48] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 11th round of wikis - https://phabricator.wikimedia.org/T308136 (10Sgs) [10:39:55] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10Sgs) 05Open→03In progress [10:41:05] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10kevinbazira) @Sgs, yes koiwiki's dataset can be used. Regarding kywiki, 17/19 models were published in this round because kywiki's tr... [10:49:32] (03PS2) 10AikoChou: revert-risk: raise http 405 when failing to fetch info from MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/935159 (https://phabricator.wikimedia.org/T341008) [10:51:00] (03CR) 10AikoChou: revert-risk: raise http 405 when failing to fetch info from MW API (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/935159 (https://phabricator.wikimedia.org/T341008) (owner: 10AikoChou) [10:58:56] (03CR) 10Elukey: revert-risk: raise http 405 when failing to fetch info from MW API (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/935159 (https://phabricator.wikimedia.org/T341008) (owner: 10AikoChou) [11:00:01] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10Sgs) @kevinbazira thank you. I updated the configuration for //koiwiki// as well. [11:00:13] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 11th round of wikis - https://phabricator.wikimedia.org/T308136 (10Sgs) p:05Triage→03Medium [11:01:02] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 11th round of wikis - https://phabricator.wikimedia.org/T308136 (10Sgs) 05Open→03In progress [11:28:27] * elukey lunch [11:42:05] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 11th round of wikis - https://phabricator.wikimedia.org/T308136 (10Sgs) I ran this script for adding the link-recommendation task type and populating the excluded sections entries: `lang=bash for WIKI... [11:45:06] (03PS24) 10Kevin Bazira: Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) [11:48:29] (03CR) 10Kevin Bazira: Set up production and test images for the recommendation-api migration (032 comments) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) (owner: 10Kevin Bazira) [12:13:02] going for a late lunch. after that I'll deploy revscoring model servers (goodfaith and damaging) on both ml-serve eqiad and codfw. [13:01:37] I'm getting an error with the testwiki isvc regarding knative that a revision is missing [13:01:37] ``` [13:01:37] Warning InternalError 2m45s v1beta1Controllers fails to reconcile predictor: fails to update knative service: Operation cannot be fulfilled on services.serving.knative.dev "testwiki-goodfaith-predictor-default": the object has been modified; please apply your changes to the latest version and try again [13:01:37] ``` [13:01:37] elukey: any hints? [13:01:50] this is on eqiad [13:04:38] Good morning all [13:04:53] I've officially celebrated America's independence without being killed by fireworks [13:07:36] Hey Chris! hope you had a great time! fireworks are crazy! [13:08:24] elukey: nevermind the above seems to have been resolved [13:08:31] my neighborhood is famous for everyone firing off their own fireworks (which is definitely illegal in a city) so every yera it is like absolute madness [13:09:38] chrisalbon_: nice :D [13:09:43] isaranto: are the pods up? [13:09:52] yep [13:10:23] okok perfect [13:11:13] ah ok there was some update to do to all the other pods [13:11:21] to since they are a lot, it takes a bit [13:13:25] ack [13:13:32] yes there were image and chart updates [13:37:39] aiko_: really nice work in https://phabricator.wikimedia.org/T340822#8989704 ! [13:47:50] elukey: thank you :D [13:54:15] TIL https://quarry.wmcloud.org/ [13:54:46] wow! great work aiko_ [13:54:52] 🚀 [14:00:07] (03PS3) 10AikoChou: revert-risk: raise http 400 when failing to fetch info from MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/935159 (https://phabricator.wikimedia.org/T341008) [14:08:01] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 12th round of wikis - https://phabricator.wikimedia.org/T308137 (10Sgs) a:05kostajh→03Sgs [14:09:54] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 12th round of wikis - https://phabricator.wikimedia.org/T308137 (10Sgs) [14:10:51] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 12th round of wikis - https://phabricator.wikimedia.org/T308137 (10Sgs) 05Open→03In progress [14:11:52] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 12th round of wikis - https://phabricator.wikimedia.org/T308137 (10Sgs) p:05Triage→03Medium [14:33:24] ml-etcd1003 will also briefly go down [14:40:56] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-notice: Deploy "add a link" to 12th round of wikis - https://phabricator.wikimedia.org/T308137 (10Sgs) I ran this script for adding the link-recommendation task type and populating the excluded sections entrie... [15:16:06] (03CR) 10Elukey: [C: 03+1] revert-risk: raise http 400 when failing to fetch info from MW API (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/935159 (https://phabricator.wikimedia.org/T341008) (owner: 10AikoChou) [15:35:43] 10Machine-Learning-Team, 10Research (FY2022-23-Research-April-June): (stretch) Deploy multilingual readability model to LiftWing - https://phabricator.wikimedia.org/T334182 (10achou) The readability model has been deployed to LiftWing staging. It is available via an internal endpoint. Test the model: ` aikoch... [15:41:22] 10Machine-Learning-Team: Revert Risk multi-lingual model performance and reliability may need a review - https://phabricator.wikimedia.org/T340822 (10elukey) >>! In T340822#8984654, @Trokhymovych wrote: > I have reviewed the [[ https://pastebin.com/3jj5FZxk | logs ]] with errors from the multilingual model, and... [16:04:04] going afk folks! o/ [16:07:07] we are going to w8 for wmf.16 train deployment to finish before we deploy ores extension to testwiki [16:07:12] going afk as well [16:09:04] o/ bye Luca and Ilias, have a nice evening! [16:13:18] (03CR) 10AikoChou: [C: 03+2] "Thanks for the review!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/935159 (https://phabricator.wikimedia.org/T341008) (owner: 10AikoChou) [16:16:55] (03Merged) 10jenkins-bot: revert-risk: raise http 400 when failing to fetch info from MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/935159 (https://phabricator.wikimedia.org/T341008) (owner: 10AikoChou) [17:31:46] 10Machine-Learning-Team, 10ContentTranslation, 10Research, 10Epic: Verify if the Python recommendation API can support the use cases of the nodejs one - https://phabricator.wikimedia.org/T340854 (10Isaac) @elukey sure anytime! for what it's worth, as part of an analysis a few years back, I translated the l...