[01:56:20] (03CR) 10Eamedina: [C:03+2] "LGTM" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1149449 (https://phabricator.wikimedia.org/T395026) (owner: 10Sbisson) [01:56:34] (03CR) 10Eamedina: [C:03+2] Make SearchRecommender inherit from BaseRecommender [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1148932 (owner: 10Sbisson) [01:57:03] (03Merged) 10jenkins-bot: Fully encode URLs for GET requests [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1149449 (https://phabricator.wikimedia.org/T395026) (owner: 10Sbisson) [01:58:10] (03Merged) 10jenkins-bot: Make SearchRecommender inherit from BaseRecommender [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1148932 (owner: 10Sbisson) [06:39:38] Good morning! [06:47:20] Good morning [07:02:13] morning! [07:12:24] good morning [08:06:00] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Data-Persistence, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): DBA Review of Tables that ORES Extension will create - https://phabricator.wikimedia.org/T391103#10851404 (10Marostegui) [08:35:58] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Data-Persistence, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): DBA Review of Tables that ORES Extension will create - https://phabricator.wikimedia.org/T391103#10851463 (10isarantopoulos) From the above initial list the extension is... [09:15:25] 06Machine-Learning-Team: Deploy peacock/tone check model to production - https://phabricator.wikimedia.org/T394779#10851589 (10gkyziridis) [09:29:15] georgekyz: o/ has the api gw change been deployed? I tried `curl -i https://api.wikimedia.org/service/lw/inference/v1/models/edit-check-staging:predict -X POST -d '{ "instances": [{"lang": "en", "check_type": "peacock", "original_text": "", "modified_text": "this is a great example of work"}]}'` but it doesnt work [09:29:45] ok the old one still works [09:29:54] the edit-check-staging [09:30:16] fwiw, I am not aware of a API GW change being pushed recently. [09:32:12] 06Machine-Learning-Team: Deploy peacock/tone check model to production - https://phabricator.wikimedia.org/T394779#10851642 (10isarantopoulos) After the discussing we had with the editing team, the following things are pending: - change the "check_type": "peacock", to "check_type": "tone", in the request/respons... [09:32:42] o/ Tobias! [09:32:44] this is the change I'm referring to https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1149375 [09:33:13] yeah, saw the commit. checking the diff on deploy1003 [09:33:40] georgekyz: we should sync with Tobias (or another SRE) to merge and deploy the changes as MLEs don't have access to that [09:33:52] or do we? I don't recall atm [09:34:18] No, you don't. It's running on the main k8s cluster [09:34:56] I'll coordinate with Georgios and Hugh (there's also a chart update) [09:35:27] I see that I can do a helmfile diff, don't know if I can also sync [09:36:08] klausman: lemme know when it is time to deploy so I can try to run helfmile sync on staging [09:36:15] *helmfile [09:37:17] ack, will do [09:37:32] Danke! [09:44:33] (03PS1) 10Ilias Sarantopoulos: edit-check: rename peacock check to tone check [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149620 (https://phabricator.wikimedia.org/T394779) [09:49:14] klausman: ping me when you have time. [09:49:46] isaranto, georgekyz: hugh is fine with us pushing. [09:49:53] georgekyz: I implemented the change --^ hope I didn't step on your toes :D [09:50:14] I can try to sync to see if I have permissions then [09:50:41] (03CR) 10CI reject: [V:04-1] edit-check: rename peacock check to tone check [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149620 (https://phabricator.wikimedia.org/T394779) (owner: 10Ilias Sarantopoulos) [09:50:45] Note that nominally, the main cluster has a staging instance in both eiad and codfw, but only the eqiad one is really used AIUI [09:51:10] ack [09:51:44] isaranto: no problem I think we need to change also the `model_name` at the settings [09:51:46] But this would DTRT: helmfile -e staging -i diff --context= [09:52:00] er, add a 3 at the end for the diff context [09:52:57] If you then replace diff with apply, it will show the diff again, but ask for confirmation to push (that's what -i does) [09:53:08] ok it seems that I can sync -- staging synced [09:53:14] nice! [09:53:28] isaranto: Do you want me to fix the tests at: https://gerrit.wikimedia.org/r/1149620 [09:53:29] can confirm that there is no diff anymore [09:53:59] georgekyz: I'll fix it, thanks for checking [09:54:01] I see one 4c pod running [09:54:20] Let's let it run for 5m, see if it explodes. [09:56:17] (03PS1) 10Ilias Sarantopoulos: edit-check: rename peacock check to tone check [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149622 (https://phabricator.wikimedia.org/T394779) [09:57:14] I updated the patch, hope it works now. georgekyz why would we change the model_name in settings? afaiu it is edit-check so i would stay the same [09:57:19] (03CR) 10Gkyziridis: "Thank you for working on this." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149620 (https://phabricator.wikimedia.org/T394779) (owner: 10Ilias Sarantopoulos) [09:58:02] ah you created an other one ? [09:58:03] staging looks good. Feel free to push on codfw. (or I can do it) [10:00:24] (03PS1) 10Ilias Sarantopoulos: edit-check: rename peacock check to tone check [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149626 (https://phabricator.wikimedia.org/T394779) [10:01:08] georgekyz: sorry I'm gonna fix it and ping you. don't know why or how I created an extra patch [10:01:13] klausman: pushing to codfw! [10:03:37] Looks good so far [10:05:07] 06Machine-Learning-Team, 10ContentTranslation: Suggestion: Add AI-assisted post-editing for machine-translated content in Content Translation tool - https://phabricator.wikimedia.org/T395111 (10Wahbarz01) 03NEW [10:07:08] (03PS2) 10Ilias Sarantopoulos: edit-check: rename peacock check to tone check [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149620 (https://phabricator.wikimedia.org/T394779) [10:07:24] (03Abandoned) 10Ilias Sarantopoulos: edit-check: rename peacock check to tone check [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149622 (https://phabricator.wikimedia.org/T394779) (owner: 10Ilias Sarantopoulos) [10:08:50] (03CR) 10Ilias Sarantopoulos: "I changed the tests as well to mention tone instead of peacock." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149620 (https://phabricator.wikimedia.org/T394779) (owner: 10Ilias Sarantopoulos) [10:14:00] shall I proceed with eqiad as well? [10:14:09] yeah, go ahead [10:14:52] (03CR) 10Gkyziridis: "Done" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149620 (https://phabricator.wikimedia.org/T394779) (owner: 10Ilias Sarantopoulos) [10:15:02] (03CR) 10Gkyziridis: [C:03+1] edit-check: rename peacock check to tone check [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149620 (https://phabricator.wikimedia.org/T394779) (owner: 10Ilias Sarantopoulos) [10:15:25] done and it works already! [10:15:36] 07artificial-intelligence, 06Machine-Learning-Team, 10ContentTranslation, 10Community Wishlist (Translations): Global AI-based translation and post-editing engine for Content Translation - https://phabricator.wikimedia.org/T395111#10851744 (10Wahbarz01) [10:17:49] I confirm! Works already! Thank you folks! [10:17:54] https://www.irccloud.com/pastebin/848AM4Vs/ [10:18:24] np! [10:18:41] I'm heading for lunch and a few errands, bbiab [10:18:42] isaranto: Should we merge this: https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1149620 in order to deploy the new image ? [10:19:20] (03CR) 10Ilias Sarantopoulos: [C:03+2] edit-check: rename peacock check to tone check [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149620 (https://phabricator.wikimedia.org/T394779) (owner: 10Ilias Sarantopoulos) [10:19:28] nice [10:19:32] done! [10:20:03] (03Merged) 10jenkins-bot: edit-check: rename peacock check to tone check [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149620 (https://phabricator.wikimedia.org/T394779) (owner: 10Ilias Sarantopoulos) [10:20:04] I can do the rest if you want [10:20:07] klausman: thanks for the help & watching my back <3, have a nice lunch! [10:20:14] georgekyz: yes please! [10:22:22] <3 [10:24:09] 06Machine-Learning-Team, 13Patch-For-Review: Deploy peacock/tone check model to production - https://phabricator.wikimedia.org/T394779#10851756 (10gkyziridis) >>! In T394779#10851642, @isarantopoulos wrote: > After the discussing we had with the editing team, the following things are pending: > - change the "c... [10:33:36] I added also httpbb tests https://gerrit.wikimedia.org/r/c/operations/puppet/+/1149634 feel free to add more as you see fit [10:51:44] I’m planning to merge the patch bumping Python target version to 3.9 (https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1148282), local tests show that everything runs as smoothly as before [10:52:22] please let me know if you have any concerns of it introducing merge conflicts with any ongoing urgent work. If not, I’ll hit the merge in in 1h [10:52:26] go gooooo ! [10:52:38] try a rebase first and then you can merge now! [10:53:57] (03PS3) 10Bartosz Wójtowicz: ci: Bump Python target-version from 3.7 to 3.9. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1148282 (https://phabricator.wikimedia.org/T393865) [10:55:10] oki, I rebased and will merge after checking CI if there aren't any concerns ;-) [10:57:06] 👍 [11:06:57] bartosz: Well done your patch is totally greeeen [11:07:15] Yeah suprisingly without out-of-storage this time :D Hitting the merge [11:07:18] 🎉 [11:07:21] niceee [11:07:32] (03CR) 10Bartosz Wójtowicz: [C:03+2] ci: Bump Python target-version from 3.7 to 3.9. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1148282 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [11:08:27] georgekyz: while checking the httpbb tests I noticed that there was no codfw deployment for edit check. could you sync to codfw as well. Otherwise users might be getting failures when making requests to the API GW as it uses the discovery endpoint which could route the request to either DCs [11:08:37] it is the first time I see the `gate-submit` and the `post-merge` running massively for all models! [11:08:53] *I forgot the questionmark (?) in my question above [11:11:11] isaranto: Done [11:11:58] you meant `ml-serve-codfw sync` on prod right ? [11:12:33] `ml-staging-codfw` was synced yesterday for edit-check [11:13:21] thnx! [11:13:58] yes I meant prod [11:14:21] alright, it works: [11:14:25] https://www.irccloud.com/pastebin/JrRvm2ni/ [11:15:52] Whenever you have time lets review this: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1149632 [11:17:43] done [11:19:47] (03Merged) 10jenkins-bot: ci: Bump Python target-version from 3.7 to 3.9. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1148282 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [11:35:02] both `gate-and-submit` and `postmerge` finished successfully, thank you team for help and reviews on those pre-commit patches ❤️ [11:35:53] Nice work bartosz ! <3 [11:41:18] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Data-Persistence, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): DBA Review of Tables that ORES Extension will create - https://phabricator.wikimedia.org/T391103#10851929 (10Ladsgroup) >Could someone run the maintain-views script for t... [11:41:57] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Data-Persistence, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): DBA Review of Tables that ORES Extension will create - https://phabricator.wikimedia.org/T391103#10851930 (10Ladsgroup) >>! In T391103#10851929, @Ladsgroup wrote: >>Could... [11:45:23] now that is a lot of staging deployments.... [11:45:49] deployments in general I mean but for today and since it is Friday we can stick with staging [11:47:47] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Data-Persistence, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): DBA Review of Tables that ORES Extension will create - https://phabricator.wikimedia.org/T391103#10851932 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.updat... [11:51:50] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Data-Persistence, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): DBA Review of Tables that ORES Extension will create - https://phabricator.wikimedia.org/T391103#10851938 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.updat... [11:56:19] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Data-Persistence, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): DBA Review of Tables that ORES Extension will create - https://phabricator.wikimedia.org/T391103#10851957 (10Ladsgroup) I ran it with multiple db options and only lawiki... [12:01:40] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06cloud-services-team, 06Data-Persistence, and 2 others: DBA Review of Tables that ORES Extension will create - https://phabricator.wikimedia.org/T391103#10852001 (10Marostegui) [12:10:12] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Data-Persistence, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): DBA Review of Tables that ORES Extension will create - https://phabricator.wikimedia.org/T391103#10852043 (10taavi) I split that to T395122 since this task is going to ge... [12:10:58] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Data-Persistence, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): DBA Review of Tables that ORES Extension will create - https://phabricator.wikimedia.org/T391103#10852052 (10isarantopoulos) Yes that makes sense. Once again thank you both! [12:31:21] (03Abandoned) 10Ilias Sarantopoulos: edit-check: rename peacock check to tone check [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1149626 (https://phabricator.wikimedia.org/T394779) (owner: 10Ilias Sarantopoulos) [12:32:12] going to deploy edit-check latest changes [12:33:49] ack! [12:43:00] Deployed! [12:44:04] wehoo [12:46:54] it works! I updated the ticket: https://phabricator.wikimedia.org/T394779#10851756 [12:48:01] I will also post the update in the slack channel [12:49:13] great work, thanks! [13:13:50] kevinbazira: if you want we can take a look together at the notebooks for simple article summaries on Monday [13:14:20] isaranto: sure sure, np! [15:47:51] * isaranto afk [17:02:05] 07artificial-intelligence, 10ContentTranslation: Global AI-based translation and post-editing engine for Content Translation - https://phabricator.wikimedia.org/T395111#10853176 (10Aklapper) @Wahbarz01: Thanks for reporting this. Please use the feature request form (linked from the top of the task creation pag... [18:33:10] 07artificial-intelligence, 10ContentTranslation: Global AI-based translation and post-editing engine for Content Translation - https://phabricator.wikimedia.org/T395111#10853469 (10Wahbarz01) >>! In T395111#10853176, @Aklapper wrote: > @Wahbarz01: Thanks for reporting this. Please use the feature request form... [18:44:31] (03PS1) 10Sbisson: Fix: Filter out section recommendations with no missing sections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1149719 (https://phabricator.wikimedia.org/T394441) [19:28:29] 07artificial-intelligence, 10ContentTranslation: Add AI-based translation and post-editing to Content Translation tool - https://phabricator.wikimedia.org/T395160#10853600 (10Pppery) [19:29:14] 07artificial-intelligence, 10ContentTranslation: Add AI-based translation and post-editing to Content Translation tool - https://phabricator.wikimedia.org/T395160#10853601 (10Pppery) [19:29:59] 07artificial-intelligence, 10ContentTranslation: Global AI-based translation and post-editing engine for Content Translation - https://phabricator.wikimedia.org/T395111#10853618 (10Pppery) →14Duplicate dup:03T395160 [19:30:01] 07artificial-intelligence, 10ContentTranslation: Add AI-based translation and post-editing to Content Translation tool - https://phabricator.wikimedia.org/T395160#10853620 (10Pppery) [19:42:48] 07artificial-intelligence, 10ContentTranslation: Add AI-based translation and post-editing to Content Translation tool - https://phabricator.wikimedia.org/T395160#10853673 (10Wahbarz01)