[06:54:57] hello! [06:55:22] going to deploy the ores extension UI to idwiki in a bit 🤞 [07:14:38] Good morning! succes! [07:27:28] yyeees it works fine! [07:27:49] 06Machine-Learning-Team: Migrate all Lift Wing k8s workers to Bookworm and containerd - https://phabricator.wikimedia.org/T387854#10862241 (10elukey) 05Open→03Resolved We are done! [07:28:11] you may find the filter here https://id.wikipedia.org/wiki/Istimewa:Perubahan_terbaru?namespace=0&limit=50&days=0.0833&urlversion=2 [07:30:33] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10862249 (10isarantopoulos) > rc.rc_timestamp > rc.rc_timestamp - INTERVAL 30 DAY; This is a great query 🤦 This would... [07:33:01] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10862260 (10isarantopoulos) The filter is live on idwiki 🎉 I cross checked during the deployment that the filters wor... [07:35:19] Hey, you can find meeting notes with Fabian here: https://phabricator.wikimedia.org/T393474#10848501 and with Micheal here: https://phabricator.wikimedia.org/T393474#10857007 . Please feel free to let me know if you have any suggestions! 💗 [07:38:00] thank you sooo much for being transparent and sharing these Ozge, helps a ton! [07:57:15] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10862353 (10isarantopoulos) I've noticed that a message is not translated and fallbacks to english although a translat... [08:04:26] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10862371 (10isarantopoulos) [08:04:28] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06DBA, 10MediaWiki-Recent-changes, and 2 others: [Epic] Recent Changes ORES Enabled Revert Risk Powered Filters Rollout Plan - https://phabricator.wikimedia.org/T391964#10862372 (10isarantopoulos) [08:17:19] hey folks! I just sent a series of code reviews for the knative migration of ml-serve-eqiad and cleanup [08:17:38] it is a lot I know but the move is a bit complex :) [08:17:48] lemme know if you want to have a meeting to discuss etc.. [08:18:07] but we already used the procedure elsewhere, the extra bit now is a final cleanup of unused settings [08:28:04] 06Machine-Learning-Team, 10Add-Link, 06Growth-Team, 05Goal: Q4 24-25 Goal: Investigate Add-a-link model training and deployment - https://phabricator.wikimedia.org/T393474#10862469 (10OKarakaya-WMF) - How to use research dataset environment on notebook. Following the [readme](https://gitlab.wikimedia.org/r... [08:39:45] 06Machine-Learning-Team, 13Patch-For-Review: Deploy peacock/tone check model to production - https://phabricator.wikimedia.org/T394779#10862557 (10gkyziridis) 05Open→03Resolved [08:40:14] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: Run analysis to retrieve thresholds for high impact wikis to deploy recent changes revert risk language agnostic filters to - https://phabricator.wikimedia.org/T392148#10862561 (10gkyziridis) 05Open→03Resolved [08:49:47] o/ elukey we'll check them today and ping you if we need more context. thanks! [08:51:55] oookkk [09:16:10] elukey: is there a particular order you want them reviewed in? [09:19:47] klausman: o/ yes they are already ordered in the chain, I think it makes sense to do a gradual rollout like that one (at least, this is what we did for codfw) [09:20:00] Yeah, sounds good. [09:20:02] it is important though that we are on the same page about the changes that we are making [09:20:12] so please ask if anything is not clear (even the broader team) [09:20:19] Ack [10:13:33] 06Machine-Learning-Team, 10Add-Link, 06Growth-Team, 05Goal: Q4 24-25 Goal: Investigate Add-a-link model training and deployment - https://phabricator.wikimedia.org/T393474#10862842 (10OKarakaya-WMF) Hi @SuchetaG and @isarantopoulos , I'm sharing some topics that we can have in the decision brief below.... [10:16:11] * isaranto afk - bbl [10:50:42] * klausman lunch [11:43:53] 06Machine-Learning-Team, 10EditCheck, 10Editing-team (Tracking): Publish summary of lessons from Tone Check MVP - https://phabricator.wikimedia.org/T393109#10863122 (10Aklapper) [11:44:35] 06Machine-Learning-Team, 10EditCheck, 10VisualEditor, 07Chinese-Sites, 10Editing-team (Tracking): Prepare annotool for Tone Check model evaluation - https://phabricator.wikimedia.org/T392324#10863125 (10Aklapper) [11:45:47] 06Machine-Learning-Team, 10EditCheck, 10VisualEditor, 10Editing-team (Tracking): Evaluate efficacy of Tone Check model output (internal review) - https://phabricator.wikimedia.org/T384651#10863144 (10Aklapper) [11:46:36] 06Machine-Learning-Team, 13Patch-For-Review: Deploy tone check model to production - https://phabricator.wikimedia.org/T394779#10863154 (10Aklapper) [12:31:21] knative images upgraded on ml-serve-eqiad [12:31:44] no changes, it is just to backport patches etc.. [12:31:53] no setting changes I meant [12:33:05] all pods up [12:33:38] second step is to allow seccomp for ml-serve-eqiad (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1151597) [12:34:39] and done as well [12:34:53] now the next step would be https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1151600 [12:35:26] that involves deploying all namespaces, in both staging (to verify that a cleanup works correctly) and ml-serve-eqiad [12:35:57] since it is a big one it needs to be scheduled so we don't do anything on the clusters for say a couple of hours [12:47:04] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06DBA, 10MediaWiki-Recent-changes, and 2 others: [Epic] Recent Changes ORES Enabled Revert Risk Powered Filters Rollout Plan - https://phabricator.wikimedia.org/T391964#10863336 (10isarantopoulos) From the above initial list the extension is already ena... [12:49:14] isaranto: could you take a look at patch updating docker images on staging when you'll have free time? 🥺 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1150696 [12:52:05] since georgekyz has already reviewed your free to merge.. I'm in a meeting with George and he said that he has double checked the images [12:52:18] elukey: thanks for the rollout! [12:52:27] bartosz: I think some revscoring models are missing though [12:52:41] bartosz: We can check together in a while [12:52:42] I can comment later, I am in meetings now :) [12:56:49] isaranto: Thank you, I was also a little confused with the revscoring models so I'm happy to check together with georgekyz: a little later 🙌 [12:57:30] ok, thanks! sorry I should have mentioned this morning that you don't need to wait for me [12:59:54] no worries! [13:00:57] sry be there in 2' [13:03:11] klausman: np! We'd need a 2-hours window (more or less) to rollout https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1151600, should we do it any of these days? [13:04:37] Yeah, sounds good. I am out tomorrow, and we probably don't want to do it on Firday, so how does Monday sound? [13:05:31] sure makes sense, could you make sure that your team knows etc?? [13:06:24] aye, will do [13:22:50] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10863561 (10gkyziridis) ===== Deploying Ores Extension I was thinking that since we will deploy on batches, I would l... [13:25:16] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06DBA, 10MediaWiki-Recent-changes, and 2 others: [Epic] Recent Changes ORES Enabled Revert Risk Powered Filters Rollout Plan - https://phabricator.wikimedia.org/T391964#10863571 (10gkyziridis) ===== Deploying Ores Extension 3 Batches I was thinking that... [13:52:37] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban): [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10863709 (10isarantopoulos) Actually the same happens with the title of the filter: `ores-rcfilters-revertrisklanguage... [14:00:35] (03PS1) 10Kgraessle: [Spike] Investigate why filtering wasn't working on testwiki [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1151700 (https://phabricator.wikimedia.org/T395256) [14:20:13] (03PS1) 10Gkyziridis: ores-extension: Add extra logging [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1151706 (https://phabricator.wikimedia.org/T395253) [14:24:11] isaranto: I tried to add some extra logging logic in the backfill script `maintenance/PopulateDatabase.php`. I am not sure how to test it tho, have a look when you have time: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ORES/+/1151706?tab=checks [14:26:32] (03PS2) 10Gkyziridis: ores-extension: Add extra logging [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1151706 (https://phabricator.wikimedia.org/T395253) [14:30:56] Folks, should we first get an approval on a patch and then hit the `Schedule backport of this change` button ? [14:32:11] ok I'm back from meetings [14:33:11] georgekyz: we can test the backfill script locally. A nice approach would be to use a model running locally so you can manipulate the responses as you wish [14:34:25] isaranto: alright I am on it [14:36:38] re: approvals before backport. I'm not sure but the only reason to schedule a backport without an approval would be if you are in a hurry and want to do it tomorrow right? [14:37:56] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban), 13Patch-For-Review: [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10863980 (10Kgraessle) I tried to fix the highlighting issue, but wasn't able to get the `ext.or... [14:44:23] isaranto: I am definitely not in a hurry especially for tomorrow :P [14:46:26] hehe [14:46:38] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Moderator-Tools-Team, 10Wikimedia-Extension-setup, 10Wikimedia-Site-requests: Install ORES extension on idwiki - https://phabricator.wikimedia.org/T382171#10864035 (10Kgraessle) [14:46:51] if you need help with testing the backfill script I'm available [14:59:31] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban), 13Patch-For-Review: [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10864120 (10isarantopoulos) Ok got it. If the [[ https://wikitech.wikimedia.org/wiki/Deployments... [15:12:48] isaranto: I think the script is working, but I do not have revisions so the output is: [15:12:54] ``` [15:12:55] Processing 1 revisions [15:12:55] ScoreFetcher errored for 2: RevisionNotFound [15:12:55] Finished processing the revisions [15:12:55] Revisions successfully scored: 1 [15:12:55] Total errors: 1 [15:12:55] HTTP 4xx errors: 0 [15:12:55] HTTP 5xx errors: 0 [15:12:56] ``` [15:13:42] bartosz: I just reviewed and left a comment .you were right regarding the missing image tags. they don't exist in all values-ml-staging-codfw.yaml files as if it is missing the production one is used [15:14:40] georgekyz: you could manually add some to the database. [15:14:56] in the above example shouldn't the 4xx counter be 1 since the revision was not found? [15:15:22] for 5xx you can try to make requests to a url that doesn't exist [15:18:00] isaranto: I see, thank you! ❤️ I was also suspecting that we're using the defaults defined in values.yaml for both production and staging in those cases. Will merge the current patch and update the values.yaml in the future patch targeting production updates [15:18:28] sounds great! [15:19:15] hmmm 👍 [15:19:46] it is trying to capture regex in the message that's why it was 0 [15:21:30] bartosz: happy merging 🤞 [15:36:44] I've merged the MR and started working on the syncing, it seems I've stumbled across first issue with articlequality model, which is now unhappy and hitting `_catboost.CatBoostError: catboost/libs/model/model_import_interface.h:19: Model file doesn't exist: /mnt/models/catboost_model.cbm` [15:36:51] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Wikipedia-Android-App-Backlog, 05FY2024-25 WE4.2, 10Moderator-Tools-Team (Kanban): Add revertrisk-language-agnostic to RecentChanges filters - https://phabricator.wikimedia.org/T348298#10864391 (10Kgraessle) [15:36:56] The `model.pkl` file exists in the STORAGE_URI it's pointing to, but maybe the problem is the name mismatch between `model.pkl` and `catboost_model.cbm`? [15:48:25] ^ I've found the issue, we need to update STORAGE_URI as well to one containing both model files. Will submit a small patch [15:48:45] 06Machine-Learning-Team, 13Patch-For-Review: Deploy tone check model to production - https://phabricator.wikimedia.org/T394779#10864504 (10isarantopoulos) 05Resolved→03Open > [] Update the WIP patch in VisualEditor to adapt to the above 2 changes Looking at https://gerrit.wikimedia.org/r/c/mediawiki/extens... [15:50:22] ok, got it! [15:50:46] bartosz: you can browse the public repository for the models here. https://analytics.wikimedia.org/published/wmf-ml-models/articlequality/language-agnostic/ [15:51:15] I guess we should go with https://analytics.wikimedia.org/published/wmf-ml-models/articlequality/language-agnostic/20250425125943/ ozge_ could validate this perhaps. [15:51:35] just add Ozge as a reviewer and we can figure this out on Friday. [15:54:45] isaranto: thanks, I didn't know I can explore them easily like this 🙌 I've noticed that we use the one you shared in inference-services, I'll create a patch and we can fix this on Friday! [15:57:48] Going afk folks , have a nice afternoon/rest of day! [15:58:15] And enjoy the holiday tomorrow-- at least those who have a holiday :) [16:02:31] enjoy the rest of the day and see you on Friday! [16:04:56] enjoy the rest of your day ilias [16:05:16] Have a nice evening and see you on Friday! [16:06:07] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Wikipedia-Android-App-Backlog, 05FY2024-25 WE4.2, 10Moderator-Tools-Team (Kanban): Ensure all ORES i18n messages are available for wikis to add revert risk language agnostic filters to - https://phabricator.wikimedia.org/T395481 (10Kgraessle) 03NEW [16:07:11] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban), 13Patch-For-Review: [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10864692 (10Kgraessle) >>! In T395256#10864120, @isarantopoulos wrote: > Ok got it. If the [[ ht... [16:08:53] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Wikipedia-Android-App-Backlog, 05FY2024-25 WE4.2, 10Moderator-Tools-Team (Kanban): Ensure all ORES i18n messages are available for wikis to add revert risk language agnostic filters to - https://phabricator.wikimedia.org/T395481#10864712 (10Kgraessle... [16:33:14] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Wikipedia-Android-App-Backlog, 05FY2024-25 WE4.2, 10Moderator-Tools-Team (Kanban): Ensure all ORES i18n messages are available for wikis to add revert risk language agnostic filters to - https://phabricator.wikimedia.org/T395481#10864861 (10Nemoralis... [16:35:59] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Wikipedia-Android-App-Backlog, 05FY2024-25 WE4.2, 10Moderator-Tools-Team (Kanban): Ensure all ORES i18n messages are available for wikis to add revert risk language agnostic filters to - https://phabricator.wikimedia.org/T395481#10864865 (10A_smart_k... [16:37:04] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Wikipedia-Android-App-Backlog, 05FY2024-25 WE4.2, 10Moderator-Tools-Team (Kanban): Ensure all ORES i18n messages are available for wikis to add revert risk language agnostic filters to - https://phabricator.wikimedia.org/T395481#10864873 (10Kgraessle) [18:56:21] 10Lift-Wing, 06Machine-Learning-Team, 10EditCheck: Create SLO dashboard for tone (peacock) check model - https://phabricator.wikimedia.org/T390706#10865498 (10ppelberg) [21:29:04] (03PS5) 10Kgraessle: Fix highlighting for revertrisklanguageagnostic model. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1151700 (https://phabricator.wikimedia.org/T395256) [21:29:04] (03CR) 10Kgraessle: "recheck" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1151700 (https://phabricator.wikimedia.org/T395256) (owner: 10Kgraessle) [21:32:13] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban), 13Patch-For-Review: [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10865969 (10PatchDemoBot) Test wiki **created** on [[ https://patchdemo.wmcloud.org | Patch demo... [21:33:28] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban), 13Patch-For-Review: [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10865984 (10PatchDemoBot) Test wiki on [[ https://patchdemo.wmcloud.org | Patch demo ]] by KGra... [21:35:09] (03PS6) 10Kgraessle: Fix highlighting for revertrisklanguageagnostic model. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1151700 (https://phabricator.wikimedia.org/T395256) [21:50:11] (03PS6) 10Kgraessle: Fix highlighting for revertrisklanguageagnostic model. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1151700 (https://phabricator.wikimedia.org/T395256) [21:51:51] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban), 13Patch-For-Review: [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10866017 (10Kgraessle) Ok, I have a [[ https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ORE...