[08:17:46] morning folks :) [08:31:28] mooorning! [08:33:04] aiko: 欢迎回来! [08:33:28] (I knew only the last part, I had to look up the first one :D) [09:02:24] 06Machine-Learning-Team, 10ORES: Train and test newcomer quality models for deployment in ORES - https://phabricator.wikimedia.org/T123095#11310773 (10Aklapper) p:05Medium→03Low [09:20:45] 06Machine-Learning-Team, 07Essential-Work, 13Patch-For-Review: Update kserve to 0.15.2 - https://phabricator.wikimedia.org/T367048#11310819 (10klausman) The issue on ml-lab1001 is (was, now) that docker did not use the big multi-TB filesystem as storage for images, but it does now. I copied the Dockerfile fr... [09:23:28] 06Machine-Learning-Team, 07Essential-Work, 13Patch-For-Review: Update kserve to 0.15.2 - https://phabricator.wikimedia.org/T367048#11310821 (10klausman) {F67626690} And here is the complete build log of the run mentioned above. [09:54:24] elukey: yay 我回来了! -> this should be easy for you then :D [10:06:08] that is "I am back!" right? [10:09:31] elukey: if you `docker-pkg -c config.yaml build images/ --select '*amd-vllm*'` how long does it take for you overall? [10:10:41] o/ [10:10:56] I haven't tried, but IIRC it is a very expensive build, it requires a ton of ram [10:11:03] so it may take a while [10:13:03] it seems to be very slow for me to download apt packages and i'm unsure if it's just me or it's a general problem [10:14:06] dpogorzelski: what patch are you using specifically? Just so i can try and replicate [10:14:17] (i have a 1gbps up/down fiber at home so should be plenty) [10:14:41] 1146891 I presume? [10:14:46] lemme confirm [10:15:10] yea [10:15:28] giving it a whirl now [10:15:56] building amd-pytorch-common was less <1M [10:16:50] and package fetches from http://mirrors.wikimedia.org/debian seem to be decently fast (I have gigabit as well) [10:16:55] https://www.irccloud.com/pastebin/krfNLknv/ [10:17:55] klausman: Dawid is using https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1146891 [10:17:56] The bookworm-wikimedia/thirdparty/amd-rocm63 stuff takes a bit longer, because the packages are huge [10:18:00] that may be very heavy [10:18:36] but i mean 187mb package like rocsparse amd64 3.3.0.60300-39~22.04 should be a blip on a gbps connection [10:18:50] it takes longer than i would expect [10:19:04] so was. just wondering if it's only me or others too :P [10:19:38] or maybe it's the mirror that is clogged [10:20:53] looking at my network monitoring, I am getting about 80Mbit peak out of the mirror [10:21:04] not much [10:21:45] Off of Steam, I usualy get > 10x that >:) [10:22:06] i'll run a few experiments with maybe regular debian mirrors to compare [10:22:47] 06Machine-Learning-Team, 10EditCheck, 10VisualEditor: [SPIKE] Define process for validating Tone Check model eval data for languages staff members do not speak - https://phabricator.wikimedia.org/T407155#11311202 (10achou) [10:22:51] package fetches are now complete here. [10:23:01] So that took ~6m [10:23:12] still going here, so perhaps i'm limited by other factors [10:23:14] thx [10:24:35] if you are hitting our apt repos, you are definitely rate limited using an external IP [10:24:40] finished fetching now and unpacking is decent speed despite running amd on arm via emulation. so i'll check things [10:24:41] https://i.imgur.com/tpXkNfG.png [10:25:19] elukey: aha, gotcha [11:18:50] 06Machine-Learning-Team, 10Recommendation-API: Collection data unavailable in several rec-api hosts - https://phabricator.wikimedia.org/T406854#11311369 (10Nikerabbit) [11:37:01] 06Machine-Learning-Team, 10EditCheck, 10VisualEditor: [SPIKE] Define process for validating Tone Check model eval data for languages staff members do not speak - https://phabricator.wikimedia.org/T407155#11311431 (10achou) Our process for collecting eval data differs from the training/eval data collection pr... [12:07:16] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models - https://phabricator.wikimedia.org/T408388 (10DMburugu) 03NEW [12:08:13] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models - https://phabricator.wikimedia.org/T408388#11311513 (10DMburugu) [12:08:17] 06Machine-Learning-Team, 05Goal: Q1 FY2025-26 Goal: Enable volunteer evaluation of Tone Check model in additional languages - https://phabricator.wikimedia.org/T400423#11311515 (10achou) @Sucheta-Salgaonkar-WMF, George is OoO this week, let me update the investigation here. Progress update on the hypothesis... [12:08:41] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models - https://phabricator.wikimedia.org/T408388#11311517 (10DMburugu) [12:11:05] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models - https://phabricator.wikimedia.org/T408388#11311533 (10DMburugu) [12:12:53] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models - https://phabricator.wikimedia.org/T408388#11311537 (10DMburugu) [12:30:11] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models - https://phabricator.wikimedia.org/T408388#11311579 (10Samwalton9-WMF) > Create a l... [12:49:25] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models - https://phabricator.wikimedia.org/T408388#11311612 (10DMburugu) Thanks for that. W... [12:54:17] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models - https://phabricator.wikimedia.org/T408388#11311627 (10isarantopoulos) Will you be... [12:54:29] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models - https://phabricator.wikimedia.org/T408388#11311630 (10DMburugu) I see there are wi... [12:57:31] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models - https://phabricator.wikimedia.org/T408388#11311651 (10DMburugu) @isarantopoulos Ye... [14:38:13] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models - https://phabricator.wikimedia.org/T408388#11312067 (10DMburugu) [14:58:18] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312153 (10Eevans) [15:01:15] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312156 (10Eevans) [15:03:22] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312163 (10Eevans) [15:07:02] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312184 (10Eevans) [15:10:15] is there a way to build these images from home without being rate limited? [15:12:57] I don't think so. I mean, one could try and cheat by tunneling the requests to the package server through SSH, but I doubt SRE would be fond of that. The only other option I see is to have a local caching proxy [15:13:53] probably having a reliable ml-lab host to test with a GPU is the best bet, I know that Kevin has some troubles as well with bandwidth and local compute power when it comes to test those images [15:14:57] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312246 (10Ottomata) I like it! Some field naming suggestions: - `paragraph_i... [15:16:17] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312271 (10Eevans) [15:18:39] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312293 (10achou) > QUESTION: What is Tone Suggestion Generator (as referenced... [15:32:04] ml-serve2001 has been down for a while, if you have time later one please give it a check :) [15:36:21] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312488 (10Eevans) >>! In T401021#11312293, @achou wrote: >> QUESTION: What is... [15:38:39] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312493 (10Ottomata) > consumes the mediawiki.page_content_change.v1 events (tr... [15:44:34] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312545 (10Ottomata) > But also, it kind of is cache isn't? I'm not sure if it... [15:45:01] happy to take a look but have to finish the getting access+getting root parts of onboarding [15:54:14] 06Machine-Learning-Team, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Machine Learning host migrations - https://phabricator.wikimedia.org/T405647#11312569 (10klausman) >>! In T405647#11298600, @RobH wrote: > Please note this migration has shifted from Oct 15th start date to Nov 1 start date. I'll be avail... [16:01:40] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312595 (10Michael) >>! In T401021#11307866, @Eevans wrote: > I've created a dr... [16:06:10] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312636 (10Eevans) >>! In T401021#11312545, @Ottomata wrote: >> But also, it ki... [16:10:34] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312654 (10Eevans) >>! In T401021#11312595, @Michael wrote: >>>! In T401021#113... [21:07:30] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11313785 (10Eevans) >>! In T401021#11313193, @Michael wrote: >>>! In T401021#113... [23:47:36] (03PS1) 10Zabe: Emit deprecation warnings when a model is configured with types [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1199087 (https://phabricator.wikimedia.org/T74157)