[00:00:31] (03PS1) 10Tim Starling: Don't pass a TitleValue to WatchedItemStore [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1199088 [00:12:46] (03CR) 10CI reject: [V:04-1] Don't pass a TitleValue to WatchedItemStore [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1199088 (owner: 10Tim Starling) [00:30:04] (03CR) 10Tim Starling: [C:03+2] Emit deprecation warnings when a model is configured with types [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1199087 (https://phabricator.wikimedia.org/T74157) (owner: 10Zabe) [00:43:12] (03Merged) 10jenkins-bot: Emit deprecation warnings when a model is configured with types [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1199087 (https://phabricator.wikimedia.org/T74157) (owner: 10Zabe) [08:10:29] 06Machine-Learning-Team: DIMM_A2 errors for ml-serve2001 - https://phabricator.wikimedia.org/T408516 (10elukey) 03NEW [08:11:23] 06Machine-Learning-Team: DIMM_A2 errors for ml-serve2001 - https://phabricator.wikimedia.org/T408516#11317528 (10ops-monitoring-bot) Host ml-serve2001 powercycled by elukey@cumin2002 with reason: None [08:12:06] o/ testing a new cookbook --^ [08:18:32] 06Machine-Learning-Team: DIMM_A2 errors for ml-serve2001 - https://phabricator.wikimedia.org/T408516#11317532 (10ops-monitoring-bot) Host ml-serve2001 powercycled by elukey@cumin2002 with reason: None [08:30:04] 06Machine-Learning-Team: DIMM_A2 errors for ml-serve2001 - https://phabricator.wikimedia.org/T408516#11317575 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by elukey@cumin1003 depool for host ml-serve2001.codfw.wmnet completed: - ml-serve2001.codfw.wmnet (**PASS**) - Host ml-serve2... [08:30:17] cordoned the node, will do some tests [08:31:19] (03PS2) 10Tim Starling: Don't pass a TitleValue to WatchedItemStore [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1199088 [08:32:33] morning! [08:33:10] elukey: yes that means I'm back! [08:33:36] \o/ [08:34:06] 早上好！ [08:42:58] 早上好!! [08:54:07] 06Machine-Learning-Team: DIMM_A2 errors for ml-serve2001 - https://phabricator.wikimedia.org/T408516#11317664 (10ops-monitoring-bot) Host ml-serve2001 powercycled by elukey@cumin2002 with reason: Testing powercycle cookbook [09:49:30] 06Machine-Learning-Team: Initial task generation and ingestion to Cassandra and Search weight tags - https://phabricator.wikimedia.org/T408533 (10achou) 03NEW [09:50:11] 06Machine-Learning-Team, 05Goal: Q1 FY2025-26 Goal: Task generation engine for Revise Tone task - https://phabricator.wikimedia.org/T408341#11317924 (10achou) [09:50:12] 06Machine-Learning-Team: Initial task generation and ingestion to Cassandra and Search weight tags - https://phabricator.wikimedia.org/T408533#11317923 (10achou) [10:15:54] 06Machine-Learning-Team: Initial task generation and ingestion to Cassandra and Search weight tags - https://phabricator.wikimedia.org/T408533#11318030 (10achou) [10:19:57] 06Machine-Learning-Team: Create a Tone Suggestion Generator in LiftWing - https://phabricator.wikimedia.org/T408538 (10achou) 03NEW [10:20:28] 06Machine-Learning-Team: Create a Tone Suggestion Generator in LiftWing - https://phabricator.wikimedia.org/T408538#11318049 (10achou) [10:20:38] 06Machine-Learning-Team, 05Goal: Q1 FY2025-26 Goal: Task generation engine for Revise Tone task - https://phabricator.wikimedia.org/T408341#11318050 (10achou) [11:30:47] 06Machine-Learning-Team: DIMM_A2 errors for ml-serve2001 - https://phabricator.wikimedia.org/T408516#11318272 (10ops-monitoring-bot) Host ml-serve2001 powercycled by elukey@cumin2002 with reason: Testing powercycle cookbook [13:28:28] (03CR) 10Samtar: [C:03+2] Don't pass a TitleValue to WatchedItemStore [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1199088 (owner: 10Tim Starling) [13:40:59] (03Merged) 10jenkins-bot: Don't pass a TitleValue to WatchedItemStore [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1199088 (owner: 10Tim Starling) [14:10:12] 10Lift-Wing, 06Machine-Learning-Team, 10Wikidata, 06Wikimedia Enterprise, 10Wikimedia Enterprise - Content Integrity: Request to host Wikidata Revert Risk on Lift Wing - https://phabricator.wikimedia.org/T406179#11318838 (10kevinbazira) Thank you for sharing this information, @Trokhymovych. In T333125,... [15:11:37] 06Machine-Learning-Team, 06DC-Ops, 10ops-codfw: DIMM_A2 errors for ml-serve2001 - https://phabricator.wikimedia.org/T408516#11319244 (10elukey) [15:12:22] 06Machine-Learning-Team, 06DC-Ops, 10ops-codfw: DIMM_A2 errors for ml-serve2001 - https://phabricator.wikimedia.org/T408516#11319258 (10elukey) The host is up after a powercycle, but it is still not serving any traffic. Adding dcops if they want to investigate it further, giving the numerous occurrences of t... [15:13:04] 06Machine-Learning-Team, 05Goal: Q1 FY2025-26 Goal: Task generation engine for Revise Tone task - https://phabricator.wikimedia.org/T408341#11319264 (10achou) [15:13:34] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11319263 (10achou) [15:21:39] 06Machine-Learning-Team, 06DC-Ops, 10ops-codfw: DIMM_A2 errors for ml-serve2001 - https://phabricator.wikimedia.org/T408516#11319327 (10Jhancock.wm) @elukey is it depooled? i wanna check some things out that might require some reboots. [15:24:16] 06Machine-Learning-Team, 06DC-Ops, 10ops-codfw: DIMM_A2 errors for ml-serve2001 - https://phabricator.wikimedia.org/T408516#11319338 (10elukey) @Jhancock.wm yep you can go ahead! Thanks :) [16:39:43] 06Machine-Learning-Team: Create a Tone Suggestion Generator in LiftWing - https://phabricator.wikimedia.org/T408538#11319935 (10achou) @BWojtowicz-WMF Here is [[ https://gitlab.wikimedia.org/repos/machine-learning/exploratory-notebook/-/blob/main/tone-check/task_generation_script.ipynb | a notebook ]] that demon... [16:55:53] 06Machine-Learning-Team, 13Patch-For-Review: Experiment with amd-smi and the new AMD GPUs MI300x - https://phabricator.wikimedia.org/T403697#11320117 (10elukey) Created also https://github.com/ROCm/amdsmi/issues/134 since ROCm 7.0.2 seems to have a different GPU usage format when partitions are used. [16:58:27] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: AI/ML Infrastructure Request: Assistance in Rolling out Revert Risk to wikis that don't have damaging/goodfaith models - https://phabricator.wikimedia.org/T408607 (10DMburugu) 03NEW [16:59:06] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: AI/ML Infrastructure Request: Assistance in Rolling out Revert Risk to wikis that don't have damaging/goodfaith models - https://phabricator.wikimedia.org/T408607#11320143 (10DMburugu) [16:59:07] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 10PersonalDashboard: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models - https://phabricator.wikimedia.org/T408388#11320144 (10DMburugu) [17:04:27] 06Machine-Learning-Team: Create a Tone Suggestion Generator in LiftWing - https://phabricator.wikimedia.org/T408538#11320158 (10achou) Hi @Ottomata, I have a question -- how can I get a sample of the "mediawiki_page_content_change_v1" event from production Kafka clusters? I don't see the stream in https://stream... [17:31:09] 06Machine-Learning-Team: Create a Tone Suggestion Generator in LiftWing - https://phabricator.wikimedia.org/T408538#11320322 (10Ottomata) Ya, currently codfw is the active datacenter, so only its topic will have real data. Try: ` $ kafkacat -C -b kafka-jumbo1010.eqiad.wmnet:9092 -o -10 -c 10 -t codfw.mediawiki... [18:34:13] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11320625 (10Michael) >>! In T401021#11313785, @Eevans wrote: >>>! In T401021#113... [19:02:02] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11320756 (10achou) I think the key questions to clarify are: - What does model_... [20:18:52] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11321018 (10achou) >>! In T401021#11313270, @Ottomata wrote: > ...and also back... [20:51:07] 06Machine-Learning-Team, 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11321183 (10Ottomata) > IIUC, you're okay with not naming this table more specif...