[02:52:01] 06Machine-Learning-Team, 10Research-engineering, 06Research (FY2025-26-Research-October-December): Share code between Research & ML teams - https://phabricator.wikimedia.org/T398974#11288346 (10fkaelin) Weekly updates - The suggested contributions are described in this [[ https://docs.google.com/document/d/1... [04:46:17] 06Machine-Learning-Team, 07Essential-Work: Merge tone-check pipeline DAGs into a single DAG for simplified orchestration - https://phabricator.wikimedia.org/T407212#11288359 (10kevinbazira) With the configuratoins described in T407212#11280133, the `tone_check_training_dag` ran end-to-end in production just li... [04:47:37] 06Machine-Learning-Team, 07Essential-Work: Merge tone-check pipeline DAGs into a single DAG for simplified orchestration - https://phabricator.wikimedia.org/T407212#11288360 (10kevinbazira) 05Open→03Resolved [06:17:45] 06Machine-Learning-Team, 07Essential-Work: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288441 (10kevinbazira) @brouberol, thank you for enabling the triggerer process in Airflow. I tested it using [[ https://gitlab... [07:07:21] 06Machine-Learning-Team, 07Essential-Work: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288489 (10brouberol) Hmm, it appears that we're affected by https://github.com/apache/airflow/issues/34644, which has no resolu... [07:11:01] hello! [07:20:42] 06Machine-Learning-Team, 07Essential-Work: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288496 (10brouberol) Can you try to pass the `in_cluster=True` argument to the `WMFKubernetesPodOperator` that also has `deferr... [07:29:22] good morning [07:44:28] good morning! ☀️ [07:49:12] 06Machine-Learning-Team, 07Essential-Work: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288531 (10kevinbazira) >>! In T406958#11288496, @brouberol wrote: > Can you try to pass the `in_cluster=True` argument to the `... [08:10:12] 06Machine-Learning-Team, 07Essential-Work: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288565 (10brouberol) Ah, that's progress! That seems to be the pod trying to call the kubernetes API, which is prevented by a N... [08:16:15] 06Machine-Learning-Team, 07Essential-Work: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288583 (10brouberol) Also note that now that we have fine-grained labels on GPU models and capabilities, you can drop the `affi... [08:40:22] 06Machine-Learning-Team, 07Essential-Work: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288668 (10brouberol) Ah wait, I think I read the error wrong! ` Trigger emitted an error event, failing the task: Cannot connec... [08:46:45] 06Machine-Learning-Team, 07Essential-Work: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288699 (10brouberol) I now see that the triggerer is allowed to hit the kubernetes API by the network policy, but is denied //p... [08:49:27] 06Machine-Learning-Team, 07Essential-Work: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288744 (10brouberol) ` brouberol@deploy2002:~$ k get pod -l release=dev-kevinbazira,component=triggerer -o json | jq '.items[0]... [08:52:09] 06Machine-Learning-Team, 07Essential-Work: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288763 (10brouberol) That seems to have done the trick! ` [08:54:45] 06Machine-Learning-Team, 07Essential-Work: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288779 (10kevinbazira) Super! The DAG task with a deferrable operator has now succeeded in the `airflow-devenv`. {F66765931 wid... [09:01:43] 06Machine-Learning-Team, 07Essential-Work, 13Patch-For-Review: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288814 (10brouberol) Feel free to destroy/redeploy your whole devenv in about ~5 minutes, and check wheth... [09:26:00] 06Machine-Learning-Team, 07Essential-Work, 13Patch-For-Review: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288858 (10kevinbazira) >>! In T406958#11288814, @brouberol wrote: > Feel free to destroy/redeploy your wh... [09:27:00] 06Machine-Learning-Team, 07Essential-Work, 13Patch-For-Review: Enable Airflow triggerer process for deferrable operators in airflow-ml and airflow-devenv - https://phabricator.wikimedia.org/T406958#11288860 (10brouberol) Np, good to know it worked! [09:29:12] https://slack-status.com/ There seems to be some issues with Slack today [09:30:58] AWS is suffering from issues atm, so there are a couple of ripple effects across the internet atm [09:41:33] yeap there are many of them, slack, signal, perplexityAI as well [09:42:52] brouberol: Whenever you find some time please leave your thoughts and ideas on this ticket: https://phabricator.wikimedia.org/T406217#11284799 , I am currently working on that one and your ideas/help would be very helpful. [09:59:01] I'll have a look this afternoon 👍 [10:26:01] brouberol: Thnx a lot! [11:12:18] 06Machine-Learning-Team, 13Patch-For-Review: Export retrained Tone-check model to an S3 bucket - https://phabricator.wikimedia.org/T406217#11289067 (10brouberol) > Do we need specific permissions for copying files from the PVC to an S3 bucket on the POD side and/or in the container itself? No. The only thing y... [11:27:11] 06Machine-Learning-Team, 13Patch-For-Review: Export retrained Tone-check model to an S3 bucket - https://phabricator.wikimedia.org/T406217#11289085 (10gkyziridis) > Q: How do you find this architecture to build the model_export functionality on the side of ml-piplines repo and not in the Airflow-DAGs repo? > A... [11:39:32] 06Machine-Learning-Team, 13Patch-For-Review: Export retrained Tone-check model to an S3 bucket - https://phabricator.wikimedia.org/T406217#11289134 (10brouberol) > Can I attach the PVC using the PythonOperator in order to use the wmf_airflow_common.clients.s3 and do the job using the "thanos_swift" connection?... [11:51:45] 06Machine-Learning-Team, 13Patch-For-Review: Export retrained Tone-check model to an S3 bucket - https://phabricator.wikimedia.org/T406217#11289154 (10gkyziridis) > Can I do the opposite? Use the wmf_airflow_common.clients.s3 directly in WMFKubernetesPodOperator ? > To do this, you'd need to have your Kuberne... [11:54:04] 06Machine-Learning-Team, 13Patch-For-Review: Export retrained Tone-check model to an S3 bucket - https://phabricator.wikimedia.org/T406217#11289157 (10brouberol) Yep, that //should// work! [12:39:39] (03PS2) 10Bartosz Wójtowicz: articletopic: Remove the transformer code. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1190570 (https://phabricator.wikimedia.org/T404294) [12:41:48] o/ I'm looking for a review on removing the legacy code from article topic model :D The CI jobs got removed already so we can progress with removing the code from inference-services repo https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1190570 [12:49:45] looking ... [12:50:54] (03PS3) 10Bartosz Wójtowicz: articletopic: Remove the transformer code. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1190570 (https://phabricator.wikimedia.org/T404294) [13:01:16] (03CR) 10Kevin Bazira: "There are still references of the articletopic transformer in the Makefile:" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1190570 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [13:02:44] thank you for the review Kevin <3 [13:04:44] np :) [13:06:28] (03PS4) 10Bartosz Wójtowicz: articletopic: Remove the transformer code. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1190570 (https://phabricator.wikimedia.org/T404294) [13:07:01] (03CR) 10Bartosz Wójtowicz: "I updated the later part of the Makefile, but missed this one, thank you! Now I've updated this part as well and tested locally to confirm" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1190570 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [13:11:56] (03CR) 10Kevin Bazira: articletopic: Remove the transformer code. (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1190570 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [13:17:17] (03PS5) 10Bartosz Wójtowicz: articletopic: Remove the transformer code. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1190570 (https://phabricator.wikimedia.org/T404294) [13:18:28] (03CR) 10Bartosz Wójtowicz: articletopic: Remove the transformer code. (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1190570 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [13:20:08] (03CR) 10Kevin Bazira: [C:03+1] "LGTM! Thank you for working on this." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1190570 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [13:26:18] (03CR) 10Bartosz Wójtowicz: [C:03+2] articletopic: Remove the transformer code. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1190570 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [13:29:10] (03Merged) 10jenkins-bot: articletopic: Remove the transformer code. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1190570 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [14:26:55] 06Machine-Learning-Team, 13Patch-For-Review: Export retrained Tone-check model to an S3 bucket - https://phabricator.wikimedia.org/T406217#11289889 (10gkyziridis) === Update === I moved all the code and logic into the [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/export_model_to_s3... [15:12:06] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team: Introduce case sensitivity to machine learning model for Add a Link - https://phabricator.wikimedia.org/T405185#11290144 (10Trizek-WMF) Thank you Kirsten. So far, I only heard about this at English Wikipedia. [17:10:54] 06Machine-Learning-Team, 10Research-engineering, 06Research (FY2025-26-Research-October-December): Share code between Research & ML teams - https://phabricator.wikimedia.org/T398974#11290666 (10Ottomata) Very cool! The 'Command API' sounds like a very useful thing outside of just ML and Research jobs. As... [18:28:22] 10Lift-Wing: LiftWing fiwiki-damaging model returning 500 - https://phabricator.wikimedia.org/T407784 (104shadoww) 03NEW