[06:03:14] 06Data-Engineering, 06Product-Analytics: Allow curl commands from Airflow BashOperator - https://phabricator.wikimedia.org/T392288#10763657 (10brouberol) So, let's sum things up a bit. If you're using the `PythonOperator`, you're executing a python function in an airflow pod, as we're using the KubernetesExecu... [06:15:33] 06Data-Engineering, 06Product-Analytics: Allow curl commands from Airflow BashOperator - https://phabricator.wikimedia.org/T392288#10763677 (10brouberol) > Direct use of BashOperator and PythonOperator in this way should be discouraged :/ This makes the Airflow scheduler do job work This I don't understand th... [06:50:14] (03CR) 10Filippo Giunchedi: "Summary from the meeting me, Hasan and Lucas had yesterday:" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [08:03:17] (03PS1) 10Majavah: refinery-core: iputils: Update Cloud VPS IP space [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1138675 (https://phabricator.wikimedia.org/T392468) [08:52:46] 06Data-Engineering, 06Data-Platform-SRE, 06Java-Scala-Standardization, 10Discovery-Search (2025.04.11 - 2025.05.02), 13Patch-For-Review: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all depen... - https://phabricator.wikimedia.org/T367405#10763917 [09:06:13] 06Data-Engineering, 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10763981 (10taavi) >>! In T392468#10762365, @Ottomata wrote: > @taavi thanks! What's the timeline for this? We've just made it possible to... [11:11:17] 10Data-Engineering (Q4 2025 April 1st - June 30th): [Refine DAG Improvement] Add Parameter to Reduce Spark Driver Logs in Skein Log Collection - https://phabricator.wikimedia.org/T381074#10764383 (10Antoine_Quhen) 05In progress→03Resolved The custom logger config is currently active on Refine staging dag. [11:58:09] 06Data-Engineering: Airflow mapped tasks UI & metrics - https://phabricator.wikimedia.org/T357430#10764461 (10Antoine_Quhen) 05Open→03Resolved Closing the ticket: • UI: The improvements have already been implemented with the upgrade to Airflow 2.5. • Metrics: The legacy Grafana dashboard for Airflow task... [13:05:48] 06Data-Engineering, 06Discovery-Search, 06Infrastructure-Foundations, 10Data-Platform-SRE (2025-04-12 - 2025-05-02): Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#10764743 (10MoritzMuehlenhoff) >>! In T390860#10746399, @Volans wrote: > As reported on the parent t... [13:08:30] 06Data-Engineering, 10Data-Platform-SRE (2025-04-12 - 2025-05-02), 13Patch-For-Review, 10Sustainability (Incident Followup): airflow: Consider restricting the rights for airflow deployers to destroy postgresql clusters - https://phabricator.wikimedia.org/T391348#10764748 (10brouberol) a:03brouberol [13:08:39] 06Data-Engineering, 10Data-Platform-SRE (2025-04-12 - 2025-05-02), 13Patch-For-Review, 10Sustainability (Incident Followup): airflow: Consider restricting the rights for airflow deployers to destroy postgresql clusters - https://phabricator.wikimedia.org/T391348#10764752 (10brouberol) 05Open→03In progre... [13:21:37] (03PS3) 10Mforns: Add MinT for Readers stream to sanitization allow list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1133484 (https://phabricator.wikimedia.org/T372724) (owner: 10KCVelaga) [13:21:49] 06Data-Engineering, 06Data-Platform-SRE, 06Java-Scala-Standardization, 10Discovery-Search (2025.04.11 - 2025.05.02), 13Patch-For-Review: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all depen... - https://phabricator.wikimedia.org/T367405#10764795 [13:21:58] (03CR) 10Mforns: [V:03+2 C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1133484 (https://phabricator.wikimedia.org/T372724) (owner: 10KCVelaga) [13:22:28] (03CR) 10Mforns: [V:03+2 C:03+2] Add MinT for Readers stream to sanitization allow list (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1133484 (https://phabricator.wikimedia.org/T372724) (owner: 10KCVelaga) [13:35:02] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Create table and pyspark job to produce wmf_content.mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T391282#10764842 (10xcollazo) Backfilled the table via: ` hostname -f an-launcher1002.eqiad.wmn... [13:52:54] 06Data-Engineering, 06Product-Analytics: Allow curl commands from Airflow BashOperator - https://phabricator.wikimedia.org/T392288#10764874 (10Ottomata) > If you're using the PythonOperator, you're executing a python function in an airflow pod, as we're using the KubernetesExecutor by default Oh! So this aut... [14:07:27] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Add data quality metrics to mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T392494#10764918 (10xcollazo) Example runs via Presto: Counts: ` presto:wmf_content> select count(1) as count from mediawiki_content_curren... [14:56:33] (03CR) 10Hasan Akgün (WMDE): "I have tested it and it's confirmed that statsd-exporter aggregates results as expected, we can proceed with it." [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [14:57:55] (03CR) 10Lucas Werkmeister (WMDE): "Great! Here’s the Phabricator task for setting up statsd-exporter: T392599" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [15:52:58] 06Data-Engineering, 06Discovery-Search, 06Infrastructure-Foundations, 10Data-Platform-SRE (2025-04-12 - 2025-05-02): Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#10765552 (10bking) Not to get too far off-topic, but have y'all considered something like [[ https:/... [18:35:32] 10Data-Engineering (Q4 2025 April 1st - June 30th): NEW BUG REPORT significantly increased edit revert rate for 2025-03 edits; Android, iOS, Mobile Web, Other - https://phabricator.wikimedia.org/T391708#10766091 (10xcollazo) From [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_request... [18:41:37] 06Data-Engineering: Strange statistics for language variants for languages other than zh and sr - https://phabricator.wikimedia.org/T392624 (10Amire80) 03NEW [18:44:02] 10Data-Engineering (Q4 2025 April 1st - June 30th): NEW BUG REPORT significantly increased edit revert rate for 2025-03 edits; Android, iOS, Mobile Web, Other - https://phabricator.wikimedia.org/T391708#10766137 (10xcollazo) >>! In T391708#10763120, @Ahoelzl wrote: > ... > We are attempting now a re-run of the 2... [18:48:09] 10Data-Engineering (Q4 2025 April 1st - June 30th): NEW BUG REPORT significantly increased edit revert rate for 2025-03 edits; Android, iOS, Mobile Web, Other - https://phabricator.wikimedia.org/T391708#10766158 (10xcollazo) Run #4 details: [[ https://airflow.wikimedia.org/dags/mediawiki_history_denormalize/gri... [18:53:17] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Create Airflow pipeline to produce wmf_content.mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T391283#10766162 (10xcollazo) Two successful runs of the Airflow job so far. 🎉 🎉 🎉 I do want to test out running it wi... [20:26:43] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Investigate artifact mismatch error when running mw_content_merge_events_to_mw_content_history_daily - https://phabricator.wikimedia.org/T391123#10766469 (10xcollazo) [[ https://github.com/apache/hadoop/blob/965fd380006fa78b2315668fbc... [20:46:42] 06Data-Engineering: Strange statistics for language variants for languages other than zh and sr - https://phabricator.wikimedia.org/T392624#10766525 (10Isaac) Adding some context from an [[https://wikimedia.slack.com/archives/CSV483812/p1745219869302509|internal discussion]] to hopefully help whoever picks this... [20:49:02] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab: FY 24-25 SDS 2.4.9 CDN Synthetic Beacon: EventGate & Varnish: update to receive events from beacon event v2 - https://phabricator.wikimedia.org/T391959#10766527 (10dr0ptp4kt) [20:51:27] 06Data-Engineering: Strange statistics for language variants for languages other than zh and sr - https://phabricator.wikimedia.org/T392624#10766552 (10Amire80) >>! In T392624#10766525, @Isaac wrote: > Adding some context from an [[https://wikimedia.slack.com/archives/CSV483812/p1745219869302509|internal discuss... [20:52:46] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab: FY 24-25 SDS 2.4.9 CDN Synthetic Beacon: EventGate & Varnish: update to receive events from beacon event v2 - https://phabricator.wikimedia.org/T391959#10766554 (10dr0ptp4kt) [20:53:52] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab: FY 24-25 SDS 2.4.9 CDN Synthetic Beacon: EventGate & Varnish: update to receive events from beacon event v2 - https://phabricator.wikimedia.org/T391959#10766556 (10dr0ptp4kt)