[00:31:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06SRE Observability, 13Patch-For-Review: Finalize and/or standardize metrics - https://phabricator.wikimedia.org/T438383#12409413 (10AKhatun_WMF) So in sum: We are keeping prefix. When we have a service utils this will be wildly simplified and automa... [00:45:00] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Create article feature counts dashboard - https://phabricator.wikimedia.org/T439065#12409426 (10AKhatun_WMF) [00:45:01] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Create dashboard(s) - https://phabricator.wikimedia.org/T438755#12409428 (10AKhatun_WMF) →14Duplicate dup:03T439065 [01:58:38] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027: Backfill and update schedule of gcp_gsc_export.max_date_sitedata_daily_aggregate - https://phabricator.wikimedia.org/T436999#12409478 (10nshahquinn-wmf) I dug into the 2026-03-01 data a bit and everything looked completely n... [02:25:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:50:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:59:50] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Handling Flagged Revs for Reference Counts in Attribution API - https://phabricator.wikimedia.org/T435557#12409550 (10AKhatun_WMF) We had some discussions T433607#12355443. Here's the summary and update: Previous state: > As of design review of 20 Aug,... [03:30:25] 06Data-Engineering, 06Data-Platform-SRE: Request for a dedicated Airflow instance for Commercial Partnerships (Enterprise) - https://phabricator.wikimedia.org/T440539 (10JArguello-WMF) 03NEW [03:30:46] 06Data-Engineering, 06Data-Platform-SRE, 06Wikimedia Enterprise: Request for a dedicated Airflow instance for Commercial Partnerships (Enterprise) - https://phabricator.wikimedia.org/T440539#12409594 (10JArguello-WMF) [03:40:43] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Create article feature counts dashboard - https://phabricator.wikimedia.org/T439065#12409612 (10AKhatun_WMF) Stub: https://grafana-rw.wikimedia.org/d/revision-feature-counts [07:27:31] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12409895 (10APizzata-WMF) [08:39:34] 06Data-Engineering, 06Data-Platform-SRE: Enable OIDC authentication and IAM authorization on the Data Platform Ceph clusters - https://phabricator.wikimedia.org/T435470#12410133 (10BTullis) [08:52:02] 06Data-Engineering, 06Data-Platform-SRE: Conda-Analytics environments are prone to dependency conflicts and installation errors - https://phabricator.wikimedia.org/T423067#12410201 (10BTullis) [08:52:15] 06Data-Engineering, 06Data-Platform-SRE: Conda-Analytics environments are prone to dependency conflicts and installation errors - https://phabricator.wikimedia.org/T423067#12410204 (10BTullis) [09:39:50] 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Conda-Analytics has package conflict when trying to install R with key packages (R-Arrow and R-Stringi) - https://phabricator.wikimedia.org/T391911#12410383 (10BTullis) I believe that this now works in `conda-analy... [09:39:57] 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Conda-Analytics has package conflict when trying to install R with key packages (R-Arrow and R-Stringi) - https://phabricator.wikimedia.org/T391911#12410385 (10BTullis) a:03BTullis [09:57:49] 06Data-Engineering, 06Data-Platform-SRE: Conda-Analytics environments stuck with very outdated packages - https://phabricator.wikimedia.org/T423052#12410437 (10BTullis) I have implemented a change to the way that the `conda-analytics` and `conda-analytics-next` packages are built, which I hope will improve the... [10:08:39] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Implement brownout logic for dumps v1 on dumps web interface - https://phabricator.wikimedia.org/T437461#12410464 (10BTullis) 05Open→03Resolved I checked the CPU/load for the servers after... [10:11:13] 10Data-Engineering-Roadmap, 07Epic: dbt automated initial backfill - https://phabricator.wikimedia.org/T438322#12410491 (10JAllemandou) We spent some time discussing this with @GGoncalves-WMF and arrived at an idea we were both happy with: * Models should run in both `incremental` and `non-incremental` fashio... [11:06:22] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Conda-Analytics environments are prone to dependency conflicts and installation errors - https://phabricator.wikimedia.org/T423067#12410771 (10BTullis) a:03BTullis [11:08:05] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Conda-Analytics environments are prone to dependency conflicts and installation errors - https://phabricator.wikimedia.org/T423067#12410785 (10BTullis) p:05Triage→03Medium [11:08:23] 06Data-Engineering, 06Data-Platform-SRE: Conda-Analytics environments stuck with very outdated packages - https://phabricator.wikimedia.org/T423052#12410789 (10BTullis) p:05Triage→03Medium [11:17:27] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA: Move SpecialUnusedTemplates computation to Hadoop - https://phabricator.wikimedia.org/T422443#12410814 (10Zabe) It seems the DAG works now and https://analytics.wikimedia.org/published/datasets/querypage/UnusedTemplates/commonswiki.json looks good. Thanks! [11:33:55] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Conda-Analytics environments are prone to dependency conflicts and installation errors - https://phabricator.wikimedia.org/T423067#12410884 (10BTullis) I have now pushed out version 0.0.42 of `conda-analytics`, which I hope will mitigate some o... [12:07:07] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10SRE-SLO: page_change SLO windows - https://phabricator.wikimedia.org/T438054#12411038 (10APizzata-WMF) Hey @RLazarus, thank you for the message and have fun on your PTO! The monthly data points are unfortunately a limitation of our systems. The sou... [13:00:49] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Handling Flagged Revs for Reference Counts in Attribution API - https://phabricator.wikimedia.org/T435557#12411299 (10Ottomata) > LAC clients would need a mechanism to wait async Hm, you know...MW PHP clients could submit a new LinkedArtifactPrecomputeJ... [13:23:35] (03PS9) 10Joal: Add spark ops metrics derived from event logs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1344687 (https://phabricator.wikimedia.org/T297231) [13:23:38] 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Conda-Analytics has package conflict when trying to install R with key packages (R-Arrow and R-Stringi) - https://phabricator.wikimedia.org/T391911#12411389 (10BTullis) 05Open→03Resolved I have also tested... [13:31:53] (03CR) 10CI reject: [V:04-1] Add spark ops metrics derived from event logs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1344687 (https://phabricator.wikimedia.org/T297231) (owner: 10Joal) [14:05:17] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Conda-Analytics environments are prone to dependency conflicts and installation errors - https://phabricator.wikimedia.org/T423067#12411626 (10BTullis) * T391911 is now fixed, so `conda install r-base r-arrow r-stringi` now works. * T435745 is... [14:05:39] (03CR) 10Joal: "> Would it make sense to build this in Spark 3.5 instead of 3.1?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1344687 (https://phabricator.wikimedia.org/T297231) (owner: 10Joal) [14:19:20] (03PS10) 10Joal: Add spark ops metrics derived from event logs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1344687 (https://phabricator.wikimedia.org/T297231) [14:19:25] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Handling Flagged Revs for Reference Counts in Attribution API - https://phabricator.wikimedia.org/T435557#12411708 (10AKhatun_WMF) Maybe no `RevisionLinkedArtifactComputedEvent` for such compute calls, because they are essentially API calls? Something si... [14:23:08] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Handling Flagged Revs for Reference Counts in Attribution API - https://phabricator.wikimedia.org/T435557#12411752 (10Ottomata) Yeha, I think we may still emit RevisionLinkedArtifactComputedEvent, but with the '[[ https://gerrit.wikimedia.org/g/mediawiki... [14:24:01] (03PS2) 10Joal: Add spark_stage_metrics table creation script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345860 (https://phabricator.wikimedia.org/T297231) [14:27:03] (03CR) 10Joal: Add spark_stage_metrics table creation script (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345860 (https://phabricator.wikimedia.org/T297231) (owner: 10Joal) [15:40:52] 06Data-Engineering, 06Data-Engineering-Radar, 07Essential-Work, 10Event-Platform, 10Test Kitchen (Experiment Platform Sprint 31): [EventGate] Perform GeoIP lookup at event ingestion time - https://phabricator.wikimedia.org/T434312#12412455 (10KReid-WMF) [15:42:30] 06Data-Engineering, 06Data-Engineering-Radar, 10Event-Platform, 07OKR-Work, 10Test Kitchen (Experiment Platform Sprint 31): [EventGate] Perform GeoIP lookup at event ingestion time - https://phabricator.wikimedia.org/T434312#12412468 (10JVanderhoop-WMF) [15:45:04] 06Data-Engineering, 06Data-Engineering-Radar, 10Event-Platform, 07OKR-Work, 10Test Kitchen (Experiment Platform Sprint 31): [EventGate] Perform GeoIP lookup at event ingestion time - https://phabricator.wikimedia.org/T434312#12412487 (10JVanderhoop-WMF) [16:04:40] (03PS11) 10Joal: Add spark ops metrics derived from event logs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1344687 (https://phabricator.wikimedia.org/T297231) [16:09:32] (03CR) 10Joal: Add spark ops metrics derived from event logs (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1344687 (https://phabricator.wikimedia.org/T297231) (owner: 10Joal) [16:17:23] 06Data-Engineering, 06Data-Engineering-Icebox, 13Patch-Needs-Improvement: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924#12412640 (10dcausse) Recording some limitations about the current schema evolution strategy: Even if refine were to... [16:48:50] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05acl*data-engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work, 13Patch-For-Review: Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12412838 (10atsuko) Fixed high-load on dag-processor, now all D... [17:39:53] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Include v2 dumps into mirrors rsync configuration - https://phabricator.wikimedia.org/T437463#12413064 (10xcollazo) >>! In T437463#12385980, @xcollazo wrote: >> Would we be able to easily move the cutoff to December 2026 instead? >... [17:42:16] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Movement-Insights (FY26-27 H1): dbt-jobs backfill: second-week retention metric - https://phabricator.wikimedia.org/T440249#12413079 (10amastilovic) Backfill completed, but I had to break it into two pieces: `2020-01-01` to `2023-01-01`, and the seco... [17:42:28] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Movement-Insights (FY26-27 H1): dbt-jobs backfill: second-week retention metric - https://phabricator.wikimedia.org/T440249#12413083 (10amastilovic) [18:02:55] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Current ruwiki History File Export run has failed 2 times, looks like it has struggled in the last couple months - https://phabricator.wikimedia.org/T440219#12413163 (10xcollazo) wikidatawiki failed today (2026-10-08) with the same error. Details at `ya... [18:05:07] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): ruwiki and wikidatawiki History File Export failures due to shuffle service rpc timeouts - https://phabricator.wikimedia.org/T440219#12413169 (10xcollazo) [18:11:31] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05MW-1.47-notes (1.47.0-wmf.23; 2026-10-06), 13Patch-For-Review: Linked Artifact Cache - Automated MediaWiki JobQueue precompute - https://phabricator.wikimedia.org/T436695#12413177 (10Ottomata) Oh! Today I realized that we can use a distinct job ty... [18:12:25] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Remove the spark_shuffle_3_3 and spark_shuffle_3_4 YARN aux services - https://phabricator.wikimedia.org/T440229#12413181 (10xcollazo) We just had a costly `wikidatawiki` file export failure due to shuffle service timeouts a... [18:23:36] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05MW-1.47-notes (1.47.0-wmf.23; 2026-10-06), 13Patch-For-Review: Linked Artifact Cache - Automated MediaWiki JobQueue precompute - https://phabricator.wikimedia.org/T436695#12413213 (10AKhatun_WMF) {meme, src=therockapplauds} [18:57:54] 06Data-Engineering: File Exports: Read one frozen Iceberg snapshot for each export run - https://phabricator.wikimedia.org/T439759#12413385 (10xcollazo) **Design so far** It changes the flow in the task description in two ways: a separate DAG task creates the tag, and the tag has `RETAIN 45 DAYS`, not 60. - A... [19:04:55] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Move the Spark 3.3.2 jobs to Spark 3.5.8 - https://phabricator.wikimedia.org/T440645 (10xcollazo) 03NEW [19:05:01] 06Data-Engineering, 07Essential-Work: Move the Spark 3.3.2 jobs to Spark 3.5.8 - https://phabricator.wikimedia.org/T440645#12413433 (10xcollazo) [19:06:54] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Remove the spark_shuffle_3_3 and spark_shuffle_3_4 YARN aux services - https://phabricator.wikimedia.org/T440229#12413439 (10xcollazo) Opened {T440645}. As can be seen over there, I under estimated the amount of work. I thou... [19:14:29] 06Data-Engineering, 07Essential-Work: Move the Spark 3.3.2 jobs to Spark 3.5.8 - https://phabricator.wikimedia.org/T440645#12413516 (10xcollazo) We read the Spark History Server to find all apps that ran on Spark 3.3 or 3.4 from 2026-08-09 to 2026-10-08. The History Server keeps event logs for 60 days. We grou... [19:15:10] (03PS1) 10Snwachukwu: Fix user_is_system in edit_per_editor_per_page_daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1353380 (https://phabricator.wikimedia.org/T434217) [19:15:43] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Remove the spark_shuffle_3_3 and spark_shuffle_3_4 YARN aux services - https://phabricator.wikimedia.org/T440229#12413520 (10xcollazo) The TL;DR from spark_shuffle_3_4 for now is: we can safely remove `spark_shuffle_3_4`. [19:18:49] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: wmf_contributors.edit_per_editor_per_page_daily claims `Martin Urbanec` is a system user - https://phabricator.wikimedia.org/T434217#12413540 (10Snwachukwu) The min date for `event_sanitized.mediawiki_page_change_v1` is `2024-12-30`... [19:26:32] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: wmf_contributors.edit_per_editor_per_page_daily claims `Martin Urbanec` is a system user - https://phabricator.wikimedia.org/T434217#12413555 (10Snwachukwu) In my patch I have followed this precedence for getting user_is_system valu... [19:30:32] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Implement 7z compression for v2 history dumps - https://phabricator.wikimedia.org/T437454#12413604 (10xcollazo) 05In progress→03Resolved [19:31:52] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: hdfs-rsync: Skip incomplete source directories - https://phabricator.wikimedia.org/T440383#12413622 (10xcollazo) As per discussion at https://gitlab.wikimedia.org/repos/data-engineering/hdfs-tools/-/merge_requests/5, holding a bit o... [19:37:00] 06Data-Engineering: File Exports: Consider storing the exports in Ceph instead of HDFS - https://phabricator.wikimedia.org/T439733#12413645 (10xcollazo) 05Open→03Resolved a:03xcollazo > With 170 days, HDFS keeps 6 runs. One run is 9.1 to 9.3 TB logical. This is 81.5 to 83.4 TB on-disk with replication... [20:09:15] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: wmf_contributors.edit_per_editor_per_page_daily claims `Martin Urbanec` is a system user - https://phabricator.wikimedia.org/T434217#12413763 (10xcollazo) a:05Snwachukwu→03None @Snwachukwu, before we go forward with changes, wan... [20:09:56] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: wmf_contributors.edit_per_editor_per_page_daily claims `Martin Urbanec` is a system user - https://phabricator.wikimedia.org/T434217#12413782 (10xcollazo) a:03Snwachukwu [20:14:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: wmf_contributors.edit_per_editor_per_page_daily claims `Martin Urbanec` is a system user - https://phabricator.wikimedia.org/T434217#12413790 (10Ottomata) > However because it's sanitized we revision.editor.user_central_id is NULL F... [20:40:43] 06Data-Engineering: Revisit thresholds for the Human-Bot Page View Count Anomaly alert - https://phabricator.wikimedia.org/T438957#12413892 (10dr0ptp4kt) ^ Patch merged, so next step will be any of the KAPOW integration and reinstating of alerts along with it. [22:52:52] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Platform, 10EditCheck, 10VisualEditor, and 3 others: Implement sampling requirements for Headless VE technical pilot - https://phabricator.wikimedia.org/T433024#12414218 (10DLynch) [23:50:57] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Define and document the earliest start_date for dbt DAGs - https://phabricator.wikimedia.org/T438276#12414321 (10amastilovic) I believe this ticket is now resolved.