[00:37:52] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Define and document the earliest start_date for dbt DAGs - https://phabricator.wikimedia.org/T438276#12372228 (10amastilovic) > Ideally, if a model owner forgets to specify start_date, this would be caught at MR review time or, failing that, cause a fail... [02:26:53] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Define and document the earliest start_date for dbt DAGs - https://phabricator.wikimedia.org/T438276#12372343 (10nshahquinn-wmf) >>! In T438276#12372228, @amastilovic wrote: > `permit_large_backfill` is possible, but I wonder if the utility of it is grea... [05:55:39] 06Data-Engineering, 10Kafka-Infrastructure, 06SRE, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Monitor kafka-configurator runs and pending changes - https://phabricator.wikimedia.org/T439506 (10RKemper) 03NEW [06:01:28] 14Analytics-Radar, 06Data-Engineering, 10Kafka-Infrastructure, 06serviceops-radar, and 5 others: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12372533 (10RKemper) [07:28:10] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA: Move SpecialUnusedTemplates computation to Hadoop - https://phabricator.wikimedia.org/T422443#12372702 (10JAllemandou) I have checked the job, and the problem is deeper than just memory resources. The job was suffering from data-skew and Spark couldn't opt... [07:28:58] (03PS1) 10Joal: Add spark_stage_metrics table creation script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345860 (https://phabricator.wikimedia.org/T297231) [07:34:55] (03PS1) 10Joal: Update querypage queries with join optimizations [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345865 (https://phabricator.wikimedia.org/T422443) [07:36:32] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 13Patch-For-Review: Move SpecialUnusedTemplates computation to Hadoop - https://phabricator.wikimedia.org/T422443#12372720 (10JAllemandou) One mopre note: I have paused the airflow job to prevent alerts to be sent while testing. Let's not forget to unpau... [08:02:41] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Observability-Metrics, 13Patch-For-Review: [Infra] Mechanism to get Apache Spark operational metrics - https://phabricator.wikimedia.org/T297231#12372783 (10JAllemandou) Good progress to report here. Reading existing EventLogs is a way better idea... [08:13:53] (03CR) 10Joal: "Tested on the cluster!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1344687 (https://phabricator.wikimedia.org/T297231) (owner: 10Joal) [08:17:37] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Include v2 dumps into mirrors rsync configuration - https://phabricator.wikimedia.org/T437463#12372843 (10GGoncalves-WMF) Excellent analysis, thank you! I'd like to see @HCoplin-WMF 's thoughts but my answers are: > Do you agree with rule 2 (cutoff dat... [08:41:05] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Define and document the earliest start_date for dbt DAGs - https://phabricator.wikimedia.org/T438276#12372930 (10GGoncalves-WMF) Sounds good to me too! [09:03:03] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Observability-Metrics, 13Patch-For-Review: [Infra] Mechanism to get Apache Spark operational metrics - https://phabricator.wikimedia.org/T297231#12373012 (10JAllemandou) Had a chat with @Antoine_Quhen and he had great suggestions: * Add some job c... [09:17:44] (03CR) 10Aqu: [V:03+2] Declare new table Google Sitedata Aggregate [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345123 (https://phabricator.wikimedia.org/T436999) (owner: 10Aqu)