[07:01:55] 06Data-Engineering, 06Data-Platform-SRE: Add relevant kafka clusters to defined airflow connections in puppet - https://phabricator.wikimedia.org/T379676#10315393 (10brouberol) Could we also make sure we add the relevant connection to https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/... [10:14:51] (03CR) 10Gehel: Extraction of RefineHelper (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [11:30:50] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Update druid config to automatically drop unused segments - https://phabricator.wikimedia.org/T376118#10316015 (10BTullis) @JAllemandou - I've applied the changes to an-test-druid1001, but the coordinator w... [11:41:30] 06Data-Engineering, 10CirrusSearch, 10Structured Data Engineering, 06Structured-Data-Backlog, 03Discovery-Search (Current work): Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#10316047 (10pfischer) 05In progress→03Stalled @Cparle, we... [12:03:33] 06Data-Engineering, 10CirrusSearch, 10Structured Data Engineering, 06Structured-Data-Backlog, 03Discovery-Search (Current work): Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#10316248 (10Cparle) Fantastic news! Thank you @pfischer ! [12:04:02] 06Data-Engineering, 10CirrusSearch, 10Structured Data Engineering, 06Structured-Data-Backlog, 03Discovery-Search (Current work): Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#10316249 (10Cparle) Are those open questions in the ticket des... [12:33:39] 06Data-Engineering, 06Data-Platform-SRE, 07Epic: Upgrade Hadoop to version 3.3.x and Hive to version 3.1.x - https://phabricator.wikimedia.org/T379385#10316356 (10BTullis) [12:48:37] 06Data-Engineering, 06Data-Platform-SRE: Draft a project plan for the Hadoop version 3 upgrade - https://phabricator.wikimedia.org/T379748 (10BTullis) 03NEW [12:48:44] 06Data-Engineering, 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Draft a project plan for the Hadoop version 3 upgrade - https://phabricator.wikimedia.org/T379748#10316388 (10BTullis) [12:48:54] 06Data-Engineering, 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Draft a project plan for the Hadoop version 3 upgrade - https://phabricator.wikimedia.org/T379748#10316389 (10BTullis) p:05Triage→03Medium [12:49:52] 06Data-Engineering, 06Data-Platform-SRE: Investigate Hadoop 3 container support with reference to Airflow deployment pipelines - https://phabricator.wikimedia.org/T288247#10316391 (10BTullis) p:05Triage→03Medium [13:11:24] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29), 13Patch-For-Review: Update druid config to automatically drop unused segments - https://phabricator.wikimedia.org/T376118#10316445 (10BTullis) @JAllemandou - I've now successfully applied the settings to... [14:03:43] 06Data-Engineering, 10CirrusSearch, 10Structured Data Engineering, 06Structured-Data-Backlog, 03Discovery-Search (Current work): Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#10316748 (10pfischer) [14:03:44] (03CR) 10Ottomata: "BTW, thank you for your advice Gehel!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [14:26:26] 06Data-Engineering, 10CirrusSearch, 10Structured Data Engineering, 06Structured-Data-Backlog, 03Discovery-Search (Current work): Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#10316875 (10pfischer) @BTullis, I would appreciate your feedba... [14:44:19] 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search: Add relevant kafka clusters to defined airflow connections in puppet - https://phabricator.wikimedia.org/T379676#10316981 (10Ottomata) [15:07:51] 10Quarry, 06cloud-services-team: Switch to using prefix puppet instead of direct-on-instance puppet - https://phabricator.wikimedia.org/T289531#10317154 (10rook) 05Open→03Declined [15:07:59] 10Quarry, 06cloud-services-team, 07Epic: Productionize quarry a bit - https://phabricator.wikimedia.org/T288982#10317156 (10rook) 05Open→03Resolved [15:11:09] 10Quarry, 06cloud-services-team: Switch to using prefix puppet instead of direct-on-instance puppet - https://phabricator.wikimedia.org/T289531#10317141 (10rook) The move to k8s appears to have made this ticket mostly unactionable. [15:12:03] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data Pipelines, 10Data-Catalog: Integrate Spark with DataHub with lineage - https://phabricator.wikimedia.org/T306896#10317173 (10Ahoelzl) [15:12:23] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10317177 (10Ladsgroup) Thanks to @CDanis for P71034 Now we know what script is refusing to update its config. @bvibber... [15:15:31] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10317199 (10ABran-WMF) thanks for the help @Ladsgroup @CDanis! @bvibber please let me know if I can help! [15:21:44] 06Data-Engineering, 06Movement-Insights, 10Wmfdata-Python, 07Documentation: Publish HTML docs for Wmfdata-Python on doc.wikimedia.org - https://phabricator.wikimedia.org/T298178#10317227 (10apaskulin) Sphinx plus the Furo theme is a great option! If you're looking for something simpler, you could consider... [16:01:14] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Traffic, 13Patch-For-Review: Rollout haproxykafka on all hosts - https://phabricator.wikimedia.org/T378578#10317529 (10Fabfur) [19:36:14] 10Data-Engineering (Q2 2024 October 1st - December 31th): Implement automatic sync of refinery HQL files to HDFS - https://phabricator.wikimedia.org/T365659#10318876 (10Ottomata) [20:56:37] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data Pipelines, 10Data-Catalog: Integrate Spark with DataHub with lineage - https://phabricator.wikimedia.org/T306896#10319322 (10Ahoelzl) First integration results: https://datahub.wikimedia.org/tasks/urn:li:dataJob:(urn:li:dataFlow:(spark,airflow_... [21:06:40] 10Data-Engineering (Q2 2024 October 1st - December 31th): Make it possible to select the DAG deployment method - https://phabricator.wikimedia.org/T379279#10319355 (10Ahoelzl) [21:11:38] 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search, 10Dumps 2.0: Add relevant kafka clusters to defined airflow connections in puppet - https://phabricator.wikimedia.org/T379676#10319375 (10Ahoelzl) [21:12:01] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform-SRE, 06Discovery-Search, 10Dumps 2.0: Add relevant kafka clusters to defined airflow connections in puppet - https://phabricator.wikimedia.org/T379676#10319376 (10Ahoelzl)