[00:24:49] 10Data-Engineering (Q4 2024 April 1st - June 30th): Migrate and re-deploy eventgate using new service runner - https://phabricator.wikimedia.org/T361768 (10Ahoelzl) 03NEW [00:25:14] 10Data-Engineering (Q4 2024 April 1st - June 30th): Migrate and re-deploy eventstreams using new service runner - https://phabricator.wikimedia.org/T361769 (10Ahoelzl) 03NEW [00:25:40] 10Data-Engineering (Q4 2024 April 1st - June 30th): Migrate and re-deploy eventstreams using new service runner - https://phabricator.wikimedia.org/T361769#9686646 (10Ahoelzl) [00:28:20] 10Data-Engineering (Q4 2024 April 1st - June 30th): Support metrics platform backend migration to new service runner - https://phabricator.wikimedia.org/T361770 (10Ahoelzl) 03NEW [00:28:32] 10Data-Engineering (Q4 2024 April 1st - June 30th): Support metrics platform backend migration to new service runner - https://phabricator.wikimedia.org/T361770#9686669 (10Ahoelzl) [00:32:50] 06Data-Engineering, 10Event-Platform: [NEEDS GROOMING] Orchestrate gobblin ingestion task with Airflow - https://phabricator.wikimedia.org/T361094#9686676 (10Ahoelzl) [00:33:53] 06Data-Engineering, 10CirrusSearch, 03Discovery-Search (Current work): [Search Update Pipeline] Source streams for private wikis - https://phabricator.wikimedia.org/T346046#9686677 (10Ahoelzl) [01:15:46] 06Data-Engineering, 10CirrusSearch, 03Discovery-Search (Current work): [Search Update Pipeline] Source streams for private wikis - https://phabricator.wikimedia.org/T346046#9686713 (10lbowmaker) @Ahoelzl - what implementation is the 13 point estimate based on? [06:47:46] karma [08:26:28] 06Data-Engineering, 06Data-Platform-SRE, 10Scap, 07git-lfs, 10Release-Engineering-Team (Now this 🫠): analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#9687234 (10hashar) [08:27:09] 14Analytics, 14Analytics-Kanban, 07git-lfs, 13Patch-For-Review: 14Update git lfs on stat1006/7 - 14https://phabricator.wikimedia.org/T214089#9687240 (10hashar) [08:48:30] 06Data-Engineering, 06Data-Platform, 13Patch-For-Review: Add movement insights group/users to MWH denormalize job alerts - https://phabricator.wikimedia.org/T357472#9687319 (10CodeReviewBot) joal opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/642 Update analytics MW... [08:49:14] 06Data-Engineering, 06Data-Platform, 13Patch-For-Review: Add movement insights group/users to MWH denormalize job alerts - https://phabricator.wikimedia.org/T357472#9687332 (10JAllemandou) Done using the airflow variable. I also sent a PR to have the defaults set: https://gitlab.wikimedia.org/repos/data-engi... [09:14:00] (03PS3) 10Aqu: Add CLI to create or update Iceberg tables [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [09:20:15] (03CR) 10CI reject: [V:04-1] Add CLI to create or update Iceberg tables [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) (owner: 10Aqu) [09:42:26] (03PS4) 10Aqu: Add CLI to create or update Iceberg tables [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [10:21:45] (03PS3) 10Gmodena: Add webrequest_frontend schema and reifine. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1012656 (https://phabricator.wikimedia.org/T314956) [10:41:36] 06Data-Engineering, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 10 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#9687858 (10Nikerabbit) [12:57:29] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117#9688020 (10CodeReviewBot) gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/64... [12:57:41] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: [Event Platform] Declare webrequest as an Event Platform stream - https://phabricator.wikimedia.org/T314956#9688021 (10CodeReviewBot) gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/643 Draft: analytic... [13:09:55] 06Data-Engineering, 13Patch-For-Review: [NEEDS GROOMING][SPIKE] Extract refine schema management into a dedicated tool - https://phabricator.wikimedia.org/T356762#9688385 (10JAllemandou) I have been wondering about how to organize this code. I was not willing to replicate the `DataFrameToHive` pattern due to t... [13:10:15] (03CR) 10Gmodena: Add gobblin job webrequest_frontend_rc0 (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983926 (https://phabricator.wikimedia.org/T314956) (owner: 10Ottomata) [13:10:39] 06Data-Engineering, 06Data-Platform-SRE, 10Scap, 07git-lfs, and 2 others: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#9688404 (10hashar) @Sfaci mentioned the [[ https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/ | analytics-refinery-update... [13:13:57] (03CR) 10Milimetric: "These comments are all the same thing - suggesting that we pass table names as params. I lean slightly towards doing it, but it's up to y" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1016796 (https://phabricator.wikimedia.org/T358681) (owner: 10Xcollazo) [13:16:44] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117#9688466 (10gmodena) >>! In T351117#9638903, @Fabfur wrote: > @gmodena you should have some more data to play with now, while... [13:44:53] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1003:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1003:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [14:40:40] 06Data-Engineering, 06Tech-Docs-Team, 05Goal: Redesign Data Platform docs on Wikitech - https://phabricator.wikimedia.org/T350911#9688695 (10TBurmeister) Status update: - [[ https://docs.google.com/document/d/1c397XdwdxPmhPCmiW0mn0P8S6csUP1hFi9GCNkCNaOQ/edit#heading=h.x0k7omatgbpc | Timeline ]] established (... [15:13:08] 10Quarry: Update helm for quarry on pr - https://phabricator.wikimedia.org/T349031#9689068 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/quarry/pull/33 [15:13:10] Starting build #103 for job analytics-refinery-update-jars-docker [15:13:15] Project analytics-refinery-update-jars-docker build #103: 04STILL FAILING in 4.6 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/103/ [15:14:24] Starting build #104 for job analytics-refinery-update-jars-docker [15:14:29] Project analytics-refinery-update-jars-docker build #104: 04STILL FAILING in 4.7 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/104/ [15:44:29] 10Quarry: Update helm for quarry on pr - https://phabricator.wikimedia.org/T349031#9689174 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/quarry/pull/33 [15:45:00] 10Quarry: Update helm for quarry on pr - https://phabricator.wikimedia.org/T349031#9689176 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/quarry/pull/35 [15:47:02] 10Quarry: Update helm for quarry on pr - https://phabricator.wikimedia.org/T349031#9689185 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/quarry/pull/35 [15:52:37] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Event Platform] [Spike] Develop a concept to apply Metrics Platform configurations to event stream configurations - https://phabricator.wikimedia.org/T361853 (10Ahoelzl) 03NEW [15:53:04] 10Quarry: 14Update helm for quarry on pr - 14https://phabricator.wikimedia.org/T349031#9689215 (10rook) 05Open→03Resolved a:03rook [15:53:30] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Event Platform] [Spike] Develop a concept to apply Metrics Platform configurations to event stream configurations - https://phabricator.wikimedia.org/T361853#9689220 (10Ahoelzl) [15:57:43] 06Data-Engineering, 13Patch-For-Review: Extract refine schema management into a dedicated tool - https://phabricator.wikimedia.org/T356762#9689240 (10Antoine_Quhen) a:03Antoine_Quhen [16:04:53] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1003:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1003:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [16:10:32] Starting build #105 for job analytics-refinery-update-jars-docker [16:10:37] Project analytics-refinery-update-jars-docker build #105: 04STILL FAILING in 5.4 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/105/ [16:19:48] (03CR) 10Joal: "Adding to Andrew's comments" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) (owner: 10Aqu) [17:13:15] Starting build #106 for job analytics-refinery-update-jars-docker [17:14:42] Project analytics-refinery-update-jars-docker build #106: 04STILL FAILING in 1 min 26 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/106/ [17:35:58] 06Data-Engineering, 13Patch-For-Review: Extract refine schema management into a dedicated tool - https://phabricator.wikimedia.org/T356762#9689821 (10Ottomata) > Should we have 2 lib files, one for schema and one for data, for both Hive and Iceberg? Or one file doing both as it is now? Either could work! I th... [17:45:26] (03PS1) 10Ahmon Dancy: Convert remaining git-fat files to git-lfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1017110 (https://phabricator.wikimedia.org/T328472) [17:48:56] (03CR) 10Ahmon Dancy: [C:03+2] Convert remaining git-fat files to git-lfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1017110 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [17:49:01] (03CR) 10Ahmon Dancy: [V:03+2 C:03+2] Convert remaining git-fat files to git-lfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1017110 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [17:54:41] (03PS1) 10Ahmon Dancy: bin/update-refinery-source-jars: git-fat to git-lfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1017113 (https://phabricator.wikimedia.org/T328472) [17:55:37] (03CR) 10Ahmon Dancy: [V:03+2 C:03+2] bin/update-refinery-source-jars: git-fat to git-lfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1017113 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [17:57:07] Starting build #107 for job analytics-refinery-update-jars-docker [17:58:36] Project analytics-refinery-update-jars-docker build #107: 04STILL FAILING in 1 min 28 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/107/ [18:02:00] (03PS1) 10Ahmon Dancy: bin/update-refinery-source-jars: Pass --local to git lfs install [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1017114 (https://phabricator.wikimedia.org/T328472) [18:02:24] (03CR) 10Ahmon Dancy: [V:03+2 C:03+2] bin/update-refinery-source-jars: Pass --local to git lfs install [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1017114 (https://phabricator.wikimedia.org/T328472) (owner: 10Ahmon Dancy) [18:02:50] Starting build #108 for job analytics-refinery-update-jars-docker [18:04:36] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.34 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1017074 [18:04:37] Yippee, build fixed! [18:04:37] Project analytics-refinery-update-jars-docker build #108: 09FIXED in 1 min 47 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/108/ [18:08:43] 06Data-Engineering, 06Data-Platform-SRE, 10Scap, 07git-lfs, and 2 others: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#9689959 (10dancy) @Sfaci The issue with the `analytics-refinery-update-jars-docker` job should be resolved now. https://integration.wikimedia.org/ci/... [18:10:09] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9689962 (10apaskulin) [18:14:11] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Event Platform] [Spike] Understand and document the details and conflicts between Datasets Config, Refine refactor, Dynamic EventStreamConfig, and Metrics Platform Instrumentation Configurator - https://phabricator.wikimedia.org/T361853#9690004 (10Ottomata) [18:17:10] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Event Platform] [Spike] Understand and document the details and conflicts between Datasets Config, Refine refactor, Dynamic EventStreamConfig, and Metrics Platform Instrumentation Configurator - https://phabricator.wikimedia.org/T361853#9690019 (10Ottomata) [18:17:11] 06Data-Engineering, 07Epic: Dataset Config Store - https://phabricator.wikimedia.org/T354557#9690021 (10Ottomata) [18:17:42] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Datasets Config][Spike] Understand and document the details and conflicts between Datasets Config, Refine refactor, Dynamic EventStreamConfig, and Metrics Platform Instrumentation Configurator - https://phabricator.wikimedia.org/T361853#9690023 (10Ottomata) [18:19:42] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9690033 (10apaskulin) [18:21:25] 06Data-Engineering, 06Data-Platform-SRE, 10Scap, 07git-lfs, and 2 others: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#9690038 (10hashar) antoine-approve [19:17:00] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Datasets Config][Spike] Understand and document the details and conflicts between Datasets Config, Refine refactor, Dynamic EventStreamConfig, and Metrics Platform Instrumentation Configurator - https://phabricator.wikimedia.org/T361853#9690280 (10Ottomata) Pe... [19:17:08] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 07Spike: [SPIKE] Can we express Event Platform configs in config store? - https://phabricator.wikimedia.org/T361017#9690282 (10Ottomata) [19:17:09] 06Data-Engineering, 07Epic: Dataset Config Store - https://phabricator.wikimedia.org/T354557#9690283 (10Ottomata) [20:23:32] 06Data-Engineering, 06Data-Platform-SRE, 10Scap, 07git-lfs, and 2 others: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#9690668 (10Sfaci) The job finished great! I can keep working on the deployment that was blocked for this. Thank you very much!! [20:38:59] 06Data-Engineering-Radar, 06Traffic, 13Patch-For-Review: Lock-in Varnish and VarnishKafka versions - https://phabricator.wikimedia.org/T304617#9690726 (10CodeReviewBot) brett opened https://gitlab.wikimedia.org/repos/sre/varnishkafka/-/merge_requests/4 Release 1.1.0-4 [20:49:22] 06Data-Engineering-Radar, 06Traffic, 13Patch-For-Review: Lock-in Varnish and VarnishKafka versions - https://phabricator.wikimedia.org/T304617#9690746 (10CodeReviewBot) brett merged https://gitlab.wikimedia.org/repos/sre/varnishkafka/-/merge_requests/4 Release 1.1.0-4 [20:49:43] 06Data-Engineering-Radar, 06Traffic, 13Patch-For-Review: Lock-in Varnish and VarnishKafka versions - https://phabricator.wikimedia.org/T304617#9690747 (10CodeReviewBot) brett closed https://gitlab.wikimedia.org/repos/sre/varnishkafka/-/merge_requests/3 Release 1.1.0-4 [21:24:39] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9690908 (10apaskulin) [21:24:50] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9690910 (10apaskulin) [21:24:56] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9690902 (10apaskulin) [21:26:27] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9690930 (10apaskulin) [21:26:39] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9690935 (10apaskulin) [22:14:38] (03PS1) 10Aleksandar Mastilovic: WMCS HQL scripts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1017161