[00:27:25] 06Data-Engineering, 03Discovery-Search (Current work), 13Patch-For-Review: Datahub - ingest Hive discovery database - https://phabricator.wikimedia.org/T374118#10194422 (10tchin) I did a manual ingestion and was able to see the tables on datahub if I access it directly through a url https://datahub.wikimedia... [08:07:15] PROBLEM - eventlogging Varnishkafka log producer on cp3066 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [08:07:16] PROBLEM - Webrequests Varnishkafka log producer on cp3066 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [08:07:17] PROBLEM - statsv Varnishkafka log producer on cp3066 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [08:17:17] RECOVERY - eventlogging Varnishkafka log producer on cp3066 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [08:17:17] RECOVERY - Webrequests Varnishkafka log producer on cp3066 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [08:17:17] RECOVERY - statsv Varnishkafka log producer on cp3066 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [09:40:35] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Update druid config to automatically drop unused segments - https://phabricator.wikimedia.org/T376118#10195097 (10JAllemandou) >>! In T376118#10193763, @BTullis wrote: > Is there some overlap between this tic... [09:41:22] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Update druid config to automatically drop unused segments - https://phabricator.wikimedia.org/T376118#10195099 (10JAllemandou) [09:43:09] (03CR) 10Joal: [C:03+1] "Thank you for this @mforns 😊" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1077098 (https://phabricator.wikimedia.org/T376212) (owner: 10Mforns) [10:19:26] 06Data-Engineering, 10Dumps 2.0, 10Event-Platform: [SPIKE] how can we support Spark producer/consumers in Event Platform - https://phabricator.wikimedia.org/T374341#10195293 (10pfischer) @Ottomata, I pushed a CR: https://gerrit.wikimedia.org/r/1077113, but my attempt to simply wrap the kafka sink seems a dea... [11:05:00] 06Data-Engineering, 06Data Products, 06DBA, 10wikitech.wikimedia.org: Please drop globalblocks table from labswiki - https://phabricator.wikimedia.org/T375783#10195398 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup I dropped it on master with replication. Let me know if I messed up something. [12:10:29] 06Data-Engineering, 06Data Products, 06DBA, 06Trust and Safety Product Team, and 2 others: Update gb_address index on the globalblocks table - https://phabricator.wikimedia.org/T376125#10195547 (10Ladsgroup) 05Open→03Resolved p:05Triage→03High a:03Ladsgroup [12:46:48] (03CR) 10Kosta Harlan: [C:03+1] "LGTM. I'll hold off on merging it until we have a proposed patch in IPInfo that makes use of the new schema, though." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1077035 (https://phabricator.wikimedia.org/T356105) (owner: 10Máté Szabó) [14:25:09] 06Data-Engineering, 06Data-Platform-SRE: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 - https://phabricator.wikimedia.org/T338057#10196149 (10xcollazo) [14:30:51] 06Data-Engineering, 06Data-Platform-SRE: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 - https://phabricator.wikimedia.org/T338057#10196162 (10xcollazo) I had discussed this a bit in description, [[ https://iceberg.apache.org/multi-engine-support/#apache-spark |... [14:39:31] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review, 10Web Team Essential Work 2024 (Migrate to new Event Platform), 10Web-Team-Backlog (FY2024-25 Q2 Sprint 1): Delete redundant mobile- and desktopwebuiactions event in WikimediaEvents - https://phabricator.wikimedia.org/T376065#10196182 (10KSarabia-W... [14:48:48] 06Data-Engineering, 10Dumps 2.0 (Kanban Board): [Iceberg Migration] Extend Iceberg table maintenance mechanism to support data rewrite - https://phabricator.wikimedia.org/T373694#10196208 (10xcollazo) Copypasting from MR for completeness: > In this MR, we expand the maintenance mechanism introduced on !806 to... [15:09:58] (03CR) 10Ebernhardson: "This repository has moved, schema updates need to be at https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary" [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/594098 (https://phabricator.wikimedia.org/T374918) (owner: 10DCausse) [16:10:53] (03CR) 10Milimetric: [C:03+2] Coalesce anomaly detection queries to output just 1 file [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1077098 (https://phabricator.wikimedia.org/T376212) (owner: 10Mforns) [16:10:57] (03CR) 10Milimetric: [V:03+2 C:03+2] Coalesce anomaly detection queries to output just 1 file [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1077098 (https://phabricator.wikimedia.org/T376212) (owner: 10Mforns) [16:44:05] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for Seanleong-WMDE - https://phabricator.wikimedia.org/T376034#10196987 (10kamila) p:05Triage→03Medium a:03kamila [17:09:01] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for Seanleong-WMDE - https://phabricator.wikimedia.org/T376034#10197148 (10kamila) [18:18:05] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for Seanleong-WMDE - https://phabricator.wikimedia.org/T376034#10197359 (10KFrancis) Hi all, confirmingI have an NDA on file for Sean, thanks! [18:20:15] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for Seanleong-WMDE - https://phabricator.wikimedia.org/T376034#10197363 (10Dzahn) Thanks, Katie! @kamila When WMDE staff is onboarded it normally always comes with the LDAP groups `nda`... [18:45:39] 06Data-Engineering, 06Data Products, 06DBA, 10GlobalBlocking, and 6 others: Allow autoblocks on IP addresses in the globalblocks table for IP addresses which are already globally blocked - https://phabricator.wikimedia.org/T376052#10197443 (10Dreamy_Jazz) 05Open→03Resolved The associated schema cha... [21:00:05] 10Data-Engineering (Q1 2024 July 1st - September 30th): [gobblin-wmf] Update parent POM to a more recent version, and make sure that the verify maven goal passes - https://phabricator.wikimedia.org/T375390#10197866 (10amastilovic) 05Open→03Resolved