[00:09:47] 10Data-Engineering (Q1 2024 July 1st - September 30th): Create gitlab ci npm publish pipeline and job in workflow_utils gitlab_ci_templates - https://phabricator.wikimedia.org/T366537#10205511 (10tchin) 05Open→03Resolved [00:12:54] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Publish Data Engineering maintained NodeJS packages to GitLab and use them in depender code - https://phabricator.wikimedia.org/T366612#10205514 (10tchin) [00:13:23] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Publish Data Engineering maintained NodeJS packages to GitLab and use them in depender code - https://phabricator.wikimedia.org/T366612#10205515 (10tchin) Seems like we missed [[ https://github.com/wikimedia/KafkaSSE/tree/master | Kafk... [07:41:21] (03CR) 10DCausse: [C:03+2] SearchSatisfaction: Add virtualPageView action [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1077731 (https://phabricator.wikimedia.org/T375387) (owner: 10Ebernhardson) [07:41:48] (03Merged) 10jenkins-bot: SearchSatisfaction: Add virtualPageView action [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1077731 (https://phabricator.wikimedia.org/T375387) (owner: 10Ebernhardson) [08:36:21] (03Abandoned) 10Aqu: Fix parse_user_agent following Refine refactoring [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1075145 (https://phabricator.wikimedia.org/T369845) (owner: 10Aqu) [12:17:13] 06Data-Engineering, 06Data-Platform, 06Product-Analytics: Investigate how Airflow dynamic task mapping works with traditional operators like SparkSqlOperator - https://phabricator.wikimedia.org/T376613 (10KCVelaga_WMF) 03NEW [12:18:37] 06Data-Engineering, 06Data-Platform, 06Product-Analytics: Investigate how Airflow dynamic task mapping works with traditional operators like SparkSqlOperator - https://phabricator.wikimedia.org/T376613#10206516 (10KCVelaga_WMF) @xcollazo @amastilovic created an initial ticket from our conversation on GitLab,... [12:18:59] 06Data-Engineering, 06Data-Platform, 06Product-Analytics: Investigate how Airflow dynamic task mapping works with traditional operators like SparkSqlOperator - https://phabricator.wikimedia.org/T376613#10206517 (10KCVelaga_WMF) [19:53:09] 06Data-Engineering, 10Event-Platform: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026#10208155 (10Ottomata) [19:53:11] 14Analytics, 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: Adopt conventions for server receive and client/event timestamps in non analytics event schemas - https://phabricator.wikimedia.org/T267648#10208156 (10Ottomata) [19:55:37] 06Data-Engineering, 10Event-Platform: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026#10208161 (10Ottomata) Added T267648 as a parent task. We might want to merge these together, as we will effectively be accomplishing T267648 by completing this task. [20:09:43] 06Data-Engineering, 10Cassandra, 10Data Pipelines, 10Data-Platform-SRE (2024.09.28 - 2024.10.18), 13Patch-For-Review: Create puppet resource for adding/updating/deleting secrets or other small files on HDFS - https://phabricator.wikimedia.org/T323692#10208207 (10Ottomata) Interesting! Good point. Okay! [20:18:52] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): Implement Airflow Dataset class for RestExternalTaskSensor - https://phabricator.wikimedia.org/T372647#10208259 (10amastilovic) I've published a draft MR that, as far as I can tell, implements most of support for the produced_b... [20:35:29] 06Data-Engineering, 06Data-Platform-SRE, 03Discovery-Search (Current work): Unable to find ingested tables in datahub - https://phabricator.wikimedia.org/T376657 (10EBernhardson) 03NEW [20:36:08] 06Data-Engineering, 10Dumps 2.0, 10Event-Platform: [SPIKE] how can we support Spark producer/consumers in Event Platform - https://phabricator.wikimedia.org/T374341#10208368 (10Ottomata) Amazing! > I would not block on the lack of support for spark-streaming Agree. If it is easy to do, then sure, but I don... [20:39:23] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for Seanleong-WMDE - https://phabricator.wikimedia.org/T376034#10208401 (10Ottomata) Approved! [21:02:30] (03CR) 10Ottomata: [C:03+1] Add /mediawiki/wikibase/entity/rdf_change/2.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/594098 (https://phabricator.wikimedia.org/T374918) (owner: 10DCausse) [22:23:07] 10Data-Engineering (Q1 2024 July 1st - September 30th): Migrate and re-deploy eventstreams using service-utils - https://phabricator.wikimedia.org/T361769#10208814 (10Ahoelzl) @tchin can you detail the remaining work to complete this task and get eventstreams deployed in production? Thanks.