[08:01:51] 06Data-Engineering, 06Data Products, 06Traffic, 13Patch-For-Review: Prepare puppet configuration to send haproxy logs to haproxykafka socket - https://phabricator.wikimedia.org/T374473#10143322 (10gmodena) @fabfur thanks for the heads up. We'll need to coordinate a bit on roll out, because the previous ing... [08:17:50] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data Pipelines, 10Data-Catalog, 13Patch-For-Review: Spike: Integrate Spark with DataHub with lineage - https://phabricator.wikimedia.org/T306896#10143350 (10JAllemandou) We had set the name with date information on purpose, to facilitate identifying... [10:04:08] 06Data-Engineering, 06Discovery-Search: Decide how to make datasets owned by analytics-search-users also readable by analytics-privatedata-users - https://phabricator.wikimedia.org/T374637#10143588 (10JAllemandou) The reason we originally started using different users/groups was to silo permissions. I think it... [10:18:40] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10143600 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=2c4e88b1-20d2-451e-a304-a07930594799) set by jynus@cumin1002 for 3 days, 0:00:00... [10:42:44] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Migrate Event Platform Schema Respositories to Gitlab - https://phabricator.wikimedia.org/T366836#10143715 (10gmodena) @Snwachukwu might you need inspiration on drafting a contribution policy for the repos your are... [11:39:48] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Migrate Event Platform Schema Respositories to Gitlab - https://phabricator.wikimedia.org/T366836#10143890 (10gmodena) We mirror repos (read-only) from Gerrit to GitHub, is there a way we can do the same to Gitlab?... [14:16:40] (03PS1) 10Ottomata: POC: can we represent openmetric, prometheus,druid data model in an event? [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1072757 (https://phabricator.wikimedia.org/T180105) [14:17:04] (03CR) 10CI reject: [V:04-1] POC: can we represent openmetric, prometheus,druid data model in an event? [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1072757 (https://phabricator.wikimedia.org/T180105) (owner: 10Ottomata) [14:23:47] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data Pipelines, 10Data-Catalog, 13Patch-For-Review: Spike: Integrate Spark with DataHub with lineage - https://phabricator.wikimedia.org/T306896#10144458 (10Ottomata) Nice! that sounds good! That is going to be a job specific setting then, so hm.... [15:31:46] 06Data-Engineering, 06cloud-services-team, 05Cloud-Services-Origin-User: WMCS-roots paging responsibilities - https://phabricator.wikimedia.org/T344608#10144604 (10fnegri) 05Open→03Declined In {T344599}, it was decided members of wmcs-roots should //not// have root access to wiki replicas hosts (clou... [15:40:19] 06Data-Engineering, 05Cloud-Services-Origin-User, 10cloud-services-team (FY2024/2025-Q1-Q2): WMCS-roots paging responsibilities - https://phabricator.wikimedia.org/T344608#10144665 (10fnegri) [15:40:26] 06Data-Engineering, 05Cloud-Services-Origin-User, 10cloud-services-team (FY2024/2025-Q1-Q2): WMCS-roots paging responsibilities - https://phabricator.wikimedia.org/T344608#10144666 (10fnegri) a:03fnegri [15:54:31] 06Data-Engineering, 10Cassandra, 10Data Pipelines, 10Data-Platform-SRE (2024.09.06 - 2024.09.27), 13Patch-For-Review: Create puppet defined type for adding/updating/deleting secrets or other small files on HDFS - https://phabricator.wikimedia.org/T323692#10144686 (10BTullis) I've had another think about... [16:08:18] 06Data-Engineering, 10Cassandra, 10Data Pipelines, 10Data-Platform-SRE (2024.09.06 - 2024.09.27), 13Patch-For-Review: Create puppet defined type for adding/updating/deleting secrets or other small files on HDFS - https://phabricator.wikimedia.org/T323692#10144733 (10Ottomata) Sure, that might work nicely... [17:06:46] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Migrate Event Platform Schema Respositories to Gitlab - https://phabricator.wikimedia.org/T366836#10144950 (10Snwachukwu) A question was raised regarding keeping permissions of current gerrit secondary schema reposi... [17:09:45] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Migrate Event Platform Schema Respositories to Gitlab - https://phabricator.wikimedia.org/T366836#10144963 (10Snwachukwu) Thanks for the material to contributing policy @gmodena. It's helpful [17:38:40] 14Analytics, 06Data-Engineering, 10Observability-Logging, 06SRE, and 2 others: Integrate Event Platform and ECS logs - https://phabricator.wikimedia.org/T291645#10145027 (10EBernhardson) This would have been useful to debug T374662, aggregating the times out of elasticsearch is a bit hard as it would have... [17:39:43] 06Data-Engineering, 10MediaWiki-extensions-WikimediaEvents, 10Observability-Metrics, 10Event-Platform, and 3 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10145033 (10Ottomata) FWIW, my interest here is similar to my interest in {T291645}. Getting a... [17:42:19] 14Analytics, 06Data-Engineering, 10Observability-Logging, 06SRE, and 2 others: Integrate Event Platform and ECS logs - https://phabricator.wikimedia.org/T291645#10145036 (10CDanis) Similar but different: {T304373} [18:33:34] 10Data-Engineering (Q1 2024 July 1st - September 30th): Handle Late-Arrived Events from Gobblin into Airflow triggered Refine - https://phabricator.wikimedia.org/T370665#10145369 (10Ahoelzl) [18:33:42] 10Data-Engineering (Q1 2024 July 1st - September 30th): [Data Quality] Improve Superset visualizations - https://phabricator.wikimedia.org/T372678#10145372 (10Ahoelzl) a:03Ahoelzl [20:55:09] 06Data-Engineering, 10MediaWiki-extensions-WikimediaEvents, 10Observability-Metrics, 10Event-Platform, and 3 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10145732 (10colewhite) In the interest of a completeness, including a description of the curren... [20:56:25] 10Data-Engineering (Q1 2024 July 1st - September 30th): [Refine Refactoring] Detect inactive event streams / Refine datasets using data recency thresholds - https://phabricator.wikimedia.org/T361498#10145735 (10Ahoelzl) a:05lbowmaker→03JAllemandou [20:58:34] 10Data-Engineering (Q1 2024 July 1st - September 30th): [Refine Refactoring] Detect inactive event streams / Refine datasets using data recency thresholds - https://phabricator.wikimedia.org/T361498#10145746 (10Ahoelzl) [22:09:49] 06Data-Engineering, 06Discovery-Search, 13Patch-For-Review: Decide how to make datasets owned by analytics-search-users also readable by analytics-privatedata-users - https://phabricator.wikimedia.org/T374637#10145951 (10EBernhardson) For future files created by spark: * we set `spark.hadoop.fs.permissions.u...