[05:03:48] (03PS1) 10Tim Starling: Port demux.py to C++ [analytics/udplog] - 10https://gerrit.wikimedia.org/r/1119598 (https://phabricator.wikimedia.org/T386421) [08:42:44] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 10fundraising-tech-ops, 07Epic: TLS connection for hive-standalone-metaserver with minio - https://phabricator.wikimedia.org/T385031#10551833 (10Gehel) [08:57:58] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization: Create Gitlab CI templates for JVM packages - https://phabricator.wikimedia.org/T386406#10551884 (10Gehel) There is [[ https://docs.google.com/document/d/1g7opvXdWDWgOCnbMX9eYIVCSXYDiMpScy072LN1-AhI/edit?tab=t.0 | some documentation ]] of... [14:31:45] 10Data-Engineering (Q3 2024 January 1st - March 31th): Update eventstreams Grafana Dashboards to use histogram for router metrics - https://phabricator.wikimedia.org/T386204#10552552 (10Ottomata) The GC stuff could be fixed by {T350180}. Could we do that in service-utils? [14:36:47] (03PS1) 10Gerrit maintenance bot: Add syl.wikipedia to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1119725 (https://phabricator.wikimedia.org/T386464) [15:03:10] 06Data-Engineering, 06Language and Product Localization, 10MediaWiki-extensions-Translate, 06MW-Interfaces-Team, and 3 others: Intermittent JobQueueError due to "Unable to deliver all events: 500: Internal Server Error" - https://phabricator.wikimedia.org/T386138#10552656 (10Ottomata) Investigating. @hnow... [15:59:13] 06Data-Engineering, 06Language and Product Localization, 10MediaWiki-extensions-Translate, 06MW-Interfaces-Team, and 3 others: Intermittent JobQueueError due to "Unable to deliver all events: 500: Internal Server Error" - https://phabricator.wikimedia.org/T386138#10552856 (10Ottomata) When eventgate report... [16:18:44] 06Data-Engineering, 06Language and Product Localization, 10MediaWiki-extensions-Translate, 06MW-Interfaces-Team, and 3 others: Intermittent JobQueueError due to "Unable to deliver all events: 500: Internal Server Error" - https://phabricator.wikimedia.org/T386138#10552911 (10Ottomata) All pods now respond... [16:19:22] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Language and Product Localization, 10MediaWiki-extensions-Translate, 06MW-Interfaces-Team, and 3 others: Intermittent JobQueueError due to "Unable to deliver all events: 500: Internal Server Error" - https://phabricator.wikimedia.org/T386138#10552914 (... [16:19:39] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Language and Product Localization, 10MediaWiki-extensions-Translate, 06MW-Interfaces-Team, and 3 others: Intermittent JobQueueError due to "Unable to deliver all events: 500: Internal Server Error" - https://phabricator.wikimedia.org/T386138#10552916 (... [16:36:09] 10Data-Engineering (Q3 2024 January 1st - March 31th): Validate `pageview` and `unique_devices` generated from `webrequest_frontend` - https://phabricator.wikimedia.org/T386343#10552985 (10JAllemandou) [16:46:27] 06Data-Engineering, 06Data-Engineering-Icebox, 10Event-Platform: [EventGate] Failures when getting stream config from MediaWiki API - https://phabricator.wikimedia.org/T286793#10553024 (10Ottomata) 05Open→03Declined I'm going to be bold and decline this. It has been years, and some things have change... [17:51:32] 10Data-Engineering (Q3 2024 January 1st - March 31th), 07Essential-Work: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10553310 (10Ahoelzl) @mforns @Gehel we only have access to our logs, that's been a constraint from the beginning. [17:54:09] 10Data-Engineering (Q3 2024 January 1st - March 31th), 07Essential-Work: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10553314 (10Ahoelzl) Initial dashboard with aggregated counts available here: https://superset.wikimedia.org/superset/dashboard/2f06e4cb-86a1-4a5a-ba04-b5... [18:21:41] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Data Pipelines, 10Data-Catalog: Integrate Spark with DataHub with lineage - https://phabricator.wikimedia.org/T306896#10553366 (10Ahoelzl) Column-level data lineage of the newly deployed Spark jobs can be queried and navigated in DataHub: https://datah... [18:27:04] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Data Pipelines, 10Data-Catalog: Integrate Spark with DataHub with lineage (Data-Engineering) - https://phabricator.wikimedia.org/T306896#10553382 (10Ahoelzl) [18:27:56] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Data Pipelines, 10Data-Catalog: Integrate Spark with DataHub with lineage (Data-Engineering) - https://phabricator.wikimedia.org/T306896#10553384 (10Ahoelzl) [19:51:09] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 07Epic: DomainEvents - Broadcasting and receiving cross-process events - https://phabricator.wikimedia.org/T379935#10553647 (10Ottomata) Daniel and I have had some discussions on how we might accomplish i... [19:52:37] 10Data-Engineering (Q3 2024 January 1st - March 31th): Update eventstreams Grafana Dashboards to use histogram for router metrics - https://phabricator.wikimedia.org/T386204#10553648 (10tchin) 05In progress→03Resolved Garbage collection is enabled by default, but it seems that the default metrics are als... [19:59:29] 10Data-Engineering (Q3 2024 January 1st - March 31th), 07Essential-Work: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10553683 (10Ahoelzl) **Update:** parser and aggregation code has been extensively test. [20:20:58] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 07Epic: DomainEvents - Broadcasting and receiving cross-process events - https://phabricator.wikimedia.org/T379935#10553701 (10Ottomata) @daniel Q for you: How might the listeners register with this `$as... [21:44:23] 06Data-Engineering, 10ChangeProp, 10observability, 10service-runner, 10Event-Platform: Upgrade prom-client in NodeJS service-runner and enable collectDefaultMetrics - https://phabricator.wikimedia.org/T350180#10553965 (10Ottomata) 05Open→03Declined This is now supported in the new [[ https://gitl... [21:44:24] 10Data-Engineering (Q3 2024 January 1st - March 31th): Update eventstreams Grafana Dashboards to use histogram for router metrics - https://phabricator.wikimedia.org/T386204#10553967 (10Ottomata) Great! I closed that other task then. Thanks!