[01:24:51] 10Analytics-Canonical-Data, 06Movement-Insights: Document fields of canonical wiki dataset - https://phabricator.wikimedia.org/T371766#10240638 (10nshahquinn-wmf) a:03nshahquinn-wmf [08:27:14] 06Data-Engineering, 10CampaignEvents, 10EntitySchema, 10JsonConfig, and 14 others: Add namespace descriptions for Special:NamespaceInfo in WMF-deployed extensions - https://phabricator.wikimedia.org/T373070#10240959 (10Msz2001) [09:51:45] 06Data-Engineering, 03Discovery-Search (Current work), 10Dumps 2.0 (Kanban Board), 10Event-Platform, 13Patch-For-Review: Bump eventutilities to support flink 1.19 - https://phabricator.wikimedia.org/T377130#10241164 (10gmodena) @dcausse flink-kafka-connector 3.3.0 just released, with support to Flink 1.2... [10:05:43] 06Data-Engineering, 10Data-Platform-SRE (2024.09.28 - 2024.10.18), 03Discovery-Search (Current work): Unable to find ingested tables in datahub - https://phabricator.wikimedia.org/T376657#10241253 (10BTullis) When I do a search for //discovery// in DataHub I can see 17 hive tables returned. {F57623685} I exp... [10:31:32] 10Quarry: Quarry shows error: This web service cannot be reached - https://phabricator.wikimedia.org/T375988#10241364 (10rook) >>! In T375988#10236341, @GTrang wrote: >>>! In T375988#10186586, @rook wrote: >> Quarry is working again. Though I didn't have time to investigate what is happening so this may happen a... [10:36:29] 06Data-Engineering, 10Temporary accounts, 10Event-Platform: Prepare EventBus for temp accounts - https://phabricator.wikimedia.org/T374811#10241370 (10gmodena) I've synced on this with @Milimetric and the team responsible for the analytics bits of the temp account feature. We will not add fields to legacy s... [10:36:45] 06Data-Engineering, 10Temporary accounts, 10Event-Platform: Prepare EventBus for temp accounts - https://phabricator.wikimedia.org/T374811#10241371 (10gmodena) a:03gmodena [10:42:26] 06Data-Engineering, 06Data Products, 06Data-Platform, 06Movement-Insights, and 2 others: Temporary Accounts Initiative (IP Masking) - Add user_is_temp to data tables - https://phabricator.wikimedia.org/T356701#10241389 (10gmodena) >>! In T356701#10231536, @Ottomata wrote: > Hm, interesting! TIL about MW's... [10:55:13] 06Data-Engineering, 10Data-Engineering-Jupyter, 10CAS-SSO, 07Epic: Improve the JupyterHub services and use CAS/SSO - https://phabricator.wikimedia.org/T260386#10241444 (10BTullis) [10:55:51] 06Data-Engineering, 10Data-Engineering-Jupyter, 10CAS-SSO, 06Data-Platform-SRE, 07Epic: Improve the JupyterHub services and use CAS/SSO - https://phabricator.wikimedia.org/T260386#10241446 (10BTullis) [11:05:28] 06Data-Engineering, 10Temporary accounts, 10Event-Platform: Prepare EventBus for temp accounts - https://phabricator.wikimedia.org/T374811#10241499 (10Milimetric) >>! In T374811#10241370, @gmodena wrote: > I've synced on this with @Milimetric and the team responsible for the analytics bits of the temp accoun... [13:09:57] 06Data-Engineering, 03Discovery-Search (Current work), 10Dumps 2.0 (Kanban Board), 10Event-Platform, 13Patch-For-Review: Bump eventutilities to support flink 1.19 - https://phabricator.wikimedia.org/T377130#10241834 (10gmodena) > I've updated (not yet pushed) the linked CR to address method migration fro... [14:08:53] 10Quarry: Quarry shows error: This web service cannot be reached - https://phabricator.wikimedia.org/T375988#10242045 (10GTrang) 05Open→03Resolved And now Quarry is working again. [14:41:58] 06Data-Engineering, 10MediaWiki-extensions-WikimediaEvents, 10Observability-Metrics, 10Event-Platform, and 3 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10242226 (10gmodena) >>>! In T355837#10230184, @gmodena wrote: >> Will this add significant tra... [15:10:44] 06Data-Engineering, 10Cassandra, 10Data Pipelines, 10Data-Platform-SRE (2024.10.19 - 2024.11.08), 13Patch-For-Review: Create puppet resource for adding/updating/deleting secrets or other small files on HDFS - https://phabricator.wikimedia.org/T323692#10242327 (10BTullis) [15:10:48] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Movement-Insights, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): 2024-10-10 Data Loss Incident - webrequest Hive table - https://phabricator.wikimedia.org/T376882#10242331 (10BTullis) [15:11:09] 06Data-Engineering, 10Data-Platform-SRE (2024.10.19 - 2024.11.08), 03Discovery-Search (Current work): Unable to find ingested tables in datahub - https://phabricator.wikimedia.org/T376657#10242329 (10BTullis) [15:13:20] 06Data-Engineering, 10Prod-Kubernetes, 06serviceops, 10Data-Platform-SRE (2024.10.19 - 2024.11.08), and 3 others: Migrate Search Platform-owned helm charts to Calico Network Policies - https://phabricator.wikimedia.org/T373195#10242351 (10BTullis) [15:13:27] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Deploy the HDFS synchronizer service to the dse-k8s cluster - https://phabricator.wikimedia.org/T371994#10242357 (10BTullis) [15:14:07] 06Data-Engineering, 06Discovery-Search, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Upload an image with flink-k8s-operator version that supports flink 1.19 - https://phabricator.wikimedia.org/T377137#10242364 (10BTullis) [15:14:44] 06Data-Engineering, 06Discovery-Search, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Create and distribute a flink base image with flink 1.19.1 - https://phabricator.wikimedia.org/T377134#10242366 (10BTullis) [15:16:15] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.10.19 - 2024.11.08), 13Patch-For-Review: Update druid config to automatically drop unused segments - https://phabricator.wikimedia.org/T376118#10242380 (10BTullis) [15:16:43] 06Data-Engineering, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Some wikibase tables not available in commonswiki_p - https://phabricator.wikimedia.org/T298452#10242396 (10BTullis) [15:17:46] 06Data-Engineering, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#10242416 (10BTullis) [15:30:39] 06Data-Engineering, 10Data-Platform-SRE (2024.10.19 - 2024.11.08), 03Discovery-Search (Current work): Unable to find ingested tables in datahub - https://phabricator.wikimedia.org/T376657#10242489 (10EBernhardson) Curious, i can't say what changed but today I'm getting search results that I wasn't a week ago... [15:45:25] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Movement-Insights, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): 2024-10-10 Data Loss Incident - webrequest Hive table - https://phabricator.wikimedia.org/T376882#10242558 (10BTullis) I think that #data-engineering has this all in-hand now, is that... [15:47:21] 06Data-Engineering, 06Data Products: [refine] Add support for extra partitioning - https://phabricator.wikimedia.org/T377600 (10mforns) 03NEW [15:48:28] 06Data-Engineering, 06Data Products: [refine] Add support for extra partitioning - https://phabricator.wikimedia.org/T377600#10242589 (10mforns) [16:43:07] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Movement-Insights, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): 2024-10-10 Data Loss Incident - webrequest Hive table - https://phabricator.wikimedia.org/T376882#10242794 (10Ahoelzl) a:03JAllemandou [16:49:33] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: [Event Platform] Declare webrequest as an Event Platform stream - https://phabricator.wikimedia.org/T314956#10242822 (10gmodena) > I agree though that we will not block haproxy migration on this, and can prioritize this separately in a future date. >... [17:20:03] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Metrics Platform: Generate $wgEventLoggingStreamNames from $wgEventStreams - https://phabricator.wikimedia.org/T303602#10242922 (10phuedx) a:05phuedx→03None [17:45:30] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform, 10Temporary accounts: Add MW table 'cu_log' to data lake - https://phabricator.wikimedia.org/T364398#10242984 (10Ahoelzl) DPE DE started working on the table ingestion. [17:46:21] 06Data-Engineering, 10Data Pipelines: Improve Airflow DAG testing process - https://phabricator.wikimedia.org/T368944#10242987 (10Ahoelzl) [18:50:39] 06Data-Engineering, 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.27; 2024-10-15), 13Patch-For-Review, and 2 others: Delete redundant mobile- and desktopwebuiactions event in WikimediaEvents - https://phabricator.wikimedia.org/T376065#10243128 (10Edtadros) [18:54:32] 06Data-Engineering, 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.27; 2024-10-15), 13Patch-For-Review, and 2 others: Delete redundant mobile- and desktopwebuiactions event in WikimediaEvents - https://phabricator.wikimedia.org/T376065#10243137 (10Edtadros) [18:57:53] 06Data-Engineering, 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.27; 2024-10-15), 13Patch-For-Review, and 2 others: Delete redundant mobile- and desktopwebuiactions event in WikimediaEvents - https://phabricator.wikimedia.org/T376065#10243154 (10Edtadros) a:05Edtadros→03None Test Result - Prod Status... [18:58:31] 06Data-Engineering, 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.27; 2024-10-15), 13Patch-For-Review, and 2 others: Delete redundant mobile- and desktopwebuiactions event in WikimediaEvents - https://phabricator.wikimedia.org/T376065#10243160 (10Edtadros) [19:05:46] 06Data-Engineering, 10Event-Platform, 10Web Team Essential Work 2024 (Migrate to new Event Platform), 10Web-Team-Backlog (FY2024-25 Q2 Sprint 2): Deprecate use of desktop- and mobilewebuiactions in Event Platform - https://phabricator.wikimedia.org/T368678#10243169 (10Edtadros) Test Result - Prod Status:... [19:05:50] 06Data-Engineering, 10Event-Platform, 10Web Team Essential Work 2024 (Migrate to new Event Platform), 10Web-Team-Backlog (FY2024-25 Q2 Sprint 2): Deprecate use of desktop- and mobilewebuiactions in Event Platform - https://phabricator.wikimedia.org/T368678#10243175 (10Edtadros) [19:25:08] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform, 06Traffic, 10Data Products (Data Products (Data Products Sprint 21 🪂)), 13Patch-For-Review: NEW BUG REPORT - Issues in calculation logic for unique devices tables - https://phabricator.wikimedia.org/T375527#10243240 (10Milimetric) [20:39:33] 10Analytics-Canonical-Data, 06Movement-Insights: Document fields of canonical wiki dataset - https://phabricator.wikimedia.org/T371766#10243542 (10nshahquinn-wmf) 05Open→03Resolved Done in [this commit](https://gitlab.wikimedia.org/repos/movement-insights/canonical-data/-/commit/93ce69dd80f4443ec2a47b2...