[04:55:18] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop gb_by from globalblocks table - https://phabricator.wikimedia.org/T370394#10021370 (10Marostegui) Just to be clear, can this be done or is it blocked on something? [06:18:26] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10021430 (10Marostegui) [06:18:44] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10021433 (10Marostegui) [09:57:28] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop gb_by from globalblocks table - https://phabricator.wikimedia.org/T370394#10022156 (10Zabe) This can be done from my perspective. Nothing is writing to that column for quite some time, nothing seems to be using it from a codesearch perspective a... [10:21:35] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop gb_by from globalblocks table - https://phabricator.wikimedia.org/T370394#10022310 (10Marostegui) This is done ` # /home/marostegui/section s7 | grep -v clouddb1021 | while read host port; do echo "$host"; db-mysql $host:$port centralauth -e... [10:22:33] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop gb_by from globalblocks table - https://phabricator.wikimedia.org/T370394#10022311 (10Marostegui) 05Open→03Resolved a:05ABran-WMF→03Marostegui [13:23:57] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data Products, 10Metrics Platform Backlog: [MPIC] Analyse risk of potential performance issues with static approach to stream configuration - https://phabricator.wikimedia.org/T366627#10022878 (10Antoine_Quhen) We have prepared some work to Refine raw... [13:57:23] 06Data-Engineering, 10CheckUser, 06Data Products, 06DBA, 07Schema-change-in-production: Remove cuc_actiontext, cuc_only_for_read_old, and cuc_private from cu_changes on WMF wikis - https://phabricator.wikimedia.org/T370903#10022999 (10Ottomata) @VirginiaPoundstone yes, the one @Dreamy_Jazz created and su... [14:09:09] !log rerunning airflow mediawiki_history_check_denormalize dag as down stream task after rerunning mediawiki_history_denormalize dag [14:09:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:48:57] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#10023341 (10Ottomata... [14:53:43] 06Data-Engineering, 06Data Products, 10GlobalBlocking, 06Trust and Safety Product Team, and 2 others: Add gb_enable_autoblock and gb_auto to the globalblocks table - https://phabricator.wikimedia.org/T371268 (10Dreamy_Jazz) 03NEW [14:54:26] 06Data-Engineering, 06Data Products, 10GlobalBlocking, 06Trust and Safety Product Team, and 2 others: Add gb_enable_autoblock and gb_autoblock_parent to the globalblocks table - https://phabricator.wikimedia.org/T371268#10023434 (10Dreamy_Jazz) [14:54:32] 06Data-Engineering, 06Data Products, 10GlobalBlocking, 06Trust and Safety Product Team, and 2 others: Add gb_enable_autoblock and gb_autoblock_parent to the globalblocks table - https://phabricator.wikimedia.org/T371268#10023431 (10Dreamy_Jazz) [14:56:23] 07Analytics-Data-Problem, 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform, 06Movement-Insights: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions - https://phabricator.wikimedia.org/T369851#10023438 (10Snwachukwu) So I reran **//mediawiki_history_denormaliz... [14:57:28] 07Analytics-Data-Problem, 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform, 06Movement-Insights: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions - https://phabricator.wikimedia.org/T369851#10023443 (10Snwachukwu) Next steps would be to rerun any affected d... [15:06:16] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Observability-Tracing: service-utils helper for trace header propagation - https://phabricator.wikimedia.org/T371120#10023462 (10tchin) [15:11:41] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 16), and 2 others: Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#10023497 (10Krinkle) 👍 This is great! **First impressions**... [15:11:45] 14Analytics-Radar, 06Data-Engineering-Icebox, 10ChangeProp, 10MassMessage, 10WMF-JobQueue: The mass-message queue reports 0 when there are still queued messages - https://phabricator.wikimedia.org/T209899#10023493 (10Legoktm) Related: {T58878}. I'm not sure what the best solution is here. The number is... [15:22:45] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 16), and 2 others: Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#10023536 (10Milimetric) great, moving this to get deployed.... [15:25:02] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 17), and 2 others: Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#10023558 (10Milimetric) [15:41:42] 06Data-Engineering, 06Data Products, 10GlobalBlocking, 06Trust and Safety Product Team, and 2 others: Add gb_enable_autoblock and gb_autoblock_parent to the globalblocks table - https://phabricator.wikimedia.org/T371268#10023644 (10Dreamy_Jazz) [15:42:27] 06Data-Engineering, 06Data Products, 10GlobalBlocking, 06Trust and Safety Product Team, and 2 others: Add gb_enable_autoblock and gb_autoblock_parent_id to the globalblocks table - https://phabricator.wikimedia.org/T371268#10023646 (10Dreamy_Jazz) [16:11:43] 14Analytics, 06Data-Engineering-Icebox: Augment NEL reports with a computed timestamp-of-generation - https://phabricator.wikimedia.org/T266886#10023754 (10colewhite) Are these reports currently in Logstash or are they in Hive? [16:26:15] 06Data-Engineering, 06Data Products, 06DBA, 10GlobalBlocking, and 4 others: Add gb_enable_autoblock and gb_autoblock_parent_id to the globalblocks table - https://phabricator.wikimedia.org/T371268#10023825 (10Dreamy_Jazz) [16:27:34] 06Data-Engineering, 06Data Products, 06DBA, 10GlobalBlocking, and 4 others: Add gb_enable_autoblock and gb_autoblock_parent_id to the globalblocks table - https://phabricator.wikimedia.org/T371268#10023828 (10Dreamy_Jazz) [16:46:33] 14Analytics, 06Data-Engineering-Icebox: Augment NEL reports with a computed timestamp-of-generation - https://phabricator.wikimedia.org/T266886#10023994 (10Ottomata) Logstash: ` curl -s 'https://meta.wikimedia.org/w/api.php?action=streamconfigs&format=json&formatversion=2&streams=w3c.reportingapi.network_erro... [16:52:17] 14Analytics, 06Data-Engineering-Icebox: Augment NEL reports with a computed timestamp-of-generation - https://phabricator.wikimedia.org/T266886#10024039 (10CDanis) Yep, Logstash presently, although it would be nice if we had them in Hive some day as well :) [18:44:02] 06Data-Engineering, 10MediaWiki-extensions-General, 10Event-Platform: Update code comment links to Meta-Wiki schemas to new event platform - https://phabricator.wikimedia.org/T371305#10024722 (10Pppery) a:05Pppery→03None [18:44:56] 06Data-Engineering, 10MediaWiki-extensions-General, 07Documentation, 10Event-Platform: Update code comment links to Meta-Wiki schemas to new event platform - https://phabricator.wikimedia.org/T371305#10024728 (10Pppery) [18:57:14] 06Data-Engineering, 10MediaWiki-extensions-General, 07Documentation, 10Event-Platform: Update code comment links to Meta-Wiki schemas to new event platform - https://phabricator.wikimedia.org/T371305#10024804 (10Ottomata) [18:57:18] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10MediaWiki-extensions-EventLogging, 10Event-Platform, 13Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230#10024805 (10Ottomata) [20:01:39] 06Data-Engineering, 10CheckUser, 06Data Products, 06DBA, 07Schema-change-in-production: Remove cuc_actiontext, cuc_only_for_read_old, and cuc_private from cu_changes on WMF wikis - https://phabricator.wikimedia.org/T370903#10025146 (10VirginiaPoundstone) @milimetric also flagged that this will break this... [20:01:55] 06Data-Engineering, 10CheckUser, 06Data Products, 06DBA, 07Schema-change-in-production: Update sqoop code to remove cuc_actiontext from query and table - https://phabricator.wikimedia.org/T371319 (10Milimetric) 03NEW [20:03:45] 06Data-Engineering: [opsweek] Airflow DAGs with Spark jobs should always include Spark tuning variables - https://phabricator.wikimedia.org/T343154#10025177 (10Ottomata) If we do this: we should refactor the DagProperties names to be prefixed with `spark_`, so that it is clear what they are for. Also, an Airf... [20:06:06] 06Data-Engineering, 10CheckUser, 06Data Products, 06DBA, 07Schema-change-in-production: Update sqoop code to remove cuc_actiontext from query and table - https://phabricator.wikimedia.org/T371319#10025181 (10Milimetric) a:05Ladsgroup→03None [20:06:09] 14Analytics, 06Data-Engineering-Icebox: Add a "latest" partition to Hive tables - https://phabricator.wikimedia.org/T252148#10025186 (10Ottomata) 05Open→03Declined Being bold and declining. In an Iceberg world, this won't be needed. [20:16:36] 06Data-Engineering, 10CheckUser, 06Data Products, 06DBA, 07Schema-change-in-production: Update sqoop code to remove cuc_actiontext from query and table - https://phabricator.wikimedia.org/T371319#10025267 (10Milimetric) →14Duplicate dup:03T371099 [20:18:01] 06Data-Engineering, 10Data Products (Data Products Sprint 17), 13Patch-For-Review, 10Trust and Safety Product Sprint (Sprint Koto (July 15 - July 26)): No longer use removed cuc_actiontext column in analytics/refinery - https://phabricator.wikimedia.org/T371099#10025264 (10Milimetric) [20:18:51] 06Data-Engineering, 10Data Products (Data Products Sprint 17), 13Patch-For-Review, 10Trust and Safety Product Sprint (Sprint Koto (July 15 - July 26)): No longer use removed cuc_actiontext column in analytics/refinery - https://phabricator.wikimedia.org/T371099#10025269 (10Milimetric) Adding this to our Sp... [21:00:35] 14Analytics, 06Data-Engineering-Icebox, 10Observability-Logging, 10SRE Observability (FY2024/2025-Q1): Augment NEL reports with a computed timestamp-of-generation - https://phabricator.wikimedia.org/T266886#10025465 (10colewhite) a:03colewhite I think this is doable for Logstash. I'll have a go at it.