[01:08:18] 06Data-Engineering, 06Data Products, 06Data-Platform: eswiki most viewed pages from Spain 2015-2024 - https://phabricator.wikimedia.org/T378909 (10Platonides) 03NEW [09:31:54] 06Data-Engineering, 06Data Products: Sqoop all mysql tables from production replicas instead of CloudDB replicas - https://phabricator.wikimedia.org/T378923 (10JAllemandou) 03NEW [09:51:16] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, 10MW-1.44-notes (1.44.0-wmf.2; 2024-11-05): Implement DomainEventDispatcher (baseline) - https://phabricator.wikimedia.org/T377229#10287699 (10daniel) [09:57:01] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, 10MW-1.44-notes (1.44.0-wmf.2; 2024-11-05): Implement DomainEventDispatcher (baseline) - https://phabricator.wikimedia.org/T377229#10287767 (10daniel) [09:57:31] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, 10MW-1.44-notes (1.44.0-wmf.2; 2024-11-05): Implement DomainEventDispatcher (baseline) - https://phabricator.wikimedia.org/T377229#10287769 (10daniel) [09:57:40] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, 10MW-1.44-notes (1.44.0-wmf.2; 2024-11-05): Implement DomainEventDispatcher (baseline) - https://phabricator.wikimedia.org/T377229#10287772 (10daniel) [10:09:41] 06Data-Engineering, 06MW-Interfaces-Team: Explore mechanism for publishing domain events - https://phabricator.wikimedia.org/T378933 (10daniel) 03NEW [10:14:55] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, 10MW-1.44-notes (1.44.0-wmf.2; 2024-11-05): Implement DomainEventDispatcher (baseline) - https://phabricator.wikimedia.org/T377229#10287849 (10daniel) [10:22:10] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, 10MW-1.44-notes (1.44.0-wmf.2; 2024-11-05): Implement DomainEventDispatcher (baseline) - https://phabricator.wikimedia.org/T377229#10287894 (10daniel) [10:27:45] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10287915 (10Ladsgroup) [10:27:57] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, 10MW-1.44-notes (1.44.0-wmf.2; 2024-11-05): Implement DomainEventDispatcher (baseline) - https://phabricator.wikimedia.org/T377229#10287917 (10daniel) 05Open→03Resolved [10:29:53] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10287933 (10Ladsgroup) I explicitly skipped sanitarium master of s8 since if I start the schema change, the replication... [12:08:08] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10288227 (10Marostegui) >>! In T367856#10287933, @Ladsgroup wrote: > I explicitly skipped sanitarium master of s8 since... [13:40:00] 06Data-Engineering, 10Data Products (Data Products (Data Products Sprint 21 🪂)), 07Documentation, 10Event-Platform: Render human-readable schemas on schema.wikimedia.org - https://phabricator.wikimedia.org/T376841#10288493 (10Milimetric) a:03Milimetric [13:52:22] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10288606 (10Marostegui) Thank you @BTullis - I have started the schema change. I did a quick test on the test cluster a... [13:58:28] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10288792 (10Marostegui) [14:01:44] 06Data-Engineering, 06Machine-Learning-Team, 06Research, 10Event-Platform: Expose revision revert risk scores in EventStreams - https://phabricator.wikimedia.org/T326179#10288798 (10Ottomata) [14:26:05] 10Quarry: update build-and-push - https://phabricator.wikimedia.org/T378978 (10rook) 03NEW [14:26:21] 10Quarry, 10PAWS: update github action - https://phabricator.wikimedia.org/T348873#10288933 (10rook) [14:26:45] 10Quarry: update build-and-push - https://phabricator.wikimedia.org/T378978#10288936 (10rook) [14:26:50] 10Quarry, 10PAWS: update github action - https://phabricator.wikimedia.org/T348873#10288937 (10rook) [14:27:00] 10Data-Engineering (Q2 2024 October 1st - December 31th), 13Patch-For-Review: [Refine Refactoring] Refine jobs should be scheduled by Airflow: deployment - https://phabricator.wikimedia.org/T369845#10288938 (10Antoine_Quhen) **1/ About the ESC source** Currently, I have a script that reads the current ESC JSO... [14:27:10] 10Quarry, 10PAWS: update github action - https://phabricator.wikimedia.org/T348873#10288939 (10rook) 05Open→03Resolved [14:31:13] 06Data-Engineering, 06Data Products, 06DBA, 10GlobalBlocking, 07Schema-change-in-production: Drop gbw_address and gbw_target_central_id from the global_block_whitelist table on WMF wikis - https://phabricator.wikimedia.org/T378747#10288956 (10Ladsgroup) 05Open→03Resolved [14:58:28] 10Data-Engineering (Q2 2024 October 1st - December 31th), 13Patch-For-Review: [Refine Refactoring] Refine jobs should be scheduled by Airflow: deployment - https://phabricator.wikimedia.org/T369845#10289064 (10Ottomata) > But keeping those 171 custom blocks statically in the MediaWiki-config repo is OK. Still... [15:11:55] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10CheckUser, 06Privacy Engineering: Add cu_log_event and cu_private_event CheckUser tables to data lake - https://phabricator.wikimedia.org/T376752#10289141 (10Snwachukwu) a:03Snwachukwu [15:24:20] 14Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 06Java-Scala-Standardization, 03Discovery-Search (Current work), 10Release-Engineering-Team (Radar): Validate CI integration so that Ci can release Maven artifacts on user's de... - https://phabricator.wikimedia.org/T367403#10289184 [15:25:02] 06Data-Engineering, 06Data-Platform-SRE, 10SRE-Access-Requests: Request Kerberos identity for jsn.sherman - https://phabricator.wikimedia.org/T378786#10289193 (10Ottomata) [15:59:45] 10Data-Engineering (Q2 2024 October 1st - December 31th), 13Patch-For-Review: [Refine Refactoring] Refine jobs should be scheduled by Airflow: deployment - https://phabricator.wikimedia.org/T369845#10289438 (10Ottomata) Re **4/ Refined table that won't Refine with new process**: **eventgate_main_error_valida... [17:42:11] 10Data-Engineering (Q2 2024 October 1st - December 31th): [SPIKE] Learn and document how to use Flink-CDC from MediaWiki MariaDB locally - https://phabricator.wikimedia.org/T373144#10290058 (10Ottomata) Awesome! Yeah I was thinking of trying that next too! So MariaDb -> Debezium -> Kafka -> Flink CDC Iceberg or... [17:42:41] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data Pipelines, 10Data-Catalog: Upgrade to Spark 3.2 to support Spark lineage for Iceberg tables - https://phabricator.wikimedia.org/T378899#10290059 (10Ottomata) [17:43:24] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data Pipelines, 10Data-Catalog: Upgrade to Spark 3.2 to support Spark lineage for Iceberg tables - https://phabricator.wikimedia.org/T378899#10290063 (10Ottomata) [17:43:27] 06Data-Engineering, 06Data-Platform-SRE: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 - https://phabricator.wikimedia.org/T338057#10290064 (10Ottomata) [17:46:05] 06Data-Engineering, 06Data-Platform-SRE: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 - https://phabricator.wikimedia.org/T338057#10290072 (10Ottomata) [17:49:13] 06Data-Engineering, 06Data Products: Sqoop all mysql tables from production replicas instead of CloudDB replicas - https://phabricator.wikimedia.org/T378923#10290095 (10Ottomata) Agree. [18:28:04] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Wikidata, 10Multi-Content-Revisions (Deployment), 13Patch-For-Review: MCR schema migration stage 4: Migrate External Store URLs (wmf production) - https://phabricator.wikimedia.org/T183490#10290312 (10Ahoelzl) [18:53:26] (03CR) 10Mforns: [C:04-1] Add items for event sanitization regarding iOS edit (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1083192 (https://phabricator.wikimedia.org/T377259) (owner: 10GOlson) [19:31:01] 10Data-Engineering (Q2 2024 October 1st - December 31th), 07Epic, 13Patch-For-Review: [Maintenance] Safeguard VarnishKafka to HAProxy analytics transition - https://phabricator.wikimedia.org/T354694#10290480 (10gmodena) [19:46:14] 10Data-Engineering (Q2 2024 October 1st - December 31th): [SPIKE] Learn and document how to use Flink-CDC from MediaWiki MariaDB locally - https://phabricator.wikimedia.org/T373144#10290534 (10NoZeroDay) Thank you! I have the first part working (Maria DB -> Debezium -> Kafka) so will be commencing work on the Ka... [20:07:18] 10Data-Engineering (Q2 2024 October 1st - December 31th): Implement a data retention policy for webrequest_frontend datasets - https://phabricator.wikimedia.org/T379024 (10gmodena) 03NEW [20:09:42] 10Data-Engineering (Q2 2024 October 1st - December 31th), 07Epic, 13Patch-For-Review: [Maintenance] Safeguard VarnishKafka to HAProxy analytics transition - https://phabricator.wikimedia.org/T354694#10290623 (10gmodena) [20:14:46] (03CR) 10Tsevener: Add items for event sanitization regarding iOS edit (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1083192 (https://phabricator.wikimedia.org/T377259) (owner: 10GOlson) [20:24:20] (03PS4) 10GOlson: Add items for event sanitization regarding iOS edit, update keep to hash. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1083192 (https://phabricator.wikimedia.org/T377259) [20:26:04] (03CR) 10Tsevener: [C:03+1] Add items for event sanitization regarding iOS edit, update keep to hash. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1083192 (https://phabricator.wikimedia.org/T377259) (owner: 10GOlson) [20:28:57] (03CR) 10GOlson: "Done! ✔" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1083192 (https://phabricator.wikimedia.org/T377259) (owner: 10GOlson) [23:05:27] 06Data-Engineering, 06Movement-Insights, 10Wmfdata-Python, 10GitLab (Project Migration): Move Wmfdata-Python from Github to Gitlab - https://phabricator.wikimedia.org/T304544#10291106 (10nshahquinn-wmf) [23:10:34] 06Data-Engineering, 10Wikipedia-Android-App-Backlog (Android Release - FY2024-25): Revert Metrics Platform changes that were made for the Recommended Content feature - https://phabricator.wikimedia.org/T379031 (10cooltey) 03NEW [23:16:40] 06Data-Engineering, 06Movement-Insights, 10Wmfdata-Python, 07Documentation: Create a proper documentation microsite for Wmfdata-Python - https://phabricator.wikimedia.org/T298178#10291130 (10nshahquinn-wmf) [23:17:09] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics, 10Wmfdata-Python: Add testing framework to wmfdata-python - https://phabricator.wikimedia.org/T349531#10291131 (10nshahquinn-wmf) [23:23:17] 06Data-Engineering, 06Product-Analytics, 10Wmfdata-Python: Enable wmfdata-py to access MariaDB replicas on the cluster - https://phabricator.wikimedia.org/T340467#10291144 (10nshahquinn-wmf) [23:24:20] 06Data-Engineering, 06Movement-Insights, 10Wikidata, 10Wikidata Analytics, 10Wmfdata-Python: Add linter and formatter to wmfdata-python (and link check) - https://phabricator.wikimedia.org/T348999#10291136 (10nshahquinn-wmf) a:05AndrewTavis_WMDE→03fkaelin @fkaelin has started [working on this](https:... [23:24:41] 06Data-Engineering, 10Wmfdata-Python: Remove Wmfdata's custom update-notification code - https://phabricator.wikimedia.org/T346706#10291145 (10nshahquinn-wmf) [23:25:31] 06Data-Engineering, 06Product-Analytics, 10Wmfdata-Python: Set up Wmfdata-Python test suite to run automatically - https://phabricator.wikimedia.org/T304547#10291159 (10nshahquinn-wmf) [23:29:42] 06Data-Engineering, 06Movement-Insights, 10Wikidata, 10Wikidata Analytics, 10Wmfdata-Python: Add linter and formatter to wmfdata-python (and link check) - https://phabricator.wikimedia.org/T348999#10291184 (10nshahquinn-wmf) [23:31:00] 14Analytics-Kanban, 06Data-Engineering, 06Product-Analytics, 10Wmfdata-Python: wmfdata.mariadb relies on analytics-mysql being available - https://phabricator.wikimedia.org/T292479#10291213 (10nshahquinn-wmf) [23:31:57] 06Data-Engineering, 06Product-Analytics, 10Wmfdata-Python: Retrieve host & port info when connecting to MariaDB replicas on the cluster - https://phabricator.wikimedia.org/T340472#10291219 (10nshahquinn-wmf) [23:34:41] 06Data-Engineering, 06Product-Analytics, 10Wmfdata-Python: Let user specify cnf to use when connecting to MariaDB - https://phabricator.wikimedia.org/T340469#10291224 (10nshahquinn-wmf) [23:34:43] 06Data-Engineering, 06Movement-Insights, 10Wmfdata-Python, 10GitLab (Project Migration): Move Wmfdata-Python from Github to Gitlab - https://phabricator.wikimedia.org/T304544#10291225 (10nshahquinn-wmf)