[06:16:53] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Release-Engineering-Team: Allow JavaScript errors to fail CI builds - https://phabricator.wikimedia.org/T318902#10487439 (10Jdlrobson) Per the web team's quarterly grooming, these tasks are being removed from the team's backlog. [08:42:53] 06Data-Engineering, 06Language and Product Localization: Shut down the Language Reportcard - https://phabricator.wikimedia.org/T384409#10487835 (10Nikerabbit) p:05Triage→03Low @KCVelaga_WMF do you have an idea who can do this? [08:49:18] 06Data-Engineering, 06Language and Product Localization: Shut down the Language Reportcard - https://phabricator.wikimedia.org/T384409#10487849 (10KCVelaga_WMF) That should probably be Data Engineering. @Milimetric I see you have created the [[ https://meta.wikimedia.org/wiki/Config:Dashiki:LanguageReportcard... [10:44:34] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10Data-Platform-SRE (2025.01.11 - 2025.01.31), 13Patch-For-Review: Data Platform access streamlining for WMDE staff - https://phabricator.wikimedia.org/T381824#10488103 (10jcrespo) 05Open→03Resolved This is now applied. [11:00:07] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Release-Engineering-Team: Allow JavaScript errors to fail CI builds - https://phabricator.wikimedia.org/T318902#10488198 (10kostajh) >>! In T318902#10480670, @Jdlrobson wrote: > FWIW Right now, if code was committed that caused JS errors this would l... [11:49:13] 06Data-Engineering, 06Data-Persistence, 10Dumps-Generation, 10Data-Platform-SRE (2025.01.11 - 2025.01.31), 13Patch-For-Review: Switch dumps 1.0 processes to use the analytics MariadB replicas (dbstore100[7-9]) - https://phabricator.wikimedia.org/T382947#10488307 (10BTullis) @joe - Sorry to trouble you. I... [12:06:06] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Change page.page_links_updated to fixed-length timestamp in wmf wikis - https://phabricator.wikimedia.org/T371742#10488440 (10Ladsgroup) [12:11:55] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592 (10Ladsgroup) 03NEW [12:15:43] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592#10488491 (10Marostegui) p:05Triage→03Medium a:03Marostegui I will do the schema change and @Ladsgroup will take care of the table creation. [12:31:34] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592#10488541 (10Marostegui) [12:32:43] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592#10488544 (10Marostegui) [12:38:12] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592#10488566 (10Marostegui) [12:45:04] 06Data-Engineering, 06Data-Persistence, 10Dumps-Generation, 10Data-Platform-SRE (2025.01.11 - 2025.01.31), 13Patch-For-Review: Switch dumps 1.0 processes to use the analytics MariadB replicas (dbstore100[7-9]) - https://phabricator.wikimedia.org/T382947#10488575 (10BTullis) I've started a dump from snaps... [13:36:49] 06Data-Engineering: Delete reportupdater jobs data/puppet-code - https://phabricator.wikimedia.org/T358210#10488757 (10JAllemandou) 05Open→03Resolved a:03JAllemandou I think this has been completed indeed. closing. [13:37:36] 06Data-Engineering, 06Language and Product Localization, 07Essential-Work: Shut down the Language Reportcard - https://phabricator.wikimedia.org/T384409#10488762 (10Milimetric) 05Open→03Resolved a:03Milimetric Turning it off and documenting here. * delete language-reportcard [[ https://horizon.wik... [13:37:53] 06Data-Engineering, 06Language and Product Localization, 07Essential-Work: Shut down the Language Reportcard - https://phabricator.wikimedia.org/T384409#10488766 (10Milimetric) [13:41:04] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Movement-Insights: Temporarily Extend Retention Window for webrequest tables - https://phabricator.wikimedia.org/T375943#10488768 (10JAllemandou) 05Open→03Resolved This has been done,. Resolving. [13:52:41] 10Data-Engineering (Q3 2024 January 1st - March 31th): HDFS capacity needs data engineering and platform users - https://phabricator.wikimedia.org/T384100#10488784 (10JAllemandou) We store 3 month of webrequest data and this sums up to almost 200Tb of data (without replication). If we wish to grow this to 6 mont... [13:54:11] (03PS1) 10Santiago Faci: Adding yue.wikiquote manually to the allow list. An alert raised with unexpected data related to this wiki [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1113808 [13:55:13] (03PS2) 10Santiago Faci: Adding yue.wikiquote manually to the allow list. An alert raised with unexpected data related to this wiki: https://groups.google.com/a/wikimedia.org/g/data-engineering-alerts/c/_rWKxpAXgVg [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1113808 [13:56:45] (03PS3) 10Santiago Faci: Adding yue.wikiquote manually to the allow list. An alert raised with unexpected data related to this wiki [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1113808 [13:57:26] (03PS4) 10Santiago Faci: Adding yue.wikiquote manually to the allow list. An alert raised with unexpected data related to this wiki: https://groups.google.com/a/wikimedia.org/g/data-engineering-alerts/c/_rWKxpAXgVg [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1113808 [13:57:27] 06Data-Engineering, 10Commons-Impact-Metrics-Requests: Request for Images_from_Wiki_Loves_Africa_2024 - https://phabricator.wikimedia.org/T381352#10488791 (10Anthere) Hello. Thanks. Now I just need to figure out exactly how those things work :) [13:58:22] (03CR) 10Joal: [C:03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1113808 (owner: 10Santiago Faci) [14:03:56] (03CR) 10Santiago Faci: [C:03+2] Adding yue.wikiquote manually to the allow list. An alert raised with unexpected data related to this wiki: https://groups.google.com/a/wiki [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1113808 (owner: 10Santiago Faci) [14:04:19] (03CR) 10Santiago Faci: [V:03+2 C:03+2] Adding yue.wikiquote manually to the allow list. An alert raised with unexpected data related to this wiki: https://groups.google.com/a/wiki [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1113808 (owner: 10Santiago Faci) [14:28:49] (03PS5) 10Aqu: Refine transforms minimalist events support [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1112392 (https://phabricator.wikimedia.org/T383914) [14:28:58] (03CR) 10CI reject: [V:04-1] Refine transforms minimalist events support [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1112392 (https://phabricator.wikimedia.org/T383914) (owner: 10Aqu) [14:32:57] 06Data-Engineering, 06Movement-Insights, 13Patch-For-Review: Backfill and recalculate unique devices data from July 2024 to present - https://phabricator.wikimedia.org/T378852#10488902 (10JAllemandou) [14:33:26] 06Data-Engineering, 06Movement-Insights, 13Patch-For-Review: Backfill and recalculate unique devices data from July 2024 to present - https://phabricator.wikimedia.org/T378852#10488903 (10JAllemandou) 05Open→03Resolved And, this is done! all jobs have been backfilled. [14:42:06] 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, 05MW-1.39-notes, and 3 others: WE 5.4 KR - Hypothesis 5.4.6 - Q3 FY24/25 - Validate Dumps 1.0 compatibility with PHP 8.1 - https://phabricator.wikimedia.org/T382484#10488926 (10BTullis) My `enwiki` dumps seems to have stopped making progress at:... [14:59:27] 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, 05MW-1.39-notes, and 3 others: WE 5.4 KR - Hypothesis 5.4.6 - Q3 FY24/25 - Validate Dumps 1.0 compatibility with PHP 8.1 - https://phabricator.wikimedia.org/T382484#10489015 (10BTullis) This is what the process tree looks like at the moment, for... [15:09:31] 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, 05MW-1.39-notes, and 3 others: WE 5.4 KR - Hypothesis 5.4.6 - Q3 FY24/25 - Validate Dumps 1.0 compatibility with PHP 8.1 - https://phabricator.wikimedia.org/T382484#10489072 (10BTullis) I think the reason is that flow has been disabled on enwiki... [15:12:11] 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, 05MW-1.39-notes, and 3 others: WE 5.4 KR - Hypothesis 5.4.6 - Q3 FY24/25 - Validate Dumps 1.0 compatibility with PHP 8.1 - https://phabricator.wikimedia.org/T382484#10489078 (10xcollazo) >>! In T382484#10489072, @BTullis wrote: > I think the re... [15:12:14] !log [data lake temp accounts] re-ran DAG mediawiki_history_check_denormalized for 2024-12 [15:12:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:13:42] 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, 05MW-1.39-notes, and 3 others: WE 5.4 KR - Hypothesis 5.4.6 - Q3 FY24/25 - Validate Dumps 1.0 compatibility with PHP 8.1 - https://phabricator.wikimedia.org/T382484#10489084 (10BTullis) Cool, thanks @xcollazo - For now, I have skipped it by manu... [15:30:59] 06Data-Engineering, 10Dumps-Generation: 20241201 wikidatawiki xml dump not progressing - https://phabricator.wikimedia.org/T382084#10489170 (10Ottomata) 05Open→03Resolved a:03Ottomata According to @milimetric this is resolved. [15:54:00] 06Data-Engineering, 06Research, 10Event-Platform: Implement stream of HTML content on mw.page_change event - https://phabricator.wikimedia.org/T360794#10489396 (10VirginiaPoundstone) @leila I know that @Ahoelzl will be meeting next week with @XiaoXiao-WMF to talk through the next steps here. Please think abo... [15:54:16] 06Data-Engineering: [SPIKE] Learn and document how to use Flink-CDC from MediaWiki MariaDB locally - https://phabricator.wikimedia.org/T373144#10489397 (10NoZeroDay) @Ottomata Hi! I've added a writeup with a narrative of the project. I've also seen that Paimon v1.0 was released - I will be testing it out shortly... [17:07:11] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Event-Platform, 13Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230#10489779 (10Ottomata) [17:07:33] 06Data-Engineering, 06Product-Analytics, 10Event-Platform: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163#10489782 (10Ottomata) [17:17:50] (03PS6) 10Aqu: Refine transforms minimalist events support [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1112392 (https://phabricator.wikimedia.org/T383914) [17:17:56] (03CR) 10CI reject: [V:04-1] Refine transforms minimalist events support [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1112392 (https://phabricator.wikimedia.org/T383914) (owner: 10Aqu) [17:20:10] (03PS7) 10Aqu: Refine transforms minimalist events support [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1112392 (https://phabricator.wikimedia.org/T383914) [17:22:49] 10Data-Engineering (Q3 2024 January 1st - March 31th): Write documentation on usage of RestExternalTaskSensor - https://phabricator.wikimedia.org/T378000#10489847 (10amastilovic) The documentation is on WikiTech's Airflow Developer guide: https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Develope... [17:38:05] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Product-Analytics, 10Event-Platform: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163#10489908 (10Ottomata) [17:38:28] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10MediaWiki-extensions-EventLogging, 10Event-Platform, 13Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230#10489911 (10Ottomata) [17:39:01] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10EventStreams, 10Event-Platform: EventStreams: kafka key should be serialized as a string - https://phabricator.wikimedia.org/T373689#10489917 (10Ottomata) [17:39:23] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06MediaWiki-Engineering, 10MediaWiki-General, 07Wikimedia-production-error: PHP Unknown error: EventLoggingLegacyConverter: Failed proxying legacy EventLogging event query string to WMF Event Platform ... - https://phabricator.wikimedia.org/T383939#10489919 [17:39:39] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06MediaWiki-Engineering, 10MediaWiki-General, 07Wikimedia-production-error: PHP Unknown error: EventLoggingLegacyConverter: Failed proxying legacy EventLogging event query string to WMF Event Platform ... - https://phabricator.wikimedia.org/T383939#10489920 [17:41:11] 06Data-Engineering, 10Dumps 2.0: Airflow job to do monthly XML dumps - https://phabricator.wikimedia.org/T384381#10489927 (10Ottomata) [18:25:20] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Product-Analytics, 10Event-Platform, 13Patch-For-Review: Enable Event Platform instruments to opt out of collecting User-Agent data - https://phabricator.wikimedia.org/T382173#10490062 (10Ottomata) [18:25:29] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Product-Analytics, 10Event-Platform, 13Patch-For-Review: Enable Event Platform instruments to opt out of collecting User-Agent data - https://phabricator.wikimedia.org/T382173#10490063 (10Ottomata) p:05Triage→03High a:03Ottomata [19:27:45] 06Data-Engineering, 10Commons-Impact-Metrics-Requests: Request for Images_from_Wiki_Loves_Africa_2024 - https://phabricator.wikimedia.org/T381352#10490343 (10GFontenelle_WMF) Dear @Anthere, I'm actually glad you made this comment! It is an opportunity for me to share a few things. I'm not sure if you are... [19:30:37] !log [data lake temp accounts] re-ran DAG mediawiki_history_reduced for 2024-12 [19:30:39] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:12:37] !log [data lake temp accounts] re-ran DAG edit_hourly for 2024-12 [20:12:39] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:32:17] 06Data-Engineering, 06Product-Analytics, 06Trust and Safety Product Team: Add mediawiki_product_metrics_incident_reporting_system_interaction to the sanitization allowlist - https://phabricator.wikimedia.org/T384650 (10cchen) 03NEW [20:32:55] 06Data-Engineering, 06Trust and Safety Product Team, 10Product-Analytics (Kanban): Add mediawiki_product_metrics_incident_reporting_system_interaction to the sanitization allowlist - https://phabricator.wikimedia.org/T384650#10490508 (10cchen) [20:33:03] 06Data-Engineering, 06Trust and Safety Product Team, 10Product-Analytics (Kanban): Add mediawiki_product_metrics_incident_reporting_system_interaction to the sanitization allowlist - https://phabricator.wikimedia.org/T384650#10490509 (10cchen) p:05Triage→03Medium [21:29:53] !log [data lake temp accounts] re-ran DAG druid_load_edit_hourly for 2024-12 [21:29:54] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:37:21] 10Data-Engineering (Q3 2024 January 1st - March 31th): Identify Internal Users of MediaWiki Wikitext Tables - https://phabricator.wikimedia.org/T383743#10490642 (10Ahoelzl) a:03Snwachukwu [21:37:50] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Data-Platform-SRE, 07Epic: HDFS capacity needs FY24/25 - https://phabricator.wikimedia.org/T384098#10490643 (10Ahoelzl) p:05Triage→03High [21:58:53] 06Data-Engineering: [SPIKE] Learn and document how to use Flink-CDC from MediaWiki MariaDB locally - https://phabricator.wikimedia.org/T373144#10490687 (10Ottomata) Great! Thanks for the update! [22:05:23] (03CR) 10Ottomata: [C:03+1] Refine transforms minimalist events support (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1112392 (https://phabricator.wikimedia.org/T383914) (owner: 10Aqu)