[07:02:53] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backup pageview data before backfilling, announce backfilling progress - https://phabricator.wikimedia.org/T404151#11166099 (10JAllemandou) [07:25:03] (03CR) 10Joal: "One last nit an it can be tested. As I wrote in the previous comment, I don't know if our sqoop utility knows how to connect to centralaut" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1184925 (https://phabricator.wikimedia.org/T389666) (owner: 10Ottomata) [07:53:26] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11166187 (10FCeratto-WMF) [08:21:45] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11166253 (10FCeratto-WMF) p:05Triage→03High [08:43:02] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11166278 (10FCeratto-WMF) [10:07:00] 06Data-Engineering, 10Product-Analytics (Kanban): Follow-up analysis to understand usage of dumps to inform v2 rollout - https://phabricator.wikimedia.org/T402963#11166571 (10GGoncalves-WMF) That is a good question, I was just quoting @BTullis 's wisdom in the original ticket (T383175#10440220). Ben, can you... [11:20:41] 06Data-Engineering, 10Product-Analytics (Kanban): Follow-up analysis to understand usage of dumps to inform v2 rollout - https://phabricator.wikimedia.org/T402963#11166855 (10BTullis) >>! In T402963#11166570, @GGoncalves-WMF wrote: > That is a good question, I was just quoting @BTullis 's wisdom in the origina... [11:48:17] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop cl_to and cl_collation from categorylinks in wmf production - https://phabricator.wikimedia.org/T402925#11166939 (10Ladsgroup) [11:55:10] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Improve handling of new stream onboarding in Refine - https://phabricator.wikimedia.org/T402186#11166951 (10Antoine_Quhen) 05In progress→03Resolved The fix is in production. Notice the line `Retrieved 201 stream(s) from canary_events at 2025-09-1... [13:15:30] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11167273 (10FCeratto-WMF) [13:18:08] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Airflow jobs to do monthly XML dumps - https://phabricator.wikimedia.org/T384381#11167281 (10BTullis) OK, got it. Thanks @xcollazo for the explanation. [14:54:05] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): Audit Moderator Tools applications for use of the "m." sub-domain - https://phabricator.wikimedia.org/T404207 (10Kgraessle) 03NEW [15:02:24] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 07Essential-Work: OpsWeek: Contact and/or remove dead dump mirrors - https://phabricator.wikimedia.org/T402987#11167975 (10xcollazo) As of now: C3SL -> Still working on getting it back.. We opened an ipv6 ticket for them: {T404202}. PDApps -> Email ne... [16:05:55] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11168272 (10Ottomata) Let us know how/when we can help, we are grooming tasks and are going to put this in on our radar. [16:06:55] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it - https://phabricator.wikimedia.org/T388793#11168276 (10Antoine_Quhen) I asked questions to the Iceberg community: * Mailing list: https://lists.apache.o... [16:10:22] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Mitigate consequences of Gobblin hiccups generating late events and alerts - https://phabricator.wikimedia.org/T402324#11168299 (10Ahoelzl) [16:10:29] 06Data-Engineering, 06Experimentation Lab, 10MediaWiki-extensions-WikimediaEvents: Prepare mediawiki-client-error Logstash dashboards for mobile subdomain sunsetting - https://phabricator.wikimedia.org/T400852#11168301 (10Ottomata) Linking [[ https://wikimedia.slack.com/archives/CSV483812/p1757439160310229 |... [16:11:24] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list August 2025 - https://phabricator.wikimedia.org/T402874#11168305 (10Ahoelzl) [16:13:45] 06Data-Engineering, 10BetaFeatures, 06cloud-services-team, 06Data-Platform-SRE, 10Data-Services: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11168322 (10Ottomata) [16:16:44] 06Data-Engineering, 10Data Pipelines, 06Data-Platform-SRE, 13Patch-For-Review: AirFlow DAGs: unable to publish deb via CI - https://phabricator.wikimedia.org/T403016#11168368 (10Ottomata) [16:17:33] 06Data-Engineering, 10Data Pipelines, 06Data-Platform-SRE, 13Patch-For-Review: AirFlow DAGs: unable to publish deb via CI - https://phabricator.wikimedia.org/T403016#11168371 (10Ottomata) @amastilovic @brouberol I think this might be related to your poetry work. Can you comment and decline if so? [16:18:59] 06Data-Engineering, 10MediaWiki-Core-Revision-backend, 10MediaWiki-DomainEvents, 10Event-Platform, and 2 others: MediaWiki\Revision\RevisionAccessException: Unable to load fresh row for rev_id: {rev_id} - https://phabricator.wikimedia.org/T400380#11168378 (10Ottomata) p:05Triage→03Medium [16:19:15] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Core-Revision-backend, 10MediaWiki-DomainEvents, 10Event-Platform, and 2 others: MediaWiki\Revision\RevisionAccessException: Unable to load fresh row for rev_id: {rev_id} - https://phabricator.wikimedia.org/T400380#11168382 (10Ottomata) [16:19:21] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168384 (10Ahoelzl) Related: https://phabricator.wikimedia.org/T402963 [16:24:39] 06Data-Engineering, 06SRE, 06Traffic: Add pageview information to turnilo's webrequest_sampled_live (is_pageview is always "-") - https://phabricator.wikimedia.org/T402612#11168439 (10Ottomata) @JAllemandou can you weigh in here? @CDanis def possible to reuse the logic in Java, but it would probably be a la... [16:28:26] 06Data-Engineering, 06Traffic, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168454 (10Ottomata) There is probably a reason why not, but could we just put dumps.wm.org behind the CDN? Then requests to it w... [16:30:39] 06Data-Engineering, 10Multi-Content-Revisions, 07Schema-change: Replace page.page_content_model with id reference to content_models table - https://phabricator.wikimedia.org/T403211#11168489 (10Ottomata) If/when this is done, Data Engineering team will need to make some changes to our sqoop logic. [16:31:09] 06Data-Engineering, 06Data-Engineering-Radar, 10Multi-Content-Revisions, 07Schema-change: Replace page.page_content_model with id reference to content_models table - https://phabricator.wikimedia.org/T403211#11168506 (10Ottomata) [16:32:08] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Change row format of cx_corpora - https://phabricator.wikimedia.org/T403362#11168529 (10Ottomata) [16:34:23] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06DBA, 07Schema-change-in-production: Add default value for afl_ip and remove default value for afl_ip_hex in abuse_filter_log table - https://phabricator.wikimedia.org/T401906#11168541 (10FCeratto-WMF) [16:39:47] 06Data-Engineering, 06Traffic, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168570 (10taavi) >>! In T403159#11168454, @Ottomata wrote: > There is probably a reason why not, but could we just put dumps.wm.o... [16:53:39] 06Data-Engineering, 06Traffic, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168700 (10Ottomata) TIL you can point rsync at dumps.wikmedia.org (if we allow you to). https://phabricator.wikimedia.org/T30655... [17:04:02] 06Data-Engineering, 06Traffic, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168760 (10ssingh) This has been brought up recently in other discussions as well and Traffic is happy to triage it if required bu... [17:09:25] 06Data-Engineering, 06Traffic, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168807 (10BTullis) How about if we did something like this? ` (1) nginx -> (2) rsyslog -> (3) kafka-logging (5) kaf... [17:13:28] 06Data-Engineering, 06Data-Platform-SRE, 06Traffic, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168825 (10BTullis) [17:19:08] 06Data-Engineering, 06Data-Platform-SRE, 06Traffic, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168854 (10BTullis) Alternatively, we could just use an `hdfs-rsync` (or similar) process on the clouddumps... [18:00:29] 06Data-Engineering, 10Data Pipelines, 06Data-Platform-SRE, 13Patch-For-Review: AirFlow DAGs: unable to publish deb via CI - https://phabricator.wikimedia.org/T403016#11169048 (10brouberol) 05Open→03Declined There's no need to publish an airlfow deb file anymore, now that: - the airflow environment... [18:00:52] 06Data-Engineering, 10Data Pipelines, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 13Patch-For-Review: AirFlow DAGs: unable to publish deb via CI - https://phabricator.wikimedia.org/T403016#11169053 (10brouberol) 05Declined→03Resolved [18:00:54] 06Data-Engineering, 10Data Pipelines, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 13Patch-For-Review: AirFlow DAGs: unable to publish deb via CI - https://phabricator.wikimedia.org/T403016#11169052 (10brouberol) [18:35:35] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): QA features on the new mobile URLs - https://phabricator.wikimedia.org/T403638#11169332 (10vaughnwalters) [19:16:21] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence: Data Persistence Design Review: Global Editor Metrics (YiR, Impact Module, App Activity Tab) - https://phabricator.wikimedia.org/T401260#11169555 (10Ottomata) [19:38:16] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backup pageview data before backfilling, announce backfilling progress - https://phabricator.wikimedia.org/T404151#11169635 (10JAllemandou) [19:38:38] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backup pageview data before backfilling, announce backfilling progress - https://phabricator.wikimedia.org/T404151#11169636 (10JAllemandou) [19:48:56] 06Data-Engineering, 06SRE, 06Traffic: Add pageview information to turnilo's webrequest_sampled_live (is_pageview is always "-") - https://phabricator.wikimedia.org/T402612#11169688 (10JAllemandou) Building a JAR containing the `is_pageview` logic with a few dependencies as possible is easy. My little researc... [20:26:11] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence: Data Persistence Design Review: Global Editor Metrics (YiR, Impact Module, App Activity Tab) - https://phabricator.wikimedia.org/T401260#11169809 (10Ottomata) Alright! first draft of design review document is here: https://wikitec... [20:28:22] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence: Data Persistence Design Review: Global Editor Metrics (YiR, Impact Module, App Activity Tab) - https://phabricator.wikimedia.org/T401260#11169819 (10Ottomata) [20:38:45] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: Global Editor Metrics for YiR, Apps Activity Tab, and Growth Impact Module - https://phabricator.wikimedia.org/T403660#11169873 (10Ottomata) Re comment T403660#11145572 ab... [21:00:37] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: Global Editor Metrics for YiR, Apps Activity Tab, and Growth Impact Module - https://phabricator.wikimedia.org/T403660#11169950 (10Ottomata) Dear product owners (@Seddon,... [21:53:47] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Serve mobile and desktop variants through the same URL (unified mobile routing) - https://phabricator.wikimedia.org/T214998#11170105 (10Jdlrobson-WMF) [22:41:40] 06Data-Engineering, 06Data-Engineering-Radar, 10XAnalytics, 10MobileFrontend (Core PHP), and 3 others: PHP Deprecated: urldecode(): Passing null to parameter #1 ($string) of type string is deprecated / PHP Warning: Undefined array key 1 - https://phabricator.wikimedia.org/T400522#11170305 (10Jdlrobson-WMF) [23:02:58] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list August 2025 - https://phabricator.wikimedia.org/T402874#11170381 (10GFontenelle_WMF) 05Open→03Resolved [23:28:32] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking), 10Wikipedia-iOS-App-Backlog (iOS Release FY2025-26): [QA Task] Verify iOS compatability with removal of m. subdomain on test wiki - https://phabricator.wikimedia.org/T404275 (10Seddon) 03NEW [23:28:58] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): [QA Task] Verify Android app compatability with removal of m. subdomain on test wiki - https://phabricator.wikimedia.org/T404276 (10Seddon) 03NEW