[07:22:34] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Script to maintain airflow-dags/config/datasets.yaml - https://phabricator.wikimedia.org/T435216#12229734 (10AndrewTavis_WMDE) [07:24:48] 14Analytics-Radar, 06Data-Engineering, 10Kafka-Infrastructure, 06serviceops-radar, and 4 others: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12229735 (10RKemper) [07:26:00] 07Analytics-Data-Problem, 06Data-Engineering: Pageview data missing since 2026-08-14: AQS returns 404 and hourly pageviews dumps stopped mid-2026-08-15 - https://phabricator.wikimedia.org/T435038#12229741 (10mahmoud) FYI, it's started to come back but 2026-08-14 is still missing atm. Thanks! [08:35:27] (03PS2) 10KCVelaga: Update user_agents_info with Cloudflare Radar [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1325932 (https://phabricator.wikimedia.org/T430673) [09:07:01] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Relative Trending - Fix records always trending. - https://phabricator.wikimedia.org/T435291 (10JMonton-WMF) 03NEW [09:26:14] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Relative Trending - Fix records always trending. - https://phabricator.wikimedia.org/T435291#12230028 (10JMonton-WMF) [09:27:48] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Review remaining 90-day windows in MWHistoryDeltaWriter - https://phabricator.wikimedia.org/T428961#12230031 (10APizzata-WMF) 05Open→03Resolved [09:28:02] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Daily sqoops for logging and cu_log tables - https://phabricator.wikimedia.org/T432208#12230033 (10APizzata-WMF) 05Open→03Resolved [10:40:14] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop two columns from transcode table - https://phabricator.wikimedia.org/T435064#12230379 (10Marostegui) [11:44:42] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop two columns from transcode table - https://phabricator.wikimedia.org/T435064#12230705 (10Marostegui) [12:45:43] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop two columns from transcode table - https://phabricator.wikimedia.org/T435064#12230985 (10Marostegui) [13:03:56] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop two columns from transcode table - https://phabricator.wikimedia.org/T435064#12231153 (10Marostegui) [13:03:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 05MW-1.47-notes (1.47.0-wmf.15; 2026-08-11), 13Patch-For-Review: mediawiki.page_change.v1 events - add wikidata id for pages - https://phabricator.wikimedia.org/T428176#12231151 (10Ottomata)... [13:31:26] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop two columns from transcode table - https://phabricator.wikimedia.org/T435064#12231248 (10Marostegui) [13:45:08] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review, 10Event-Platform, 13Patch-For-Review: [Data Persistence Design Review] Relative trending pages - https://phabricator.wikimedia.org/T432352#12231296 (10Ottomata) Here's a quick estimate of write volume, with the int... [13:56:30] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 05MW-1.47-notes (1.47.0-wmf.15; 2026-08-11), 13Patch-For-Review: mediawiki.page_change.v1 events - add wikidata id for pages - https://phabricator.wikimedia.org/T428176#12231404 (10tchin) I t... [14:16:15] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12231499 (10tchin) > only captures tags for revisions I see that tags can have an associated `rc_id`, `rev_id`, and/or `log_id`, and currently w... [14:22:06] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Wikimedia Enterprise, 10Wikimedia Enterprise - Content Integrity, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), and 2 others: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#12231530 (10... [14:36:13] (03CR) 10Mforns: [C:03+1] Update user_agents_info with Cloudflare Radar (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1325932 (https://phabricator.wikimedia.org/T430673) (owner: 10KCVelaga) [14:37:29] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): data lake: git commit history - https://phabricator.wikimedia.org/T433696#12231641 (10Ahoelzl) a:05Ahoelzl→03Antoine_Quhen [14:38:27] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Analyze and correctly classify iMessage User-Agent - https://phabricator.wikimedia.org/T435110#12231643 (10mforns) I will take this, although I will prioritize KAPOW work. [14:42:36] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12231659 (10Ottomata) > Should the generic stream attempt to find/attach the recentchange and log entry to the event if the id exists? Hm, I'm n... [14:42:45] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Analyze and correctly classify iMessage User-Agent - https://phabricator.wikimedia.org/T435110#12231662 (10GGoncalves-WMF) Right, we can do this one in the context of the parent ticket, which is still part of the KAPOW roadmap, but comes after the initia... [14:51:39] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Dumps-Generation: mediawiki_content_current exports silently omit all pages edited between the snapshot cutoff and dump generation time (enwiki 2026-07-01: ~3,560 pages missing per partition file) - https://phabricator.wikimedia.org/T431872#12231703 (... [14:52:03] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Dumps-Generation: MediaWiki Content File Exports are incomplete - https://phabricator.wikimedia.org/T431723#12231706 (10APizzata-WMF) a:05APizzata-WMF→03xcollazo [15:27:23] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Java-Scala-Standardization, 07Essential-Work, 13Patch-For-Review: Publish all wmf-sparksqlclidriver-*.jar versions to GitLab Package Registry - https://phabricator.wikimedia.org/T435126#12232213 (10xcollazo) All current Spark versions now have re... [15:35:04] 06Data-Engineering, 06Product-Analytics, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Add cloudflare_radar API connection to Airflow analytics main instance - https://phabricator.wikimedia.org/T433982#12232287 (10KCVelaga_WMF) Confirming that the connection is working and the [[ https://airflow.wikimed... [15:40:40] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Investigate oversized driver logs for `mediawiki_history_denormalize` - https://phabricator.wikimedia.org/T424358#12232364 (10APizzata-WMF) 05Open→03Resolved [15:53:07] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 07Epic, 13Patch-For-Review: Incremental MediaWiki History v1 - Phase I - https://phabricator.wikimedia.org/T424350#12232444 (10Ottomata) [15:56:34] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): hive metastore was down - https://phabricator.wikimedia.org/T432507#12232456 (10BTullis) 05Open→03Resolved I'm closing this issue for now. We can resurrect it if we believe that there is a specific problem, b... [16:14:09] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Platform, 10EditCheck, 10VisualEditor, and 3 others: Implement sampling requirements for Headless VE technical pilot - https://phabricator.wikimedia.org/T433024#12232578 (10DLynch) The sampling was implemented as https://gerrit.wikimedia.org... [16:34:18] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Make scheduling and configuring dbt jobs easy - https://phabricator.wikimedia.org/T435369 (10Milimetric) 03NEW [16:41:26] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt: delay job by arbitrary amounts of time - https://phabricator.wikimedia.org/T433976#12232774 (10GGoncalves-WMF) [16:41:27] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt isolated model orchestration - https://phabricator.wikimedia.org/T430625#12232775 (10GGoncalves-WMF) [16:42:30] 06Data-Engineering, 06Product-Analytics (Kanban): Switch to user_agents_info and deprecate user_agent_compliance_classified_hourly - https://phabricator.wikimedia.org/T433992#12232781 (10Ahoelzl) @KCVelaga_WMF assuming you will be working on the migration, correct? cc @mforns [16:42:52] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Product-Analytics (Kanban): Switch to user_agents_info and deprecate user_agent_compliance_classified_hourly - https://phabricator.wikimedia.org/T433992#12232782 (10Ahoelzl) [16:44:33] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: mediawiki.page_change.v1 event - Add user first_registration_dt field - https://phabricator.wikimedia.org/T426998#12232825 (10Ottomata) [16:45:19] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Platform, 10EditCheck, 10VisualEditor, and 3 others: Implement sampling requirements for Headless VE technical pilot - https://phabricator.wikimedia.org/T433024#12232841 (10ppelberg) [16:45:26] 06Data-Engineering, 06Data-Engineering-Radar, 10Test Kitchen, 07Essential-Work, 10Event-Platform: [EventGate] Perform GeoIP lookup at event ingestion time - https://phabricator.wikimedia.org/T434312#12232849 (10Ottomata) [16:49:38] 07Analytics-Data-Problem, 06Data-Engineering: Pageview data missing since 2026-08-14: AQS returns 404 and hourly pageviews dumps stopped mid-2026-08-15 - https://phabricator.wikimedia.org/T435038#12232883 (10Ottomata) 05Open→03Resolved a:03Ottomata As far as I can tell, its all back now! ` for d in... [16:51:22] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser-SuggestedInvestigations, 06MediaWiki-API-Platform-Team, and 4 others: MediaWiki\JobQueue\Exceptions\JobQueueError: Could not enqueue jobs - https://phabricator.wikimedia.org/T434199#12232936 (10Ottomata) Moving to DE Radar. #mediawiki-api-platform-t... [16:51:28] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser-SuggestedInvestigations, 06MediaWiki-API-Platform-Team, and 4 others: MediaWiki\JobQueue\Exceptions\JobQueueError: Could not enqueue jobs - https://phabricator.wikimedia.org/T434199#12232939 (10Ottomata) [16:51:47] 06Data-Engineering, 06Product-Analytics, 06Product Safety and Integrity (Sprint Sunflower (Aug 11 - Aug 28)): Run backfills for bad faith edit reverts metrics - https://phabricator.wikimedia.org/T435372#12232948 (10Tchanders) [16:52:53] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Platform, 06Growth-Team, 05DE1.3.2 Impact + Dashboard, and 2 others: Make the AQS endpoints more usable for the ImpactModule extension - https://phabricator.wikimedia.org/T433983#12232951 (10Ahoelzl) a:03Snwachukwu [16:53:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Collaboration-Services, 06Reader Growth Team: Structured Data packages size on GitLab - https://phabricator.wikimedia.org/T435199#12232954 (10Ahoelzl) a:03APizzata-WMF [16:55:12] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop two columns from transcode table - https://phabricator.wikimedia.org/T435064#12232966 (10Ottomata) I am pretty sure we don't sqoop transcode stuff. Moving to DE Radar [16:55:21] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop two columns from transcode table - https://phabricator.wikimedia.org/T435064#12232968 (10Ottomata) [16:57:07] 06Data-Engineering, 06Traffic: Turnilo: X-Is-Browser filter doesn't work (always HTTP 500) - https://phabricator.wikimedia.org/T435152#12232984 (10Ottomata) a:03JAllemandou [16:57:17] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Traffic: Turnilo: X-Is-Browser filter doesn't work (always HTTP 500) - https://phabricator.wikimedia.org/T435152#12232986 (10Ottomata) [16:58:04] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Product-Analytics, 06Product Safety and Integrity (Sprint Sunflower (Aug 11 - Aug 28)): Run backfills for bad faith edit reverts metrics - https://phabricator.wikimedia.org/T435372#12232996 (10Ottomata) a:03amastilovic [16:58:11] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Product-Analytics, 06Product Safety and Integrity (Sprint Sunflower (Aug 11 - Aug 28)): Run backfills for bad faith edit reverts metrics - https://phabricator.wikimedia.org/T435372#12232998 (10Ottomata) [17:18:21] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Product-Analytics (Kanban): Switch to user_agents_info and deprecate user_agent_compliance_classified_hourly - https://phabricator.wikimedia.org/T433992#12233148 (10KCVelaga_WMF) Yes, that's right, I will work on this. Might need some code reviews fr... [17:22:23] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Script to maintain airflow-dags/config/datasets.yaml - https://phabricator.wikimedia.org/T435216#12233169 (10Ahoelzl) @AndrewTavis_WMDE would this be something you can work on? We are happy to review and deploy. [18:11:41] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 07Epic: Incremental MediaWiki History - v2 - https://phabricator.wikimedia.org/T432025#12233435 (10Ottomata) @ottomata's hopes for v2: - data model looks like our [[ https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary/-/tree/mast... [18:12:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:37:20] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Wikimedia Enterprise, 10Wikimedia Enterprise - Content Integrity, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), and 2 others: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#12233554 (10... [18:39:00] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review, 10Event-Platform, 13Patch-For-Review: [Data Persistence Design Review] Relative trending pages - https://phabricator.wikimedia.org/T432352#12233565 (10Ottomata) FWIW, @JMonton-WMF is working on a solution for the <... [18:42:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:54:40] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Script to maintain airflow-dags/config/datasets.yaml - https://phabricator.wikimedia.org/T435216#12233647 (10Ottomata) Q: will sorting of datasets.yaml help with scanning and understandability? I'd prefer to have a manually maintained file grouped... [19:12:43] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Delete `wmf_raw.mediawiki_content` table after a few months - https://phabricator.wikimedia.org/T427441#12233730 (10xcollazo) Confirming that `wmf_raw.mediawiki_content` is not being written to: ` spark-sql (default)> show partitions wmf_raw.mediawiki_co... [19:22:24] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Delete `wmf_raw.mediawiki_content` table after a few months - https://phabricator.wikimedia.org/T427441#12233806 (10xcollazo) ` $ hostname -f an-launcher1003.eqiad.wmnet $ sudo -u analytics bash $ hive hive (default)> describe formatted wmf_raw.mediawik... [19:22:46] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Delete `wmf_raw.mediawiki_content` table after a few months - https://phabricator.wikimedia.org/T427441#12233807 (10xcollazo) a:05APizzata-WMF→03xcollazo [19:23:27] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Delete `wmf_raw.mediawiki_content` table after a few months - https://phabricator.wikimedia.org/T427441#12233812 (10xcollazo) 05Open→03Resolved [19:54:31] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Script to maintain airflow-dags/config/datasets.yaml - https://phabricator.wikimedia.org/T435216#12233912 (10AndrewTavis_WMDE) Plan from @amastilovic and I was that I'd work on this, @Ahoelzl :) This is why I assigned myself so it's clear that it's... [20:04:33] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Script to maintain airflow-dags/config/datasets.yaml - https://phabricator.wikimedia.org/T435216#12233938 (10Ottomata) Thanks! I don't know the best approach! Sorting by instance might make more sense then simply sorting alphabetically. Okay! I'l... [20:08:57] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12233943 (10tchin) The `ChangeTagsAfterUpdateTags` hook already has the log_id. Actually the method that runs it (`ChangeTagsStore::updateTags`)... [20:13:05] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 06MediaWiki-API-Platform-Team: MediaWiki DomainEvents - Include LogEntry - https://phabricator.wikimedia.org/T427815#12233953 (10Ottomata) a:03ArielGlenn [20:13:45] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12233956 (10Ottomata) Nice! @tchin if you want to explore making a patch to create DomainEvent(s) for ChangeTags, I don't think anyone would ob... [20:17:15] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review, 10Event-Platform, 13Patch-For-Review: [Data Persistence Design Review] Relative trending pages - https://phabricator.wikimedia.org/T432352#12233968 (10Ottomata) FYI slack thread discussing this [[ https://wikimedia... [20:39:20] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Product-Analytics, 06Product Safety and Integrity (Sprint Sunflower (Aug 11 - Aug 28)): Run backfills for bad faith edit reverts metrics - https://phabricator.wikimedia.org/T435372#12234085 (10amastilovic) Tables `wmf_psi.same_day_handling_bad_fait... [20:52:32] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review, 10Event-Platform, 13Patch-For-Review: [Data Persistence Design Review] Relative trending pages - https://phabricator.wikimedia.org/T432352#12234152 (10Ottomata) cc @Ladsgroup [21:14:16] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): data lake: git commit history - https://phabricator.wikimedia.org/T433696#12234218 (10Addshore) @Antoine_Quhen If you give me some pointers for schema location and or script location / structure I can try and throw up some sort of commits for review [21:33:22] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt: per-group DAG generation - https://phabricator.wikimedia.org/T429439#12234323 (10amastilovic) [21:42:19] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt: a dedicated dbt Airflow instance - https://phabricator.wikimedia.org/T435412 (10amastilovic) 03NEW [21:59:20] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 07Epic: data lake: mailing list emails - https://phabricator.wikimedia.org/T435416 (10Addshore) 03NEW [22:00:08] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12234486 (10tchin) a:03tchin [22:11:29] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 07Epic: data lake: server admin log (sal) - https://phabricator.wikimedia.org/T435417 (10Addshore) 03NEW [22:26:46] 06Data-Engineering, 10Data-Engineering-Wikistats, 07Essential-Work: Update state management - https://phabricator.wikimedia.org/T384050#12234623 (10Volker_E) Relatively related {T386392} – maybe also to use as blueprint