[06:29:40] (03PS1) 10KCVelaga: Update partitioning of api_requests_refined to use ts column. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1344135 (https://phabricator.wikimedia.org/T438655) [07:56:49] (03CR) 10Joal: [C:03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1344135 (https://phabricator.wikimedia.org/T438655) (owner: 10KCVelaga) [09:49:02] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Wikimedia Enterprise: WME Pageviews DAG for HDFS to S3 Transfer - https://phabricator.wikimedia.org/T426017#12353986 (10SGupta-WMF) Yes , this has been implemented. [10:13:15] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06RoadToWiki, 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Mirror datasets used on the key metrics dashboard to Postgres - https://phabricator.wikimedia.org/T438586#12354051 (10Antoine_Quhen) The list of missing schemas and tables could be extra... [11:49:49] 06Data-Engineering, 06Product-Analytics: Dbt-jobs backfill: structured_edits_weekly - https://phabricator.wikimedia.org/T438196#12354568 (10MNeisler) a:03amastilovic [12:53:56] 06Data-Engineering: Revisit thresholds for the Human-Bot Page View Count Anomaly alert - https://phabricator.wikimedia.org/T438957 (10Antoine_Quhen) 03NEW [13:34:07] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Stop publishing _SUCCESS files for v2 dumps - https://phabricator.wikimedia.org/T438748#12354963 (10xcollazo) Confirmed working on a test run. [13:34:36] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): refinery-spark: Distribute HdfsFileFingerprintWriter instead of looping in the driver - https://phabricator.wikimedia.org/T438244#12354971 (10xcollazo) Confirmed working on a test run. [13:37:53] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Implement 7z compression for v2 history dumps - https://phabricator.wikimedia.org/T437454#12354983 (10xcollazo) Deployed to production, regenerated `DagProperties` to pickup new settings. Will keep this open until I see proper progress on the Oct 1 run. [13:41:45] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: mediawiki.page_change.v1 events - add wikidata id for pages - https://phabricator.wikimedia.org/T428176#12354989 (10Ottomata) It works! `lang=json "changelog_kind": "upda... [13:43:01] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Incremental MWH - MediaWiki event data source improvements - https://phabricator.wikimedia.org/T423935#12355006 (10Ottomata) [13:48:46] 06Data-Engineering, 06Data-Engineering-Radar, 06Content-Platform-Team, 06MediaWiki-API-Platform-Team, 10Event-Platform: Expose MediaWiki Parser render_id as a response header in relevant MW REST API endpoints - https://phabricator.wikimedia.org/T418792#12355054 (10Ottomata) BTW, thank you! https://g... [14:26:25] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Define and document the earliest start_date for dbt DAGs - https://phabricator.wikimedia.org/T438276#12355233 (10GGoncalves-WMF) There's two questions we're trying to answer here: 1. Should there be a minimum `start_date` for the automated initial backf... [14:37:44] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Commons, 06Data-Persistence, 07OKR-Work: Data Lake - sqoop of commons link tables needs to adapt to move to dedicated MariaDB cluster - https://phabricator.wikimedia.org/T431476#12355280 (10Ottomata) Thank you Dan! [14:59:45] 06Data-Engineering, 10AQS2.0, 06Growth-Team, 07OKR-Work (DE1 FY2026-27): AQS - Add per wiki filtering to edits/v3/per_editor - https://phabricator.wikimedia.org/T434136#12355432 (10Snwachukwu) a:03Snwachukwu [15:38:01] 06Data-Engineering, 10Event-Platform, 07good first task: EventBus - move event producers to a different extension - https://phabricator.wikimedia.org/T432050#12355737 (10Ottomata) Hi @Anushka10patel ! Both T430967 and T428176 are merged and deployed. We can proceed with moving the PageChangeEvent and UserC... [16:28:30] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): data lake: server admin log (sal) - https://phabricator.wikimedia.org/T435417#12356078 (10Ahoelzl) [16:31:39] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Sqoop globalimagelinks and filerevision - https://phabricator.wikimedia.org/T427532#12356106 (10dr0ptp4kt) >>! In T427532#12350958, @mforns wrote: > The MR has been merged. > The two datasets should be query ready every month moving... [16:33:13] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: data lake: git commit history - https://phabricator.wikimedia.org/T433696#12356119 (10Ottomata) > For a first version I'd suggest writing directly to an Iceberg table, using the same architecture for backfill and incremental. Sound... [16:38:21] (03PS2) 10Mforns: Switch Clickstream to use webrequest_v2 instead of pageview_actor. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1342767 (https://phabricator.wikimedia.org/T425668) [16:39:31] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: data lake: git commit history - https://phabricator.wikimedia.org/T433696#12356198 (10Ottomata) Given the complexity of the schema and the expertise of my friends here involved, I will let you all work out the event schema change MR... [16:40:39] (03PS3) 10Mforns: Switch Clickstream to use webrequest_v2 instead of pageview_actor. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1342767 (https://phabricator.wikimedia.org/T425668) [16:53:04] (03PS4) 10Mforns: Switch Clickstream to use webrequest_v2 instead of pageview_actor. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1342767 (https://phabricator.wikimedia.org/T425668) [16:53:12] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Movement-Insights: WE3.3 Account Creation metrics dashboard showing lower numbers than expected - https://phabricator.wikimedia.org/T438385#12356389 (10jwang) Thanks @amastilovic! For this case, do we need to manually delete rows in the table? can I... [17:03:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Connection-Team, 06Growth-Team, 10Thanks: Define the requirements for Thanks instrumentation - https://phabricator.wikimedia.org/T438493#12356428 (10Ahoelzl) p:05Triage→03High [17:04:19] 06Data-Engineering, 06Data-Engineering-Radar, 06Movement-Insights: Replace dbt model JOINs on `event_user_text` with `event_user_central_id` (movement-insights-contributors-metrics) - https://phabricator.wikimedia.org/T438573#12356436 (10Ottomata) [17:05:22] 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review, 06Product-Analytics (Kanban): Update partitioning of api_requests_refined to use ts col - https://phabricator.wikimedia.org/T438655#12356441 (10Ottomata) [17:06:03] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt-jobs backfill: new reverted-edit models - https://phabricator.wikimedia.org/T438793#12356454 (10Ahoelzl) [17:07:39] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Product-Analytics: Dbt-jobs backfill: structured_edits_weekly - https://phabricator.wikimedia.org/T438196#12356481 (10Ahoelzl) [17:10:14] 06Data-Engineering, 06Data-Platform-SRE: Matomo: Enable “AIAgents” core plugin - https://phabricator.wikimedia.org/T438862#12356491 (10Ahoelzl) @KReid-WMF FYI re Matomo deprecation [17:11:41] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE: Matomo: Enable “AIAgents” core plugin - https://phabricator.wikimedia.org/T438862#12356500 (10Ahoelzl) [17:12:47] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10AQS2.0, 06Growth-Team, 07OKR-Work (DE1 FY2026-27): AQS - Add per wiki filtering to edits/v3/per_editor - https://phabricator.wikimedia.org/T434136#12356508 (10Ahoelzl) p:05Triage→03Medium [17:12:57] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10AQS2.0, 06Growth-Team, 07OKR-Work (DE1 FY2026-27): AQS - Add per wiki filtering to edits/v3/per_editor - https://phabricator.wikimedia.org/T434136#12356510 (10Ahoelzl) Happening as part of DE1.3.10 [17:16:32] 06Data-Engineering, 10Test Kitchen: event_sanitized.serversideaccountcreation reports users that actually don't exist - https://phabricator.wikimedia.org/T429288#12356525 (10Ottomata) CC #test_kitchen for syncs on instrumentation about user account creation vs events & MWH. [17:17:24] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Create mediawiki.user_change event stream - https://phabricator.wikimedia.org/T423952#12356528 (10Ottomata) After user_change is v1 (still lots to verify/design), we could c... [17:20:22] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt-jobs backfill: new reverted-edit models - https://phabricator.wikimedia.org/T438793#12356546 (10CMyrick-WMF) @amastilovic After [[ https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/191 | MR !191 ]] is merged, could you rer... [17:28:34] PROBLEM - Check if active EventStreams endpoint is delivering messages. on alert1002 is CRITICAL: CRITICAL: No EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [17:30:28] 07Analytics-Data-Problem, 06Data-Engineering, 06Wikimedia Enterprise: Netherlands (NL) absent from country_project_page flat files since 2023-11-09 - https://phabricator.wikimedia.org/T426559#12356584 (10Ahoelzl) Pinging @HShaikh on wether this is WME maintained. [17:30:45] 06Data-Engineering, 10Datasets-General-or-Unknown, 06Wikimedia Enterprise: Missing DP files 2022-10-19.tsv, 2020-07-19.tsv, 2020-07-20.tsv - https://phabricator.wikimedia.org/T381057#12356587 (10Ahoelzl) Pinging @HShaikh on wether this is WME maintained. [17:30:51] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:32:13] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-main in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [17:34:54] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Define and document the earliest start_date for dbt DAGs - https://phabricator.wikimedia.org/T438276#12356601 (10nshahquinn-wmf) >>! In T438276#12355233, @GGoncalves-WMF wrote: > The motivation for **1** is indeed to strike a balance between convenience... [17:35:48] FIRING: [2x] EventgateLatency: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [17:39:05] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:42:42] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Wikimedia Enterprise: PageViews S3 Data Transfer MR [Enterprise] - https://phabricator.wikimedia.org/T425543#12356609 (10Ahoelzl) a:03Antoine_Quhen [17:58:24] RECOVERY - Check if active EventStreams endpoint is delivering messages. on alert1002 is OK: OK: An EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [18:00:13] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10AQS2.0, 06Growth-Team, 07OKR-Work (DE1 FY2026-27): AQS - Add per wiki filtering to edits/v3/per_editor - https://phabricator.wikimedia.org/T434136#12356652 (10Snwachukwu) > edits/v3/per_editor is backed by the mediawiki_history_reduced druid data... [18:09:34] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Optimize performance of dbt_daily upload_bandwidth_hourly task - https://phabricator.wikimedia.org/T439016#12356723 (10amastilovic) a:03mforns [18:13:24] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:17:41] RESOLVED: [2x] EventgateLatency: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [18:19:55] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:20:56] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06RoadToWiki, 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Mirror datasets used on the key metrics dashboard to Postgres - https://phabricator.wikimedia.org/T438586#12356760 (10amastilovic) OK I've created the Postgres schemas and backfilled all... [18:28:10] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:30:37] FIRING: [2x] EventgateLatency: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [18:41:10] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10AQS2.0, 06Growth-Team, 07OKR-Work (DE1 FY2026-27): AQS - Add per wiki filtering to edits/v3/per_editor - https://phabricator.wikimedia.org/T434136#12356849 (10Ottomata) Oh...right... T406069#11406936 > Instead of doing Option 2. new mediawiki_hi... [18:56:33] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Observability-Metrics: [Infra] Mechanism to get Apache Spark operational metrics - https://phabricator.wikimedia.org/T297231#12356890 (10xcollazo) @BTullis had mentioned the Spark History Service uptread in T297231#9392844. I want to revive that ide... [19:03:04] RESOLVED: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:04:07] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Observability-Metrics: [Infra] Mechanism to get Apache Spark operational metrics - https://phabricator.wikimedia.org/T297231#12356912 (10Ottomata) Joseph mentioned this to me today as well. Sounds good! [19:08:18] RESOLVED: [2x] EventgateLatency: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [21:22:25] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Platform, 10EditCheck, 10VisualEditor, and 3 others: Implement sampling requirements for Headless VE technical pilot - https://phabricator.wikimedia.org/T433024#12357280 (10jhathaway) Sampling schedule looks good to me. [22:13:02] 06Data-Engineering, 10Test Kitchen: event_sanitized.serversideaccountcreation reports users that actually don't exist - https://phabricator.wikimedia.org/T429288#12357417 (10mpopov) >>! In T429288#12034365, @Ottomata wrote: > Got it. > > Welp, I'm not really sure who to report this to. Is there an owner of t... [22:58:36] 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list for September 2026 - https://phabricator.wikimedia.org/T439050 (10GFontenelle_WMF) 03NEW