[04:20:59] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Pageviews-API: PageView tool not updated since 21 May 2026 - https://phabricator.wikimedia.org/T427171#11970492 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal I think this can be resolved? Please re-open if you disagree. [04:21:30] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Pageviews-API: PageView tool not updated since 21 May 2026 - https://phabricator.wikimedia.org/T427171#11970495 (10MusikAnimal) a:05MusikAnimal→03Ahoelzl [06:58:29] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Request for backfill of webrequest with is_api_request and ip_provenance columns - https://phabricator.wikimedia.org/T427474#11970611 (10JAllemandou) I think we're on the same page @KCVelaga_WMF :) I quickly scanned the PR you linked in the previous commen... [07:07:44] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Request for backfill of webrequest with is_api_request and ip_provenance columns - https://phabricator.wikimedia.org/T427474#11970628 (10KCVelaga_WMF) Sounds good. > I don't know what DE would consider best practice in terms of repositories management: shou... [09:22:23] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Support for Java 21 and Flink 2 - https://phabricator.wikimedia.org/T412978#11971302 (10atsuko) [12:19:11] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Test Kitchen, 07Essential-Work, 05Goal: [GOAL] Tidy up EventLogging - https://phabricator.wikimedia.org/T408059#11971764 (10Sfaci) [13:43:18] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Airflow DAGs for mediawiki_history_incremental_v1 writers - https://phabricator.wikimedia.org/T425730#11972168 (10xcollazo) 05Open→03In progress p:05Triage→03High a:03xcollazo [13:54:58] (03CR) 10Joal: [C:03+1] Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [13:55:17] (03CR) 10A-pizzata: [C:03+1] Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [13:56:50] (03PS14) 10Xcollazo: Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) [13:57:48] (03CR) 10Xcollazo: Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [13:58:15] (03CR) 10Xcollazo: [C:03+2] Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [14:00:45] (03CR) 10Joal: [C:03+2] "Merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1295404 (https://phabricator.wikimedia.org/T266374) (owner: 10Joal) [14:06:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Incremental MWH - MediaWiki event data source improvements - https://phabricator.wikimedia.org/T423935#11972344 (10Ottomata) [14:07:49] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Incremental MWH - MediaWiki event data source improvements - https://phabricator.wikimedia.org/T423935#11972354 (10Ottomata) [14:12:57] (03CR) 10CI reject: [V:04-1] Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [14:20:00] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 06MW-Interfaces-Team, 10Event-Platform: MediaWiki DomainEvents - Create new User related DomainEvents - https://phabricator.wikimedia.org/T427817 (10Ottomata) 03NEW [14:27:14] (03Merged) 10jenkins-bot: Fix mediawiki-history revert algorithm [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1295404 (https://phabricator.wikimedia.org/T266374) (owner: 10Joal) [14:58:41] (03CR) 10Xcollazo: [C:03+2] "recheck" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [15:13:00] (03Merged) 10jenkins-bot: Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [15:16:33] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10PageViewInfo, 06ServiceOps new, 10ServiceOps-SharedInfra: Migrate PageViewInfo calls away from rest-gateway - https://phabricator.wikimedia.org/T411771#11972861 (10tchin) a:03tchin [15:18:44] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10PageViewInfo, 06ServiceOps new, 10ServiceOps-SharedInfra: Migrate PageViewInfo calls away from rest-gateway - https://phabricator.wikimedia.org/T411771#11972873 (10tchin) I can take a look at this; what are the ports for the `pageviews` and `unique-d... [15:19:34] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 05MW-1.47-notes (1.47.0-wmf.4; 2026-05-26), 13Patch-For-Review: Event schemas - mediawiki user entity should be wiki aware - https://phabricator.wikimedia.org/T426198#11972875 (10Ottomata) [15:22:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 05MW-1.47-notes (1.47.0-wmf.3; 2026-05-19), 13Patch-For-Review: EventBus - consider schema versions when serializing entities - https://phabricator.wikimedia.org/T424767#11972886 (10Ottomata) [15:22:57] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: DE3.1 - Logged-out reader 21-day retention on web - https://phabricator.wikimedia.org/T424706#11972888 (10Ahoelzl) [15:24:55] (03PS2) 10Xcollazo: Ingest revision_visibility_change to populate revision_deleted_parts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290064 (https://phabricator.wikimedia.org/T425573) [15:25:03] (03CR) 10CI reject: [V:04-1] Ingest revision_visibility_change to populate revision_deleted_parts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290064 (https://phabricator.wikimedia.org/T425573) (owner: 10Xcollazo) [15:26:25] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Incremental MWH - MediaWiki event data source improvements - https://phabricator.wikimedia.org/T423935#11972898 (10Ottomata) a:03Ottomata [15:27:03] (03CR) 10Xcollazo: "recheck" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290064 (https://phabricator.wikimedia.org/T425573) (owner: 10Xcollazo) [15:36:27] (03PS3) 10Xcollazo: Ingest revision_visibility_change to populate revision_deleted_parts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290064 (https://phabricator.wikimedia.org/T425573) [15:39:11] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316#11972963 (10AKhatun_WMF) Update: - dbt-job MR: https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/70 (for some... [15:49:09] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Engineering-Radar, 06ServiceOps new, 10ServiceOps-Services-Oids, and 2 others: Make eventstreams-internal available to WMF staff without an ssh tunnel - https://phabricator.wikimedia.org/T348763#11973046 (10atsuko) 05Open→03Resolved [15:49:11] (03PS1) 10Joal: Update changelog for v0.3.16 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1295955 [15:54:29] (03CR) 10A-pizzata: [C:03+1] Update changelog for v0.3.16 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1295955 (owner: 10Joal) [15:55:04] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1295955 (owner: 10Joal) [16:08:42] 06Data-Engineering, 06Data-Platform-SRE, 10FR-Tech-Analytics, 07Epic: Enable greater integration between the DPE and FR-tech analytics stacks - https://phabricator.wikimedia.org/T416457#11973126 (10Ottomata) Just came across https://github.com/unitycatalog/unitycatalog [17:20:38] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to for - https://phabricator.wikimedia.org/T427553#11973388 (10Raine) [17:22:45] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to for - https://phabricator.wikimedia.org/T427553#11973401 (10Raine) @Milimetric @Ahoelzl @Ottomata can one of you please approve? Thanks! [17:29:24] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to for - https://phabricator.wikimedia.org/T427553#11973446 (10Raine) >>! In T427553#11973400, @Raine wrote: > @Milimetric @Ahoelzl @Ottomata can one of you please approve? Thanks! Apologies, I hadn't realised this... [17:33:04] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to for - https://phabricator.wikimedia.org/T427553#11973471 (10Milimetric) approved [18:11:06] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10PageViewInfo, 06ServiceOps new, 10ServiceOps-SharedInfra: Migrate PageViewInfo calls away from rest-gateway - https://phabricator.wikimedia.org/T411771#11973569 (10Ottomata) - page-analytics: 4793 - https://gerrit.wikimedia.org/r/plugins/gitiles/oper... [18:14:45] PROBLEM - Check if active EventStreams endpoint is delivering messages. on alert1002 is CRITICAL: CRITICAL: No EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [18:17:57] 06Data-Engineering, 10EventStreams: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839 (10JJMC89) 03NEW [18:24:53] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 05MW-1.47-notes (1.47.0-wmf.4; 2026-05-26), 13Patch-For-Review: Event schemas - mediawiki user entity should be wiki aware - https://phabricator.wikimedia.org/T426198#11973635 (10Ottomata) [18:26:05] 06Data-Engineering, 10EventStreams: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839#11973644 (10Ottomata) Hi! Indeed we recently got an automated alert about this. Mostly the service looks fine to me and I can consume events, but I did a server error or two while checki... [18:30:41] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Create API and User-Agent compliance related tables under wmf_traffic - https://phabricator.wikimedia.org/T427840 (10KCVelaga_WMF) 03NEW [18:32:11] 06Data-Engineering, 10EventStreams: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839#11973672 (10JJMC89) The most recent 502 was 2026-06-01T18:30. [18:32:59] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Create API and User-Agent compliance related tables under wmf_traffic - https://phabricator.wikimedia.org/T427840#11973678 (10KCVelaga_WMF) @amastilovic @JAllemandou can one of you help with this please? As you are closest to this work. [18:34:44] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Create API and User-Agent compliance related tables under wmf_traffic - https://phabricator.wikimedia.org/T427840#11973679 (10KCVelaga_WMF) Also, for future use cases, if it isn't an issue, can we consider enabling `analytics-product` user to be able to crea... [18:36:17] 06Data-Engineering, 10EventStreams: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839#11973681 (10Ottomata) Indeed, the service is flapping. On it... https://grafana.wikimedia.org/goto/ffnuiu4wclj40f?orgId=1 [18:38:39] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Create API and User-Agent compliance related tables under wmf_traffic - https://phabricator.wikimedia.org/T427840#11973686 (10KCVelaga_WMF) [18:44:45] RECOVERY - Check if active EventStreams endpoint is delivering messages. on alert1002 is OK: OK: An EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [19:00:40] !log roll-restarting eventstreams in eqiad - T427839 [19:00:43] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:00:43] T427839: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839 [19:15:03] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Blunderbuss doesn't replace the whole destination folder in HDFS - https://phabricator.wikimedia.org/T423573#11973821 (10amastilovic) This is now deployed and live in production (and tested in production, too!) [19:50:04] 06Data-Engineering, 10EventStreams, 13Patch-For-Review: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839#11973931 (10Ottomata) This has been happening since 2026-05-27. The only meaningful change that has happened is {T427416}. I don't know how that could affect memory... [20:15:48] (03PS4) 10Xcollazo: Ingest revision_visibility_change to populate revision_deleted_parts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290064 (https://phabricator.wikimedia.org/T425573) [20:15:56] (03CR) 10CI reject: [V:04-1] Ingest revision_visibility_change to populate revision_deleted_parts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290064 (https://phabricator.wikimedia.org/T425573) (owner: 10Xcollazo) [20:16:08] (03PS5) 10Xcollazo: Ingest revision_visibility_change to populate revision_deleted_parts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290064 (https://phabricator.wikimedia.org/T425573) [20:18:14] (03CR) 10Xcollazo: [C:03+2] Ingest revision_visibility_change to populate revision_deleted_parts (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290064 (https://phabricator.wikimedia.org/T425573) (owner: 10Xcollazo) [20:25:07] (03CR) 10Xcollazo: Add event_entity='page' to MWHistoryDeltaWriter (MERGE 5) (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290172 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [20:32:28] (03Merged) 10jenkins-bot: Ingest revision_visibility_change to populate revision_deleted_parts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290064 (https://phabricator.wikimedia.org/T425573) (owner: 10Xcollazo) [20:43:28] 06Data-Engineering, 06Data-Platform-SRE, 10EventStreams: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839#11974123 (10Ottomata) Okay, OOMs, still happening even after revert to old config. I'm going to bring back latest config, and also increase mem limits. I have to ru... [20:56:08] 06Data-Engineering, 06Data-Platform-SRE, 10EventStreams, 13Patch-For-Review: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839#11974138 (10Ottomata) I think we found it. https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1289978 may have included an acc... [21:05:12] (03PS2) 10Xcollazo: Add event_entity='page' to MWHistoryDeltaWriter (MERGE 5) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290172 (https://phabricator.wikimedia.org/T425729) [21:06:25] (03CR) 10Xcollazo: Add event_entity='page' to MWHistoryDeltaWriter (MERGE 5) (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290172 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [21:10:10] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316#11974187 (10AKhatun_WMF) Some analysis on user status (anonymous, temporary, permanent, cross-wiki) - In this metric we focus on `event_u... [21:15:40] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Engineering-Radar, 06ServiceOps new, 10ServiceOps-Services-Oids, and 2 others: Make eventstreams-internal available to WMF staff without an ssh tunnel - https://phabricator.wikimedia.org/T348763#11974200 (10atsuko) [21:15:42] 06Data-Engineering, 06Data-Platform-SRE, 10EventStreams, 13Patch-For-Review: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839#11974199 (10atsuko) [21:23:15] 06Data-Engineering, 06Data-Platform-SRE, 10EventStreams, 13Patch-For-Review: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839#11974208 (10atsuko) 05Open→03In progress p:05Triage→03Unbreak! Potential start time https://grafana.wikimedia.org/goto/cfnux6el9oruob?... [21:39:08] 06Data-Engineering, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Kanban Board), 07OKR-Work, 13Patch-For-Review: redioscope: periodically publish top clients to the data lake - https://phabricator.wikimedia.org/T424823#11974278 (10daniel) p:05Triage→03Low [21:49:18] 06Data-Engineering, 06Data-Platform-SRE, 10EventStreams: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839#11974307 (10Ottomata) Thank you!!! [21:49:34] 06Data-Engineering, 06Data-Platform-SRE, 10EventStreams: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839#11974308 (10atsuko) 05In progress→03Resolved a:03atsuko [21:52:14] (03CR) 10Joal: Add event_entity='page' to MWHistoryDeltaWriter (MERGE 5) (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290172 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [23:09:02] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 06MW-Interfaces-Team, 10Event-Platform: MediaWiki DomainEvents - Create new User related DomainEvents - https://phabricator.wikimedia.org/T427817#11974514 (10BPirkle) I'm unclear whether Data Engineering is planning to do this work, or is requesting... [23:11:54] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Convert `control_map MAP` to a typed struct in `mediawiki_history_incremental_v1` - https://phabricator.wikimedia.org/T427862 (10xcollazo) 03NEW [23:12:38] (03CR) 10Xcollazo: [C:03+2] Add event_entity='page' to MWHistoryDeltaWriter (MERGE 5) (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290172 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [23:30:34] (03CR) 10CI reject: [V:04-1] Add event_entity='page' to MWHistoryDeltaWriter (MERGE 5) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290172 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [23:36:38] (03CR) 10Xcollazo: Add event_entity='user' to MWHistoryDeltaWriter (MERGE 6) (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290945 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [23:38:32] (03CR) 10Xcollazo: [C:03+2] "recheck" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290172 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [23:39:24] (03PS2) 10Xcollazo: Add event_entity='user' to MWHistoryDeltaWriter (MERGE 6) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290945 (https://phabricator.wikimedia.org/T425729) [23:52:47] (03Merged) 10jenkins-bot: Add event_entity='page' to MWHistoryDeltaWriter (MERGE 5) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290172 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [23:59:00] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: DE3.1 - Logged-out reader 21-day retention on web - https://phabricator.wikimedia.org/T424706#11974692 (10jwang) @tchin , I discovered a few instrumentation limitations for Rounds 4–10 that could affect th...