[01:21:14] RECOVERY - Check if active EventStreams endpoint is delivering messages. on alert1002 is OK: OK: An EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [01:38:02] 06Data-Engineering, 10EventStreams, 07Wikimedia-Incident: 2026-10-26 stream.wikimedia.org 5xx errors - https://phabricator.wikimedia.org/T442462#12475695 (10herron) p:05Unbreak!→03High Hi, I just triaged a page related to this and found the backing instances in an oom crash loop. I've made a temporary m... [01:47:21] 06Data-Engineering, 10EventStreams, 07Wikimedia-Incident: 2026-10-26 stream.wikimedia.org 5xx errors - https://phabricator.wikimedia.org/T442462#12475730 (10JJMC89) {T427839} also had OOM issues [02:15:55] 06Data-Engineering, 10EventStreams, 07Wikimedia-Incident: 2026-10-26 stream.wikimedia.org 5xx errors - https://phabricator.wikimedia.org/T442462#12475802 (10tchin) Huh, afaik nothing has changed about the service recently except for a node version bump on the 23rd. I'll try to look into this [02:29:24] 06Data-Engineering, 10EventStreams, 07Wikimedia-Incident: 2026-10-26 stream.wikimedia.org 5xx errors - https://phabricator.wikimedia.org/T442462#12475809 (10JJMC89) More 5xx 2026-10-27T02:12 to 2026-10-27T02:15 and 2026-10-27T02:21 to 2026-10-27T02:25 [03:31:03] 06Data-Engineering, 10EventStreams, 07Wikimedia-Incident: 2026-10-26 stream.wikimedia.org 5xx errors - https://phabricator.wikimedia.org/T442462#12475870 (10tchin) The [[ https://grafana.wikimedia.org/d/9a06dcee-d981-4c1e-814d-83e189ac7f9f/service-utils3a-nodejs-health-metrics?orgId=1&from=now-24h&to=now&tim... [03:38:15] 06Data-Engineering, 10EventStreams, 07Wikimedia-Incident: 2026-10-26 stream.wikimedia.org 5xx errors - https://phabricator.wikimedia.org/T442462#12475874 (10SD0001) I'm also continuously running into 502 (Broken pipe) and 503 (Service unavailable) - almost every second. [05:38:20] 06Data-Engineering, 10EventStreams, 07Wikimedia-Incident: 2026-10-26 stream.wikimedia.org 5xx errors - https://phabricator.wikimedia.org/T442462#12475993 (10Marostegui) This just paged again the oncall [05:50:11] 06Data-Engineering, 10EventStreams, 07Wikimedia-Incident: 2026-10-26 stream.wikimedia.org 5xx errors - https://phabricator.wikimedia.org/T442462#12476042 (10Marostegui) We are backing showing up errors {F110940094} [06:15:08] 06Data-Engineering, 10EventStreams, 07Wikimedia-Incident: 2026-10-26 stream.wikimedia.org 5xx errors - https://phabricator.wikimedia.org/T442462#12476085 (10Marostegui) This has auto-resolved, but it clearly needs a long/medium term solution because this could come back anytime. [09:04:36] 06Data-Engineering, 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12476513 (10Sfaci) [09:31:05] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: data lake: git commit history - https://phabricator.wikimedia.org/T433696#12476599 (10Antoine_Quhen) Update: A round of reviews have been performed. Now implementing small updates before deployment. [09:32:32] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Collaboration-Services, 10Phabricator, 13Patch-For-Review: data lake: phabricator events - https://phabricator.wikimedia.org/T434047#12476600 (10Antoine_Quhen) Update: Reviews are done. We’re making minor tweaks before deployment. [10:17:33] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12476763 (10APizzata-WMF) @nshahquinn-wmf @CMyrick-WMF If that's ok with, I will close this ticket, and we can h... [11:07:31] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): ST6.3.2 mediawiki.page_change.v1 event stream SLO - https://phabricator.wikimedia.org/T436664#12476888 (10APizzata-WMF) [13:27:30] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Dumps-Generation: mediawiki-content-pipelines: Gate the File Exports on the ingest of the monthly reconcile - https://phabricator.wikimedia.org/T442415#12477756 (10xcollazo) 05Open→03In progress p:05Triage→03Medium [13:28:28] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Dumps-Generation: mediawiki-content-pipelines: Gate the File Exports on the ingest of the monthly reconcile - https://phabricator.wikimedia.org/T442415#12477764 (10xcollazo) a:03xcollazo [13:30:13] (03CR) 10Snwachukwu: "the data with new wiki_id dimension is about 1.2149 increase in the data. I tested loading to druid and it was successful. Only changed a" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1355145 (https://phabricator.wikimedia.org/T434136) (owner: 10Snwachukwu) [13:30:20] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Add wiki_id dimension to table to support poject level filtering [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1355145 (https://phabricator.wikimedia.org/T434136) (owner: 10Snwachukwu) [14:08:10] (03PS1) 10Aqu: dev-metrics: optimize git_interaction writes [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1359279 (https://phabricator.wikimedia.org/T433696) [14:09:30] (03PS2) 10Aqu: dev-metrics: optimize git_interaction writes [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1359279 (https://phabricator.wikimedia.org/T433696) [14:10:41] (03CR) 10Aqu: [V:03+2] "Yes EvolveIcebergTable is managing it." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1348792 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [14:23:19] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Platform, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12478115 (10JAllemandou) >>! In T437472#12461... [14:30:17] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Platform, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12478179 (10JAllemandou) >>! In T437472#12461... [14:39:47] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Platform, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12478231 (10JAllemandou) >>! In T437472#12462... [14:47:47] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Platform, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12478297 (10JAllemandou) >>! In T437472#12464... [14:47:50] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Platform, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12478298 (10Ottomata) > I wouldn't think of i... [14:49:55] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Platform, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12478341 (10Ottomata) > And if so, I would ra... [15:00:14] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Platform, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12478488 (10Ottomata) > when you have updated... [15:07:37] 06Data-Engineering, 10Event Metrics, 10Event-Platform: Issue with the counting of contributions linked to Wikidata via the event registration tool - https://phabricator.wikimedia.org/T442564 (10Geugeor-WMF) 03NEW [15:24:18] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12478735 (10APizzata-WMF) a:03APizzata-WMF [15:49:36] (03CR) 10A-pizzata: [C:03+2] Add PageChangeCompletenessSlo job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1357761 (https://phabricator.wikimedia.org/T440963) (owner: 10A-pizzata) [16:14:47] 06Data-Engineering, 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12479053 (10JVanderhoop-WMF) [16:32:47] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Roll out production deployment - https://phabricator.wikimedia.org/T442594 (10AKhatun_WMF) 03NEW [16:41:10] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Movement-Insights (FY26-27 H1): dbt-jobs backfill: second-week retention metric - https://phabricator.wikimedia.org/T440249#12479475 (10Mayakp.wiki) [16:49:03] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Movement-Insights (FY26-27 H1): dbt-jobs backfill: second-week retention metric - https://phabricator.wikimedia.org/T440249#12479595 (10Mayakp.wiki) 05Open→03Resolved DataQA for backfill - compared it with my db tables; everything matches up.... [17:31:36] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): KAPOW initial backfill - https://phabricator.wikimedia.org/T442233#12480169 (10mforns) [17:32:03] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): KAPOW initial backfill - https://phabricator.wikimedia.org/T442233#12480174 (10mforns) [17:53:56] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Make lambda accept non-Wikipedia projects as welll - https://phabricator.wikimedia.org/T442621 (10AKhatun_WMF) 03NEW [17:58:24] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt scheduler: Implement support for "backfill scheduling groups" - https://phabricator.wikimedia.org/T442623 (10amastilovic) 03NEW [18:00:05] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt scheduler: Implement support for "backfill scheduling groups" - https://phabricator.wikimedia.org/T442623#12480459 (10GGoncalves-WMF) [18:00:07] 10Data-Engineering-Roadmap, 07Epic: dbt automated initial backfill - https://phabricator.wikimedia.org/T438322#12480460 (10GGoncalves-WMF) [18:50:09] !log Deploying Refinery at 4a515e4 for change 1355145/T434136 Add per wiki filtering to edits/v3/per_editor [18:50:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:17:11] !log Deployed Refinery using Scap, and then deployed onto hdfs [19:17:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:50:12] (03CR) 10Xcollazo: dev-metrics: optimize git_interaction writes (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1359279 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [19:54:50] (03CR) 10Xcollazo: dev-metrics: optimize git_interaction writes (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1359279 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [21:48:48] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: User and team Hive database cleanup tracking for Fall 2026 - https://phabricator.wikimedia.org/T437733#12481855 (10xcollazo) @Hghani cleaned what could be cleaned out of `/user/hive/warehouse/hghani.db`. [21:49:18] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: User and team Hive database cleanup tracking for Fall 2026 - https://phabricator.wikimedia.org/T437733#12481857 (10xcollazo) I think we should be good here, will do a final recalculation and close.