[01:54:37] FIRING: RevisionFeatureCountsLatencyHigh: RevisionFeatureCounts is slow in codfw: p95 above 5s - https://wikitech.wikimedia.org/wiki/Revision_Feature_Counts#RevisionFeatureCountsLatencyHigh - https://grafana.wikimedia.org/d/revision-feature-counts/revision-feature-counts?orgId=1&var-site=codfw&var-prometheus=k8s&viewPanel=30 - https://alerts.wikimedia.org/?q=alertname%3DRevisionFeatureCountsLatencyHigh [01:59:37] RESOLVED: RevisionFeatureCountsLatencyHigh: RevisionFeatureCounts is slow in codfw: p95 above 5s - https://wikitech.wikimedia.org/wiki/Revision_Feature_Counts#RevisionFeatureCountsLatencyHigh - https://grafana.wikimedia.org/d/revision-feature-counts/revision-feature-counts?orgId=1&var-site=codfw&var-prometheus=k8s&viewPanel=30 - https://alerts.wikimedia.org/?q=alertname%3DRevisionFeatureCountsLatencyHigh [02:13:37] FIRING: RevisionFeatureCountsLatencyHigh: RevisionFeatureCounts is slow in codfw: p95 above 5s - https://wikitech.wikimedia.org/wiki/Revision_Feature_Counts#RevisionFeatureCountsLatencyHigh - https://grafana.wikimedia.org/d/revision-feature-counts/revision-feature-counts?orgId=1&var-site=codfw&var-prometheus=k8s&viewPanel=30 - https://alerts.wikimedia.org/?q=alertname%3DRevisionFeatureCountsLatencyHigh [02:18:37] RESOLVED: RevisionFeatureCountsLatencyHigh: RevisionFeatureCounts is slow in codfw: p95 above 5s - https://wikitech.wikimedia.org/wiki/Revision_Feature_Counts#RevisionFeatureCountsLatencyHigh - https://grafana.wikimedia.org/d/revision-feature-counts/revision-feature-counts?orgId=1&var-site=codfw&var-prometheus=k8s&viewPanel=30 - https://alerts.wikimedia.org/?q=alertname%3DRevisionFeatureCountsLatencyHigh [08:07:56] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Epic: Make developer metrics related data sets available in the data lake - https://phabricator.wikimedia.org/T433683#12488082 (10Addshore) [08:34:06] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats, 06translatewiki.net: Phrases on stats.wikimedia.org that are unavailable for translation - https://phabricator.wikimedia.org/T442809 (10Mitte27) 03NEW [08:38:52] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats, 06translatewiki.net: Incorrect capitalization in the translation on stats.wikimedia.org - https://phabricator.wikimedia.org/T442810 (10Mitte27) 03NEW [08:41:34] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Collaboration-Services, 10Phabricator, 13Patch-For-Review: data lake: phabricator events - https://phabricator.wikimedia.org/T434047#12488154 (10Addshore) Had a quick look today, and I think that all looks good to me (infact, to be honest, eve... [09:04:56] 06Data-Engineering, 10ChangeProp, 10EventStreams, 06MediaWiki-Engineering, and 14 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12488245 (10taavi) The [[ https://gitlab.wikimedia.org/repos/mediawiki/services/chart-renderer | chart-renderer ]] servic... [09:15:40] (03PS5) 10Aqu: dev-metrics: git_interaction table and merge [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1359279 (https://phabricator.wikimedia.org/T433696) [09:49:21] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12488399 (10APizzata-WMF) The issue was in the deployment pipeline that failed [[ https://gitlab.wikimedia.org/r... [10:02:26] (03CR) 10Aqu: [C:03+2] "Merging following discussion on Slack." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1359279 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [10:11:19] (03PS6) 10Aqu: dev-metrics: git_interaction table and merge [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1359279 (https://phabricator.wikimedia.org/T433696) [10:12:58] (03CR) 10Aqu: [V:03+2 C:03+2] dev-metrics: git_interaction table and merge [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1359279 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [10:23:14] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: data lake: git commit history - https://phabricator.wikimedia.org/T433696#12488505 (10Antoine_Quhen) The code was refactored to let analytics/refinery manage the table. [12:39:10] 06Data-Engineering, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Decide upon and deploy batch scheduling for Spark on dse-k8s - https://phabricator.wikimedia.org/T442726#12489361 (10BTullis) p:05Triage→03Medium a:03BTullis I have created a document that discusses some of these aspects of the scheduler... [12:40:07] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Generate pagecounts-ez data back to 2008 - https://phabricator.wikimedia.org/T188041#12489373 (10Milimetric) Doing the analysis on this, here are the gaps we found: |gap_start|gap_end|missing_hours |2008-01-27 00:00:00|2008-01-27 00:00:00|1 |2008-03-... [13:13:28] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10AQS2.0, 06Growth-Team, 07OKR-Work (DE1 FY2026-27), 13Patch-For-Review: AQS - Add per wiki filtering to edits/v3/per_editor - https://phabricator.wikimedia.org/T434136#12489491 (10Snwachukwu) The new v4 endpoint is now available at https://w... [13:21:44] 06Data-Engineering, 06Data-Platform-SRE: Decide upon and deploy batch scheduling for Spark on dse-k8s - https://phabricator.wikimedia.org/T442726#12489529 (10BTullis) a:05BTullis→03None Now that I have created the document and shared it for feedback, I feel that I can take this out of our current milestone... [13:34:53] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12489601 (10Aklapper) Hi and welcome @cfoster, could you please also connect your [SUL staff account](https:/... [13:36:46] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10AQS2.0, 06Growth-Team, 07OKR-Work (DE1 FY2026-27), 13Patch-For-Review: AQS - Add per wiki filtering to edits/v3/per_editor - https://phabricator.wikimedia.org/T434136#12489606 (10Ottomata) VERY NICE [14:36:35] (03PS2) 10Aqu: Add phabricator_interaction table and merge [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1356458 (https://phabricator.wikimedia.org/T434047) [14:47:18] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 05MW-1.47-notes (1.47.0-wmf.23; 2026-10-06), 05MW-1.48-notes (1.48.0-wmf.1; 2026-10-20), 13Patch-For-Review: Linked Artifact Cache - Automated MediaWiki JobQueue prefetch - https://phabricator.wikimedia.org/T436695#12490092 (10Ottomata) okay, ba... [14:49:26] 06Data-Engineering, 10ChangeProp, 10EventStreams, 06MediaWiki-Engineering, and 14 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12490114 (10dancy) [14:53:22] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12490130 (10Sfaci) [14:54:47] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 05MW-1.47-notes (1.47.0-wmf.23; 2026-10-06), 05MW-1.48-notes (1.48.0-wmf.1; 2026-10-20), 13Patch-For-Review: Linked Artifact Cache - Automated MediaWiki JobQueue prefetch - https://phabricator.wikimedia.org/T436695#12490159 (10Ottomata) > If we... [15:13:18] 06Data-Engineering, 06Data-Engineering-Radar, 10Event-Platform, 05MW-1.48-notes (1.48.0-wmf.1; 2026-10-20), and 2 others: [EventGate] Perform GeoIP lookup at event ingestion time - https://phabricator.wikimedia.org/T434312#12490327 (10KReid-WMF) 05Open→03Resolved [15:15:22] 06Data-Engineering, 10Experimentation Lab Roadmap, 10MediaWiki-extensions-EventLogging, 07Essential-Work, and 2 others: Deprecate EventLogging::schemaValidate() - https://phabricator.wikimedia.org/T317793#12490360 (10KReid-WMF) [15:51:44] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12490655 (10cfoster) @Aklapper just connected them and will add this to our team onboarding doc for next time... [16:15:10] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12490839 (10APizzata-WMF) Recap of all the MRs: [[ https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1359824 | stre... [16:19:59] (03CR) 10Xcollazo: dev-metrics: git_interaction table and merge (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1359279 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [16:25:36] 06Data-Engineering, 06Data-Engineering-Radar, 10Fundraising analytics stack, 07Epic: EPIC: Trino implementation - https://phabricator.wikimedia.org/T392093#12490902 (10AStein-WMF) [16:26:46] 06Data-Engineering, 06Data-Engineering-Radar, 10Fundraising analytics stack, 07Epic: EPIC: Trino implementation - https://phabricator.wikimedia.org/T392093#12490906 (10AStein-WMF) 05Open→03Resolved [16:31:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12490954 (10Ottomata) > Q: how do we test the instance? what's the deployment procedure? how do we rollback in case of errors?... [16:33:23] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12490958 (10Ottomata) @APizzata-WMF shall we do this together Monday or Tuesday? Want to schedule a time? I think you can go a... [16:38:42] 06Data-Engineering, 10ChangeProp, 10EventStreams, 10Proton, and 5 others: Migrate node-based services in production to node20 - https://phabricator.wikimedia.org/T364779#12490989 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF This is as Resolved as it will ever be. Node 20 is EOL, noth... [16:39:38] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12490994 (10APizzata-WMF) Monday sounds perfect :) I will merge, do I understand correctly that the deployment in beta is autom... [16:49:58] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Spark metrics job fails with weird exception - https://phabricator.wikimedia.org/T442905 (10JAllemandou) 03NEW [16:50:09] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Spark metrics job fails with weird exception - https://phabricator.wikimedia.org/T442905#12491037 (10JAllemandou) a:03JAllemandou [16:52:51] 06Data-Engineering, 10ChangeProp, 10EventStreams, 06MediaWiki-Engineering, and 5 others: Update change-prop to node22+ - https://phabricator.wikimedia.org/T442195#12491048 (10HCoplin-WMF) [16:53:31] 06Data-Engineering, 10ChangeProp, 10EventStreams, 06MediaWiki-Engineering, and 5 others: Update change-prop to node22+ - https://phabricator.wikimedia.org/T442195#12491051 (10HCoplin-WMF) Moving to code review! Thanks @Jdforrester-WMF <3 [17:00:47] 06Data-Engineering, 10ChangeProp, 10EventStreams, 06MediaWiki-Engineering, and 14 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12491079 (10Jdforrester-WMF) [17:03:42] 06Data-Engineering, 10ChangeProp, 10EventStreams, 06MediaWiki-Engineering, and 14 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12491112 (10Jdforrester-WMF) >>! In T393434#12488245, @taavi wrote: > The [[ https://gitlab.wikimedia.org/repos/mediawiki... [17:08:39] 06Data-Engineering, 06Data-Engineering-Icebox, 06Data-Platform-SRE: Public DataHub - https://phabricator.wikimedia.org/T366720#12491134 (10BTullis) I'm tagging with #data-platform-sre because I think that we would likely be involved in the operational side of making this world-readable and doing so safely.... [17:09:53] 10Data-Engineering-Roadmap, 06Data-Platform-SRE, 07Epic: [EPIC] Datahub Improvements - https://phabricator.wikimedia.org/T369756#12491156 (10BTullis) [17:23:10] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: mediawiki-content-pipelines: Add a Spark test harness - https://phabricator.wikimedia.org/T441886#12491241 (10xcollazo) > https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-content-pipelines/-/merge_requests/125: an up... [17:23:42] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: mediawiki-content-pipelines: Add a Spark test harness - https://phabricator.wikimedia.org/T441886#12491243 (10xcollazo) 05Open→03In progress p:05Triage→03Medium [17:57:59] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12491443 (10nshahquinn-wmf) > The issue was in the deployment pipeline that failed last week due to the work tha... [18:11:16] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: User and team Hive database cleanup tracking for Fall 2026 - https://phabricator.wikimedia.org/T437733#12491546 (10xcollazo) [18:14:48] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: User and team Hive database cleanup tracking for Fall 2026 - https://phabricator.wikimedia.org/T437733#12491555 (10xcollazo) **Result** This task reclaimed 513.4 TB of on-disk HDFS space, which includes replication. The tracked... [18:42:56] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): data lake: GitHub events - https://phabricator.wikimedia.org/T441643#12491714 (10Addshore) a:03Addshore [18:50:51] 06Data-Engineering, 06Movement-Insights (FY26-27 H1): Update Data Lake–to–Postgres data mirroring - https://phabricator.wikimedia.org/T442925 (10nshahquinn-wmf) 03NEW [18:56:08] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12491779 (10Jdforrester-WMF) [18:56:10] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12491787 (10Jdforrester-WMF) a:05tchin→03None [19:21:21] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 05MW-1.47-notes (1.47.0-wmf.23; 2026-10-06), 05MW-1.48-notes (1.48.0-wmf.1; 2026-10-20), 13Patch-For-Review: Linked Artifact Cache - Automated MediaWiki JobQueue prefetch - https://phabricator.wikimedia.org/T436695#12491946 (10AKhatun_WMF) Ok wi... [19:28:16] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12491966 (10Ottomata) Naw, we have to update the image version in horizon hiera data and then run puppet (or wait) on the VPS in... [19:29:01] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12491979 (10Ottomata) Oh wow look at these nice docs from Gabriele! https://wikitech.wikimedia.org/wiki/Event_Platform/Beta_Depl... [19:36:03] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 05MW-1.47-notes (1.47.0-wmf.23; 2026-10-06), 05MW-1.48-notes (1.48.0-wmf.1; 2026-10-20), 13Patch-For-Review: Linked Artifact Cache - Automated MediaWiki JobQueue prefetch - https://phabricator.wikimedia.org/T436695#12492045 (10Ottomata) Thanks!... [19:41:05] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Essential-Work: Build an HDFS growth dashboard based on wmf.hdfs_usage - https://phabricator.wikimedia.org/T438056#12492078 (10xcollazo) [19:41:08] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: Figure out the biggest HDFS consumers over time - https://phabricator.wikimedia.org/T437605#12492079 (10xcollazo) [20:09:40] (03PS1) 10Addshore: Add github_interaction table and merge [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1360531 (https://phabricator.wikimedia.org/T441643) [20:16:39] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: data lake: GitHub events - https://phabricator.wikimedia.org/T441643#12492232 (10Addshore) @Antoine_Quhen I've put together an initial GitHub activity pipeline following the phab pipeline's collection → HDFS staging → Spark merge... [20:31:03] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Essential-Work: Build an HDFS growth dashboard based on wmf.hdfs_usage - https://phabricator.wikimedia.org/T438056#12492277 (10xcollazo) It does look like we could use a bit of cleanup for `wmf.hdfs_usage` related artifacts: ` xcollazo@an-launcher... [21:40:41] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Movement-Insights (FY26-27 H1): dbt_scheduled: Migrate movement-insights and page-trending jobs from legacy DAGs to scheduling groups - https://phabricator.wikimedia.org/T441959#12492505 (10Mayakp.wiki) [22:15:14] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats: Issue in stats.wikimedia.org with card display in the Russian version. - https://phabricator.wikimedia.org/T442956 (10Mitte27) 03NEW [22:30:42] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats: Words are missing from translatewiki.net for translation - https://phabricator.wikimedia.org/T442958 (10Mitte27) 03NEW [22:44:59] 06Data-Engineering, 10ChangeProp, 10EventStreams, 06MediaWiki-Engineering, and 5 others: Update change-prop to node22+ - https://phabricator.wikimedia.org/T442195#12492867 (10ArielGlenn) We (well, James with me along as a sidekick) are jumping straight to node26. See https://gerrit.wikimedia.org/r/c/mediaw... [23:03:41] 06Data-Engineering: Track Gitlab artifact downloads in the data lake - https://phabricator.wikimedia.org/T442969 (10Addshore) 03NEW [23:03:52] 06Data-Engineering, 10a Wikimedia CLI: Track Gitlab artifact downloads in the data lake - https://phabricator.wikimedia.org/T442969#12492937 (10Addshore) [23:04:34] 06Data-Engineering, 10a Wikimedia CLI: Track Gitlab artifact downloads in the data lake - https://phabricator.wikimedia.org/T442969#12492940 (10Addshore) [23:27:11] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Generate pagecounts-ez data back to 2008 - https://phabricator.wikimedia.org/T188041#12492994 (10Milimetric) And a general data quality check for this range of dates (TL;DR; - all good, pretty clean): |src_year|src_month|mw|unconvertible|meta_m|not_a... [23:50:03] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Generate pagecounts-ez data back to 2008 - https://phabricator.wikimedia.org/T188041#12493007 (10Milimetric) And the final insert looks like this: * `project IS NOT NULL` to weed out garbage projects, same as joining to `canonical_data.wikis` by `pag...