[04:01:29] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12383404 (10Eric) >>! In T439644#12381049, @eniroomand wrote: > @ayounsi I confirm that I have read th... [07:30:35] volans|off: sorry I had an emergency yesterday, I was off [07:37:55] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: [Tech evaluation] Table catalog options for the S3 data lake - https://phabricator.wikimedia.org/T435604#12383702 (10JAllemandou) [07:38:42] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Platform, 06Data-Platform-SRE, 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12383703 (10JAllemandou) [07:39:33] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Platform, 06Data-Platform-SRE, 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12383705 (10JAllemandou) a:03JAllemandou [07:44:12] (03CR) 10Joal: [V:03+2 C:03+2] "I'll merge it and it'll be deployed next Tuesday as part of our train :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345865 (https://phabricator.wikimedia.org/T422443) (owner: 10Joal) [08:26:11] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12383801 (10APizzata-WMF) [08:29:14] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12383807 (10Tchanders) Ah, that should be @EMill-WMF [09:30:51] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12384124 (10APizzata-WMF) Merged the HQL table creation scripts and the weekly sqoop jobs. Would appreciate a revi... [09:51:48] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384177 (10BTullis) [10:00:04] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384208 (10BTullis) Creating the `codfw-sts-test` account. ` btullis@ceph-admin2001:~$ sudo radosgw-admin accoun... [10:00:34] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384209 (10BTullis) [10:05:21] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384223 (10BTullis) Configuring an `aws` CLI profile on ceph-admin2001. ` btullis@ceph-admin2001:~$ aws configur... [10:40:55] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384363 (10BTullis) Get the certificate for the k8s server. ` btullis@ceph-admin2001:~$ openssl s_client -connec... [10:41:33] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384366 (10BTullis) [10:45:16] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384369 (10BTullis) Here it is in more detail. ` btullis@ceph-admin2001:~$ aws --endpoint-url $RGW iam list-open... [10:47:23] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384383 (10BTullis) Creating a test bucket. ` btullis@ceph-admin2001:~$ aws --endpoint-url $RGW s3api create-buc... [11:11:07] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384447 (10BTullis) Here is the trust policy that we are creating. ` btullis@ceph-admin2001:~$ cat trust-writer.... [11:11:53] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384448 (10BTullis) [11:29:17] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384486 (10BTullis) Now I am at step 4a, which tests STS via an IAM AssumeRole operation. Then it uses these tem... [11:33:05] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384503 (10BTullis) Use these temporary keys to carry out some S3 operations. ` btullis@ceph-admin2001:~$ export... [11:36:45] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: mediawiki-event-enrichment - set deterministic meta.id - https://phabricator.wikimedia.org/T424987#12384504 (10JMonton-WMF) I'm working on it, and I was wondering, if creating a deterministic `meta.id` derived from the original `meta.... [11:48:38] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384544 (10BTullis) We can see that listing all buckets as this user fails, because it is not part of the permis... [11:49:30] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384550 (10BTullis) [11:58:17] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12384624 (10BTullis) Creating the serviceaccount in the `analytics-test` namespace in `dse-k8s-codfw` ` root@depl... [12:55:34] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12384852 (10EMill-WMF) Yes, I approve Eva for access here. And sorry @Eric! I think the fact that we... [13:08:31] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12384952 (10Eric) >>! In T439644#12384852, @EMill-WMF wrote: > And sorry @Eric! I think the fact that... [13:40:10] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12385099 (10BTullis) OK, part 4b, testing with `AssumeRoleWithWebIdentity` is failing, as expected. I checked by... [13:55:53] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12385235 (10ayounsi) 05Open→03Resolved Give it 30min for the change to propagate and you shoul... [14:03:04] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: mediawiki-event-enrichment - set deterministic meta.id - https://phabricator.wikimedia.org/T424987#12385280 (10Ottomata) I think deterministic from meta.id should work, no? Sourcing the uuid hash from other fields would probably wo... [14:30:59] 06Data-Engineering, 06Research: Add width bands to the mediarequests API - https://phabricator.wikimedia.org/T439420#12385548 (10Effeietsanders) [14:38:53] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Include v2 dumps into mirrors rsync configuration - https://phabricator.wikimedia.org/T437463#12385626 (10GGoncalves-WMF) >> Which cutoff date do you want? 2026-09-01 is available now. 2026-10-01 is the first run with v2 7z files. >... [14:56:20] 06Data-Engineering, 06ServiceOps, 10ServiceOps-SharedInfra: Create namespace for LAC lambdas - https://phabricator.wikimedia.org/T439879 (10Clement_Goubert) 03NEW [14:56:55] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Implement ArticleFeatureCounts Application - https://phabricator.wikimedia.org/T437000#12385733 (10Clement_Goubert) [14:56:56] 06Data-Engineering, 06ServiceOps, 10ServiceOps-SharedInfra: Create namespace for LAC lambdas - https://phabricator.wikimedia.org/T439879#12385734 (10Clement_Goubert) [14:57:09] 06Data-Engineering, 06ServiceOps, 10ServiceOps-SharedInfra: Create namespace for LAC lambdas - https://phabricator.wikimedia.org/T439879#12385735 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High [14:58:24] 06Data-Engineering, 06ServiceOps, 10ServiceOps-SharedInfra: Create namespace for LAC lambdas - https://phabricator.wikimedia.org/T439879#12385744 (10Clement_Goubert) [15:24:05] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05acl*data-engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12385844 (10atsuko) Moving this two work items from the [[ https://phabricator.wikime... [15:32:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05acl*data-engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12385894 (10atsuko) >>! In T435480#12297026, @Antoine_Quhen wrote: > create a small n... [15:39:26] 06Data-Engineering, 06Data-Engineering-Radar, 06MediaWiki-Media-Platform-Team, 06ServiceOps, and 3 others: DC Switchover - MW requests to EventGate cause long queuing waiting time - https://phabricator.wikimedia.org/T439552#12385959 (10CDanis) First of all, awesome to see formal ownership change so quickly... [15:42:33] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Include v2 dumps into mirrors rsync configuration - https://phabricator.wikimedia.org/T437463#12385980 (10xcollazo) > Would we be able to easily move the cutoff to December 2026 instead? Yes, easy fix. Will do. [15:50:26] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05acl*data-engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12386063 (10atsuko) >>! In T435480#12297445, @xcollazo wrote: > But I also think tha... [16:06:31] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Implement 7z compression for v2 history dumps - https://phabricator.wikimedia.org/T437454#12386227 (10xcollazo) Details on the ongoing Airflow dagrun copy pasted from a Slack message. We are currently alerting due to elevated scheduler loop: > https:/... [16:31:25] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05acl*data-engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12386315 (10xcollazo) > I believe DagProperties is working, we tested it at least on... [18:20:23] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Implement 7z compression for v2 history dumps - https://phabricator.wikimedia.org/T437454#12386885 (10xcollazo) **Follow-up on the scheduler load of the Oct 1 run** I exported 90 days of the scheduler metrics from Grafana. The range forces a 3 h step, s... [18:25:15] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12386911 (10eniroomand) @ayounsi It works, I can now see the data that was previously hidden. A follow-up question (let... [20:12:17] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05MW-1.47-notes (1.47.0-wmf.23; 2026-10-06), 13Patch-For-Review: Linked Artifact Cache - Automated MediaWiki JobQueue precompute - https://phabricator.wikimedia.org/T436695#12387223 (10Ottomata) I'm working on patch 2 and patch 3 together. patch 3 w... [20:22:02] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: data lake: git commit history - https://phabricator.wikimedia.org/T433696#12387278 (10Antoine_Quhen) Here is a first version of the Iceberg table: `wmf_dev_metrics.git_interaction` It contains all commits from all branches since beg... [20:33:06] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: gitlab-mcp-server: Add read-only MCP server for GitLab merge requests and comments - https://phabricator.wikimedia.org/T438508#12387320 (10xcollazo) >>! In T438508#12387316, @CodeReviewBot wrote: > xcollazo updated https://gitlab.wi... [21:31:36] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 07Epic, 07Essential-Work: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#12387493 (10amastilovic) I believe `datahub-cli` has already been migrated to GitLab, publishing its artifac... [21:32:45] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 07Epic, 07Essential-Work: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#12387498 (10amastilovic) [22:17:30] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Define and document the earliest start_date for dbt DAGs - https://phabricator.wikimedia.org/T438276#12387608 (10amastilovic) Style guide updated: https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/dbt/dbt-jobs_repository_style_guide#Choosing_a_sc...