[06:01:02] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Switch wmde_bi_yearly_data_cleanup_email to email all cluster users - https://phabricator.wikimedia.org/T436324 (10AndrewTavis_WMDE) 03NEW [06:03:11] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Switch wmde_bi_yearly_data_cleanup_email to email all cluster users - https://phabricator.wikimedia.org/T436324#12264185 (10AndrewTavis_WMDE) CCing engineers who have shown interest in this being more widely used: @xcollazo, @awight, @JAllemandou 👋 [06:07:30] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Switch wmde_bi_yearly_data_cleanup_email to email all cluster users - https://phabricator.wikimedia.org/T436324#12264187 (10AndrewTavis_WMDE) [06:14:23] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Switch wmde_bi_yearly_data_cleanup_email to email all cluster users - https://phabricator.wikimedia.org/T436324#12264201 (10AndrewTavis_WMDE) [06:19:48] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Switch wmde_bi_yearly_data_cleanup_email to email all cluster users - https://phabricator.wikimedia.org/T436324#12264210 (10AndrewTavis_WMDE) [06:23:43] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Switch wmde_bi_yearly_data_cleanup_email to email all cluster users - https://phabricator.wikimedia.org/T436324#12264218 (10AndrewTavis_WMDE) [06:26:05] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Switch bi-yearly data cleanup email DAG to email all cluster users - https://phabricator.wikimedia.org/T436324#12264220 (10AndrewTavis_WMDE) [06:26:51] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Switch bi-yearly data cleanup email DAG to email all cluster users - https://phabricator.wikimedia.org/T436324#12264221 (10AndrewTavis_WMDE) [06:32:06] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Switch bi-yearly data cleanup email DAG to email all cluster users - https://phabricator.wikimedia.org/T436324#12264227 (10AndrewTavis_WMDE) Question here aside from adjustments to the text as needed: where should people be directed to at the end? W... [06:39:08] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Switch bi-yearly data cleanup email DAG to email all cluster users - https://phabricator.wikimedia.org/T436324#12264241 (10AndrewTavis_WMDE) What would the preference be for sending the email? To `noreploy@wikimedia.org` and bcc all cluster users? T... [07:39:13] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Traffic, 10Wikidata, 10WMDE Analytics, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#12264348 (10Gehel) [09:29:28] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 07Essential-Work: With spill_enabled = true, Presto fails queries with CROSS JOIN - https://phabricator.wikimedia.org/T435862#12264666 (10RKemper) 05Open→03Resolved Now that this incident is resolved, it's worth circling back on the root cause.... [09:56:52] 06Data-Engineering, 10Event-Platform, 07Wikimedia-production-error: User::loadFromSession called before the end of Setup.php - https://phabricator.wikimedia.org/T436345 (10Dreamy_Jazz) 03NEW [09:59:14] 06Data-Engineering, 10Event-Platform, 07Wikimedia-production-error: User::loadFromSession called before the end of Setup.php - https://phabricator.wikimedia.org/T436345#12264806 (10Dreamy_Jazz) I would suggest deferring the serializing of the user until later in the request, maybe inside a deferred update? [10:01:02] 06Data-Engineering, 10Test Kitchen, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#12264809 (10atsuko) [10:04:17] 06Data-Engineering, 10Event-Platform, 07Wikimedia-production-error: "User::loadFromSession called before the end of Setup.php" when EventBus is serliasing user groups - https://phabricator.wikimedia.org/T436345#12264842 (10Dreamy_Jazz) [10:04:28] 06Data-Engineering, 10Event-Platform, 07Wikimedia-production-error: "User::loadFromSession called before the end of Setup.php" when EventBus is serializing user groups - https://phabricator.wikimedia.org/T436345#12264843 (10Dreamy_Jazz) [10:27:43] 06Data-Engineering, 10Test Kitchen, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#12264915 (10atsuko) @mpopov what users should I add to `analytics-experiment-users` group? [11:26:06] 06Data-Engineering, 10Test Kitchen, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#12265126 (10atsuko) Asked @MoritzMuehlenhoff to help with creating a LDAP group. Initial member: `mpopov` Group descrip... [11:39:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:39:43] 06Data-Engineering, 10Test Kitchen, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#12265172 (10MoritzMuehlenhoff) >>! In T416709#12265125, @atsuko wrote: > Asked @MoritzMuehlenhoff to help with creating... [11:49:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [12:59:00] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Metrics Superset dashboard access for all WMF - https://phabricator.wikimedia.org/T431697#12265395 (10Gehel) [12:59:18] 06Data-Engineering, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Enable Ceph S3 locations for Hive Metastore tables - https://phabricator.wikimedia.org/T425673#12265403 (10Gehel) [12:59:34] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Wikimedia Enterprise, 10Wikimedia Enterprise - Content Integrity, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), and 2 others: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#12265405 (10... [13:00:03] 06Data-Engineering, 06Data-Engineering-Radar, 06Privacy Engineering, 06Security-Team, and 3 others: Privacy review of x1 tables in preparation of adding them to wikireplicas - https://phabricator.wikimedia.org/T415219#12265419 (10Gehel) [13:00:07] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Platform, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Debmonitor checks failing for docker-registry.discovery.wmnet/repos/data-engineering/eventstreams:v0.19.0 - https://phabricator.wikimedia.org/T432824#12265423 (10Gehel) [13:00:19] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Optimize enqueueing of refine_webrequest_hourly pipeline - https://phabricator.wikimedia.org/T419050#12265429 (10Gehel) [13:00:30] 06Data-Engineering, 06Data-Engineering-Radar, 06cloud-services-team, 06Data-Persistence, and 4 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#12265431 (10Gehel) [13:00:44] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work: [Infra] Carry out end-user testing of spark on kubernetes - https://phabricator.wikimedia.org/T412925#12265437 (10Gehel) [13:00:54] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 4 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#12265439 (10Gehel) [13:01:43] 06Data-Engineering, 10Test Kitchen, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#12265455 (10Gehel) [13:02:15] 14Analytics-Radar, 06Data-Engineering, 10Kafka-Infrastructure, 06serviceops-radar, and 4 others: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12265465 (10Gehel) [13:03:22] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): implement script to move data from P&T data lake to FR Tech data lake - https://phabricator.wikimedia.org/T425133#12265496 (10Gehel) [13:04:40] 06Data-Engineering, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): [Tech evaluation] Table catalog options for the S3 data lake - https://phabricator.wikimedia.org/T435604#12265531 (10Gehel) [13:15:52] 06Data-Engineering, 10Test Kitchen, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#12265613 (10mpopov) >>! In T416709#12264915, @atsuko wrote: > @mpopov what users should I add to `analytics-experiment-... [13:19:31] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 07Epic: EPIC: DYK Archive - https://phabricator.wikimedia.org/T430319#12265617 (10Snwachukwu) [13:20:35] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 10Event-Platform: Enable kubernetes.operator.savepoint.dispose-on-delete by default - https://phabricator.wikimedia.org/T432455#12265639 (10Gehel) [13:20:47] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: [Infra] Add a presto query logger - https://phabricator.wikimedia.org/T269832#12265642 (10Gehel) [13:20:53] 06Data-Engineering, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Task Tries and Logs for Airflow DAGs sometimes unavailable - https://phabricator.wikimedia.org/T419162#12265644 (10Gehel) [13:22:12] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work: Superset "track job" button leads to broken URL - https://phabricator.wikimedia.org/T410149#12265683 (10Gehel) [13:22:59] 06Data-Engineering, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work: ERROR AsyncEventQueue: Listener DatahubSparkListener threw an exception - https://phabricator.wikimedia.org/T400207#12265701 (10Gehel) [13:23:28] 06Data-Engineering, 10BetaFeatures, 06cloud-services-team, 10Data-Services, and 6 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#12265711 (10Gehel) [13:23:36] 06Data-Engineering, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Delete orphan EventLogging topic `eventlogging_HomepageModulet` - https://phabricator.wikimedia.org/T429017#12265717 (10Gehel) [13:24:44] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Impact Module Metrics - Sliding 30 Day Top-K Pages for Editor - https://phabricator.wikimedia.org/T436368 (10Snwachukwu) 03NEW [13:27:08] 06Data-Engineering, 10Test Kitchen, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#12265768 (10atsuko) `airflow-experiment-platform-ops`: users added, see [[ https://ldap.toolforge.org/group/airflow-exp... [13:38:11] 06Data-Engineering, 10Test Kitchen, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#12265823 (10atsuko) @mpopov diff [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328595 | 1328595 ]] is pendin... [14:15:05] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Hypothesis: Reference Counts [Attribution API] - https://phabricator.wikimedia.org/T432604#12265975 (10AKhatun_WMF) [15:28:36] 06Data-Engineering, 10WMDE Analytics: [Analytics] [Tech Debt] Switch bi-yearly data cleanup email DAG to email all cluster users - https://phabricator.wikimedia.org/T436324#12266283 (10xcollazo) >The emails are public in data.yaml anyway, but maybe still bcc? Ideally we would have an email group that auto upda... [15:53:27] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12266340 (10tchin) I wonder if the domain event should capture `ChangeTagAfterDelete` (the tag itself was deleted) since i... [15:58:04] 14Data-Engineering (Q4 FS25/26 April 1st - June 30st), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: DE3.1 - Logged-out Wikipedia 21-day retention on mobile web - https://phabricator.wikimedia.org/T424706#12266349 (10JAnstee_WMF) Weekly Status update: * Needs confirmation of implementation sign-off * Activ... [16:00:39] 14Data-Engineering (Q4 FS25/26 April 1st - June 30st), 05Metrics-Sprint-2026-2027: DE3.2 - Logged-in Wikipedia reader 2nd week retention on web - https://phabricator.wikimedia.org/T424708#12266354 (10JAnstee_WMF) Status Update: * Implementation needs approval confirmation * Active Data Issue: Yes (using l... [16:06:18] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12266363 (10xcollazo) > I wonder if the domain event should capture `ChangeTagAfterDelete` (the tag itself was deleted) A... [17:06:15] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12266526 (10tchin) >>! In T434143#12266363, @xcollazo wrote: > Do you know if tags themselves have an id at all, or are th... [17:09:28] 06Data-Engineering, 10Test Kitchen, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#12266539 (10atsuko) [17:13:25] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Make scheduling and configuring dbt jobs easy - https://phabricator.wikimedia.org/T435369#12266542 (10amastilovic) 05Open→03Resolved [17:13:52] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt isolated model orchestration - https://phabricator.wikimedia.org/T430625#12266545 (10amastilovic) @GGoncalves-WMF I think this ticket can be resolved/closed now. [17:16:34] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): implement script to move data from P&T data lake to FR Tech data lake - https://phabricator.wikimedia.org/T425133#12266549 (10AStein-WMF) This is complete! we have 2 dags moving data to FR Tech's data lake... [17:16:43] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): implement script to move data from P&T data lake to FR Tech data lake - https://phabricator.wikimedia.org/T425133#12266550 (10AStein-WMF) 05Open→03Resolved [17:26:12] 06Data-Engineering, 10Test Kitchen, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#12266571 (10atsuko) [18:01:04] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12266661 (10xcollazo) >>! In T434143#12266526, @tchin wrote: >>>! In T434143#12266363, @xcollazo wrote: >> Do you know if... [18:32:26] (03PS4) 10Mforns: Switch pageview_actor consumers to use webrequest_v2 instead. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1318283 (https://phabricator.wikimedia.org/T425668) [19:07:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:37:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:15:48] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Move refinery HQL files to a new GitLab repository - https://phabricator.wikimedia.org/T432378#12267121 (10Milimetric) >>! In T432378#12138797, @amastilovic wrote: >>>! In T432378#12138142, @Ottomata wrote: >> Any reason not to use... [20:35:51] 06Data-Engineering, 06Data-Engineering-Radar, 06Content-Platform-Team, 06MediaWiki-API-Platform-Team, 10Event-Platform: Expose MediaWiki Parser render_id as a response header in relevant MW REST API endpoints - https://phabricator.wikimedia.org/T418792#12267201 (10cscott) 05Open→03Resolved a:03c... [20:46:20] (03PS5) 10Mforns: Switch pageview_actor consumers to use webrequest_v2 instead. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1318283 (https://phabricator.wikimedia.org/T425668) [21:07:07] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:47:07] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:05:02] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Update DBT Airflow operator's task notes only in case of failure - https://phabricator.wikimedia.org/T436410 (10amastilovic) 03NEW [23:31:41] (03PS6) 10Mforns: Switch pageview_actor consumers to use webrequest_v2 instead. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1318283 (https://phabricator.wikimedia.org/T425668) [23:32:56] (03CR) 10Mforns: [V:03+2] "I've tested these jobs very carefully, and found a couple issues with the code, that I fixed. I think this is ready for final review." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1318283 (https://phabricator.wikimedia.org/T425668) (owner: 10Mforns)