[07:14:03] (03CR) 10Joal: [C:03+1] "Code looks good, one question in file." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1335154 (https://phabricator.wikimedia.org/T431476) (owner: 10Ottomata) [08:39:51] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Large wiki dumps leak temporary files - https://phabricator.wikimedia.org/T437267 (10brouberol) 03NEW p:05Triage→03Medium [08:39:56] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Large wiki dumps leak temporary files - https://phabricator.wikimedia.org/T437267#12296453 (10brouberol) [08:39:58] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Large wiki dumps leak temporary files - https://phabricator.wikimedia.org/T437267#12296455 (10brouberol) 05Open→03In progress [08:41:39] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Large wiki dumps leak temporary files - https://phabricator.wikimedia.org/T437267#12296459 (10brouberol) [08:43:23] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Large wiki dumps leak temporary files - https://phabricator.wikimedia.org/T437267#12296462 (10brouberol) Once we've ensure we're deleting //future// temporary file leftovers, we'll also need to remove the old ones as well. [09:26:17] 06Data-Engineering, 06Data-Platform-SRE: [Tech evaluation] How we import the MediaWiki databases into the S3 data lake - https://phabricator.wikimedia.org/T437079#12296625 (10JAllemandou) >>! In T437079#12294644, @BTullis wrote: > Thanks, yes I agree that this was a bit wooly and vague. How does this version o... [09:51:37] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Backfill wmf_objective_metrics.heartbeat_active_editors_weekly - https://phabricator.wikimedia.org/T437118#12296723 (10APizzata-WMF) The table is backfilled: ` spark.sql(""" select dt, count(*) from wmf_ob... [10:08:14] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Large wiki dumps leak temporary files - https://phabricator.wikimedia.org/T437267#12296770 (10brouberol) I confirm that this is working! ` www-data@mediawiki-dumps-legacy-dump-toolbox-69b887b98b-q56k8:/mnt/dumpsdata/xmldat... [10:08:50] (03CR) 10Joal: "Some changes and some performance validation needed." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1318283 (https://phabricator.wikimedia.org/T425668) (owner: 10Mforns) [10:11:11] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Large wiki dumps leak temporary files - https://phabricator.wikimedia.org/T437267#12296781 (10brouberol) {F102202159} We can see that this had a small effect, as the wikis responsible for the largest temporary files (wikid... [10:12:30] (03CR) 10Joal: "Actually, one change needed :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1318283 (https://phabricator.wikimedia.org/T425668) (owner: 10Mforns) [10:16:28] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Large wiki dumps leak temporary files - https://phabricator.wikimedia.org/T437267#12296810 (10brouberol) ` www-data@mediawiki-dumps-legacy-dump-toolbox-69b887b98b-q56k8:/mnt/dumpsdata/xmldatadumps/temp$ find . -name "*-202... [10:16:41] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Large wiki dumps leak temporary files - https://phabricator.wikimedia.org/T437267#12296814 (10brouberol) 05In progress→03Resolved [10:18:25] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): 7za is OOMkilled when dumps are running on a Bookworm-based image - https://phabricator.wikimedia.org/T436913#12296815 (10brouberol) Half of the failing dumps have now completed. I'm going to close. [10:18:28] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): 7za is OOMkilled when dumps are running on a Bookworm-based image - https://phabricator.wikimedia.org/T436913#12296817 (10brouberol) 05Open→03Resolved [10:20:21] 06Data-Engineering, 10Experimentation Lab Roadmap, 06MediaWiki-Core-Platform-Team, 10MediaWiki-extensions-EventLogging, and 4 others: [EPIC] Deprecate EventLogging::logEvent() - https://phabricator.wikimedia.org/T318263#12296819 (10phuedx) [10:20:51] 06Data-Engineering, 10Experimentation Lab Roadmap, 06MediaWiki-Core-Platform-Team, 10MediaWiki-extensions-EventLogging, and 3 others: Deprecate EventLogging::logEvent() - https://phabricator.wikimedia.org/T318263#12296821 (10phuedx) [10:25:56] (03CR) 10Mforns: [V:03+2] Switch pageview_actor consumers to use webrequest_v2 instead. (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1318283 (https://phabricator.wikimedia.org/T425668) (owner: 10Mforns) [10:39:19] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: mediawiki-event-enrichment page_html_change job should use X-MediaWiki-Render-Id header instead of etag - https://phabricator.wikimedia.org/T420217#12296891 (10JMonton-WMF) 05Open→03In progress a:03JMonton-WMF [10:41:20] 10Data-Engineering-Roadmap, 07Epic: Create, orchestrate and backfill wmf.projectview_legacy_hourly table - https://phabricator.wikimedia.org/T437277 (10mforns) 03NEW [10:41:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Engineering-Roadmap, 07Epic: Create, orchestrate and backfill wmf.projectview_legacy_hourly table - https://phabricator.wikimedia.org/T437277#12296910 (10mforns) [10:43:14] (03PS1) 10Mforns: Add query to generate projectview_legacy_hourly data for KAPOW cutover [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1337882 (https://phabricator.wikimedia.org/T437277) [10:44:27] (03CR) 10Mforns: [V:03+2] "I tested this query by generating an hour of projectview data," [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1337882 (https://phabricator.wikimedia.org/T437277) (owner: 10Mforns) [11:04:34] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 10Test Kitchen, 07Technical-Debt: Non-deterministic unit test "streamInSample() - session sampling resets" - https://phabricator.wikimedia.org/T304379#12296949 (10phuedx) [11:31:21] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE: Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12297026 (10Antoine_Quhen) Thanks for driving the Airflow upgrade @atsuko! @JAllemandou, overall I agree with the method for determining the short list of D... [13:02:06] (03CR) 10Joal: [C:03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1337882 (https://phabricator.wikimedia.org/T437277) (owner: 10Mforns) [13:44:17] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE: Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12297445 (10xcollazo) >>! In T435480#12270730, @JAllemandou wrote: > I asked Claude to help me with the categorization, and I think it did a great job: https... [14:09:21] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE: Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12297546 (10JAllemandou) a:05JAllemandou→03Ahoelzl Thanks @Antoine_Quhen and @xcollazo for your comments. To my understanding, @atsuko is already tackli... [14:11:52] 06Data-Engineering, 06Product-Analytics, 06Product Safety and Integrity (Sprint Heather (Aug 31 - Sept 18)): Dbt backfill: signal_driven_handling_bad_faith_accounts - https://phabricator.wikimedia.org/T437095#12297581 (10Tchanders) [14:31:36] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 07OKR-Work (WE1 FY2025-26): [Spike] Adding access_method metadata to moderator action event streams - https://phabricator.wikimedia.org/T419019#12297709 (10Ottomata) > mediawiki_browser_js just a stand-in for VisualEditor More general than VisualEditor... [14:35:13] 10Data-Engineering-Roadmap, 06Data-Persistence, 10MediaWiki-Page-derived-data, 07Community-Wishlist, and 2 others: Add page_is_redirect/page_namespace/page_title index - https://phabricator.wikimedia.org/T387537#12297738 (10Ottomata) I think also related: https://meta.wikimedia.org/wiki/Community_Wishlist... [14:36:59] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 07OKR-Work (WE1 FY2025-26): [Spike] Adding access_method metadata to moderator action event streams - https://phabricator.wikimedia.org/T419019#12297743 (10GGoncalves-WMF) > We are kind of just guessing what would be useful here. This is better than th... [14:38:33] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12297745 (10tchin) I prefer we avoid fat events and have people do the enrichment themselves, but the old event stream alr... [14:48:18] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Commons, 06Data-Persistence, 07OKR-Work, 13Patch-For-Review: Data Lake - sqoop of commons link tables needs to adapt to move to dedicated MariaDB cluster - https://phabricator.wikimedia.org/T431476#12297788 (10Ottomata) FWIW, there are surely M... [15:07:29] 06Data-Engineering, 06Product-Analytics: Create a new database for WMF traffic related metrics - https://phabricator.wikimedia.org/T436678#12297941 (10Ottomata) @GGoncalves-WMF FWIW, either of your suggestions is fine with me. In lieu of objections, I would love if you could be the decider on this one! TY! [15:22:20] 06Data-Engineering, 10Experimentation Lab Roadmap, 06MediaWiki-Core-Platform-Team, 10MediaWiki-extensions-EventLogging, and 4 others: Deprecate EventLogging::logEvent() - https://phabricator.wikimedia.org/T318263#12298010 (10KReid-WMF) p:05Triage→03Medium [15:24:53] (03CR) 10Ottomata: sqoop-mediawiki-tables - add a DB_MAPPING_TABLE_OVERRIDES (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1335154 (https://phabricator.wikimedia.org/T431476) (owner: 10Ottomata) [15:50:26] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Make Airflow Druid infrastructure more failproof and resilient - https://phabricator.wikimedia.org/T437310 (10amastilovic) 03NEW [16:49:40] 06Data-Engineering, 06Product-Analytics, 06Product Safety and Integrity (Sprint Heather (Aug 31 - Sept 18)): Dbt backfill: signal_driven_handling_bad_faith_accounts - https://phabricator.wikimedia.org/T437095#12298868 (10cchen) [16:55:23] 10Data-Engineering-Roadmap, 07Epic: Write technical documentation - https://phabricator.wikimedia.org/T437332 (10mforns) 03NEW [16:55:42] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Engineering-Roadmap, 07Epic: Write technical documentation - https://phabricator.wikimedia.org/T437332#12298922 (10mforns) [16:55:59] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Engineering-Roadmap, 07Epic: Write technical documentation for KAPOW - https://phabricator.wikimedia.org/T437332#12298927 (10mforns) [18:18:33] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12299381 (10Ottomata) Good points. Hm. Perhaps, we should take a minute to be explicit about the cases where we need loggi... [18:44:24] 06Data-Engineering, 06Data-Platform-SRE: [Tech evaluation] How we ingest the Kafka streams into the S3 data lake - https://phabricator.wikimedia.org/T437080#12299527 (10Ottomata) Thank you Ben! Some comments: > Does it understand our event schemas? Let's change this to "Can it understand our event schemas".... [18:45:54] 06Data-Engineering, 06Data-Platform-SRE: [Tech evaluation] How we ingest the Kafka streams into the S3 data lake - https://phabricator.wikimedia.org/T437080#12299533 (10Ottomata) [18:46:40] 06Data-Engineering, 06Data-Engineering-Radar, 06Traffic: Add backend information to webrequests in data lake - https://phabricator.wikimedia.org/T436223#12299535 (10ssingh) This is something that Traffic would undertake in theory, correct. I will need to check with the team and particularly @Fabfur so will f... [19:03:25] 06Data-Engineering, 06Data-Platform-SRE: [Tech evaluation] How we ingest the Kafka streams into the S3 data lake - https://phabricator.wikimedia.org/T437080#12299630 (10Ottomata) [19:03:39] 06Data-Engineering, 06Data-Platform-SRE: [Tech evaluation] How we ingest the Kafka streams into the S3 data lake - https://phabricator.wikimedia.org/T437080#12299631 (10Ottomata) > Can it run alongside what we have? I was bold and removed this. I hope that is okay! [19:04:07] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:04:09] 06Data-Engineering, 06Data-Platform-SRE: [Tech evaluation] How we import the MediaWiki databases into the S3 data lake - https://phabricator.wikimedia.org/T437079#12299632 (10Ottomata) [19:04:28] 06Data-Engineering, 06Data-Platform-SRE: [Tech evaluation] How we import the MediaWiki databases into the S3 data lake - https://phabricator.wikimedia.org/T437079#12299634 (10Ottomata) > Can it run alongside what we have? I was bold and removed this following T437080#12299630. [19:06:31] 06Data-Engineering, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): [Tech evaluation] Table catalog options for the S3 data lake - https://phabricator.wikimedia.org/T435604#12299646 (10Ottomata) I think this tech Table Catalog/Metastore eval is less coupled than T437080 and T437079 are. We can probably make pro... [19:13:50] (03PS7) 10Mforns: Switch pageview_actor consumers to use webrequest_v2 instead. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1318283 (https://phabricator.wikimedia.org/T425668) [19:26:48] (03CR) 10Mforns: [V:03+2] Switch pageview_actor consumers to use webrequest_v2 instead. (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1318283 (https://phabricator.wikimedia.org/T425668) (owner: 10Mforns) [19:28:23] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Make Airflow Druid infrastructure more failproof and resilient - https://phabricator.wikimedia.org/T437310#12299800 (10brouberol) We cannot use LVS, as Druid is deployed in the analytics VLAN. I think we can however do 2 things: - define a simple round-r... [19:29:20] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Make Airflow Druid infrastructure more failproof and resilient - https://phabricator.wikimedia.org/T437310#12299805 (10brouberol) [19:29:21] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Make Airflow Druid infrastructure more failproof and resilient - https://phabricator.wikimedia.org/T437310#12299806 (10brouberol) 05Open→03In progress [19:38:00] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12299850 (10Ottomata) >> I wonder if the domain event should capture ChangeTagAfterDelete (the tag itself was deleted) > A... [19:39:18] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Make Airflow Druid infrastructure more failproof and resilient - https://phabricator.wikimedia.org/T437310#12299870 (10brouberol) For the public druid cluster, @Stevemunene had deployed an LVS endpoint/VIP... [19:48:50] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12299923 (10Ottomata) change_tags can be associated with revisions, logs, and recentchanges. In this task description, I... [19:54:07] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:06:07] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:07:16] 06Data-Engineering, 06Product-Analytics, 06Product Safety and Integrity (Sprint Heather (Aug 31 - Sept 18)): Dbt backfill: signal_driven_handling_bad_faith_accounts - https://phabricator.wikimedia.org/T437095#12300277 (10amastilovic) Backfill completed. [21:42:44] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Java-Scala-Standardization, 07Essential-Work, 13Patch-For-Review: Decouple airflow-dags repository from Archiva - https://phabricator.wikimedia.org/T434160#12300479 (10amastilovic) [21:43:06] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Java-Scala-Standardization, 07Essential-Work, 13Patch-For-Review: Decouple airflow-dags repository from Archiva - https://phabricator.wikimedia.org/T434160#12300480 (10amastilovic) Task completed! [22:06:07] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly