[02:05:26] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10Kafka-Infrastructure: beta - Run a single Kafka cluster in beta - https://phabricator.wikimedia.org/T437541 (10Ottomata) 03NEW [02:05:54] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Cloud-VPS (Debian Bullseye Deprecation), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12306081 (10Ottomata) Semi related: {T437541} [03:18:11] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Load Google Search Console AI impressions into the data lake - https://phabricator.wikimedia.org/T436573#12306191 (10Ahoelzl) We might have hit a road block here. My latest investigations suggest that ai impressions are subsumed into the overall url impr... [07:09:19] 06Data-Engineering, 10Data-Platform, 06Data-Platform-SRE, 07Epic: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12306453 (10JAllemandou) [07:13:49] 06Data-Engineering, 10Data-Platform, 06Data-Platform-SRE, 07Epic: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12306458 (10JAllemandou) >>! In T437472#12304529, @Ottomata wrote: >> Judge components, not projects. The parts that matter are the I... [10:03:20] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Research: Add new referrer-class for AI chatbots/LLMs to clickstream dataset - https://phabricator.wikimedia.org/T428136#12306984 (10Miriam) I will let @Khantstop confirm, but this is for a public data release and we won't necessarily need this chang... [10:21:49] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Growth-Team, 07Browser-Support-Apple-Safari: Investigate how Safari in iOS 17 and macOS Sonoma will impact URLs generated in Wikimedia sites - https://phabricator.wikimedia.org/T338571#12307062 (10Urbanecm_WMF) 05Open→03Declined... [10:28:15] 06Data-Engineering, 06Data-Engineering-Radar, 10Data-Platform, 06Growth-Team, and 6 others: Image Suggestions uses AI-generated images from Commons when adding images on English Wikipedia - https://phabricator.wikimedia.org/T422513#12307076 (10GGoncalves-WMF) Agree with Growth as product management owner.... [11:33:56] 06Data-Engineering, 10Dumps-Generation, 06serviceops-deprecated, 06SRE, and 2 others: conf* hosts ran out of disk space due to log spam - https://phabricator.wikimedia.org/T322360#12307345 (10MLechvien-WMF) [11:39:27] 14Analytics, 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06SRE, and 3 others: kafka eqiad cluster keeps crashing - https://phabricator.wikimedia.org/T199353#12307475 (10MLechvien-WMF) [11:39:29] 14Analytics, 06Data-Engineering, 06SRE, 10Wikidata, and 4 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451#12307471 (10MLechvien-WMF) [13:30:54] 06Data-Engineering, 10Experimentation Lab Roadmap, 10MediaWiki-extensions-EventLogging, 10Notifications (Echo), and 3 others: Deprecate EventLogging::logEvent() - https://phabricator.wikimedia.org/T318263#12308010 (10JTweed-WMF) [13:31:07] 06Data-Engineering, 10Experimentation Lab Roadmap, 10MediaWiki-extensions-EventLogging, 10Notifications (Echo), and 4 others: Deprecate EventLogging::logEvent() - https://phabricator.wikimedia.org/T318263#12308011 (10JTweed-WMF) [14:01:22] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Backfilled data lake dataset of page current/latest revision HTML - https://phabricator.wikimedia.org/T426347#12308117 (10Ottomata) [14:03:34] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Backfilled data lake dataset of page current/latest revision HTML - https://phabricator.wikimedia.org/T426347#12308124 (10Ottomata) [14:15:33] (03PS9) 10Snwachukwu: Add wmf_content.mediawiki_html_content_current_v0 table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1319873 (https://phabricator.wikimedia.org/T426347) [14:28:22] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Figure out the biggest HDFS consumers over time - https://phabricator.wikimedia.org/T437605 (10xcollazo) 03NEW [14:36:00] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Figure out the biggest HDFS consumers over time - https://phabricator.wikimedia.org/T437605#12308379 (10xcollazo) Context from other related work: {T417694} {T431391} {T381707} [14:46:22] (03PS1) 10Astein: add centralnoticebannerhistory_events hql [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1338956 [14:49:46] 06Data-Engineering: please merge hql/fr_tech/centralnoticebannerhistory_events.hql - https://phabricator.wikimedia.org/T437608 (10AStein-WMF) 03NEW [14:58:03] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Incremental MediaWiki History v2 - data requests - https://phabricator.wikimedia.org/T437051#12308539 (10Isaac) > But, we can't reconcile that. Ahh too expensive to update all the records? Or just on principle you d... [15:00:29] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel - https://phabricator.wikimedia.org/T425734#12308551 (10xcollazo) Answering @APizzata-WMF questions: ==== Q1: events with `event_t... [15:16:55] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Incremental MediaWiki History v2 - data requests - https://phabricator.wikimedia.org/T437051#12308620 (10xcollazo) >Some stats on how Mediawiki (MW) and Mediawiki History (MWH) align on this edit count value Very i... [15:24:18] 10Data-Engineering-Roadmap, 07Epic: Epic: dbt DPE work - https://phabricator.wikimedia.org/T416679#12308708 (10GGoncalves-WMF) [15:25:20] 06Data-Engineering, 07Epic: Epic: dbt Roadmap - https://phabricator.wikimedia.org/T437618 (10GGoncalves-WMF) 03NEW [15:25:51] 06Data-Engineering, 07Epic: Epic: dbt Roadmap - https://phabricator.wikimedia.org/T437618#12308737 (10GGoncalves-WMF) [15:25:52] 10Data-Engineering-Roadmap, 07Epic: Epic: dbt DPE work - https://phabricator.wikimedia.org/T416679#12308738 (10GGoncalves-WMF) [15:26:52] 06Data-Engineering, 07Epic: Epic: dbt Backlog - https://phabricator.wikimedia.org/T437619 (10GGoncalves-WMF) 03NEW [15:27:14] 06Data-Engineering, 07Epic: Epic: dbt Backlog - https://phabricator.wikimedia.org/T437619#12308756 (10GGoncalves-WMF) [15:27:16] 10Data-Engineering-Roadmap, 07Epic: Epic: dbt DPE work - https://phabricator.wikimedia.org/T416679#12308757 (10GGoncalves-WMF) [15:28:51] 06Data-Engineering, 07Epic: Epic: dbt Roadmap - https://phabricator.wikimedia.org/T437618#12308765 (10GGoncalves-WMF) [15:28:52] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt standardized workflow and repository structure - https://phabricator.wikimedia.org/T430622#12308764 (10GGoncalves-WMF) [15:28:53] 10Data-Engineering-Roadmap, 07Epic: Epic: dbt DPE work - https://phabricator.wikimedia.org/T416679#12308766 (10GGoncalves-WMF) [15:29:29] 06Data-Engineering, 07Epic: Epic: dbt Roadmap - https://phabricator.wikimedia.org/T437618#12308771 (10GGoncalves-WMF) [15:29:30] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt isolated model orchestration - https://phabricator.wikimedia.org/T430625#12308770 (10GGoncalves-WMF) [15:29:31] 10Data-Engineering-Roadmap, 07Epic: Epic: dbt DPE work - https://phabricator.wikimedia.org/T416679#12308772 (10GGoncalves-WMF) [15:29:44] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt data tests validate production data - https://phabricator.wikimedia.org/T430786#12308773 (10GGoncalves-WMF) [15:29:46] 06Data-Engineering, 07Epic: Epic: dbt Roadmap - https://phabricator.wikimedia.org/T437618#12308774 (10GGoncalves-WMF) [15:29:47] 10Data-Engineering-Roadmap, 07Epic: Epic: dbt DPE work - https://phabricator.wikimedia.org/T416679#12308775 (10GGoncalves-WMF) [15:34:58] (03PS1) 10Xcollazo: MWHistoryDeltaWriter: align fields with mediawiki_history [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1338971 (https://phabricator.wikimedia.org/T425734) [15:36:14] 06Data-Engineering: Investigate using --partial-parse for further isolating dbt models in different scheduling groups - https://phabricator.wikimedia.org/T436834#12308819 (10GGoncalves-WMF) [15:36:15] 06Data-Engineering, 07Epic: Epic: dbt Backlog - https://phabricator.wikimedia.org/T437619#12308820 (10GGoncalves-WMF) [15:40:34] (03CR) 10Ottomata: [C:03+2] add centralnoticebannerhistory_events hql [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1338956 (owner: 10Astein) [15:40:38] (03CR) 10Ottomata: [V:03+2 C:03+2] add centralnoticebannerhistory_events hql [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1338956 (owner: 10Astein) [15:53:30] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Incremental MediaWiki History v2 - data requests - https://phabricator.wikimedia.org/T437051#12308916 (10Isaac) > I think we independently found this issue and we discuss it a bit at T425734#12308551. TL;DR: we are... [15:55:47] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel - https://phabricator.wikimedia.org/T425734#12308941 (10Isaac) > It still matters, because size() on this col... [16:04:25] (03CR) 10Mforns: [V:03+2 C:03+2] "Merging!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1337882 (https://phabricator.wikimedia.org/T437277) (owner: 10Mforns) [16:38:48] (03CR) 10Joal: [V:03+2 C:03+2] "Merge for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1308104 (https://phabricator.wikimedia.org/T431256) (owner: 10Joal) [17:50:20] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Engineering-Roadmap, 07Epic, 13Patch-For-Review: Create, orchestrate and backfill wmf.projectview_legacy_hourly table - https://phabricator.wikimedia.org/T437277#12309395 (10mforns) I created the legacy table, copied all the existing product... [17:50:27] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Engineering-Roadmap, 07Epic, 13Patch-For-Review: Create, orchestrate and backfill wmf.projectview_legacy_hourly table - https://phabricator.wikimedia.org/T437277#12309396 (10mforns) [18:12:48] !log Deploying Refinery at 92b5c04e [18:12:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:15:28] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10Kafka-Infrastructure: beta - Run a single Kafka cluster in beta - https://phabricator.wikimedia.org/T437541#12309448 (10tchin) I moved eventstreams and eventgate to consume from main. Seems to work fine and @Sfaci inadvertently helped me test it. I complet... [18:18:58] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Capture relevant mediawiki tags changes in event stream(s) - https://phabricator.wikimedia.org/T434143#12309479 (10Ottomata) [[ https://wikimedia.slack.com/archives/CLKDS4MG9/p1788980748134409 | From slack ]], it... [18:26:44] 06Data-Engineering: please merge hql/fr_tech/centralnoticebannerhistory_events.hql - https://phabricator.wikimedia.org/T437608#12309499 (10Ottomata) a:03Ottomata [18:26:45] 06Data-Engineering: please merge hql/fr_tech/centralnoticebannerhistory_events.hql - https://phabricator.wikimedia.org/T437608#12309500 (10Ottomata) 05Open→03Resolved [18:36:05] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Figure out the biggest HDFS consumers over time - https://phabricator.wikimedia.org/T437605#12309537 (10xcollazo) **TL;DR** - The 0.5 PB/month rate is the Jul-to-Aug rate. It is not the trend. The 6-month rate is 0.152 PB/month. The 23-month rate is 0.1... [18:36:35] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Add a new `is_api` field to webrequest_sampled in druid - https://phabricator.wikimedia.org/T437638 (10JAllemandou) 03NEW [18:49:26] (03PS1) 10Joal: Add transform to kafka-druid webrequest_sampled [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1339067 (https://phabricator.wikimedia.org/T437638) [19:35:45] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Incremental MediaWiki History v2 - data requests - https://phabricator.wikimedia.org/T437051#12309753 (10Ottomata) > But, we can't reconcile that. >> Ahh too expensive to update all the records? Or just on principle... [19:37:41] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10Kafka-Infrastructure: beta - Run a single Kafka cluster in beta - https://phabricator.wikimedia.org/T437541#12309778 (10Ottomata) I think we can keep kafka-logging; we don't mirror from it into jumbo (by default) like we did from kafka main. Also kafka lo... [19:37:50] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10Kafka-Infrastructure: beta - Run a single event platform Kafka cluster in beta - https://phabricator.wikimedia.org/T437541#12309782 (10Ottomata) [19:40:05] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Add a new `is_api` field to webrequest_sampled in druid - https://phabricator.wikimedia.org/T437638#12309808 (10Ottomata) Hm, how urgent is this? We are working with RDS to redefine the meaning of 'access_method'. It would be nice... [20:03:02] (03CR) 10Xcollazo: "This table could eventually drift due to missing events. Is that ok?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1319873 (https://phabricator.wikimedia.org/T426347) (owner: 10Snwachukwu) [20:14:34] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Backfilled data lake dataset of page current/latest revision HTML - https://phabricator.wikimedia.org/T426347#12309892 (10xcollazo) As per [[ https://wikimedia.slack.com/archives/C05RHK7PS6Q/p1789065939534419?thr... [20:27:40] 06Data-Engineering, 06Data-Engineering-Radar, 10Data-Platform, 06Growth-Team, and 6 others: Image Suggestions uses AI-generated images from Commons when adding images on English Wikipedia - https://phabricator.wikimedia.org/T422513#12309930 (10KStoller-WMF) [20:47:21] (03CR) 10Snwachukwu: "The backfill is done via internal REST API endpoint. There is a python script for it. There was a discussion about the best place to keep " [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1319873 (https://phabricator.wikimedia.org/T426347) (owner: 10Snwachukwu) [20:49:09] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Figure out the biggest HDFS consumers over time - https://phabricator.wikimedia.org/T437605#12310014 (10Isaac) Thanks for tackling this! I personally think it's not so bad for us to do some cleaning semi-regularly. Quick thoughts: * Reduce the number of... [21:10:10] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:25:10] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly