[07:00:14] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10AbuseFilter, 06DBA, 07Schema-change-in-production: Drop the afl_ip column and the afl_ip_timestamp index from the abuse_filter_log table - https://phabricator.wikimedia.org/T407997#11348021 (10Marostegui) [07:01:46] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10AbuseFilter, 06DBA, 07Schema-change-in-production: Drop the afl_ip column and the afl_ip_timestamp index from the abuse_filter_log table - https://phabricator.wikimedia.org/T407997#11348026 (10Marostegui) [07:46:56] (03CR) 10Joal: [C:03+2] "LGTM - Merging" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1202334 (https://phabricator.wikimedia.org/T406000) (owner: 10Aleksandar Mastilovic) [07:54:53] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 07Essential-Work, 06Movement-Insights (FY25-26 H1): NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11348197 (10JAllemandou) Starting backfilling `pageview_hourly` hive ta... [08:02:47] (03Merged) 10jenkins-bot: Add appropriate NULL checks to computeForTuples [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1202334 (https://phabricator.wikimedia.org/T406000) (owner: 10Aleksandar Mastilovic) [09:32:53] 06Data-Engineering, 06Data-Engineering-Radar, 06Discovery-Search, 06Infrastructure-Foundations, and 3 others: Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#11348586 (10elukey) 05Open→03Resolved Spicerack deployed, thanks all for the work! [11:27:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:32:17] RESOLVED: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [12:09:41] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11349324 (10Michael) >>! In T401021#11343856... [13:55:39] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform, 13Patch-For-Review: [Event Platform] eventutilites-python: improve consistency guarantees of async process functions - https://phabricator.wikimedia.org/T347282#11349631 (10Ottomata) [13:56:01] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform, 13Patch-For-Review: [Event Platform] eventutilites-python: improve consistency guarantees of async process functions - https://phabricator.wikimedia.org/T347282#11349643 (10Ottomata) [13:58:43] (03PS1) 10Phuedx: Remove mediawiki.wikistories_* santization allowlist entries [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1202718 (https://phabricator.wikimedia.org/T408178) [14:03:30] !log Test Kitchen edge-unique experiment updates - adds: hcaptcha-on-french-wikipedia, fy2025-26-we3.1-image-browsing-ab-test; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:03:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:03:32] Sorry, I'm a non-interactive bot. . [14:03:40] !log Test Kitchen mw-user experiment updates - adds: growthexperiments-get-started-notification, synth-aa-test-mw-php; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:03:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:03:42] Sorry, I'm a non-interactive bot. . [14:04:21] (sorry about the irc logspam, stopping the bot) [14:20:42] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Iceberg Merge strategies with dbt - https://phabricator.wikimedia.org/T409099#11349767 (10JAllemandou) We met with @JMonton-WMF and found that DBT supports building new materializations: https://docs.getdbt.com/guides/create-new-materializations From a... [14:22:36] !log Test Kitchen edge-unique experiments (poll 1) - adds: fy2025-26-we3.1-image-browsing-ab-test, hcaptcha-on-french-wikipedia; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:22:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:22:46] !log Test Kitchen mw-user experiment (poll 1) - adds: synth-aa-test-mw-php, growthexperiments-get-started-notification; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:22:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:30:09] !log Test Kitchen mw-user experiment (poll 23) - adds: none; removes: synth-aa-test-mw-php; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:30:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:36:20] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Iceberg Merge strategies with dbt - https://phabricator.wikimedia.org/T409099#11349883 (10JAllemandou) I looked on the internet, and the idempotent-pipeline problem in DBT is a known and documented issue. My preferred article is https://techblog.finate... [14:40:19] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Cite, and 4 others: Remove Reference Previews tracking metrics - https://phabricator.wikimedia.org/T242127#11349912 (10phuedx) [14:42:15] !log Test Kitchen edge-unique experiments (poll 1) - adds: hcaptcha-on-french-wikipedia, fy2025-26-we3.1-image-browsing-ab-test; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:42:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:42:25] !log Test Kitchen mw-user experiment (poll 1) - adds: growthexperiments-get-started-notification; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:42:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:47:47] !log Test Kitchen edge-unique experiments (poll 1) - adds: fy2025-26-we3.1-image-browsing-ab-test, hcaptcha-on-french-wikipedia; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:47:49] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:47:57] !log Test Kitchen mw-user experiment (poll 1) - adds: growthexperiments-get-started-notification; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:47:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:03:42] FIRING: AlertLintProblem: Linting problems found for EventgateProduceRateStop - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:11:55] 06Data-Engineering, 10MobileFrontend, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog, 06Content-Transform-Team (Work In Progress): Add page_id and namespace to X-Analytics header in Mobile App requests (2025 remake) - https://phabricator.wikimedia.org/T409358#11350038 (10cscott) [15:17:29] 06Data-Engineering, 10MobileFrontend, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog, 06Content-Transform-Team (Work In Progress): Add page_id and namespace to X-Analytics header in Mobile App requests (2025 remake) - https://phabricator.wikimedia.org/T409358#11350083 (10cscott) I'm assumin... [15:18:42] FIRING: [2x] AlertLintProblem: Linting problems found for EventgateProduceRateStop - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:18:42] FIRING: [2x] AlertLintProblem: Linting problems found for EventgateProduceRateStop - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:30:28] 06Data-Engineering, 10Event-Platform: mediawiki.page_change.v1 - add a page type field - https://phabricator.wikimedia.org/T409462 (10Ottomata) 03NEW [16:30:45] 06Data-Engineering, 10Event-Platform: mediawiki.page_change.v1 event - add a page type field - https://phabricator.wikimedia.org/T409462#11350485 (10Ottomata) [16:31:57] 06Data-Engineering, 10Event-Platform: mediawiki.page_change.v1 event - add a page type field - https://phabricator.wikimedia.org/T409462#11350498 (10Ottomata) [16:33:42] RESOLVED: [2x] AlertLintProblem: Linting problems found for EventgateProduceRateStop - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:35:34] 06Data-Engineering, 10Event-Platform: mediawiki.page_change.v1 event - add a 'new revision created' field - https://phabricator.wikimedia.org/T409464 (10Ottomata) 03NEW [16:36:25] 06Data-Engineering, 06MW-Interfaces-Team, 10Event-Platform: mediawiki.page_change.v1 event stream - Investigate mistmatched meta.dt and dt (and rev_dt) fields - https://phabricator.wikimedia.org/T409105#11350539 (10Ottomata) If these are all caused by imports (are they? we should check for sure), then we sho... [16:39:24] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 3 others: Global Editor Metrics - HTTP API endpoints - https://phabricator.wikimedia.org/T405041#11350544 (10Ottomata) Update for most recent API endpoints: ` GE... [17:56:04] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Implement the data layout, UI, and documentation for the XML file export - https://phabricator.wikimedia.org/T401022#11350873 (10xcollazo) >>! In T401022#11350430, @CodeReviewBot wrote: > xcollazo **merged** https://gitlab.wikimed... [19:00:44] 06Data-Engineering, 10CirrusSearch, 06Data-Platform-SRE, 10DPE-Mediawiki-Content, and 4 others: Source the CirrusSearch index dumps from hadoop instead of a MW maintenance script - https://phabricator.wikimedia.org/T366248#11351150 (10EBernhardson) >>! In T366248#11343661, @brouberol wrote: > @EBernhardson... [19:19:47] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review: Global Editor Metrics - Data Persistence Design Review - https://phabricator.wikimedia.org/T401260#11351192 (10Ottomata) [19:33:20] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 10ConfirmEdit (CAPTCHA extension), 06Data-Persistence, and 3 others: Add columns to store associated log ID or revision ID that caused a signal to match - https://phabricator.wikimedia.org/T409093#11351255 (10Ladsgroup) The schema itself is fine. I t... [19:47:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:47:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [20:12:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [20:12:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [20:45:50] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 10ConfirmEdit (CAPTCHA extension), 06Data-Persistence, and 3 others: Add columns to store associated log ID or revision ID that caused a signal to match - https://phabricator.wikimedia.org/T409093#11351494 (10Dreamy_Jazz) >>! In T409093#11351255, @La... [21:41:26] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): mediawiki_history_dumps failure - https://phabricator.wikimedia.org/T409470#11351729 (10xcollazo) [21:41:43] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): mediawiki_history_dumps failure - https://phabricator.wikimedia.org/T409470#11351731 (10xcollazo) CC @JAllemandou [23:16:26] !log Test Kitchen edge-unique experiments (poll 1517) - adds: xlab-mw-module-loaded-v2; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [23:16:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [23:22:48] !log Test Kitchen edge-unique experiments (poll 1536) - adds: none; removes: none; fields: xlab-mw-module-loaded-v2 - xLab/MPIC/TK tips at https://w.wiki/FwuD [23:22:49] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log