[00:04:36] RESOLVED: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:04:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:33:26] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12012935 (10RKemper) a:03RKemper Starting on this [03:23:35] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12012980 (10RKemper) @JMonton-WMF Okay, all 20 topics are deleted (each verified empty beforehand).... [06:21:15] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12013159 (10JMonton-WMF) Hi @RKemper, thanks for the detailed explanation! (1) I'm checking it but... [09:37:45] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: mw_content_reconcile_mw_content_history_monthly failed on rerun - https://phabricator.wikimedia.org/T428999#12013647 (10APizzata-WMF) error found in the logs of `application_1778834239566_465645` (wiki `zh_min_nanwiki`): ` error:... [10:00:57] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: mw_content_reconcile_mw_content_history_monthly failed on rerun - https://phabricator.wikimedia.org/T428999#12013719 (10xcollazo) I suspect this one is `zh_min_nanwiki ` bad data. Suggested follow up checks: Are there other insta... [10:50:23] (03CR) 10Xcollazo: [C:03+2] MWHistoryDeltaWriter: fix CAST format bug and stale comment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1300876 (https://phabricator.wikimedia.org/T428503) (owner: 10Xcollazo) [10:57:27] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#12013846 (10xcollazo) **Schema update — what changed since the original spec** Since the spec we... [11:00:47] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#12013875 (10xcollazo) **Current schema for `mediawiki_history_incremental_v1` (full reference)**... [11:03:56] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#12013880 (10xcollazo) 05Open→03Resolved (I'm now boldly closing this phase I task. There... [11:06:25] (03Merged) 10jenkins-bot: MWHistoryDeltaWriter: fix CAST format bug and stale comment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1300876 (https://phabricator.wikimedia.org/T428503) (owner: 10Xcollazo) [11:10:31] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add row_update_dt watermark column to mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T428503#12013899 (10xcollazo) (https://gerrit.wikimedia.org/r/c/analytics/refinery/source/... [11:13:57] !log Test Kitchen edge-unique experiments (poll 1) - adds: logged-out-retention-round12, share-highlight, we-1-8-mobile-account-menu, account-creation-reading-list-cta, logged-out-retention-round13, attribution-research-2026-06-run, logged-out-retention-round15, ncs-cors-test, logged-out-retention-round14; removes: none; fields: none - TK tips at https://w.wiki/FwuD [11:14:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:14:17] !log Test Kitchen edge-unique experiments (poll 2) - adds: logged-out-retention-round12, share-highlight, we-1-8-mobile-account-menu, account-creation-reading-list-cta, logged-out-retention-round13, attribution-research-2026-06-run, logged-out-retention-round15, ncs-cors-test, logged-out-retention-round14; removes: none; fields: none - TK tips at https://w.wiki/FwuD [11:14:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:14:37] !log Test Kitchen edge-unique experiments (poll 3) - adds: logged-out-retention-round12, share-highlight, we-1-8-mobile-account-menu, account-creation-reading-list-cta, logged-out-retention-round13, attribution-research-2026-06-run, logged-out-retention-round15, ncs-cors-test, logged-out-retention-round14; removes: none; fields: none - TK tips at https://w.wiki/FwuD [11:14:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:14:57] !log Test Kitchen edge-unique experiments (poll 4) - adds: logged-out-retention-round12, share-highlight, we-1-8-mobile-account-menu, account-creation-reading-list-cta, logged-out-retention-round13, attribution-research-2026-06-run, logged-out-retention-round15, ncs-cors-test, logged-out-retention-round14; removes: none; fields: none - TK tips at https://w.wiki/FwuD [11:15:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:40:42] 06Data-Engineering: Delete orphan EventLogging topic `eventlogging_HomepageModulet` - https://phabricator.wikimedia.org/T429017 (10Sgs) 03NEW [11:53:21] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: mw_content_reconcile_mw_content_history_monthly failed on rerun - https://phabricator.wikimedia.org/T428999#12014074 (10APizzata-WMF) Thanks for the suggestions @xcollazo! > Are there other instances of empty string page_titles on... [12:39:47] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Add row_update_dt watermark column to mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T428503#12014184 (10xcollazo) a:05xcollazo→03APizzata-WMF [12:53:07] 06Data-Engineering, 06Data-Platform-SRE: Delete orphan EventLogging topic `eventlogging_HomepageModulet` - https://phabricator.wikimedia.org/T429017#12014198 (10Ottomata) Hi TY! There are many kafka topics that are typos or are unused. Cleaning them up never hurts, but in practice it doesn't matter very mu... [13:01:15] !log Test Kitchen experiment (poll 1) - adds: incident_reporting_system_interaction, fy25-26-we-1-7-8-suggestion-mode-beta, we-1-10-articleguidance-v1, logged-out-retention-round15, attribution-research-2026-06-run, logged-out-retention-round14, ncs-cors-test, growthexperiments-revise-tone, logged-out-retention-round12, share-highlight, account-creation-reading-list-cta, we-1-8-mobile-account-menu, logged-out-retention-round13; remov [13:01:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:16:20] !log Test Kitchen experiment (poll 1) - adds: we-1-8-mobile-account-menu, incident_reporting_system_interaction, growthexperiments-revise-tone, fy25-26-we-1-7-8-suggestion-mode-beta, we-1-10-articleguidance-v1, logged-out-retention-round12, attribution-research-2026-06-run, share-highlight, ncs-cors-test, logged-out-retention-round14, logged-out-retention-round15, logged-out-retention-round13, acc... [13:16:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log