[01:17:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:17:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [05:17:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [05:17:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [05:47:54] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11577308 (10Marostegui) [05:48:07] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11577309 (10Marostegui) [05:49:49] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11577315 (10Marostegui) [06:03:45] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577337 (10Marostegui) [06:09:44] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577347 (10Marostegui) [06:16:38] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577362 (10Marostegui) [06:20:06] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11577365 (10Marostegui) [06:20:13] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11577366 (10Marostegui) [06:20:22] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11577367 (10Marostegui) 05Open→03Resolved Finally all done! [06:20:29] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11577370 (10Marostegui) 05Open→03Resolved Finally all done! [06:23:30] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577395 (10Marostegui) [06:25:46] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577413 (10Marostegui) [06:27:02] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577417 (10Marostegui) [06:30:50] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577433 (10Marostegui) [06:51:50] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577472 (10Marostegui) [06:56:17] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577475 (10Marostegui) [06:59:47] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577476 (10Marostegui) [07:09:40] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577505 (10Marostegui) [09:17:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [09:17:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [09:48:48] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577792 (10Marostegui) [09:54:03] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11577826 (10Marostegui) @Zabe any ETA on when I can start with s4? [10:13:36] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: Use wmf.mediawiki_history as baseline for slo completeness - https://phabricator.wikimedia.org/T416312 (10APizzata-WMF) 03NEW [10:37:15] 06Data-Engineering: Fix db_cleaner for small instances - https://phabricator.wikimedia.org/T416315 (10Antoine_Quhen) 03NEW [10:49:55] 06Data-Engineering, 06Machine-Learning-Team, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11578057 (10achou) Answering @gkyziridis's questions: > … what kind of optimization do you have in mind? I meant o... [10:59:20] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 07Technical-Debt, 06Test Kitchen (Experiment Platform Sprint 19): Migrate 1 instrument using mw.eventLog.newInstrument() to mw.xLab.getInstrument() - https://phabricator.wikimedia.org/T408096#11578099 (10phuedx) > [] After the in... [11:00:25] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 07Technical-Debt, 06Test Kitchen (Experiment Platform Sprint 19): Migrate 1 instrument using mw.eventLog.newInstrument() to mw.xLab.getInstrument() - https://phabricator.wikimedia.org/T408096#11578106 (10phuedx) [11:03:22] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11578125 (10Marostegui) [11:20:29] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11578187 (10Zabe) >>! In T415786#11577826, @Marostegui wrote: > @Zabe any ETA on when I can start with s4? The script will need another ~2 weeks to finish on... [11:22:44] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11578198 (10Marostegui) >>! In T415786#11578187, @Zabe wrote: >>>! In T415786#11577826, @Marostegui wrote: >> @Zabe any ETA on when I can start with s4? > >... [11:47:11] !log Restarting druid-middlemanager services on an-druid100[3-7] - removing the extra GC logging for T415799 [11:47:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:47:28] T415799: Since upgrade Druid realtime-ingestion tasks replication fails - https://phabricator.wikimedia.org/T415799 [12:08:53] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11578319 (10Marostegui) [12:49:14] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Reader Growth Team, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog, and 2 others: Add page_id and namespace to X-Analytics header in Mobile App requests (2025 remake) - https://phabricator.wikimedia.org/T409358#11578430 (10Jgiannelos... [13:08:38] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 13Patch-For-Review: Batch index webrequest_sampled data in Druid - https://phabricator.wikimedia.org/T415359#11578504 (10JAllemandou) [13:09:01] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1235740 (https://phabricator.wikimedia.org/T415359) (owner: 10Joal) [13:17:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:17:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [14:56:48] 06Data-Engineering, 07Sustainability: Move some analytics jobs to day time in Virginia - https://phabricator.wikimedia.org/T384166#11578943 (10Ladsgroup) >>! In T384166#11565913, @xcollazo wrote: > Since the vast majority of analytic jobs are scheduled via Airflow, we could certainly change the `schedule` cron... [15:14:12] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11579004 (10Marostegui) [16:00:06] (03CR) 10Joal: [C:03+1] "The code is good, some comments would be great :)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1235830 (https://phabricator.wikimedia.org/T416176) (owner: 10Xcollazo) [16:50:45] 06Data-Engineering, 06Data-Engineering-Radar, 06Test Kitchen, 07Essential-Work, 10Event-Platform: X-Experiment-Enrollments EventGate handling reinforcement for MalformedHeaderError cases - https://phabricator.wikimedia.org/T409106#11579564 (10KReid-WMF) [16:54:29] 06Data-Engineering, 06Data-Engineering-Radar, 06Test Kitchen, 07Essential-Work, 10Event-Platform: More strictly validate X-Experiment-Enrollments-Header - https://phabricator.wikimedia.org/T401198#11579587 (10KReid-WMF) →14Duplicate dup:03T409106 [16:54:31] 06Data-Engineering, 06Data-Engineering-Radar, 06Test Kitchen, 07Essential-Work, 10Event-Platform: X-Experiment-Enrollments EventGate handling reinforcement for MalformedHeaderError cases - https://phabricator.wikimedia.org/T409106#11579589 (10KReid-WMF) [16:55:30] 06Data-Engineering, 06Data-Engineering-Radar, 06Test Kitchen, 07Essential-Work, 10Event-Platform: X-Experiment-Enrollments EventGate handling reinforcement for MalformedHeaderError cases - https://phabricator.wikimedia.org/T409106#11579596 (10KReid-WMF) See {T401198} for additional considerations. [16:57:42] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Technical-Debt: Deprecate and remove mw.eventLog.submitInteraction() - https://phabricator.wikimedia.org/T415362#11579603 (10JVanderhoop-WMF) p:05Triage→03High [17:02:23] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work, 07Technical-Debt: Decouple mw.xLab.getInstrument() and mw.eventLog.newInstrument() - https://phabricator.wikimedia.org/T408094#11579634 (10JVanderhoop-WMF) p:05Triage→03High [17:17:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:17:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:17:52] (03PS1) 10Snwachukwu: Migrate cu_changes table to use cuua_text in new cu_usergent table. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1236347 (https://phabricator.wikimedia.org/T361210) [18:26:57] (03PS2) 10Xcollazo: MediawikiDumper: fix filenames to include end revision when covering a single page. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1235830 (https://phabricator.wikimedia.org/T416176) [18:28:06] (03CR) 10Xcollazo: "@joal@wikimedia.org for your re-review." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1235830 (https://phabricator.wikimedia.org/T416176) (owner: 10Xcollazo) [19:10:35] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13): Batch index webrequest_sampled data in Druid - https://phabricator.wikimedia.org/T415359#11580377 (10JAllemandou) All code has been merged but I paused the job as the newly indexed data shows a discrepancy... [19:17:32] (03PS1) 10Joal: Fix druid webrequest_sampled temporary data job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1236351 (https://phabricator.wikimedia.org/T415359) [19:17:58] (03CR) 10Joal: [V:03+2 C:03+2] "Hotfix, merging" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1236351 (https://phabricator.wikimedia.org/T415359) (owner: 10Joal) [19:23:58] (03CR) 10Joal: [C:03+1] "This is great :) Thanks a lot Xabriel!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1235830 (https://phabricator.wikimedia.org/T416176) (owner: 10Xcollazo) [19:38:44] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13): Batch index webrequest_sampled data in Druid - https://phabricator.wikimedia.org/T415359#11580520 (10JAllemandou) Bug found, I had inverted typed for the two dedicated fields, making `time_firstbyte` a bigi... [19:39:16] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13): Batch index webrequest_sampled data in Druid - https://phabricator.wikimedia.org/T415359#11580534 (10JAllemandou) 05Open→03Resolved [19:44:59] (03CR) 10Xcollazo: [C:03+2] MediawikiDumper: fix filenames to include end revision when covering a single page. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1235830 (https://phabricator.wikimedia.org/T416176) (owner: 10Xcollazo) [19:55:05] 06Data-Engineering, 06Machine-Learning-Team, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11580587 (10Ottomata) > we may need to produce predictions to a separate stream instead of mediawiki.page_revert_r... [19:59:24] (03Merged) 10jenkins-bot: MediawikiDumper: fix filenames to include end revision when covering a single page. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1235830 (https://phabricator.wikimedia.org/T416176) (owner: 10Xcollazo) [20:54:00] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): File Export files with revision ranges do not show end revision in filename - https://phabricator.wikimedia.org/T416176#11580897 (10xcollazo) @Msz2001, the fix for this issue has been merged, but most of the wikis have already been exported for the 2026-0... [21:17:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:17:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:53:55] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10MediaWiki-extensions-WikimediaEvents, 07Essential-Work, and 3 others: Update name in EventLogging Extension and WikimediaEvents Extension - https://phabricator.wikimedia.org/T407904#11581258 (10Sfaci) [22:54:35] 06Data-Engineering, 10Dumps-Generation: SQL metadata files problem - https://phabricator.wikimedia.org/T416416 (10Novem_Linguae) 03NEW [22:55:07] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10MediaWiki-extensions-WikimediaEvents, 07Essential-Work, and 3 others: Update name in EventLogging Extension and WikimediaEvents Extension - https://phabricator.wikimedia.org/T407904#11581274 (10Sfaci) [22:58:24] 06Data-Engineering, 10Dumps-Generation: SQL metadata files problem - https://phabricator.wikimedia.org/T416416#11581290 (10Novem_Linguae)