[00:23:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:23:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [00:28:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:28:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [00:29:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:29:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [00:34:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:34:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [00:36:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:36:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [00:41:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:41:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [00:45:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:45:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:00:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:00:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:05:09] 10Analytics-Canonical-Data, 06Movement-Insights, 06Product-Analytics: Create a structured list of Wikimedia projects' creation and closure dates - https://phabricator.wikimedia.org/T336999#11523697 (10nshahquinn-wmf) In the WMF Slack, @Michael just asked about getting wiki creation dates for some newer wikis... [01:06:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:06:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:21:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:21:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:23:56] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Technical assessment of AQS framework - https://phabricator.wikimedia.org/T414109#11523723 (10AKhatun_WMF) ##### Assessment [Here is an initial technical assessment](https://docs.google.com/document/d/1YQnLB0aUCMampd0O4qxifRl54GcCHutRDZOdEoPKiF4/edit?usp=... [01:25:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:25:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:30:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:30:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:31:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:31:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:41:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:41:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:42:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:42:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:52:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:52:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:55:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:55:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:00:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:00:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:26:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:26:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:31:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:31:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:33:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:33:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:48:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:48:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [03:05:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [03:05:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [03:10:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [03:10:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [03:32:20] 06Data-Engineering: [refine] Add support for custom Hive Iceberg partitioning - https://phabricator.wikimedia.org/T377600#11523793 (10Ottomata) [03:55:09] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Technical assessment of AQS framework - https://phabricator.wikimedia.org/T414109#11523819 (10Ottomata) Nice! I like the 4. Go vs Python/NodeJS section. As we consider if we want to invest in AQS, we should probably add one more option: **MediaWiki**... [03:55:31] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0: Technical assessment of AQS framework - https://phabricator.wikimedia.org/T414109#11523823 (10Ottomata) [04:12:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [04:12:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [04:22:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [04:22:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [04:38:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [04:38:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [04:43:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [04:43:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [05:42:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [05:42:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [05:57:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [05:57:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:14:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:14:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:19:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:19:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:21:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:21:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:29:15] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add il_target_id to imagelinks table in wmf production - https://phabricator.wikimedia.org/T413525#11523902 (10Marostegui) [06:31:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:31:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:37:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:37:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:42:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:42:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [07:28:50] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11523955 (10Marostegui) [07:32:11] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11523958 (10Marostegui) [08:46:51] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 13Patch-For-Review, and 2 others: Migrate 1 instrument using mw.eventLog.newInstrument() to mw.xLab.getInstrument() - https://phabricator.wikimedia.org/T408096#11524098 (10phuedx) Thanks @mpopov. I've updated [[ https://mpic.wikimed... [08:51:34] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add il_target_id to imagelinks table in wmf production - https://phabricator.wikimedia.org/T413525#11524104 (10Tacsipacsi) Thanks @Marostegui and @fnegri for figuring out what’s wrong and fixing the replicas! >>! In T413525#11521490, @fnegri wrote: >... [09:01:11] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add il_target_id to imagelinks table in wmf production - https://phabricator.wikimedia.org/T413525#11524133 (10Marostegui) >>! In T413525#11524104, @Tacsipacsi wrote: > Thanks @Marostegui and @fnegri for figuring out what’s wrong and fixing the replic... [09:30:12] 06Data-Engineering, 06Data-Platform-SRE, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for hmonroy - https://phabricator.wikimedia.org/T414375#11524193 (10JMeybohm) >>! In T414375#11523067, @HMonroy wrote: > @JMeybohm Hi! I'm trying a query wmf.mediawiki_history in superset. I'm... [10:02:40] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11524261 (10elukey) ` elukey@krb1002:~$ cat puppetserver_keytabs puppetserver1001.eqiad.wmnet,create_princ... [10:06:06] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 10Wikidata, 10Wikidata Analytics: NEW FEATURE REQUEST: sqoop (all) user properties from mariadb to wmf_raw.mediawiki_user_properties - https://phabricator.wikimedia.org/T323456#11524274 (10AndrewTavis_WMDE) Hi all 👋 Quick check in here to s... [11:08:01] 06Data-Engineering, 06Data-Persistence, 10GlobalBlocking, 06Product Safety and Integrity, and 2 others: Change type of gb_address to VARBINARY - https://phabricator.wikimedia.org/T414211#11524413 (10Dreamy_Jazz) [11:17:34] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11524439 (10elukey) While reviewing the last patch with Moritz, he made me realize that we'd need Java 8 on... [11:20:51] 06Data-Engineering, 06Data-Persistence, 10GlobalBlocking, 06Product Safety and Integrity, and 2 others: Change type of gb_address to VARBINARY - https://phabricator.wikimedia.org/T414211#11524453 (10Dreamy_Jazz) [13:43:58] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11524840 (10elukey) I had a chat with Ben about options for S3, and there is another possible road: > I w... [14:04:47] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11524915 (10BTullis) >>! In T402512#11139224, @brouberol wrote: > We could do this, however there's a fixed... [14:18:28] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11524969 (10MoritzMuehlenhoff) >>! In T402512#11524915, @BTullis wrote: >>>! In T402512#11139224, @broubero... [14:20:53] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11524987 (10BTullis) >>! In T402512#11524969, @MoritzMuehlenhoff wrote: >> How about [[https://airflow-oper... [15:04:44] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11525272 (10elukey) @BTullis let's proceed, thanks! I can also create them in case, lemme know what works best. [15:25:04] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: duplicated page_title in mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T413888#11525368 (10xcollazo) Fix has been deployed to production and ran via 2026-01-14 run of [[ https://airflow.wikimedia.org/dags/mw_conten... [15:50:23] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: duplicated page_title in mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T413888#11525444 (10xcollazo) A cursory look on the remaining 447 duplicate titles suggest that these are recently deleted and undeleted articl... [15:51:01] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: duplicated page_title in mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T413888#11525446 (10xcollazo) 05Open→03In progress p:05Triage→03Medium [17:01:46] (03PS1) 10Xcollazo: Drop DDL HQL for mediawiki_wikitext_* tables. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1227377 [17:02:24] (03PS2) 10Xcollazo: Drop DDL HQL for mediawiki_wikitext_* tables. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1227377 (https://phabricator.wikimedia.org/T396031) [17:09:51] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11525741 (10xcollazo) Ran the following to drop tables: ` a... [17:11:00] !log dropped deprecated wmf.mediawiki_wikitext_* tables. T396031. [17:11:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:11:04] T396031: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031 [17:21:15] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11525774 (10xcollazo) [17:25:17] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Investigate reasons for remaining inconsistencies - https://phabricator.wikimedia.org/T385112#11525780 (10xcollazo) 05In progress→03Resolved I'm being bold and closing this long standing ticket. We... [17:27:41] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: Another instance of duplicate rows on wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T404975#11525788 (10xcollazo) 05In progress→03Resolved Follow up work being done on T410431. Closing this... [18:04:22] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Add data-steward-alerts mail to anomaly_detection_traffic_distribution_daily DAG - https://phabricator.wikimedia.org/T414714 (10Ahoelzl) 03NEW [18:59:58] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for kareid - https://phabricator.wikimedia.org/T413364#11526053 (10thcipriani) >>! In T413364#11521115, @JMeybohm wrote: > @thcipriani this needs sign-off from you as the approver for the... [19:48:11] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07OKR-Work: SDS 2.2.6 Improve experiment event data data lake management - https://phabricator.wikimedia.org/T414105#11526261 (10xcollazo) Summary of meeting with @mpopov: * Mikhail explained to me the basic model of GrowthBook. * We went over an examp... [21:33:30] !log Test Kitchen edge-unique experiments (poll 5451) - adds: test-kitchen-mw-module-loaded; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [21:33:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:33:40] !log Test Kitchen mw-user experiment (poll 5451) - adds: synth-aa-test-mw-php-tk; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [21:33:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:34:41] !log Test Kitchen mw-user experiment (poll 5454) - adds: none; removes: synth-aa-test-mw-php-tk; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [21:34:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:34:51] !log Test Kitchen edge-unique experiments (poll 5455) - adds: none; removes: test-kitchen-mw-module-loaded; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [21:34:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:37:02] !log Test Kitchen mw-user experiment (poll 5461) - adds: synth-aa-test-mw-php-tk; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [21:37:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:37:12] !log Test Kitchen edge-unique experiments (poll 5462) - adds: test-kitchen-mw-module-loaded; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [21:37:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:46:28] 14Data-Engineering-Kanban, 10Cassandra, 06Data-Platform-SRE: Properly add aqsloader user (w/ secrets) - https://phabricator.wikimedia.org/T305600#11526835 (10Eevans) [21:51:22] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Persistence, 10Data-Persistence-Design-Review: Global Editor Metrics - Data Persistence Design Review - https://phabricator.wikimedia.org/T401260#11526869 (10Eevans) [21:51:41] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11526873 (10Eevans)