[00:04:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:04:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [00:59:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:59:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:04:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:04:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:11:59] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027: Backfill and update schedule of gcp_gsc_export.max_date_sitedata_daily_aggregate - https://phabricator.wikimedia.org/T436999#12288165 (10Ahoelzl) p:05Triage→03High a:03Ahoelzl [02:01:23] 10Data-Engineering-Roadmap, 06Data-Persistence, 10MediaWiki-Page-derived-data, 07Community-Wishlist, and 2 others: Add page_is_redirect/page_namespace/page_title index - https://phabricator.wikimedia.org/T387537#12288184 (10Pppery) Not sure what wish was being referenced in T387537#10780663, however see ht... [02:14:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:14:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:19:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:19:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [03:04:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:09:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:19:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:19:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [07:09:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [07:09:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [09:24:02] 06Data-Engineering, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): [Tech evaluation] Table catalog options for the S3 data lake - https://phabricator.wikimedia.org/T435604#12288785 (10BTullis) a:03JAllemandou Assigning to @JAllemandou for guidance at this stage. [10:15:19] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel - https://phabricator.wikimedia.org/T425734#12288965 (10APizzata-WMF) During the first phase of the analysis @JAllemandou and I not... [11:09:51] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [11:09:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:34:31] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: MediaWiki state change events - emit global edit count - https://phabricator.wikimedia.org/T430967#12289820 (10Ottomata) [13:34:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:34:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:42:51] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History: Incremental MediaWiki History v2 - data requests - https://phabricator.wikimedia.org/T437051 (10Ottomata) 03NEW [13:45:12] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Ops week: Daily firing of sla miss alert for Mediarequest dag - https://phabricator.wikimedia.org/T437052 (10Snwachukwu) 03NEW [13:46:07] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History: Incremental MediaWiki History v2 - data requests - https://phabricator.wikimedia.org/T437051#12289885 (10Ottomata) [13:47:20] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Ops week: Daily firing of sla miss alert for Mediarequest dag - https://phabricator.wikimedia.org/T437052#12289887 (10Snwachukwu) Fix: I added an early-month grace period to snapshot selection. Any run before the 6th of the month use the already-availab... [13:49:27] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Impact Module Metrics - Sliding 30 Day Top-K Pages for Editor - https://phabricator.wikimedia.org/T436368#12289893 (10Snwachukwu) [13:49:48] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Impact Module Metrics - Sliding 30 Day Top-K Pages for Editor - https://phabricator.wikimedia.org/T436368#12289894 (10Snwachukwu) [13:54:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Ops week: Daily firing of sla miss alert for Mediarequest dag - https://phabricator.wikimedia.org/T437052#12289896 (10Snwachukwu) 05Open→03Resolved [13:55:49] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Ops Week: Increase wait time of webrequest_actor_metrics_hourly sensor task before it fails - https://phabricator.wikimedia.org/T430247#12289898 (10Snwachukwu) 05Open→03Resolved [14:03:15] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): wmf_contributors.edit_per_editor_per_page_daily claims `Martin Urbanec` is a system user - https://phabricator.wikimedia.org/T434217#12289921 (10Ottomata) Ah good finds! Hm, couple of thoughts: After the backfill, this table will have had new data load... [14:24:45] 06Data-Engineering, 06Product-Analytics: Airflow DAG: refine_api_requests_hourly - use artifact approach for rules.json - https://phabricator.wikimedia.org/T437058 (10KCVelaga_WMF) 03NEW [14:51:11] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Commons, 06Data-Persistence, 07OKR-Work, 13Patch-For-Review: Data Lake - sqoop of commons link tables needs to adapt to move to dedicated MariaDB cluster - https://phabricator.wikimedia.org/T431476#12290131 (10Ladsgroup) If this is about sqoops... [15:04:13] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): wmf_contributors.edit_per_editor_per_page_daily claims `Martin Urbanec` is a system user - https://phabricator.wikimedia.org/T434217#12290190 (10Snwachukwu) @Ottomata if i use `event.mediawiki_page_change_v1`, We have 253,367 editors that are in the back... [17:13:07] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:18:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:19:42] 06Data-Engineering, 06Data-Platform-SRE: [Tech evaluation] How we import the MediaWiki databases into the S3 data lake - https://phabricator.wikimedia.org/T437079 (10BTullis) 03NEW [17:35:33] 06Data-Engineering, 06Data-Platform-SRE: Data Lake ingestion after Hadoop: decide the future of Sqoop and Gobblin - https://phabricator.wikimedia.org/T435489#12290829 (10BTullis) [17:55:24] (03PS7) 10Snwachukwu: Add wmf_content.mediawiki_html_content_current_v0 table and backfill [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1319873 [18:24:33] (03PS1) 10Snwachukwu: Fix pageview_per_editor_per_page_daily inflating view counts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1335995 (https://phabricator.wikimedia.org/T434113) [18:29:52] (03PS2) 10Snwachukwu: Fix pageview_per_editor_per_page_daily inflating view counts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1335995 (https://phabricator.wikimedia.org/T434113) [18:36:27] (03PS3) 10Snwachukwu: Fix pageview_per_editor_per_page_daily inflating view counts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1335995 (https://phabricator.wikimedia.org/T434113) [18:56:47] (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1335995 (https://phabricator.wikimedia.org/T434113) (owner: 10Snwachukwu) [19:31:16] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History: Incremental MediaWiki History v2 - data requests - https://phabricator.wikimedia.org/T437051#12291108 (10Isaac) A few that I haven't linked up yet as child tasks to v2: * Non-trivial but I'll mention for the sake of completeness: {T434709} * Expand block... [19:56:28] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: mediawiki.page_change.v1 events - add wikidata id for pages - https://phabricator.wikimedia.org/T428176#12291249 (10gmodena) >>! In T428176#12228480, @Ottomata wrote: > Wow... [21:47:33] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE: add x4 to the MediaWiki replicas - https://phabricator.wikimedia.org/T423980#12291608 (10bd808) https://lists.wikimedia.org/hyperkitty/list/cloud@lists.wikimedia.org/thread/E4RXH3F3WTBWSGZDN6BZPJGEUY2I... [21:49:12] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE: add x4 to the MediaWiki replicas - https://phabricator.wikimedia.org/T423980#12291630 (10bd808) p:05Triage→03High I know that priority is subjective and should be controlled by people doing the wor... [21:58:38] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE: add x4 to the MediaWiki replicas - https://phabricator.wikimedia.org/T423980#12291649 (10Novem_Linguae) This ticket is for putting x4 on the private mediawiki replicas (e.g. stat1008). It sounds like y... [22:01:54] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE: add x4 to the MediaWiki replicas - https://phabricator.wikimedia.org/T423980#12291652 (10Novem_Linguae) {T431309} looks related to putting x4 on the wiki replicas, but doesnt quite cover everything. Pe... [22:23:06] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE: add x4 to the MediaWiki replicas - https://phabricator.wikimedia.org/T423980#12291671 (10bd808) >>! In T423980#12291649, @Novem_Linguae wrote: > This ticket is for putting x4 on the private mediawiki r... [23:26:26] 06Data-Engineering: Backfill wmf_objective_metrics.heartbeat_active_editors for June 2026 - https://phabricator.wikimedia.org/T436886#12291790 (10nshahquinn-wmf) 05Open→03Declined This table has been superseded by `wmf_objective_metrics.heartbeat_active_editors_weekly`. [23:28:35] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Backfill wmf_objective_metrics.heartbeat_active_editors_weekly - https://phabricator.wikimedia.org/T437118 (10nshahquinn-wmf) 03NEW