[00:01:31] 06Data-Engineering, 10Event-Platform: Remove eventlogging_ReferencePreviews* streams - https://phabricator.wikimedia.org/T409446#11453373 (10Ahoelzl) p:05Triage→03Medium a:03JMonton-WMF [00:03:40] 06Data-Engineering: Request Kerberos identity for Tchanders - https://phabricator.wikimedia.org/T411860#11453377 (10Ahoelzl) Approved. [00:03:56] 06Data-Engineering, 06Data-Platform-SRE: Request Kerberos identity for Tchanders - https://phabricator.wikimedia.org/T411860#11453378 (10Ahoelzl) [00:04:13] 06Data-Engineering, 06Data-Platform-SRE: Request Kerberos identity for Tchanders - https://phabricator.wikimedia.org/T411860#11453379 (10Ahoelzl) p:05Triage→03High [00:18:04] 06Data-Engineering: Airflow main performance instance optimization - https://phabricator.wikimedia.org/T411988#11453394 (10Ahoelzl) p:05Triage→03High [01:24:51] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:24:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [03:29:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [03:29:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [03:30:06] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [03:30:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [04:22:18] 06Data-Engineering, 10Dumps-Generation: Update dump mirror rsync allowlist to reflect new IP address for Scatter - https://phabricator.wikimedia.org/T409006#11453792 (10Harej) It works! Thank you [07:25:27] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Infrastructure-Foundations, 10netops: Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11453887 (10ayounsi) I removed the "Traffic direction" option from my previous comment, as Nokia replied saying that the... [07:28:56] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Infrastructure-Foundations, 10netops: Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11453895 (10ayounsi) @xcollazo what's the reason for the failure ? No disk space ? [07:30:21] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [07:30:21] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [09:01:01] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Increase partitions of mediawiki.content_history_reconcile.v1 - https://phabricator.wikimedia.org/T411598#11454057 (10JMonton-WMF) Thanks for the info @xcollazo. This is actually a good example for this ticket, even if we increase to 20 TaskManagers te... [11:30:21] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [11:30:21] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:21:27] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 06DBA, 13Patch-For-Review, and 2 others: Add sic_updated_timestamp column and associated indexes to the cusi_case table - https://phabricator.wikimedia.org/T411821#11454551 (10kostajh) [13:36:25] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Fix reconcile bug where user_id is not being populated correctly. - https://phabricator.wikimedia.org/T411803#11454602 (10xcollazo) Resuming all MW Content DAGs. [13:41:29] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11454623 (10Marostegui) [13:41:40] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11454628 (10Marostegui) [14:01:21] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Infrastructure-Foundations, 10netops: Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11454673 (10xcollazo) >>! In T412443#11453895, @ayounsi wrote: > @xcollazo what's the reason for the failure ? No disk s... [15:30:21] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:30:21] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:50:10] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.8 Deploy 100% sample rate instrument to a small wiki - https://phabricator.wikimedia.org/T412529 (10tchin) 03NEW [16:00:06] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:00:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:07:06] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:07:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:12:16] !log Test Kitchen edge-unique experiments (poll 12323) - adds: none; removes: fy25-26-we-1-1-19-mobile-section-dead-end; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [16:12:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:48:40] (03CR) 10Xcollazo: "It looks like we pivoted to have this DDL at https://gitlab.wikimedia.org/htriedman/wme-pageviews/-/merge_requests/1." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1215635 (https://phabricator.wikimedia.org/T409601) (owner: 10Snwachukwu) [17:27:26] !log Test Kitchen edge-unique experiments (poll 12547) - adds: fy25-26-we-1-1-19-mobile-section-dead-end-phase-2; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [17:27:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:58:56] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06MW-Interfaces-Team, 10Event-Platform: mediawiki.page_change.v1 event stream - Investigate mistmatched meta.dt and dt (and rev_dt) fields - https://phabricator.wikimedia.org/T409105#11455817 (10xcollazo) I think our model definitely has a gap if w... [20:07:21] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [20:07:21] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag