[00:15:13] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Build a set of configurable pre-scheduled DBT Airflow DAGs executing dbt-jobs models - https://phabricator.wikimedia.org/T419925 (10amastilovic) 03NEW [00:15:52] 14Analytics, 10Wikimedia-Affiliates-Data-Portal: Santuraki foundation founder - https://phabricator.wikimedia.org/T419926 (10Santurakifoundationfounder) 03NEW [01:03:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:03:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:08:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:08:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [05:08:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [05:08:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [08:45:49] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026-03-06 - 2026-03-27), 13Patch-For-Review: Make canary-events use a single airflow task per dg-run instead of one per stream - https://phabricator.wikimedia.org/T419540#11705462 (10Gehel) 05Open→03Resolved [08:52:35] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-03-06 - 2026-03-27), 13Patch-For-Review: Requesting Kerberos access for SCardenas (WMF) - https://phabricator.wikimedia.org/T418664#11705467 (10Gehel) Shell request is tracked on T419932 [09:08:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [09:08:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [10:38:09] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Build a set of configurable pre-scheduled DBT Airflow DAGs executing dbt-jobs models - https://phabricator.wikimedia.org/T419925#11705729 (10GGoncalves-WMF) Nice! This looks good functionality-wise, and thanks for writing out the process from a client tea... [10:43:41] 06Data-Engineering, 10MediaWiki-extensions-WikimediaEvents, 10Event-Platform: mw.track doesn't send beacons on leaving page in Chrome - https://phabricator.wikimedia.org/T419956 (10mszwarc) 03NEW [11:28:25] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Build a set of configurable pre-scheduled DBT Airflow DAGs executing dbt-jobs models - https://phabricator.wikimedia.org/T419925#11706046 (10JMonton-WMF) I think the conversation about these different setups and all their details can be summarized in two... [11:44:51] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Library for unified_diffs - https://phabricator.wikimedia.org/T419969 (10JMonton-WMF) 03NEW [12:59:31] 06Data-Engineering, 06ServiceOps new, 10ServiceOps-Mediawiki, 10ServiceOps-Upgrades-Hardware, 07Schema-change: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980 (10Raine) 03NEW [13:08:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:08:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:38:02] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): druid_load_webrequest_sampled_live_hourly SerDe error in singular DAG run - https://phabricator.wikimedia.org/T419121#11706564 (10Antoine_Quhen) A patch is in preparation to modify the root cause here: https://gerrit.wikimedia.org/g/operations/puppet/... [13:40:39] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Add new API rate limiting fields from webrequest_logs to Turnilo view - https://phabricator.wikimedia.org/T419736#11706586 (10HCoplin-WMF) Definitely #2 and #3. My experience with Superset is that it is borderline unusable because the performance issues r... [14:19:38] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Fix PyFlink log levels - https://phabricator.wikimedia.org/T419997 (10JMonton-WMF) 03NEW [14:31:31] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Library for unified_diffs - https://phabricator.wikimedia.org/T419969#11706914 (10JMonton-WMF) [14:32:34] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Library for unified_diffs - https://phabricator.wikimedia.org/T419969#11706928 (10Ottomata) > and removing it when it's being read. Yeah, and we want people to be able to easily apply the diff without access to our code. As is, they... [14:42:31] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Add new API rate limiting fields from webrequest_logs to Turnilo view - https://phabricator.wikimedia.org/T419736#11707000 (10GGoncalves-WMF) Thank you! Let's talk about the other uses you have for Turnilo later on :) > However, if we're creating a dedic... [14:47:22] 06Data-Engineering, 06ServiceOps new, 10ServiceOps-Mediawiki, 10ServiceOps-Upgrades-Hardware, 07Schema-change: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11707028 (10jcrespo) DBAs: Given the interactions with clouddbs you may want to be on top of that, as this... [15:02:44] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0: Alter AQS Cassandra tables in support of video plays endpoints - https://phabricator.wikimedia.org/T420008 (10Eevans) 03NEW [15:33:37] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0, 13Patch-For-Review: Alter AQS Cassandra tables in support of video plays endpoints - https://phabricator.wikimedia.org/T420008#11707321 (10Eevans) [15:43:36] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0, 13Patch-For-Review: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11707364 (10Eevans) @Snwachukwu I've opened [[ https://gerrit.wikimedia.org/r/c/generated-data-platform/aqs/media-analytics/+/... [15:55:29] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Add Fixture Generation for KubernetesPodOperator Tasks - https://phabricator.wikimedia.org/T419889#11707407 (10xcollazo) KubernetesPodOperator Fixture Coverage — Decision Summary... [16:09:14] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions: Fix Image suggestion DagProperty values - https://phabricator.wikimedia.org/T419204#11707518 (10Michael) I think this task probably does not need input from Growth, but please let me know if I'm misreading this. [16:17:06] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Upgrade mediawiki-event-enrichment jobs to >= Flink 1.20.2 and Java 17 - https://phabricator.wikimedia.org/T408918#11707546 (10JMonton-WMF) The workaround added in event utilities has been reve... [17:08:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:08:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:55:38] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 3 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11708103 (10BTullis) Another problem relating to the wikireplica views maintenance, as discovered by @fnegri in T414240. T... [18:24:36] (03CR) 10Mforns: "Looks great to me! I left 1 refactor option, and a suggestion for naming." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1250005 (https://phabricator.wikimedia.org/T415202) (owner: 10Snwachukwu) [18:32:17] 06Data-Engineering: table_maintenance_iceberg_monthly permission issue fails task due to permission on Ivy cache artifact - https://phabricator.wikimedia.org/T418804#11708186 (10xcollazo) >>! In T418804#11678791, @xcollazo wrote: > This had happened before when we run an airflow devenv with a personal user that... [18:32:29] 06Data-Engineering: table_maintenance_iceberg_monthly permission issue fails task due to permission on Ivy cache artifact - https://phabricator.wikimedia.org/T418804#11708187 (10xcollazo) rerunning failed tasks now.. [18:45:33] 06Data-Engineering: table_maintenance_iceberg_monthly permission issue fails task due to permission on Ivy cache artifact - https://phabricator.wikimedia.org/T418804#11708214 (10xcollazo) All tasks are green now. Now the more general issue is why this continues to happen if we are supposed to [[ https://gitlab.... [19:08:46] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Add Human-Bot Alert Runbook Link to Alert Email. - https://phabricator.wikimedia.org/T420046 (10Snwachukwu) 03NEW [19:16:12] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Datasets-General-or-Unknown: Get dump mirrors to use new dumps-rsync service name - https://phabricator.wikimedia.org/T415193#11708352 (10xcollazo) [[ https://meta.wikimedia.org/w/index.php?title=Mirroring_Wikimedia_project_XML_dumps&diff=30244205&old... [19:16:44] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Datasets-General-or-Unknown: Get dump mirrors to use new dumps-rsync service name - https://phabricator.wikimedia.org/T415193#11708355 (10xcollazo) 05Open→03Resolved [19:23:08] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Datasets-General-or-Unknown: Get dump mirrors to use new dumps-rsync service name - https://phabricator.wikimedia.org/T415193#11708397 (10taavi) Thank you!! [19:30:43] FIRING: [2x] HaproxykafkaNoMessages: [WARNING] - Haproxykafka on cp4050 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaNoMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxykafkaNoMessages [19:45:43] RESOLVED: HaproxykafkaNoMessages: [WARNING] - Haproxykafka on cp4050 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaNoMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4050 - https://alerts.wikimedia.org/?q=alertname%3DHaproxykafkaNoMessages [20:27:32] 06Data-Engineering, 10Data-Platform, 06Moderator-Tools-Team, 10PersonalDashboard, 06Product-Analytics (Kanban): Personal Dashboard Instrumentation Superset Dashboard - https://phabricator.wikimedia.org/T412137#11708611 (10MNeisler) I've made the following updates to the superset [[ https://superset.wikim... [20:28:49] 06Data-Engineering, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 06Growth-Team (FY2025-26 Q3 Sprint 5), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10): New accounts created from editor anon warning redirect to welcome survey, not back to ... - https://phabricator.wikimedia.org/T419723#11708619 [21:08:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:08:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:13:22] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Schedule three new monthly DBT models for Movement Insights - https://phabricator.wikimedia.org/T420069 (10amastilovic) 03NEW [22:40:21] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10PageViewInfo, 06ServiceOps new, 10ServiceOps-SharedInfra: Migrate PageViewInfo calls away from rest-gateway - https://phabricator.wikimedia.org/T411771#11709088 (10Ahoelzl) [22:42:23] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research, 10Event-Platform, 13Patch-For-Review: Event stream with latest revision HTML & parent revision HTML diff - https://phabricator.wikimedia.org/T360794#11709091 (10Ahoelzl) p:05Triage→03High