[04:03:42] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10SRE-SLO: page_change SLO windows - https://phabricator.wikimedia.org/T438054#12388089 (10RLazarus) Thanks @APizzata-WMF! Sorry to take so long to reply; I'm about to be out of office for a couple of weeks, so leaving some thoughts here before I go, b... [04:16:47] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE, 07Epic, 13Patch-For-Review: Upgrade to Spark 3.5 - https://phabricator.wikimedia.org/T338057#12388091 (10nshahquinn-wmf) [05:46:43] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12388109 (10ayounsi) Great! Please open a new ticket, the person in charge of access requests changes every 7 days (it'... [10:22:59] 06Data-Engineering, 06Data-Engineering-Radar, 06ServiceOps, 10ServiceOps-SharedInfra: Standard helm chart for simple service-utils nodejs apps - https://phabricator.wikimedia.org/T428174#12388697 (10MLechvien-WMF) [10:43:40] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: mediawiki-event-enrichment - set deterministic meta.id - https://phabricator.wikimedia.org/T424987#12388766 (10JMonton-WMF) I've added a simple function in the mediawiki-event-enrichment repo and used it for both... [11:10:27] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: User and team Hive database cleanup tracking for Fall 2026 - https://phabricator.wikimedia.org/T437733#12388835 (10xcollazo) [12:52:26] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): mw_content_history_reconcile_enrich api call returned 503 - https://phabricator.wikimedia.org/T415264#12389123 (10JMonton-WMF) The job's HTTP connection pool is hard-coded to 12 connections, but there is a parallelism of 2, so each worker can make 24 req... [13:13:02] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Reconcile Flink Job: Too many warnings - https://phabricator.wikimedia.org/T420353#12389168 (10JMonton-WMF) The job's HTTP connection pool is hard-coded to 12 connections, but there is a parallelism of 2, so each worker can make 24 requests at once. It p... [13:28:37] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: User and team Hive database cleanup tracking for Fall 2026 - https://phabricator.wikimedia.org/T437733#12389254 (10KCVelaga_WMF) Sorry for the delay in responding here. Sharing the findings here from the Slack thread with Xabriel:... [13:57:54] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: User and team Hive database cleanup tracking for Fall 2026 - https://phabricator.wikimedia.org/T437733#12389401 (10xcollazo) >>! In T437733#12389254, @KCVelaga_WMF wrote: > ... > * The other things is the location itself. > * Even... [14:17:42] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: User and team Hive database cleanup tracking for Fall 2026 - https://phabricator.wikimedia.org/T437733#12389474 (10xcollazo) [14:29:03] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): gitlab-mcp-server: Add read-only MCP server for GitLab merge requests and comments - https://phabricator.wikimedia.org/T438508#12389567 (10Ottomata) Just tried it! It works! [14:30:14] 06Data-Engineering, 06Data-Engineering-Radar, 06ServiceOps, 10ServiceOps-SharedInfra: Standard helm chart for simple service-utils nodejs apps - https://phabricator.wikimedia.org/T428174#12389587 (10Ottomata) We could probably decline this or merge into {T436785}, intending to use that. [14:31:18] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: mediawiki-event-enrichment - set deterministic meta.id - https://phabricator.wikimedia.org/T424987#12389615 (10Ottomata) Even KISSer why not! [14:55:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:55:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:00:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:00:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:27:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:27:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:59:38] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: User and team Hive database cleanup tracking for Fall 2026 - https://phabricator.wikimedia.org/T437733#12390217 (10KCVelaga_WMF) cleared `/user/hive/warehouse/kcvelaga_staging.db` and `/user/hive/warehouse/kcvelaga_wmf_traffic.db` -... [16:05:52] (03PS1) 10Aqu: dev-metrics: git interaction [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1348792 (https://phabricator.wikimedia.org/T433696) [17:02:03] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Apply Iceberg table maintenance to all dbt marts - https://phabricator.wikimedia.org/T439692#12390599 (10Ahoelzl) [17:02:04] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Figure out the biggest HDFS consumers over time - https://phabricator.wikimedia.org/T437605#12390600 (10Ahoelzl) [17:02:14] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Implement default expiration for staging and intermediate dbt models - https://phabricator.wikimedia.org/T439690#12390602 (10Ahoelzl) [17:02:16] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Figure out the biggest HDFS consumers over time - https://phabricator.wikimedia.org/T437605#12390603 (10Ahoelzl) [18:41:57] 14Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12390852 (10dr0ptp4kt) `an-druid1005` and `an-druid1007` are full. Landing here for keyword search. I've... [18:45:25] (03CR) 10Aleksandar Mastilovic: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1348792 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [18:55:52] 14Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12390865 (10dr0ptp4kt) Here are some directory-by-directory views on storage right now to get a feel. `... [19:27:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:27:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:45:35] 14Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12391011 (10dr0ptp4kt) 05Resolved→03Open [21:03:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:08:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:23:18] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): gitlab-mcp-server: Add read-only MCP server for GitLab merge requests and comments - https://phabricator.wikimedia.org/T438508#12391603 (10Ahoelzl) 05Open→03Resolved [21:23:19] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: New Schema for Mediawiki output job - https://phabricator.wikimedia.org/T437875#12391605 (10Ahoelzl) 05Open→03Resolved [21:23:20] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): dbt standardized workflow and repository structure - https://phabricator.wikimedia.org/T430622#12391604 (10Ahoelzl) 05Open→03Resolved [21:23:23] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Implement full parity between HiveSensor and RESTExternalTaskSensor - https://phabricator.wikimedia.org/T384726#12391608 (10Ahoelzl) 05Open→03Resolved [21:23:28] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Customizable dbt scheduled dependency sensors - https://phabricator.wikimedia.org/T430579#12391611 (10Ahoelzl) 05Open→03Resolved [21:23:30] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06RoadToWiki, 05Metrics-Sprint-2026-2027: Mirror datasets used on the key metrics dashboard to Postgres - https://phabricator.wikimedia.org/T438586#12391610 (10Ahoelzl) 05Open→03Resolved [21:23:32] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10AQS2.0, 10Data-Platform, 06Growth-Team, 07OKR-Work (DE1 FY2026-27): DE1.3.2: Implement "Sliding 30 Day Top-K Pages for Editor" - https://phabricator.wikimedia.org/T437710#12391613 (10Ahoelzl) 05Open→03Resolved [21:23:33] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Commons, 06Data-Persistence, 07OKR-Work: Data Lake - sqoop of commons link tables needs to adapt to move to dedicated MariaDB cluster - https://phabricator.wikimedia.org/T431476#12391612 (10Ahoelzl) 05Open→03Resolved [21:23:42] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Movement-Insights: WE3.3 Account Creation metrics dashboard showing lower numbers than expected - https://phabricator.wikimedia.org/T438385#12391618 (10Ahoelzl) 05Open→03Resolved [21:23:46] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 07Epic, 10Event-Platform: Relative Trending - https://phabricator.wikimedia.org/T425418#12391621 (10Ahoelzl) 05Open→03Resolved [21:23:50] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: mediawiki.page_change.v1 events - add wikidata id for pages - https://phabricator.wikimedia.org/T428176#12391619 (10Ahoelzl) 05Open→03Resolved [21:23:54] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Relative Trending - Milestone 4 - Storage - https://phabricator.wikimedia.org/T431555#12391622 (10Ahoelzl) 05Open→03Resolved [21:23:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Sqoop globalimagelinks and filerevision - https://phabricator.wikimedia.org/T427532#12391624 (10Ahoelzl) 05Open→03Resolved [21:24:02] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Flink - Test `zstd` compression - https://phabricator.wikimedia.org/T429456#12391626 (10Ahoelzl) 05Open→03Resolved [21:24:06] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10EditCheck, 06Editing-team, 10VisualEditor: Server Side Edit Suggestions - Linked Artifacts Cache precompute job - https://phabricator.wikimedia.org/T432733#12391625 (10Ahoelzl) 05Open→03Resolved [21:24:12] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Relative Trending: Documentation - https://phabricator.wikimedia.org/T438832#12391628 (10Ahoelzl) 05Open→03Resolved [21:24:16] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10AQS2.0, 10Data-Platform, 06Growth-Team, 07OKR-Work (DE1 FY2026-27): top_pages_per_editor returns implausible data for metawiki:Steward requests/Global permissions - https://phabricator.wikimedia.org/T434113#12391633 (10Ahoelzl) 05Open→03... [21:24:21] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Wikimedia Enterprise: WME Pageviews DAG for HDFS to S3 Transfer - https://phabricator.wikimedia.org/T426017#12391631 (10Ahoelzl) 05In progress→03Resolved [21:35:04] 14Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12391657 (10dr0ptp4kt) Cross-linking: {T440033} seemed to play a role in (probably) forcing more data onto su... [23:27:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [23:27:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag