[00:44:36] (03PS10) 10Ottomata: spark HiveExtensions now support column COMMENTs in DDL and merge helpers [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/987195 (https://phabricator.wikimedia.org/T307040) [05:05:00] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Requesting Kerberos access for sd - https://phabricator.wikimedia.org/T405219#11212705 (10SD0001) Got the email, and have reset the temporary password. Thanks! [08:08:45] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11212954 (10brouberol) Flagging [ml: add DAG to copy tone-check training data from HDFS to PVC](https://gitlab.wikimedia.or... [08:35:36] 06Data-Engineering, 06Data-Engineering-Radar, 10Page Content Service, 10RESTBase Sunsetting, and 3 others: Add page namespace information on resource change events - https://phabricator.wikimedia.org/T387435#11213057 (10Jgiannelos) 05Open→03Invalid After talking with the team it makes more sense to... [09:22:53] 06Data-Engineering, 06Product-Analytics, 10superset.wikimedia.org: Improve Superset's error message for draft dashboards - https://phabricator.wikimedia.org/T405535#11213242 (10GGoncalves-WMF) I took a look at the upstream issue tracker and found [[ https://github.com/apache/superset/issues/14175 | this issu... [09:48:55] (03CR) 10WMDE-leszek: [C:03+2] "Looks alright. I haven't checked if there are any remainders of "graphite" string in the repository (probably not, the change looks throro" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1190705 (owner: 10Lucas Werkmeister (WMDE)) [09:49:20] (03Merged) 10jenkins-bot: Remove last traces of Graphite, send data to Prometheus only [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1190705 (owner: 10Lucas Werkmeister (WMDE)) [10:18:30] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Clean up artifacts.yaml - https://phabricator.wikimedia.org/T405379#11213504 (10JAllemandou) >>! In T405379#11210081, @Ottomata wrote: >> This might be quite onerous on ops week duty and/or folks just trying to upgrade or deploy their job. > > Yes, but... [10:24:47] 06Data-Engineering: Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11213528 (10JAllemandou) This will have an impact on the `mediawiki_history` `identity-revert` field and related fields. We need to spend time on this @Ahoelzl . [10:26:23] (03PS1) 10Lucas Werkmeister (WMDE): Remove last traces of Graphite, send data to Prometheus only [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1191319 [10:26:29] (03CR) 10Lucas Werkmeister (WMDE): [C:03+2] Remove last traces of Graphite, send data to Prometheus only [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1191319 (owner: 10Lucas Werkmeister (WMDE)) [10:26:55] (03Merged) 10jenkins-bot: Remove last traces of Graphite, send data to Prometheus only [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1191319 (owner: 10Lucas Werkmeister (WMDE)) [10:37:44] 07Analytics-Data-Problem, 06Data-Engineering: Unique devices data uses non-standard domains for Wikidata, Wikifunctions, and MediaWiki.org - https://phabricator.wikimedia.org/T405533#11213571 (10JAllemandou) >>! In T405533#11212441, @nshahquinn-wmf wrote: > How difficult would it be simply to alter the Data La... [10:53:07] (03CR) 10Joal: "small nits. I like the change of approach :)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1191153 (https://phabricator.wikimedia.org/T384381) (owner: 10Xcollazo) [10:56:06] 06Data-Engineering: Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11213627 (10Ladsgroup) We are planning to do the drop in 30 days. Would that be enough time for the team? [11:03:43] 06Data-Engineering, 06Data-Persistence, 10MediaWiki-Core-Revision-backend, 10MW-1.45-notes (1.45.0-wmf.21; 2025-09-30), and 2 others: Rethink rev_sha1 field - https://phabricator.wikimedia.org/T389026#11213664 (10Zabe) a:03Zabe [13:18:30] 06Data-Engineering, 10Event-Platform: Schema validation for EventStreamConfig - https://phabricator.wikimedia.org/T405516#11214256 (10mpopov) >>! In T405516#11211895, @Ottomata wrote: > I suppose EventStreamConfig could have the logic for validating a provided schema, and the schema could live in mediawiki-con... [13:21:48] (03PS1) 10Snwachukwu: CREATE HQL SCRIPT TO UPDATE SCOOP WIKI LIST DATA FILE. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191398 [13:23:36] (03PS2) 10Pikne: Enable Estonian language [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1191391 [13:27:57] (03PS2) 10Snwachukwu: CREATE HQL SCRIPT TO UPDATE SCOOP WIKI LIST DATA FILE. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191398 [13:30:28] (03PS3) 10Pikne: Enable Estonian language [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1191391 [13:36:10] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11214325 (10xcollazo) >This all feels very achievable, but I wonder if we might be making things difficult for ourselves by... [13:45:05] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Clean up artifacts.yaml - https://phabricator.wikimedia.org/T405379#11214361 (10Ottomata) ^ wow that is the opposite I thought your opinion was! I still prefer the stability of versioned jobs, but my opinion is not strongly held. [13:47:20] (03CR) 10Xcollazo: Memory performance improvements for MediaWikiDumper (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1191153 (https://phabricator.wikimedia.org/T384381) (owner: 10Xcollazo) [13:52:56] 06Data-Engineering: Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11214411 (10Ottomata) FWIW, the `revision.content_slots.content_sha1` field is available in `mediawiki.page_change.v1`. If we had done {T258511} and Dumps 2 reconciliation focused on recon... [13:54:19] 06Data-Engineering: Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11214430 (10Ottomata) Anyway, it seems we do sqoop the `content` table which has `content_sha1` in it. I suppose the mediawiki_history algorithm just needs to join revision with the conte... [13:55:44] (03PS2) 10Xcollazo: Memory performance improvements for MediaWikiDumper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1191153 (https://phabricator.wikimedia.org/T384381) [14:08:23] 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Discovery-Search (2025.09.05 - 2025.09.26), and 2 others: Eventutilities Flink: port SerDe tests from SUP - https://phabricator.wikimedia.org/T404597#11214512 (10pfischer) [15:19:50] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform, 10Event-Platform: Warnings from handlePageHistoryVisibilityChangedEvent in EventBus from revision deletion - https://phabricator.wikimedia.org/T403648#11214986 (10Ottomata) Indeed. PageChangeEventIngress only emits visibilty change e... [16:38:18] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform, 10Event-Platform: Warnings from handlePageHistoryVisibilityChangedEvent in EventBus from revision deletion - https://phabricator.wikimedia.org/T403648#11215446 (10Ottomata) Oh, hm. Maybe not. getAffectedRevisionIDs is more appropria... [16:39:31] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11215452 (10JAllemandou) >>! In T405360#11214325, @xcollazo wrote: >>However, hdfs-rsync does not support remote SSH/SFTP t... [16:42:00] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Clean up artifacts.yaml - https://phabricator.wikimedia.org/T405379#11215459 (10JAllemandou) >>! In T405379#11214361, @Ottomata wrote: > ^ wow that is the opposite I thought your opinion was! I change my mind of various things lately :) [16:47:26] (03CR) 10Joal: [C:03+1] "LGTM!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1191153 (https://phabricator.wikimedia.org/T384381) (owner: 10Xcollazo) [16:56:20] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform, 10Event-Platform: Warnings from handlePageHistoryVisibilityChangedEvent in EventBus from revision deletion - https://phabricator.wikimedia.org/T403648#11215544 (10Ottomata) Actually, I think having the 'current revision' as PageHisto... [16:57:31] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform, 10Event-Platform: Warnings from handlePageHistoryVisibilityChangedEvent in EventBus from revision deletion - https://phabricator.wikimedia.org/T403648#11215550 (10Ottomata) I could work around this by first checking $event->wasCurren... [16:58:04] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for ericmill - https://phabricator.wikimedia.org/T404903#11215553 (10EMill-WMF) Yes, I can see the dashboards I was intending to now! Thank you very much for everyone who helped resolve my issue, and apologies f... [17:01:55] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for ericmill - https://phabricator.wikimedia.org/T404903#11215627 (10Dzahn) Thanks @EMill-WMF for confirming that. Great! And no need to apologize. The whole thing was about how that process is confusing even f... [17:23:24] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11215807 (10Dzahn) To be pragmatic.. let's just start with the lowest level of (that type of) access and see in practice if you run into any blockers. Level... [17:29:19] 06Data-Engineering: Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11215854 (10xcollazo) This will affect `wmf_content.mediawiki_content_history_v1` as well. And it will also affect {T384382}. Since [[ https://github.com/wikimedia/mediawiki/blob/14cb7ee... [17:33:59] (03CR) 10Xcollazo: [C:03+2] Memory performance improvements for MediaWikiDumper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1191153 (https://phabricator.wikimedia.org/T384381) (owner: 10Xcollazo) [17:37:41] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11215890 (10Dzahn) 05Open→03In progress a:03Dzahn [17:41:19] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11215925 (10Dzahn) To be pragmatic I am going with a "one level upgrade" from the lowest to the second lowest level here: https:... [17:43:05] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11215948 (10Dzahn) @TLessa-WMF Could you take a look at signing L3 while the code change I uploaded is in review? Cheers [17:45:06] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11215952 (10Dzahn) [17:46:37] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11215958 (10Dzahn) [17:49:13] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11215964 (10BTracy-WMF) That sounds like the right path forward. Thank you! [17:53:14] (03PS3) 10Snwachukwu: CREATE HQL SCRIPT TO UPDATE SCOOP WIKI LIST DATA FILE. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191398 (https://phabricator.wikimedia.org/T365203) [17:54:19] (03Merged) 10jenkins-bot: Memory performance improvements for MediaWikiDumper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1191153 (https://phabricator.wikimedia.org/T384381) (owner: 10Xcollazo) [17:59:29] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11216044 (10Dzahn) @TLessa-WMF We need one more thing. Please get your manager to approve here on this ticket. Thank you [18:01:31] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11216104 (10Dzahn) Great! I am going ahead. Meanwhile.. please check the box that you have read https://wikitech.wikimedia.org/wiki/Data_Platform/Data_acce... [18:05:37] 06Data-Engineering: Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11216226 (10xcollazo) [18:05:39] 06Data-Engineering, 10DPE-Mediawiki-Content, 07Epic: Production-level file export (aka dump) of MW Content in XML - https://phabricator.wikimedia.org/T384382#11216227 (10xcollazo) [18:07:54] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: Adapt MW Content pipelines to the removal of upstream revision.rev_sha1 - https://phabricator.wikimedia.org/T405641#11216287 (10xcollazo) [18:26:23] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: Adapt MW Content pipelines to the removal of upstream revision.rev_sha1 - https://phabricator.wikimedia.org/T405641#11216516 (10xcollazo) [18:26:40] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11216519 (10BTracy-WMF) [18:27:28] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11216524 (10BTracy-WMF) I've read the guidelines and updated this request to reflect. Thanks, again. [18:29:30] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11216543 (10Dzahn) Great. Thanks! I have uploaded a code change to make it happen and it's in review now. [18:34:49] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11216626 (10Dzahn) 05Open→03In progress [18:35:12] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11216632 (10Dzahn) a:03Dzahn [18:44:23] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Investigate reasons for remaining inconsistencies - https://phabricator.wikimedia.org/T385112#11216686 (10xcollazo) Currently re-running the september full reconcile ([[ https://airflow.wikimedia.org/dags/... [19:05:03] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review: Global Editor Metrics - Data Persistence Design Review - https://phabricator.wikimedia.org/T401260#11216766 (10Ottomata) Ah, @mforns, I reran your Pageviews that have pageviews in 1 day query but... [19:24:55] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11216876 (10amastilovic) > This all feels very achievable, but I wonder if we might be making things difficult for ourselve... [19:32:18] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11216903 (10CDanis) >>! In T405360#11216876, @amastilovic wrote: > Other than copying to multiple targets using one command... [19:32:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:32:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:37:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:37:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:38:33] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-rename, 10Event-Platform, 07Wikimedia-production-error: InvalidArgumentException: No page moved from 'File:Hospital sign.svg' to 'File:MUTCD D9-2.svg' with ID 1208082 could be fou... - https://phabricator.wikimedia.org/T387695#11216926 [19:39:46] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform, 10Event-Platform, 13Patch-For-Review: Warnings from handlePageHistoryVisibilityChangedEvent in EventBus from revision deletion - https://phabricator.wikimedia.org/T403648#11216928 (10Ottomata) I submitted a patch to MW core to add... [19:43:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:43:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [20:20:08] (03PS4) 10Snwachukwu: CREATE HQL SCRIPT TO UPDATE SCOOP WIKI LIST DATA FILE. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191398 [20:25:44] 06Data-Engineering: Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667 (10mforns) 03NEW [20:40:46] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11217235 (10Dzahn) Hey @BTracy-WMF You have been added to the `analytics-privatedata-users` group as requested. Let us know if you ne... [20:43:44] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11217244 (10Dzahn) 05In progress→03Resolved You can try out Superset now. [20:44:54] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11217247 (10Dzahn) [20:48:52] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11217261 (10Dzahn) a:05Dzahn→03cmadeo Hello @cmadeo Dayforce says you are the manager of @TLessa-WMF and to complete this acc... [20:59:50] 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list September 2025 - https://phabricator.wikimedia.org/T405672 (10GFontenelle_WMF) 03NEW [21:33:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:33:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:34:06] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:34:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:42:46] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Data Pipeline - https://phabricator.wikimedia.org/T405039#11217399 (10Ahoelzl) p:05Triage→03High [22:14:06] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:14:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:16:06] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:16:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:18:37] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11217559 (10TLessa-WMF) @Dzahn document signed, thank you so much for your help! @cmadeo for context here, I am trying to be ab... [22:31:00] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Traffic, 06Movement-Insights (FY25-26 H1): NEW BUG REPORT: Investigate rise in May 2025 Reader metrics - https://phabricator.wikimedia.org/T395934#11217591 (10Mayakp.wiki) Backfill is happening in T405667 [23:09:00] 06Data-Engineering: Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11217691 (10mforns) [23:28:03] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11217710 (10BTullis) We already use `rclone` for the out-of-band backups of our postgresql clusters behind the airflow inst... [23:46:06] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [23:46:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [23:47:06] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [23:47:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag