[03:18:34] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:18:34] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:20:48] hello folks! [07:21:03] I have a change for a new eventgate stream for maps: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1167438 [07:21:15] not sure if there is anybody willing to check the change, lemme know in case :) [08:23:39] 06Data-Engineering, 06Experimentation Lab, 10Event-Platform: EventGate: Add Prometheus metric for hoisting errors - https://phabricator.wikimedia.org/T398922#10987312 (10phuedx) [11:16:27] Starting build #9 for job wikimedia-event-utilities-maven-release [11:18:34] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:24:34] Project wikimedia-event-utilities-maven-release build #9: 09SUCCESS in 8 min 7 sec: https://integration.wikimedia.org/ci/job/wikimedia-event-utilities-maven-release/9/ [13:14:01] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Dumps-Generation, 07Essential-Work: wikidata-20250707-all.json.gz is corrupted - https://phabricator.wikimedia.org/T399077#10988038 (10xcollazo) [13:52:41] FIRING: MediawikiPageContentChangeEnrichHighKafkaConsumerLag: ... [13:52:41] High Kafka consumer lag for mw_page_content_change_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichHighKafkaConsumerLag [14:17:56] RESOLVED: MediawikiPageContentChangeEnrichHighKafkaConsumerLag: ... [14:17:56] High Kafka consumer lag for mw_page_content_change_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichHighKafkaConsumerLag [14:18:37] 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list June 2025 - https://phabricator.wikimedia.org/T398149#10988403 (10mforns) This was done by Xabriel! Thanks for posting the task. [14:22:41] FIRING: [2x] MediawikiPageContentChangeEnrichHighKafkaConsumerLag: ... [14:22:41] High Kafka consumer lag for mw_page_content_change_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichHighKafkaConsumerLag [14:24:34] 07Analytics-Data-Problem, 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Movement-Insights: NEW BUG REPORT <"Domain" field issue: some domains have trailing dots> - https://phabricator.wikimedia.org/T395963#10988448 (10mforns) Agree, that they look like bots. We are working on an improvement to the bo... [14:27:41] RESOLVED: MediawikiPageContentChangeEnrichHighKafkaConsumerLag: ... [14:27:41] High Kafka consumer lag for mw_page_content_change_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichHighKafkaConsumerLag [15:15:36] 06Data-Engineering, 06DBA, 10MediaWiki-Page-derived-data, 13Patch-For-Review: Normalize categorylinks table - https://phabricator.wikimedia.org/T299951#10988691 (10Zabe) [15:18:34] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [18:26:58] 06Data-Engineering, 10Event-Platform: Automated event stream throughput alerting for important state change streams - https://phabricator.wikimedia.org/T329070#10989484 (10gmodena) [18:27:03] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Event-Platform, 13Patch-For-Review: Add alerting to eventbus and eventgate for drastic changes in event rate production. - https://phabricator.wikimedia.org/T398437#10989485 (10gmodena) [18:36:41] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to Product's Superset & Turnilo for SKivlehan - https://phabricator.wikimedia.org/T393626#10989496 (10Dzahn) Gotcha! Thanks @SKivlehan-WMF ! I did not see a new ticket. We can just reuse this one for the new request. I am adding the group requ... [18:36:52] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to Product's Superset & Turnilo for SKivlehan - https://phabricator.wikimedia.org/T393626#10989497 (10Dzahn) [18:38:44] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to Product's Superset & Turnilo for SKivlehan - https://phabricator.wikimedia.org/T393626#10989503 (10Dzahn) 05Stalled→03In progress [18:39:25] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to Product's Superset & Turnilo for SKivlehan - https://phabricator.wikimedia.org/T393626#10989504 (10Dzahn) Once access works for you feel free to just change the status to "resolved" here. [19:08:37] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10989593 (10Melderick) Confirming that this week dumps are present in https://dumps.wikimedia.org/wikidatawiki/entities/... [19:18:34] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:23:34] o/ Are historical wikimedia commons wikibase data dumps loaded anywhere in the analytics cluster? [19:31:09] hmmm, and just noticed I dont appear to be able to ssh into the stat machines any more [19:35:45] aaah https://github.com/wikimedia/operations-puppet/commit/045bef013b1cf09ceed1783402684681992ce2ef [19:38:04] further aaaaaaaaah https://phabricator.wikimedia.org/T397850 [20:41:32] 10Data-Engineering (Q4 2025 April 1st - June 30th): Data in mediawiki_page_content_change_v1 changing after being marked with _SUCCESS and _REFINE - https://phabricator.wikimedia.org/T398446#10989909 (10xcollazo) 05In progress→03Resolved [20:41:39] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Dumps-Generation: hrwiki-20250620-pagelinks.sql.gz dump not correct? - https://phabricator.wikimedia.org/T398265#10989910 (10xcollazo) 05In progress→03Resolved [20:41:50] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Data-Quality, 10DPE-Mediawiki-Content: Duplicate rows on wmf_content.mediawiki_content_history_v1 after big reconcile - https://phabricator.wikimedia.org/T397525#10989911 (10xcollazo) 05Open→03Resolved [23:18:34] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem