[03:06:16] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [03:06:16] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - . [03:06:16] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [04:54:07] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Load Google Search Console data into the Data Lake - https://phabricator.wikimedia.org/T420996#11885156 (10santhosh) > @santhosh can you share what method you used and how much of the data you ended up extracting that way? This the code I used to extract go... [05:27:46] 06Data-Engineering, 10AbuseFilter, 06DBA, 07Schema-change-in-production: Drop AbuseFilter tables from closed wikis - https://phabricator.wikimedia.org/T425066#11885168 (10Marostegui) p:05Triage→03Medium a:03Marostegui [07:06:16] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [07:06:16] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - . [07:06:16] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [07:20:45] 06Data-Engineering: 21 new wikis missing from mediawiki_history - https://phabricator.wikimedia.org/T386649#11885250 (10nshahquinn-wmf) 05Open→03Resolved [07:41:16] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Load Google Search Console data into the Data Lake - https://phabricator.wikimedia.org/T420996#11885315 (10SCherukuwada) I would very much recommend importing and keeping wikimedia.org data - at least for commons. We're likely going to need to have some visi... [07:57:49] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Set up a rolling retention window for event_sanitized.netflow - https://phabricator.wikimedia.org/T424904#11885395 (10ayounsi) It would be best to keep all historical data but aggregate it further as time goes. Do we already drop/aggregate all the fields men... [08:22:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [08:22:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [08:22:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [08:25:04] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 07Essential-Work, and 3 others: Migrate "WikiLambda API" instrument to use the Test Kitchen SDK - https://phabricator.wikimedia.org/T415254#11885491 (10Sfaci) [08:36:48] 06Data-Engineering: Migrate generated-data-platform-aqs Docker images away from Debian Bullseye - https://phabricator.wikimedia.org/T425310#11885574 (10Aklapper) [08:37:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [08:37:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [08:37:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [08:40:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [08:40:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [08:40:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:00:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:00:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:00:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:00:56] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Add new API rate limiting fields from webrequest_logs to Turnilo view - https://phabricator.wikimedia.org/T419736#11885859 (10daniel) Any progress on this? We ran into a situation with requests from the Wikipedia App getting blocked, an... [09:04:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:04:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:04:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:09:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:09:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:09:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:13:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:13:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:13:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:18:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:18:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:18:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:22:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:22:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:22:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:27:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:27:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:27:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:38:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:38:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:38:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:41:01] RESOLVED: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [09:41:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - . [09:41:01] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [09:43:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:43:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:43:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:44:11] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:44:11] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:44:11] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:44:31] FIRING: [6x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [09:44:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - . [09:44:31] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [09:47:45] FIRING: [8x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [09:47:46] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - . [09:47:46] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [09:53:56] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:53:56] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:53:56] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:58:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:58:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:58:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [10:15:39] 06Data-Engineering, 06Data-Engineering-Radar, 10Event-Platform, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11886142 (10gkyziridis) The new stream is deploye... [10:27:33] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: HTML Pipeline - Reduce max allowed size - https://phabricator.wikimedia.org/T425336 (10JMonton-WMF) 03NEW [10:28:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [10:28:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [10:28:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [10:33:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [10:33:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [10:33:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [10:38:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [10:38:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [10:38:44] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [10:39:11] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [10:39:11] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [10:39:11] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [10:43:56] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [10:43:56] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [10:43:56] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [10:47:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [10:47:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [10:47:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [10:52:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [10:52:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [10:52:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [11:01:04] 06Data-Engineering, 06Data-Engineering-Radar, 10Event-Platform, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11886231 (10isarantopoulos) 🎉 Indeed there are... [11:03:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [11:03:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [11:03:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [11:23:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [11:23:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [11:23:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [11:42:15] 06Data-Engineering, 06Data-Engineering-Radar, 10Event-Platform, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11886361 (10gkyziridis) >>! In T415892#11886231,... [12:21:44] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): brand health tracker data ingestion - https://phabricator.wikimedia.org/T425347 (10APizzata-WMF) 03NEW [12:25:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [12:25:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [12:25:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [12:28:09] (03Abandoned) 10Joal: Add kai.wikipedia to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1225039 (https://phabricator.wikimedia.org/T414237) (owner: 10Gerrit maintenance bot) [12:35:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [12:35:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [12:35:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [12:41:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [12:41:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [12:41:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [12:46:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [12:46:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [12:46:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [12:48:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [12:48:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [12:48:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [12:53:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [12:53:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [12:53:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [13:00:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [13:00:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [13:00:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [13:05:00] 06Data-Engineering: table_maintenance_iceberg_monthly permission issue fails task due to permission on Ivy cache artifact - https://phabricator.wikimedia.org/T418804#11886580 (10xcollazo) @lerickson reproed again ([[ https://wikimedia.slack.com/archives/CSV483812/p1777644371496949 | Slack thread ]]). Ok, what w... [13:05:30] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): brand health tracker data ingestion - https://phabricator.wikimedia.org/T425347#11886581 (10APizzata-WMF) Created a local script to ingest google sheet. It is not yet ready to be deployed but is a start to ingest and test. The document provided was ingeste... [13:05:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [13:05:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [13:05:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [13:06:03] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Pipeline - Reduce max allowed size - https://phabricator.wikimedia.org/T425336#11886584 (10Ottomata) > We need to reduce the max_content_size to a lower number Either that, or increase kafka max message size ag... [13:08:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [13:08:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [13:08:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [13:13:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [13:13:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [13:13:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [13:15:34] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Streaming HTML & Edit Types - productionization checklist - https://phabricator.wikimedia.org/T423920#11886616 (10Ottomata) [13:21:32] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11886631 (10AnnieKim_WMDE) Hello! Is there anything else I can or need to provide? [13:34:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [13:34:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [13:34:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [13:38:00] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Backfill datasets affected by Nov 2025 automated traffic incident - https://phabricator.wikimedia.org/T421735#11886698 (10mforns) [13:44:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [13:44:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [13:44:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [13:46:08] 06Data-Engineering: implement script to move data from P&T data lake to FR Tech data lake - https://phabricator.wikimedia.org/T425133#11886725 (10xcollazo) a:05xcollazo→03None Here is an implementation idea that @AStein-WMF, @amastilovic and I discussed that can potentially solve this use case: Use case de... [13:47:41] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list April 2026 - https://phabricator.wikimedia.org/T424607#11886731 (10mforns) This was merged and deployed! [13:47:46] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [13:47:46] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [13:47:46] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [13:56:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11886758 (10xcollazo) Even though the patch applied cleanly and DAG `refine_webrequest_hourly_text` continues to move forward, the error email... [14:01:13] (03CR) 10Mforns: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1279651 (https://phabricator.wikimedia.org/T421743) (owner: 10Snwachukwu) [14:01:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [14:01:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [14:01:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [14:06:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [14:06:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [14:06:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [14:13:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [14:13:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [14:13:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [14:23:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [14:23:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [14:23:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [14:24:11] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [14:24:11] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [14:24:11] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [14:39:11] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [14:39:11] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [14:39:11] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [14:45:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [14:45:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [14:45:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [14:50:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [14:50:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [14:50:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [14:56:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [14:56:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [14:56:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [15:04:58] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: mediawiki.page_change.v1 event - add a page type field - https://phabricator.wikimedia.org/T409462#11887020 (10Ottomata) [15:05:01] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: mediawiki.page_change.v1 event - add a 'new revision created' field - https://phabricator.wikimedia.org/T409464#11887022 (10Ottomata) [15:08:48] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [15:15:43] !log Deploy refinery to HDFS [15:15:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:18:35] 06Data-Engineering, 10Event-Platform: mediawiki-event-enrichment - set deterministic meta.id - https://phabricator.wikimedia.org/T424987#11887098 (10Ottomata) > We will still have duplicate events due to event producer resends, no? Yes. This change would prevent undetected duplicates due to reprocessing a m... [15:19:49] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11887099 (10JAllemandou) The problem is due to a forgotten step in deploying the `refinery` repo. I have just fixed that. Let's validate that... [15:23:48] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [15:26:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [15:26:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [15:26:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [15:29:22] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Pipeline - Issue with max size messages - https://phabricator.wikimedia.org/T425336#11887118 (10JMonton-WMF) [15:36:12] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats: Wiki Stats: Problem Detecting (unique) Desktop Devices since August 2025? - https://phabricator.wikimedia.org/T425359 (10Polimorph) 03NEW [15:39:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [15:39:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [15:39:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [15:41:33] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: EventBus - user entity schema should differentiate between explicit and implicit user groups - https://phabricator.wikimedia.org/T425360 (10Ottomata) 03NEW [15:49:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [15:49:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [15:49:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [15:54:03] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Edit type enrichment: Watermarks lagging - https://phabricator.wikimedia.org/T425362 (10AKhatun_WMF) 03NEW [15:55:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [15:55:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [15:55:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [16:08:45] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Dumps-Generation: when analyzing a Wikifunctions dump, parent_id in page creation revisions is sometimes 0 and sometimes None - https://phabricator.wikimedia.org/T420974#11887286 (10Ottomata) Hm! Interesting, TIL. A comment in [[ https://gerrit.wikimedi... [16:09:39] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Edit type enrichment: Watermarks lagging - https://phabricator.wikimedia.org/T425362#11887288 (10AKhatun_WMF) @Ottomata Sorry, I can't fully remember what happened when DC switchover happened. Wanted to confirm, when... [16:11:53] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Create mediawiki.user_change event stream - https://phabricator.wikimedia.org/T423952#11887296 (10Ottomata) I just encountered an annoying issue with how we have been modeling user groups in event schemas. I filed {T... [16:25:35] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Pipeline - Issue with max size messages - https://phabricator.wikimedia.org/T425336#11887339 (10JMonton-WMF) Some information that can be useful: Looking at the error topic (in the Datalake), it doesn't seem t... [16:30:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [16:30:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [16:30:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [16:32:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [16:32:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [16:32:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [16:37:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [16:37:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [16:37:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [16:41:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [16:41:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [16:41:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [16:42:46] RESOLVED: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [16:42:49] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [16:42:55] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [16:46:31] FIRING: [6x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [16:46:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [16:46:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [16:49:45] FIRING: [8x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [16:49:46] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [16:49:46] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [16:56:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [16:56:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [16:56:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [16:57:48] 06Data-Engineering: table_maintenance_iceberg_monthly permission issue fails task due to permission on Ivy cache artifact - https://phabricator.wikimedia.org/T418804#11887398 (10AKhatun_WMF) Same problem today for maintenance dags in main airflow ` Exception in thread "main" java.io.FileNotFoundException: /tmp/t... [17:01:17] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Edit type enrichment: Watermarks lagging - https://phabricator.wikimedia.org/T425362#11887401 (10Ottomata) > when DC switchover does happen, page_change_v1 will have events in codfw. html_content_change will consume from codfw.page_change... [17:04:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [17:04:47] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [17:04:47] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [17:09:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [17:09:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [17:09:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [17:14:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [17:14:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [17:14:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [17:19:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [17:19:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [17:19:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [17:24:37] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Add new API rate limiting fields from webrequest_logs to Turnilo view - https://phabricator.wikimedia.org/T419736#11887469 (10amastilovic) >>! In T419736#11885859, @daniel wrote: > Any progress on this? We ran into a situation with requ... [17:55:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [17:55:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [17:55:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [18:05:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [18:05:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [18:05:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [18:17:31] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Pytest - https://phabricator.wikimedia.org/T425369 (10Ahoelzl) 03NEW [18:19:45] RESOLVED: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [18:19:45] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [18:19:45] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [18:22:31] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [18:22:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [18:22:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [18:26:45] FIRING: [8x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [18:26:46] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [18:26:46] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [18:57:31] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [18:57:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [18:57:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [19:01:45] RESOLVED: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [19:01:46] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [19:01:46] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [19:02:31] FIRING: [6x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [19:02:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [19:02:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [19:02:45] RESOLVED: [3x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [19:02:45] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [19:02:45] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [19:07:31] FIRING: [8x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [19:07:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [19:07:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [19:07:53] FIRING: [8x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [19:07:57] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [19:08:02] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [19:10:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [19:10:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [19:10:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [19:12:31] FIRING: [9x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [19:12:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [19:12:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [19:17:31] FIRING: [12x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [19:17:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [19:17:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [19:30:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [19:30:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [19:30:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [19:43:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [19:43:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [19:43:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [19:48:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [19:48:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [19:48:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [19:53:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [19:53:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [19:53:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [19:58:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [19:58:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [19:58:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [20:01:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [20:01:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [20:01:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [20:14:48] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [20:19:48] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [20:26:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [20:26:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [20:26:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [20:30:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [20:30:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [20:30:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [20:35:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [20:35:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [20:35:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [20:39:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [20:39:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [20:39:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [20:42:30] RESOLVED: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [20:42:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [20:42:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [20:44:30] FIRING: [3x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [20:44:30] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [20:44:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [20:44:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [20:44:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [20:44:44] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [20:48:45] FIRING: [8x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [20:48:46] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [20:48:46] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [20:53:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [20:53:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [20:53:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [20:58:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [20:58:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [20:58:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [21:23:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [21:23:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [21:23:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [21:28:45] RESOLVED: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [21:28:46] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [21:28:46] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [21:33:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [21:33:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [21:33:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [21:39:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [21:39:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [21:39:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [21:40:00] FIRING: [3x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [21:40:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [21:40:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [21:44:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [21:44:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [21:44:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [21:45:07] FIRING: [8x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [21:45:10] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [21:45:16] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [21:49:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [21:49:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [21:49:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [21:54:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [21:54:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [21:54:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [21:55:11] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [21:55:11] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [21:55:11] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [21:59:56] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [21:59:56] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [21:59:59] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [22:00:10] RESOLVED: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [22:00:14] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [22:00:19] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [22:05:31] FIRING: [8x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [22:05:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [22:05:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [22:10:31] RESOLVED: [8x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [22:10:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [22:10:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [22:20:31] FIRING: [8x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [22:20:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [22:20:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [22:31:18] 06Data-Engineering, 07Sustainability: Move some analytics jobs to day time in Virginia - https://phabricator.wikimedia.org/T384166#11887979 (10Ahoelzl) Moving `daily` and `monthly` jobs will incur a 12 hour data delivery delay which will impact downstream pipelines. The estimated climate impact reduction is le... [23:33:48] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 07Epic: Incremental MediaWiki History - https://phabricator.wikimedia.org/T424350#11888091 (10Ahoelzl) [23:34:27] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 10Event-Platform, 13Patch-For-Review: Create mediawiki.user_change event stream - https://phabricator.wikimedia.org/T423952#11888093 (10Ahoelzl) [23:35:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11888094 (10Ahoelzl) [23:36:57] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Investigate oversized driver logs for `mediawiki_history_denormalize` - https://phabricator.wikimedia.org/T424358#11888095 (10Ahoelzl) [23:37:44] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Draft: Architectural design agreement: Incremental MediaWiki History - https://phabricator.wikimedia.org/T424359#11888096 (10Ahoelzl) [23:37:53] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): A more recent Spark + Iceberg will make Incremental MediaWiki History much more efficient - https://phabricator.wikimedia.org/T424381#11888097 (10Ahoelzl) [23:38:15] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 10Event-Platform: Incremental MWH - MediaWiki event data source improvements - https://phabricator.wikimedia.org/T423935#11888098 (10Ahoelzl) [23:38:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 10Event-Platform: Emit comprehensive mediawiki user block change information in an event stream - https://phabricator.wikimedia.org/T424685#11888099 (10Ahoelzl) [23:39:08] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 10Event-Platform, 13Patch-For-Review: EventBus - consider schema versions when serializing entities - https://phabricator.wikimedia.org/T424767#11888100 (10Ahoelzl) [23:39:21] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 10Event-Platform: EventBus - user entity schema should differentiate between explicit and implicit user groups - https://phabricator.wikimedia.org/T425360#11888101 (10Ahoelzl) [23:39:37] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 10Event-Platform: mediawiki.page_change event - add platform field - https://phabricator.wikimedia.org/T424887#11888102 (10Ahoelzl) [23:39:47] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 10Event-Platform: mediawiki.page_change.v1 - add revision.editor.first_edit_dt field - https://phabricator.wikimedia.org/T425029#11888103 (10Ahoelzl)