[02:51:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[02:51:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[03:49:36] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th), 10MW-1.44-notes (1.44.0-wmf.12; 2025-01-14): EventBus PageChangeHooks uses unconventional log channel name - https://phabricator.wikimedia.org/T382288#10439643 (10tstarling) 05Open→03Resolved
[08:30:09] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10Data-Platform-SRE (2024.11.30 - 2024.12.20): Data Platform access streamlining for WMDE staff - https://phabricator.wikimedia.org/T381824#10439878 (10Gehel)
[08:33:34] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE: Do performance testing of a big Hadoop Table hosted by Ceph - https://phabricator.wikimedia.org/T381416#10439902 (10Gehel) p:05Triage→03Medium
[10:29:27] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th): Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10440107 (10Ahoelzl)
[10:51:29] <wikibugs>	 (03PS1) 10KCVelaga: Bug: T373785 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1109039 (https://phabricator.wikimedia.org/T373785)
[10:53:40] <wikibugs>	 (03PS2) 10KCVelaga: Bug: T373785 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1109039 (https://phabricator.wikimedia.org/T373785)
[10:56:05] <wikibugs>	 (03PS3) 10KCVelaga: Add event_context to sanitization allowlist for Content Translation events. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1109039 (https://phabricator.wikimedia.org/T373785)
[11:01:56] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th): Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10440220 (10BTullis) Just for reference, these access logs for https://dumps.wikimedia.org are available for analysis on stat1011. ` btullis@stat1011:/srv/log/webrequest/a...
[11:10:35] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10Data Pipelines, 10Pageviews-Anomaly, and 3 others: Analyze possible bot traffic for frwiki article Cookie (informatique) - https://phabricator.wikimedia.org/T313114#10440236 (10hashar) 05Open→03Declined I have looked at the Turnillo requests mentioned...
[13:35:30] <wikibugs>	 (03CR) 10Nik Gkountas: [C:03+1] Add event_context to sanitization allowlist for Content Translation events. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1109039 (https://phabricator.wikimedia.org/T373785) (owner: 10KCVelaga)
[13:35:39] <wikibugs>	 (03PS4) 10Nik Gkountas: Add event_context to sanitization allowlist for Content Translation events. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1109039 (https://phabricator.wikimedia.org/T373785) (owner: 10KCVelaga)
[13:43:28] <elukey>	 brouberol, btullis o/ - While checking https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=alertname%3DSmartNotHealthy I noticed an issue with dse-k8s-worker1009, is it known?
[13:49:40] <brouberol>	 o> hmm, at least not to me!
[14:24:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[14:24:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[14:44:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[14:44:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[14:47:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[14:47:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[14:52:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[14:52:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[14:56:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[14:56:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[15:06:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[15:06:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[15:09:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[15:09:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[15:27:06] <wikibugs>	 06Data-Engineering, 10Data-Services, 06DBA, 06Privacy Engineering: Create views for SecurePoll db tables in Toolforge replicas - https://phabricator.wikimedia.org/T381197#10441346 (10joanna_borun)
[15:34:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[15:34:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[15:35:05] <wikibugs>	 06Data-Engineering-Icebox, 10Data-Services, 13Patch-For-Review: Log_param is redacted in wiki replica when only comment and/or user should be - https://phabricator.wikimedia.org/T301943#10441407 (10Andrew) *bump* This is a data engineering task but it's pretty simple isn't it?
[15:35:11] <wikibugs>	 06Data-Engineering-Icebox, 10Data-Services, 13Patch-For-Review: Log_param is redacted in wiki replica when only comment and/or user should be - https://phabricator.wikimedia.org/T301943#10441408 (10joanna_borun)
[15:41:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[15:41:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[15:53:34] <wikibugs>	 06Data-Engineering, 06Growth-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaEvents, and 6 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10441526 (10Michael) >>! In T355837#10438710, @Krinkle wrote: >>>! In T355837#10438413, @Micha...
[16:01:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[16:01:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[16:03:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[16:03:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[16:48:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[16:48:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[17:07:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[17:07:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[17:12:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[17:12:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[17:13:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[17:13:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[17:18:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[17:18:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[17:50:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[17:50:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[17:55:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[17:55:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[18:11:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[18:11:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[18:21:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[18:21:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[18:25:38] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board): Identify indicators to inform an SLO for event emission and intake - https://phabricator.wikimedia.org/T345195#10442231 (10xcollazo) 05Open→03Resolved
[18:27:02] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board): Implement Airflow Dataset class for RestExternalTaskSensor - https://phabricator.wikimedia.org/T372647#10442244 (10xcollazo) 05Open→03Resolved
[18:29:11] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th), 03Discovery-Search (Current work), 10Dumps 2.0 (Kanban Board): Bump eventutilities to support flink 1.20 - https://phabricator.wikimedia.org/T377130#10442258 (10xcollazo) 05Open→03Resolved
[18:30:10] <wikibugs>	 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Event Platform] Instrument EventBus with prometheus MW Statslib - https://phabricator.wikimedia.org/T363587#10442263 (10xcollazo) 05Open→03Resolved
[18:30:39] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board): Update eventutilities_python wrappers to support Flink 1.20 - https://phabricator.wikimedia.org/T374359#10442276 (10xcollazo) 05Open→03Resolved
[18:33:47] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Enable HA for the mw-content-history-reconcile-enrich flink application - https://phabricator.wikimedia.org/T375176#10442302 (10xcollazo) >>! In T375176#10437452, @xcollazo wrote: > Today we discovered t...
[18:35:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[18:35:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[18:40:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[18:40:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[18:42:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[18:42:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[18:47:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[18:47:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[19:02:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[19:02:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[19:07:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[19:07:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[19:09:32] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th), 10MediaWiki-General, 10MediaWiki-Platform-Team (Radar): Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#10442442 (10Ottomata) No raw eventlogging events coming in!...
[19:12:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[19:12:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[19:17:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[19:17:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[19:22:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[19:22:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[19:27:20] <wikibugs>	 06Data-Engineering, 06Product-Analytics, 10Event-Platform: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163#10442482 (10Ottomata)
[19:40:53] <wikibugs>	 (03CR) 10Mforns: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1106349 (https://phabricator.wikimedia.org/T379771) (owner: 10Aleksandar Mastilovic)
[19:41:34] <wikibugs>	 06Data-Engineering, 10MediaWiki-General, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Update Pingback to use the Event Platform - https://phabricator.wikimedia.org/T323828#10442524 (10Ottomata)
[19:42:08] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th), 10MediaWiki-General, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Update Pingback to use the Event Platform - https://phabricator.wikimedia.org/T323828#10442526 (10Ottomata)
[19:46:23] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th): [Data Quality] Update data_quality schemas to be compatible with Iceberg tables - https://phabricator.wikimedia.org/T356866#10442537 (10xcollazo) (Removing #dumps_2.0 tag as this is not Dumps specific work).
[19:46:36] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th): [Data Quality] Update data_quality schemas to be compatible with Iceberg tables - https://phabricator.wikimedia.org/T356866#10442539 (10xcollazo)
[19:47:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[19:47:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[19:49:02] <wikibugs>	 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list December 2024 - https://phabricator.wikimedia.org/T382740#10442544 (10mforns) This has been deployed, and the calculations have started, they should be available soon.
[19:52:04] <wikibugs>	 07Analytics-Data-Problem, 06Data-Engineering, 06Movement-Insights, 06Product-Analytics: webrequest dataset sets referer_class "unknown" instead of "external (search engine)" for origin-based referer values - https://phabricator.wikimedia.org/T383088#10442549 (10nshahquinn-wmf)
[19:53:07] <wikibugs>	 06Data-Engineering, 06Research, 10Research-engineering, 10Event-Platform: Productionized Edit Types - https://phabricator.wikimedia.org/T351225#10442553 (10leila)
[19:55:17] <wikibugs>	 06Data-Engineering, 06Research, 10Research-engineering, 10Event-Platform: Productionized Edit Types - https://phabricator.wikimedia.org/T351225#10442572 (10leila) Update on this request: we have asked Data Platform Engineering to prioritize T360794 as what we need for being able to continue the work for de...
[19:56:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[19:56:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[20:01:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[20:01:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[20:02:12] <wikibugs>	 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Event-Platform, 13Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230#10442619 (10Ottomata)
[20:06:22] <wikibugs>	 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Event-Platform, 13Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230#10442625 (10Ottomata)
[20:17:23] <wikibugs>	 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Event-Platform, 13Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230#10442650 (10Ottomata)
[20:19:22] <wikibugs>	 06Data-Engineering, 10Dumps-Generation: Wikimedia Downloads not complete - https://phabricator.wikimedia.org/T383030#10442665 (10ValterVB) To explain myself better: in https://dumps.wikimedia.org/backup-index-bydb.html they are all 'Done' except Commons, in reality, most (perhaps all) of the dumps are incomplete.
[20:54:49] <wikibugs>	 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list December 2024 - https://phabricator.wikimedia.org/T382740#10442756 (10FRomeo_WMF) Thank you
[21:10:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[21:10:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[21:15:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[21:15:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[21:31:34] <wikibugs>	 06Data-Engineering, 10decommission-hardware: Delete ganeti VM eventlog1003.eqiad.wmnet - https://phabricator.wikimedia.org/T383276 (10Ottomata) 03NEW
[21:31:56] <wikibugs>	 06Data-Engineering, 10decommission-hardware: Delete ganeti VM eventlog1003.eqiad.wmnet - https://phabricator.wikimedia.org/T383276#10442824 (10Ottomata)
[21:32:02] <wikibugs>	 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Event-Platform, 13Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230#10442825 (10Ottomata)
[21:32:48] <wikibugs>	 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Event-Platform, 13Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230#10442826 (10Ottomata)
[21:32:52] <wikibugs>	 06Data-Engineering, 06Data-Platform-SRE: Delete ganeti VM eventlog1003.eqiad.wmnet - https://phabricator.wikimedia.org/T383276#10442829 (10Ottomata)
[21:33:55] <wikibugs>	 06Data-Engineering, 06Experimentation Lab: WebClientError events have version in unexpected format - https://phabricator.wikimedia.org/T383275#10442830 (10Ottomata)
[21:34:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[21:34:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[21:36:33] <wikibugs>	 06Data-Engineering, 06Experimentation Lab: WebClientError events have version in unexpected format - https://phabricator.wikimedia.org/T383275#10442848 (10Ottomata) Hm, I'm not really sure who owns MW JS client error logging.  It was the now defunct product-data-infrastructure team, which sort of morphed into...
[21:39:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[21:39:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[21:43:24] <wikibugs>	 06Data-Engineering, 06Experimentation Lab: WebClientError events have version in unexpected format - https://phabricator.wikimedia.org/T383275#10442857 (10Ottomata) > or some kind of validation in the schema https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary/-/blob/ecbac6df4ba701deb75008...
[21:45:32] <wikibugs>	 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Event-Platform, 13Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230#10442861 (10Ottomata) I have stopped eventlogging-processor on eventlog1003 and removed relevant puppet...
[22:02:30] <wikibugs>	 06Data-Engineering, 06Experimentation Lab, 06Web-Team: WebClientError events have version in unexpected format - https://phabricator.wikimedia.org/T383275#10442910 (10Jdlrobson) Thanks I can take a look at this when I have some time. I will reach out if I hit any snags and likely for some code review!
[22:02:32] <wikibugs>	 06Data-Engineering, 10Dumps-Generation: Wikimedia Downloads not complete - https://phabricator.wikimedia.org/T383030#10442914 (10xcollazo) Thanks for the report.  We did temporary disable the `enwiki` dumps (T368098#10420647), but it should have not affected other wikis such as `itwiki`.  Will investigate.
[22:29:36] <jinxer-wm>	 FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[22:29:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[22:34:36] <jinxer-wm>	 RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ...
[22:34:36] <jinxer-wm>	 High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag
[23:27:46] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06Experimentation Lab, 10MediaWiki-extensions-EventLogging, and 5 others: Decide on how data platform wants to monitor bundle sizes - https://phabricator.wikimedia.org/T378772#10443067 (10Jdlrobson) (Note: above patches do not solve this issue and are just asso...