[00:06:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:06:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [00:50:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:50:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:10:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:10:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:12:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:12:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:17:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:17:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:22:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:22:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:42:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:42:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:43:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:43:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:53:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:53:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:54:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:54:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:59:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:59:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:02:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:02:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:12:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:12:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:16:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:16:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:21:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:21:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:22:06] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:22:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [03:02:06] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [03:02:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [03:06:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [03:06:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [03:16:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [03:16:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [03:19:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [03:19:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [03:34:34] 06Data-Engineering, 10MediaWiki-General, 07Schema-change: Change data type of namespace fields from int to smallint or mediumint - https://phabricator.wikimedia.org/T383507#10460935 (10Bugreporter) >>! In T383507#10458920, @Ladsgroup wrote: > I think we can go to smallint. We should just talk to `WSSpaces` a... [05:29:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [05:29:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [05:35:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [05:35:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [05:55:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [05:55:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [05:56:06] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [05:56:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:06:06] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:06:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:07:06] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:07:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:12:06] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:12:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:14:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:14:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:19:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:19:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:21:06] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:21:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:51:06] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:51:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:52:06] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:52:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [07:42:06] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [07:42:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [07:45:06] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox, 07Browser-Support-Apple-Safari: On analytics.wikimedia.org, Safari on iOS ignores collapsed tabs interaction - https://phabricator.wikimedia.org/T266122#10461068 (10TheDJ) This works in newer versions of the browser, and seeing the history of... [07:45:08] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox, 07Browser-Support-Apple-Safari: On analytics.wikimedia.org, Safari on iOS ignores collapsed tabs interaction - https://phabricator.wikimedia.org/T266122#10461069 (10TheDJ) 05Open→03Declined [07:45:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [07:45:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [08:39:28] 06Data-Engineering, 06Data-Engineering-Radar, 10observability, 10Revision-Slider, and 4 others: Where should keys used for stats in an extension be documented? - https://phabricator.wikimedia.org/T185111#10461163 (10fgiunchedi) 05Open→03Declined I'm politely declining this old task, which has seen... [08:41:31] 06Data-Engineering-Radar, 06SRE, 07Puppet: modules/udp2log/manifests/instance/monitoring.pp has unreachable code - https://phabricator.wikimedia.org/T152104#10461173 (10fgiunchedi) 05Open→03Invalid Manifest doesn't contain unreachable code anymore ` define udp2log::instance::monitoring( $log_dir... [08:47:18] (03CR) 10Aqu: [C:03+2] Extraction of RawRefineDataReader [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [08:54:13] (03PS3) 10Aqu: Improve Spark Logger Quietness [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1100394 (https://phabricator.wikimedia.org/T381074) [08:55:47] (03CR) 10Aqu: [V:03+2 C:03+2] Improve Spark Logger Quietness [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1100394 (https://phabricator.wikimedia.org/T381074) (owner: 10Aqu) [08:57:54] 06Data-Engineering: Fix skein/spark memory unit missfit - https://phabricator.wikimedia.org/T383589#10461215 (10dcausse) >>! In T383589#10459568, @Ottomata wrote: > @Gehel @JAllemandou @dcausse What do you think the priority here should be? > > Is there a related task? I faced this issue while migrating a hive... [09:00:30] (03PS5) 10Aqu: Add an option to ignore missing input folders in Refine [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1088346 (https://phabricator.wikimedia.org/T369845) [09:00:58] (03Merged) 10jenkins-bot: Extraction of RawRefineDataReader [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [10:35:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [10:35:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [10:38:54] (03CR) 10Aqu: [C:03+2] refactoring(RefineHelper): extract SparkSchemaLoader from RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1102386 (owner: 10Gehel) [10:39:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [10:39:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [10:49:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [10:49:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [10:50:36] (03Merged) 10jenkins-bot: refactoring(RefineHelper): extract SparkSchemaLoader from RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1102386 (owner: 10Gehel) [10:51:06] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [10:51:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [10:56:22] (03CR) 10Aqu: [C:03+2] Add an option to ignore missing input folders in Refine [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1088346 (https://phabricator.wikimedia.org/T369845) (owner: 10Aqu) [11:08:39] (03Merged) 10jenkins-bot: Add an option to ignore missing input folders in Refine [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1088346 (https://phabricator.wikimedia.org/T369845) (owner: 10Aqu) [11:26:46] 06Data-Engineering, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Delete ganeti VM eventlog1003.eqiad.wmnet - https://phabricator.wikimedia.org/T383276#10461712 (10BTullis) 05Open→03Resolved [12:01:06] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [12:01:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [12:02:06] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [12:02:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:18:15] 06Data-Engineering, 10CampaignEvents, 06Campaigns-Product-Team, 06Data-Persistence, 07Schema-change: Convert `ce_participants.cep_private` from boolean to mwtinyint - https://phabricator.wikimedia.org/T383777#10463022 (10Marostegui) [16:29:01] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Airflow operator to manage old data deletion - https://phabricator.wikimedia.org/T326826#10463087 (10xcollazo) Linking this with the work done for Iceberg table maintenance, that also does data deletion via SQL: {T338065}. [17:18:08] 06Data-Engineering, 10Commons-Impact-Metrics-Requests: Request for Images_from_Wiki_Loves_Africa_2024 - https://phabricator.wikimedia.org/T381352#10463315 (10GFontenelle_WMF) Dear @Anthere, thanks for contributing to CIM! The category you requested is now available on the 486 line. Let me know if you need anyt... [17:18:27] 06Data-Engineering, 10Commons-Impact-Metrics-Requests: Request for Images_from_Wiki_Loves_Africa_2024 - https://phabricator.wikimedia.org/T381352#10463318 (10GFontenelle_WMF) 05In progress→03Resolved [17:19:22] 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list December 2024 - https://phabricator.wikimedia.org/T382740#10463326 (10GFontenelle_WMF) 05Open→03Resolved [17:32:26] 06Data-Engineering, 06Product-Analytics, 10Event-Platform: Enable Event Platform instruments to opt out of collecting User-Agent data - https://phabricator.wikimedia.org/T382173#10463474 (10Ottomata) [17:34:28] 06Data-Engineering, 06Product-Analytics, 10Event-Platform: Enable Event Platform instruments to opt out of collecting User-Agent data - https://phabricator.wikimedia.org/T382173#10463506 (10Ottomata) [17:34:47] 06Data-Engineering: [sqoop] Check if a sqoop needs changes when one mariadb field changes from int to smallint - https://phabricator.wikimedia.org/T383803 (10mforns) 03NEW [17:35:33] 06Data-Engineering, 06Product-Analytics, 10Event-Platform: Enable Event Platform instruments to opt out of collecting User-Agent data - https://phabricator.wikimedia.org/T382173#10463520 (10Ottomata) [17:36:02] 06Data-Engineering, 10MediaWiki-General, 07Schema-change: Change data type of namespace fields from int to smallint or mediumint - https://phabricator.wikimedia.org/T383507#10463523 (10mforns) OK, created this task T383803 for that. [17:40:14] 06Data-Engineering, 06Movement-Insights: Modify the automated traffic detection to be applied at the project family level - https://phabricator.wikimedia.org/T377257#10463557 (10JAllemandou) 05Open→03Resolved This is done! [17:40:41] 06Data-Engineering, 06Movement-Insights, 13Patch-For-Review: Backfill and recalculate unique devices data from July 2024 to present - https://phabricator.wikimedia.org/T378852#10463561 (10JAllemandou) [17:47:04] 06Data-Engineering, 06Movement-Insights: Test airflow cascading-reruns for projectview-hourly dependent jobs - https://phabricator.wikimedia.org/T383804 (10JAllemandou) 03NEW [17:47:31] 06Data-Engineering, 06Movement-Insights: Test airflow cascading-reruns for projectview-hourly dependent jobs - https://phabricator.wikimedia.org/T383804#10463609 (10JAllemandou) p:05Triage→03High [19:06:16] 06Data-Engineering, 10EventStreams, 10Event-Platform, 13Patch-For-Review: EventStreams: kafka key should be serialized as a string - https://phabricator.wikimedia.org/T373689#10464024 (10Ottomata) [19:07:32] 06Data-Engineering, 10EventStreams, 10Event-Platform, 13Patch-For-Review: EventStreams: kafka key should be serialized as a string - https://phabricator.wikimedia.org/T373689#10464027 (10Ottomata) a:03Ottomata @tchin I kept answering some questions about eventstreams and was embarrassed we hadn't fixed t... [19:25:50] 06Data-Engineering: Identify Internal Users of MediaWiki Wikitext Tables - https://phabricator.wikimedia.org/T383743#10464070 (10Ahoelzl) @Ottomata what about non analytics users? [19:56:48] 06Data-Engineering, 10Event-Platform: Upgrade eventgate-wikimedia to node20 - https://phabricator.wikimedia.org/T383814 (10Ottomata) 03NEW [20:10:33] 06Data-Engineering: Identify Internal Users of MediaWiki Wikitext Tables - https://phabricator.wikimedia.org/T383743#10464183 (10Ottomata) Unless I'm mistaken, I don't think there is anything preventing us from enabling lineage there? Since each instance declares artifacts, we have to redeclare the deployment o... [20:42:04] 10Data-Engineering (Q2 2024 October 1st - December 31th), 13Patch-For-Review: Publish Data Engineering maintained NodeJS packages to GitLab and use them in depender code - https://phabricator.wikimedia.org/T366612#10464376 (10Ottomata) [20:42:05] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: Upgrade eventgate-wikimedia to node20 - https://phabricator.wikimedia.org/T383814#10464377 (10Ottomata) [20:44:22] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: Upgrade eventgate-wikimedia to node20 - https://phabricator.wikimedia.org/T383814#10464384 (10Ottomata) @tchin I'm trying to do like you did in https://gitlab.wikimedia.org/repos/data-engineering/eventgate/-/merge_requests/4, but am failing. Please h... [21:34:15] (03PS2) 10Snwachukwu: Update Geoeditors Edits Monthly to support Temp Accounts. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1092890 (https://phabricator.wikimedia.org/T379768) [21:35:47] 06Data-Engineering, 10Dumps-Generation, 06MediaWiki-Platform-Team, 06serviceops: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10464579 (10jijiki) [21:37:29] 06Data-Engineering, 10Dumps-Generation, 06MediaWiki-Platform-Team, 06serviceops: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10464582 (10jijiki) [22:03:00] (03PS3) 10Snwachukwu: Update Geoeditors Edits Monthly to support Temp Accounts. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1092890 (https://phabricator.wikimedia.org/T379768) [22:26:14] PROBLEM - Check if active EventStreams endpoint is delivering messages. on alert1002 is CRITICAL: CRITICAL: No EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [22:56:14] RECOVERY - Check if active EventStreams endpoint is delivering messages. on alert1002 is OK: OK: An EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [23:26:24] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Change page.page_links_updated to fixed-length timestamp in wmf wikis - https://phabricator.wikimedia.org/T371742#10464938 (10Ladsgroup)