[00:03:09] 10Analytics-Canonical-Data, 06Movement-Insights, 06Security-Team, 06WMF-Legal, 07SecTeam-Processed: Decide stewardship of the Country and Territory Protection List - https://phabricator.wikimedia.org/T381944#10784793 (10nshahquinn-wmf) [00:06:26] 10Analytics-Canonical-Data, 06Movement-Insights, 06Security-Team, 06WMF-Legal, 07SecTeam-Processed: Decide stewardship of the Country and Territory Protection List - https://phabricator.wikimedia.org/T381944#10784805 (10nshahquinn-wmf) >>! In T381944#10778103, @sbassett wrote: >>>! In T381944#10777768, @... [00:17:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:17:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [00:32:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:32:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:13:18] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Traffic, 10DPE HAProxy Migration: Add HAproxy termination field to webrequest - https://phabricator.wikimedia.org/T387454#10784953 (10JAllemandou) a:03JAllemandou [06:13:49] 10Data-Engineering (Q4 2025 April 1st - June 30th): Refine to Hive with Airflow – Handle Late-Arrived Events - https://phabricator.wikimedia.org/T370665#10784955 (10JAllemandou) [06:23:15] 10Data-Engineering (Q4 2025 April 1st - June 30th): Refine to Hive with Airflow – Handle Late-Arrived Events - https://phabricator.wikimedia.org/T370665#10784963 (10JAllemandou) After some thoughtful discussion here is the proposed plan: * Add an airflow step in Refine and webrequest jobs to add a `_PROCESSED`... [06:24:18] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Traffic, 10DPE HAProxy Migration: Add HAproxy termination field to webrequest - https://phabricator.wikimedia.org/T387454#10784964 (10JAllemandou) Hi @Fabfur , sorry for the late answer :) I'm adding a PR for you to validate. There are multiple places whe... [06:32:31] (03PS1) 10Joal: Add HAProxy termination_state to webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1140566 (https://phabricator.wikimedia.org/T387454) [06:52:25] 06Data-Engineering: Warning of mismatch in declarations of Webrequest schema - https://phabricator.wikimedia.org/T380916#10785005 (10JAllemandou) I have solved the issue using the following strategy with the `hive` CLI. * Find the spark schema properties in hive schema (starting with `spark.sql`) `> show create... [06:52:47] 10Data-Engineering (Q4 2025 April 1st - June 30th): Warning of mismatch in declarations of Webrequest schema - https://phabricator.wikimedia.org/T380916#10785006 (10JAllemandou) [06:53:11] 10Data-Engineering (Q4 2025 April 1st - June 30th): Warning of mismatch in declarations of Webrequest schema - https://phabricator.wikimedia.org/T380916#10785007 (10JAllemandou) @nshahquinn-wmf can you confirm the problem is solved for you please? Thank you :) [06:53:34] 10Data-Engineering (Q4 2025 April 1st - June 30th): Warning of mismatch in declarations of Webrequest schema - https://phabricator.wikimedia.org/T380916#10785008 (10JAllemandou) a:03JAllemandou [07:27:06] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for madalina - https://phabricator.wikimedia.org/T392893#10785049 (10tappof) 05Open→03Resolved a:03tappof [07:33:55] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to for - https://phabricator.wikimedia.org/T393066#10785083 (10tappof) [07:34:41] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to for - https://phabricator.wikimedia.org/T393066#10785084 (10tappof) [07:39:24] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to for - https://phabricator.wikimedia.org/T393066#10785107 (10tappof) [07:41:32] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to for - https://phabricator.wikimedia.org/T393066#10785119 (10tappof) 05Open→03In progress @Ospingou, could you please sign off on the access request? Thank you! [07:52:29] (03CR) 10Fabfur: [C:03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1140566 (https://phabricator.wikimedia.org/T387454) (owner: 10Joal) [08:48:57] 06Data-Engineering, 10service-utils: Migrate and re-deploy eventgate using new service-utils - https://phabricator.wikimedia.org/T361768#10785294 (10Jdforrester-WMF) [08:49:57] 06Data-Engineering, 10service-utils: Support experimentation platform backend migration to new service_utils - https://phabricator.wikimedia.org/T361770#10785297 (10Jdforrester-WMF) [15:45:20] !log reboot an-worker1166 after the hardware swap for T390170 [15:45:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:45:23] T390170: Upgrade an-worker hard drives from 4TB to 8TB (group 3 - rack F5) - https://phabricator.wikimedia.org/T390170 [17:07:37] 06Data-Engineering-Radar, 10BDC-Implementation, 07Epic: [MinIO] Investigate load-balancing approaches to eliminate SPOF - https://phabricator.wikimedia.org/T392249#10786845 (10Jgreen) Here's an HAProxy config that seems to work, more testing is needed: ` global maxconn 4096 user haproxy group hapro... [17:19:04] !log reboot an-worker1167 after the hardware swap for T390170 [17:19:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:19:08] T390170: Upgrade an-worker hard drives from 4TB to 8TB (group 3 - rack F5) - https://phabricator.wikimedia.org/T390170 [17:19:29] 10Data-Engineering (Q4 2025 April 1st - June 30th): RefineSanitize fails to send emails - https://phabricator.wikimedia.org/T393202 (10xcollazo) 03NEW [17:27:22] !log reboot an-worker1168 after the hardware swap for T390170 [17:27:25] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:27:25] T390170: Upgrade an-worker hard drives from 4TB to 8TB (group 3 - rack F5) - https://phabricator.wikimedia.org/T390170 [17:31:45] 14Data-Engineering (Q3 2025 January 1st - March 31th), 07Data-Platform, 06Experimentation Lab: NEW/CHANGE FEATURE REQUEST: Documentation for v1 Enterprise endpoint deprecation - https://phabricator.wikimedia.org/T389542#10786927 (10jberkel) >>! In T389542#10763021, @HShaikh wrote: > Hey @jberkel > > yes... [19:52:03] 06Data-Engineering-Radar, 10BDC-Implementation, 07Epic: [MinIO] Investigate load-balancing approaches to eliminate SPOF - https://phabricator.wikimedia.org/T392249#10787225 (10Jgreen) Improved ` global maxconn 4096 user haproxy group haproxy stats socket /run/haproxy/haproxy.sock mode 666 leve... [20:11:30] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Enable merge-on-read for wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T393012#10787267 (10xcollazo) [20:35:34] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Experimentation Lab (Experiment Platform Sprint 6), 13Patch-For-Review: FY 24-25 SDS 2.4.9 CDN Synthetic Beacon: EventGate & Varnish: update to receive events from beacon event v2 - https://phabricator.wikimedia.org/T391959#10787356 (10dr0ptp4kt) [21:33:09] 10Data-Engineering (Q4 2025 April 1st - June 30th): Warning of mismatch in declarations of Webrequest schema - https://phabricator.wikimedia.org/T380916#10787479 (10nshahquinn-wmf) @JAllemandou yes, I no longer get the warning message. Thank you! [22:36:49] 06Data-Engineering, 06Data-Platform-SRE, 10superset.wikimedia.org: Frequent filter timeouts in superset UI - https://phabricator.wikimedia.org/T393097#10787565 (10BTullis)