[03:19:25] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [06:10:44] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#11015796 (10Marostegui) >>! In T399249#11015445, @Dzahn wrote: > We got paged about db1229 and then extended the downtime. Thank you - looks like the schema... [06:24:40] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#11015816 (10Marostegui) I've opened: https://jira.mariadb.org/browse/MDEV-37270 [07:19:25] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [08:34:27] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser, 06DBA, and 2 others: Add '*_actor_ip_hex_time' indexes to 'cu_changes', 'cu_log_event', and 'cu_private_event' on WMF wikis - https://phabricator.wikimedia.org/T399728#11016049 (10FCeratto-WMF) 05Open→03In progress [11:19:25] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:52:58] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser, 06Trust and Safety Product Team, and 3 others: Add '*_actor_ip_hex_time' indexes to 'cu_changes', 'cu_log_event', and 'cu_private_event' - https://phabricator.wikimedia.org/T395683#11016587 (10Dreamy_Jazz) [11:53:00] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser, 06DBA, and 3 others: Add '*_actor_ip_hex_time' indexes to 'cu_changes', 'cu_log_event', and 'cu_private_event' on WMF wikis - https://phabricator.wikimedia.org/T399728#11016586 (10Dreamy_Jazz) [12:22:16] 06Data-Engineering, 10CheckUser, 06DBA, 06Trust and Safety Product Team, 07Schema-change: Read from cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946 (10Dreamy_Jazz) 03NEW [12:22:18] 06Data-Engineering, 10CheckUser, 06DBA, 06Trust and Safety Product Team, 07Schema-change: Read from cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016661 (10Dreamy_Jazz) [12:23:13] 06Data-Engineering, 10CheckUser, 06DBA, 06Trust and Safety Product Team, 07Schema-change: Read from cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016665 (10Dreamy_Jazz) [12:24:35] 06Data-Engineering, 10CheckUser, 06DBA, 06Trust and Safety Product Team, 07Schema-change: Read from cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016671 (10Dreamy_Jazz) Requesting DBA feedback on this. This would wait at least u... [12:25:48] 06Data-Engineering, 10CheckUser, 06Data-Persistence, 06Trust and Safety Product Team, 07Schema-change: Read from cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016688 (10Marostegui) [12:26:12] 06Data-Engineering, 10CheckUser, 06Data-Persistence, 06Trust and Safety Product Team, 07Schema-change: Read from cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016691 (10Dreamy_Jazz) [12:26:47] 06Data-Engineering, 10CheckUser, 06Data-Persistence, 06Trust and Safety Product Team, 07Schema-change: Read from cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016693 (10Dreamy_Jazz) [12:30:11] 06Data-Engineering, 10CheckUser, 06Data-Persistence, 06Trust and Safety Product Team, 07Schema-change: Read from cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016703 (10Marostegui) @Dreamy_Jazz would you be able to tell me 1 or... [12:40:10] 06Data-Engineering, 10CheckUser, 06Data-Persistence, 06Trust and Safety Product Team, 07Schema-change: Read from cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016722 (10Dreamy_Jazz) >>! In T399946#11016703, @Marostegui wrote: >... [12:52:02] 06Data-Engineering, 10CheckUser, 06Data-Persistence, 06Trust and Safety Product Team, 07Schema-change: Read from cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016744 (10Dreamy_Jazz) >>! In T399946#11016703, @Marostegui wrote: >... [12:52:55] 06Data-Engineering, 10CheckUser, 06Data-Persistence, 06Trust and Safety Product Team, 07Schema-change: Make cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016748 (10Dreamy_Jazz) [13:01:13] 10Data-Engineering (Q4 2025 April 1st - June 30th): Refine to Hive with Airflow – Post-Migration Cleanup - https://phabricator.wikimedia.org/T392698#11016793 (10Antoine_Quhen) [13:01:34] 10Data-Engineering (Q4 2025 April 1st - June 30th): Refine to Hive with Airflow – Post-Migration Cleanup - https://phabricator.wikimedia.org/T392698#11016800 (10Antoine_Quhen) 05Open→03In progress [13:03:39] 06Data-Engineering, 10CheckUser, 06Data-Persistence, 06Trust and Safety Product Team, 07Schema-change: Make cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016804 (10Marostegui) Thank you - so sizes: * cu_changes - The largest ta... [13:03:43] 10Data-Engineering (Q4 2025 April 1st - June 30th): Refine to Hive with Airflow – Kubernetes Resource Optimization - https://phabricator.wikimedia.org/T392668#11016806 (10Antoine_Quhen) Linked with T396686 [13:05:11] 10Data-Engineering (Q4 2025 April 1st - June 30th): Investigate Airflow scheduler performance on main deployment - https://phabricator.wikimedia.org/T396686#11016824 (10Antoine_Quhen) 05Open→03Resolved Since refine_to_hive has been moved into its own queue it lets some chances to other dags tasks to be p... [13:05:40] 06Data-Engineering, 10CheckUser, 06Data-Persistence, 06Trust and Safety Product Team, 07Schema-change: Make cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016831 (10Dreamy_Jazz) 05Open→03Declined > From a DBA point of I d... [13:12:35] hi analytics folks. can someone suggest/guide me on if analytics-privatedata-users is really required for "Access to Airflow data analytics"? [13:12:49] context is https://phabricator.wikimedia.org/T399899 and I am asking per my clinic duty responsibilities. [13:13:16] 06Data-Engineering, 10CheckUser, 06Data-Persistence, 06Trust and Safety Product Team, 07Schema-change: Make cuc_ip_hex, cule_ip_hex, and cupe_ip_hex not nullable and remove the default - https://phabricator.wikimedia.org/T399946#11016855 (10Marostegui) Thank you! [13:34:51] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#11016959 (10Marostegui) [13:35:45] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#11016962 (10Marostegui) [13:45:48] 10Data-Engineering (Q4 2025 April 1st - June 30th): Refine to Hive with Airflow – Switch Over day scripts - https://phabricator.wikimedia.org/T392696#11016999 (10Antoine_Quhen) 05Open→03Resolved * https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1550 * https://gitlab.wik... [13:48:11] 06Data-Engineering, 10CheckUser, 06Trust and Safety Product Team: Inform Data-Engineering about removal of cuc_ip, cule_ip, and cupe_ip - https://phabricator.wikimedia.org/T399958 (10Dreamy_Jazz) 03NEW [14:00:24] 10Data-Engineering (Q4 2025 April 1st - June 30th): Refine to Hive with Airflow – Update Refine Documentation on Wikitech - https://phabricator.wikimedia.org/T392697#11017057 (10Antoine_Quhen) [14:49:16] 10Data-Engineering (Q4 2025 April 1st - June 30th): Move more of refine_hive_hourly dag logic into RefineConfiguration - https://phabricator.wikimedia.org/T375064#11017224 (10Antoine_Quhen) The goal of this ticket remains to move more logic into RefineConfiguration. However, we now have multiple objectives: * Ce... [14:56:28] 10Data-Engineering (Q4 2025 April 1st - June 30th): Refine to Hive with Airflow – Kubernetes Resource Optimization - https://phabricator.wikimedia.org/T392668#11017244 (10Antoine_Quhen) Next steps are: * analytics_test: skipping skein and using Spark local in pod. * main: running small tasks with LocalExecutor.... [15:19:25] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [17:11:42] 06Data-Engineering, 06Growth-Team, 10GrowthExperiments, 10Discovery-Search (2025.07.04 - 2025.07.25), and 2 others: Error for mediawiki.cirrussearch-request: '' should NOT have additional properties - https://phabricator.wikimedia.org/T399965#11017531 (10EBernhardson) a:03EBernhardson [19:19:25] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:23:21] !log mw_content_merge_events_to_mw_content_history_daily precursor: refine_event_sanitized_main_immediate on mediawiki_page_content_change_v1 for 2025-07-15T06:00:00.000Z to 2025-07-15T10:00:00.000Z' [19:23:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:04:11] !log mw_content_merge_events_to_mw_content_history_daily precursor: refine_event on mediawiki_page_content_change_v1 for 2025-07-15T06:00:00.000Z to 2025-07-16T00:00:00.000Z' [20:04:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:13:34] ^^ and ^^^^ didn't really achieve anything. As part of Ops Week, this is on a Slack thread to try to figure out Monday morning when folks are back online if there's a different approach. [23:19:25] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem