[00:42:57] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Request to apply schema change in production: Add cea_country_code column to ce_address table (X1 DB) - https://phabricator.wikimedia.org/T398664 (10cmelo) 03NEW [00:53:16] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Request to apply schema change in production: Add cea_country_code column to ce_address table (X1 DB) - https://phabricator.wikimedia.org/T398664#10973802 (10cmelo) [01:20:45] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Request to apply schema change in production: Add cea_country_code column to ce_address table (X1 DB) - https://phabricator.wikimedia.org/T398664#10973811 (10Daimona) [01:26:51] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Request to apply schema change in production: Add cea_country_code column to ce_address table (X1 DB) - https://phabricator.wikimedia.org/T398664#10973815 (10Daimona) [03:18:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:18:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [08:24:57] 06Data-Engineering, 10Data-Platform-SRE (2025.06.13 - 2025.07.04): Errors from refinery-sqoop-whole-mediawiki.service - July 2025 - https://phabricator.wikimedia.org/T398602#10974134 (10BTullis) That finished successfully. Moving on to `/usr/local/bin/refinery-sqoop-mediawiki-not-history` [08:27:34] 06Data-Engineering, 10Data-Platform-SRE (2025.06.13 - 2025.07.04): Errors from refinery-sqoop-whole-mediawiki.service - July 2025 - https://phabricator.wikimedia.org/T398602#10974138 (10BTullis) It looks like we got another error from `etwiki.ipblocks` during this run. ` 2025-07-04T08:26:27 ERROR ERROR: etwik... [09:10:19] 06Data-Engineering, 10Data-Platform-SRE (2025.06.13 - 2025.07.04): Errors from refinery-sqoop-whole-mediawiki.service - July 2025 - https://phabricator.wikimedia.org/T398602#10974250 (10BTullis) We have identified a link between this chain of errors and the work on {T390767} Currently working on a fix with @JA... [09:32:44] (03PS1) 10Joal: Move sqooped ipblocks table to private [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1166354 (https://phabricator.wikimedia.org/T398602) [09:34:03] (03CR) 10Btullis: [C:03+1] "Looks good to me." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1166354 (https://phabricator.wikimedia.org/T398602) (owner: 10Joal) [09:39:11] (03PS2) 10Joal: Move sqooped ipblocks table to private [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1166354 (https://phabricator.wikimedia.org/T398602) [09:39:29] (03CR) 10Joal: [V:03+2 C:03+2] "Merging" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1166354 (https://phabricator.wikimedia.org/T398602) (owner: 10Joal) [09:57:54] 06Data-Engineering, 10Data-Platform-SRE (2025.06.13 - 2025.07.04), 13Patch-For-Review: Errors from refinery-sqoop-whole-mediawiki.service - July 2025 - https://phabricator.wikimedia.org/T398602#10974488 (10BTullis) Making progress now. @JAllemandou identified the issue with the user table and fixed it with:... [09:59:53] 06Data-Engineering, 10Data-Platform-SRE (2025.06.13 - 2025.07.04), 13Patch-For-Review: Errors from refinery-sqoop-whole-mediawiki.service - July 2025 - https://phabricator.wikimedia.org/T398602#10974490 (10BTullis) Now continuing to run this in a screen session on an-launcher1002. ` analytics@an-launcher1002... [10:00:52] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Request to apply schema change in production: Add cea_country_code column to ce_address table (X1 DB) - https://phabricator.wikimedia.org/T398664#10974494 (10Ladsgroup) 05Openā†’03Resolved a:03Ladsgroup [11:18:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:31:55] (03CR) 10Aqu: [C:04-1] "This MR is in standby." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1166132 (https://phabricator.wikimedia.org/T369845) (owner: 10Aqu) [11:36:13] 10Data-Engineering (Q4 2025 April 1st - June 30th): Facilitate automatic artifact cache warming for airflow-dags artifacts - https://phabricator.wikimedia.org/T392244#10974904 (10BTullis) >>! In T392244#10851643, @JAllemandou wrote: >>>! In T392244#10851595, @Antoine_Quhen wrote: >> Iā€™d like to hold off on decom... [11:50:01] 10Data-Engineering (Q4 2025 April 1st - June 30th): Facilitate automatic artifact cache warming for airflow-dags artifacts - https://phabricator.wikimedia.org/T392244#10974926 (10BTullis) >>! In T392244#10960339, @amastilovic wrote: > OK, so now that we have the necessary updates to the `workflow_utils` library... [14:19:27] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Dumps-Generation, 10Data-Platform-SRE (2025.06.13 - 2025.07.04), 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08): The 20250401 dumps haven't started on time because the mediawikiwiki dump from 20250320 is l... - https://phabricator.wikimedia.org/T390839#10975337 [15:18:33] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:59:48] 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Refine to Hive with Airflow ā€“ Handle Late-Arrived Events - https://phabricator.wikimedia.org/T370665#10975524 (10JAllemandou) Latest update on this before leaving for holidays. Gobblin now reports publisher metrics to Prometheus: * files... [16:04:59] 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Structured Data Engineering, and 4 others: Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#10975608 (10BTullis) [16:05:58] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 07Epic: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#10975622 (10BTullis) [16:31:08] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Platform-SRE (2025.07.05 - 2025.07.25): Provide tooling to instantiate ad-hoc temporary Airflow DEV environments - https://phabricator.wikimedia.org/T393521#10975763 (10BTullis) [16:31:19] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Platform-SRE (2025.07.05 - 2025.07.25): Provide tooling to instantiate ad-hoc temporary Airflow DEV environments - https://phabricator.wikimedia.org/T393521#10975765 (10BTullis) 05Openā†’03Resolved a:03BTullis [17:27:59] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 2 others: Hypothesis 5.2.13: EventBus Adoption of Domain Events - https://phabricator.wikimedia.org/T391254#10975858 (10gmodena) [19:18:33] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:59:12] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 10Discovery-Search (2025.06.13 - 2025.07.04): Flink: Update k8s operator to 1.12.0 - https://phabricator.wikimedia.org/T398162#10976061 (10brouberol) [20:00:21] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 10Discovery-Search (2025.06.13 - 2025.07.04): Flink: Update k8s operator to 1.12.0 - https://phabricator.wikimedia.org/T398162#10976062 (10brouberol) [23:18:33] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem