[01:52:23] 06Data-Engineering, 10AbuseFilter, 06Data-Persistence, 10MediaWiki-extensions-IPReputation, and 3 others: AbuseFilter protected variables: Make it possible for protected variable values expire when the IP address expires - https://phabricator.wikimedia.org/T390873#10711407 (10Bugreporter) As I said above,... [02:01:13] 06Data-Engineering, 10MediaWiki-DomainEvents, 10Event-Platform: Port EventBus PageChangeHooks to Domain Events - https://phabricator.wikimedia.org/T390969#10711423 (10Ottomata) > We don't need and EventIngress class per listener, but a generic one for all page state changes (`PageStateIngress`?) We had the... [07:49:53] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-Blocks, 10Multiblocks, and 5 others: Add a unique index to the block_target table - https://phabricator.wikimedia.org/T389028#10711714 (10matej_suchanek) [08:15:23] 06Data-Engineering, 06Machine-Learning-Team, 06Research, 10Event-Platform, 13Patch-For-Review: Emit revision revert risk scores as a stream and expose in EventStreams API - https://phabricator.wikimedia.org/T326179#10711809 (10achou) @Ottomata @kevinbazira I just realized we probably don't want `language... [08:18:58] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE, 06SRE, 10SRE-Access-Requests: Requesting Kerberos access for ben.buchenau - https://phabricator.wikimedia.org/T390734#10711837 (10Gehel) p:05Triage→03High [08:19:07] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Requesting Kerberos access for ben.buchenau - https://phabricator.wikimedia.org/T390734#10711839 (10Gehel) [08:26:07] 06Data-Engineering, 06Data-Platform-SRE: Create views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10711852 (10Gehel) @Bugreporter : It's not very clear what you are expecting here. Could you provide more details so we can understand what needs to be done without too much investigation? [08:26:44] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Rebuild Spark images with Bookworm / bullseye-backports deprecation - https://phabricator.wikimedia.org/T390139#10711854 (10Gehel) [08:27:05] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Rebuild Spark images with Bookworm / bullseye-backports deprecation - https://phabricator.wikimedia.org/T390139#10711855 (10Gehel) p:05Triage→03Medium [08:44:25] 06Data-Engineering, 06Data-Platform-SRE: Create views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10711897 (10Bugreporter) Currently there is no view of these three tables in Cloud, so they are not queryable in Cloud. [08:46:01] 06Data-Engineering, 10Data-Platform-SRE (2025.03.22 - 2025.04.11), 07Epic: HDFS capacity needs FY24/25 - https://phabricator.wikimedia.org/T384098#10711904 (10Gehel) 05Open→03Resolved [09:18:52] 06Data-Engineering, 06Data-Platform-SRE, 10Data-Services: Create views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10711997 (10taavi) [09:19:01] 06Data-Engineering, 06Data-Platform-SRE, 10Data-Services: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10711998 (10taavi) [09:36:21] 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: [Refine Refactoring] Refine jobs should be scheduled by Airflow: deployment - https://phabricator.wikimedia.org/T369845#10712045 (10Antoine_Quhen) The last version of the dag has been deployed on Airflow analytics_test & main. 2 errors we... [11:00:18] 10Data-Engineering (Q4 2025 April 1st - June 30th): Some search entries in wmf.webrequest have their query appended to their uri_path - https://phabricator.wikimedia.org/T383135#10712305 (10JAllemandou) This is still a thing with HAProxy data: ` scala> spark.sql(""" | SELECT | normalized_host.proje... [11:22:13] 06Data-Engineering, 10AbuseFilter, 06Data-Persistence, 10MediaWiki-extensions-IPReputation, and 3 others: AbuseFilter protected variables: Make it possible for protected variable values expire when the IP address expires - https://phabricator.wikimedia.org/T390873#10712394 (10Ladsgroup) >>! In T390873#1071... [11:35:16] 06Data-Engineering, 10MediaWiki-DomainEvents, 10Event-Platform: Port EventBus PageChangeHooks to Domain Events - https://phabricator.wikimedia.org/T390969#10712440 (10gmodena) [11:40:06] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: Product Analytics ETL Migration: Pilot (MediaSearch ETLs) - https://phabricator.wikimedia.org/T333208#10712465 (10mpopov) a:05mpopov→03None [11:43:38] 06Data-Engineering, 10MediaWiki-DomainEvents, 10Event-Platform: Port EventBus PageChangeHooks to Domain Events - https://phabricator.wikimedia.org/T390969#10712484 (10gmodena) >>! In T390969#10711423, @Ottomata wrote: >> We don't need and EventIngress class per listener, but a generic one for all page state... [12:11:52] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop afl_patrolled_by from abuse_filter_log in production - https://phabricator.wikimedia.org/T391056#10712632 (10Marostegui) a:03FCeratto-WMF [13:29:23] (03CR) 10TChin: [C:03+2] Add columns to data_quality_alerts to support inserting ResultKey [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1127967 (https://phabricator.wikimedia.org/T384962) (owner: 10TChin) [13:29:30] (03CR) 10TChin: [C:03+2] Support inserting ResultKey into DeequVerificationSuiteToDataQualityAlerts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1127964 (https://phabricator.wikimedia.org/T384962) (owner: 10TChin) [13:29:49] (03CR) 10TChin: [V:03+2 C:03+2] Add columns to data_quality_alerts to support inserting ResultKey [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1127967 (https://phabricator.wikimedia.org/T384962) (owner: 10TChin) [13:41:08] (03Merged) 10jenkins-bot: Support inserting ResultKey into DeequVerificationSuiteToDataQualityAlerts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1127964 (https://phabricator.wikimedia.org/T384962) (owner: 10TChin) [13:51:35] Starting build #30 for job analytics-refinery-maven-release [13:59:43] (03CR) 10TChin: [V:03+2 C:03+2] Add columns to data_quality_alerts to support inserting ResultKey (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1127967 (https://phabricator.wikimedia.org/T384962) (owner: 10TChin) [14:14:40] Project analytics-refinery-maven-release build #30: 09SUCCESS in 23 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/30/ [14:29:06] Starting build #27 for job analytics-refinery-update-jars [14:29:41] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.60 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1134238 [14:29:42] Project analytics-refinery-update-jars build #27: 09SUCCESS in 35 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/27/ [14:31:23] (03CR) 10TChin: [V:03+2 C:03+2] Add refinery-source jars for v0.2.60 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1134238 (owner: 10Maven-release-user) [14:43:02] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Implement alerting for wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T384962#10713108 (10tchin) Altered table: `sql ALTER TABLE wmf_data_ops.data_quality_alerts ADD COLUMNS ( da... [14:49:46] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Figure root cause of silent failures when computing metrics for mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T387033#10713133 (10xcollazo) 05Open→03Resolved [14:50:11] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: mw_content_reconcile_mw_content_history_monthly is not sensing correctly - https://phabricator.wikimedia.org/T390783#10713138 (10xcollazo) 05In progress→03Resolved [15:10:03] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 13Patch-For-Review: [Data Quality] Add ability to add tags to alerts - https://phabricator.wikimedia.org/T389162#10713232 (10tchin) [15:10:24] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Canary failure on airflow platform_eng intsance after migrating to Kubernetes - https://phabricator.wikimedia.org/T390727#10713234 (10xcollazo) Even after [[ https://gitlab.wikimedia.org/repos/data-engineering/ai... [15:35:13] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 13Patch-For-Review: [Data Quality] Add ability to add tags to alerts - https://phabricator.wikimedia.org/T389162#10713419 (10tchin) [15:44:03] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 13Patch-For-Review: [Data Quality] Add ability to add tags to alerts - https://phabricator.wikimedia.org/T389162#10713452 (10tchin) [15:51:40] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 13Patch-For-Review: [Data Quality] Add ability to add tags to alerts - https://phabricator.wikimedia.org/T389162#10713474 (10tchin) Seems to be working, `webrequest_analyzer` dag runs normally and I can... [15:54:16] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Investigate artifact mismatch error when running mw_content_merge_events_to_mw_content_history_daily - https://phabricator.wikimedia.org/T391123 (10xcollazo) 03NEW [15:54:29] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Investigate artifact mismatch error when running mw_content_merge_events_to_mw_content_history_daily - https://phabricator.wikimedia.org/T391123#10713499 (10xcollazo) a:03xcollazo [16:02:27] !log merged changes to alter refine_webrequest checker thresholds https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1217 [16:02:28] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:10:12] 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Migrate Gobblin to Airflow - https://phabricator.wikimedia.org/T390249#10713805 (10Ahoelzl) p:05Triage→03High [17:11:22] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work: Migrate Gobblin to Airflow - https://phabricator.wikimedia.org/T388246#10713820 (10Ahoelzl) →14Duplicate dup:03T390249 [17:11:26] 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Migrate Gobblin to Airflow - https://phabricator.wikimedia.org/T390249#10713822 (10Ahoelzl) [17:11:42] 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Migrate Gobblin to Airflow - https://phabricator.wikimedia.org/T390249#10713824 (10Ahoelzl) [17:13:25] 06Data-Engineering, 06Product-Analytics: Iceberg table maintenance for tables under wmf_product database - https://phabricator.wikimedia.org/T391135 (10KCVelaga_WMF) 03NEW [17:30:37] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Investigate artifact mismatch error when running mw_content_merge_events_to_mw_content_history_daily - https://phabricator.wikimedia.org/T391123#10713932 (10xcollazo) [17:36:02] 06Data-Engineering: [Iceberg Migration] Extend Iceberg table maintenance mechanism to support multiple Airflow instances - https://phabricator.wikimedia.org/T373693#10713944 (10xcollazo) [17:36:03] 06Data-Engineering, 06Product-Analytics: Iceberg table maintenance for tables under wmf_product database - https://phabricator.wikimedia.org/T391135#10713945 (10xcollazo) [17:38:37] 06Data-Engineering, 06Product-Analytics: Iceberg table maintenance for tables under wmf_product database - https://phabricator.wikimedia.org/T391135#10713958 (10KCVelaga_WMF) [17:42:56] 06Data-Engineering: [Iceberg Migration] Extend Iceberg table maintenance mechanism to support multiple Airflow instances - https://phabricator.wikimedia.org/T373693#10713964 (10xcollazo) [17:43:34] 06Data-Engineering: [Iceberg Migration] Extend Iceberg table maintenance mechanism to support multiple Airflow instances - https://phabricator.wikimedia.org/T373693#10713969 (10xcollazo) Perhaps we should do this work jointly with {T383931}? [19:14:12] 14Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content, 06Research: A dataset sensor should work independent of airflow instance  - https://phabricator.wikimedia.org/T386973#10714182 (10xcollazo) [19:15:54] !log reran commons_impact_metrics_monthly for 2025-03 after an allow list update [19:15:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:42:49] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 10Event-Platform: Port EventBus PageChangeHooks to Domain Events - https://phabricator.wikimedia.org/T390969#10714442 (10Ahoelzl) [20:43:43] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 10Event-Platform: Port EventBus PageChangeHooks to Domain Events - https://phabricator.wikimedia.org/T390969#10714443 (10Ahoelzl)