[06:17:51] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add il_target_id to imagelinks table in wmf production - https://phabricator.wikimedia.org/T413525#11535567 (10Marostegui) [06:29:53] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add il_target_id to imagelinks table in wmf production - https://phabricator.wikimedia.org/T413525#11535588 (10Marostegui) Running it on the old s4 master: db2179 [06:48:21] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11535640 (10Marostegui) [06:48:52] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11535641 (10Marostegui) [06:49:14] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11535642 (10Marostegui) [06:50:25] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11535645 (10Marostegui) [06:51:29] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11535646 (10Marostegui) [06:52:11] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add il_target_id to imagelinks table in wmf production - https://phabricator.wikimedia.org/T413525#11535657 (10Marostegui) [11:33:53] !log Test Kitchen edge-unique experiments (poll 25120) - adds: message; removes: synth-aa-test-traffic-impact; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [11:33:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:34:13] !log Test Kitchen edge-unique experiments (poll 25121) - adds: message; removes: synth-aa-test-traffic-impact; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [11:34:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:34:33] !log Test Kitchen edge-unique experiments (poll 25122) - adds: message; removes: synth-aa-test-traffic-impact; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [11:34:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:34:53] !log Test Kitchen edge-unique experiments (poll 25123) - adds: message; removes: synth-aa-test-traffic-impact; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [11:34:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:35:13] !log Test Kitchen edge-unique experiments (poll 25124) - adds: message; removes: synth-aa-test-traffic-impact; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [11:35:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:35:34] !log Test Kitchen edge-unique experiments (poll 25125) - adds: message; removes: synth-aa-test-traffic-impact; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [11:35:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:37:09] (03PS1) 10Gerrit maintenance bot: Add kaj.wikipedia to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1229091 (https://phabricator.wikimedia.org/T415038) [11:38:05] (03PS1) 10Gerrit maintenance bot: Add ppl.wikipedia to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1229095 (https://phabricator.wikimedia.org/T415046) [12:07:22] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add il_target_id to imagelinks table in wmf production - https://phabricator.wikimedia.org/T413525#11536953 (10Marostegui) [12:07:31] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add il_target_id to imagelinks table in wmf production - https://phabricator.wikimedia.org/T413525#11536954 (10Marostegui) 05Open→03Resolved This is all done [12:09:06] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11536957 (10Marostegui) [13:00:35] (03PS1) 10Santiago Faci: Updated description of a TestKitchen contextual attribute [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1229112 (https://phabricator.wikimedia.org/T414467) [13:10:11] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add il_target_id to imagelinks table in wmf production - https://phabricator.wikimedia.org/T413525#11537212 (10Ladsgroup) \o/ Thank you! [13:42:59] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Test the dbt+skein approach to running dbt Spark jobs in K8s - https://phabricator.wikimedia.org/T414784#11537372 (10amastilovic) The dbt+skein test as outlined in this ticket has been performed successfully: https://airflow-test-k8s.wikimedia.org/dags... [13:43:12] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Test the dbt+skein approach to running dbt Spark jobs in K8s - https://phabricator.wikimedia.org/T414784#11537373 (10amastilovic) [14:16:17] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Migrate refinery-drop-older-than from refinery to Airflow - https://phabricator.wikimedia.org/T414815#11537518 (10Antoine_Quhen) 05Open→03Declined Duplicate of T411999 [14:19:02] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Migrate cleanup jobs for snapshot datasets from systemd timers to Airflow - https://phabricator.wikimedia.org/T411999#11537541 (10Aklapper) [14:35:57] 06Data-Engineering, 10Data-Platform, 06Moderator-Tools-Team: Personal Dashboard Instrumentation Superset Dashboard - https://phabricator.wikimedia.org/T412137#11537582 (10jsn.sherman) [15:07:44] 06Data-Engineering, 10Data-Platform, 06Moderator-Tools-Team, 06Product-Analytics: Personal Dashboard Instrumentation Superset Dashboard - https://phabricator.wikimedia.org/T412137#11537689 (10DMburugu) [15:23:28] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11537796 (10BTullis) >>! In T402512#11529638, @elukey wrote: > Next steps: > > * DP to create the Airflow SRE instance. That's... [15:44:06] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Publish Dumps 2 to dumps.wikimedia.org and provide only monthly dumps - https://phabricator.wikimedia.org/T414389#11537920 (10xcollazo) [15:45:31] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Publish Dumps 2 to dumps.wikimedia.org and provide only monthly dumps - https://phabricator.wikimedia.org/T414389#11537929 (10xcollazo) [15:52:07] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review, 07Spike: Explore how to migrate PyFlink to Java/Scala - https://phabricator.wikimedia.org/T410266#11537969 (10Ottomata) > Please proceed with the plan! I've been reading MRs T360794 for and this has made me won... [15:53:59] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Test the dbt+skein approach to running dbt Spark jobs in K8s - https://phabricator.wikimedia.org/T414784#11537990 (10JMonton-WMF) Just a couple of comments: - About `dbt deps`: I believe Maya is not using (yet) anything from the extra packages, so if... [16:20:24] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 13Patch-For-Review, and 2 others: Migrate 1 instrument using mw.eventLog.newInstrument() to mw.xLab.getInstrument() - https://phabricator.wikimedia.org/T408096#11538115 (10phuedx) This is blocked on input from @TheresNoTime, @Samwal... [16:43:44] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): SDS 1.3.6 SPUR bot detection analysis - https://phabricator.wikimedia.org/T407103#11538277 (10Antoine_Quhen) 05In progress→03Resolved [18:38:54] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11538813 (10Marostegui) [19:02:36] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Test the dbt+skein approach to running dbt Spark jobs in K8s - https://phabricator.wikimedia.org/T414784#11538893 (10Ottomata) > SimpleSkeinOperator supports overriding the default YARN resource configuration e.g. executor memory, number of vcores etc.... [20:33:24] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: Wait till november wmf_raw.mediawiki_slots sqoop table is available, and apply origin_rev_id fix to mw_content tables - https://phabricator.wikimedia.org/T407237#11539276 (10xcollazo) Ran this fix for `mediawiki_content_history_... [20:35:33] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: Wait till november wmf_raw.mediawiki_slots sqoop table is available, and apply origin_rev_id fix to mw_content tables - https://phabricator.wikimedia.org/T407237#11539284 (10xcollazo) Ran this fix for `mediawiki_content_current_... [21:45:31] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: Wait till november wmf_raw.mediawiki_slots sqoop table is available, and apply origin_rev_id fix to mw_content tables - https://phabricator.wikimedia.org/T407237#11539579 (10xcollazo) Ran this fix for `mediawiki_revision_history... [21:46:22] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Rewrite wmf_content.mediawiki_content_*_v1 tables with a new column for origin_rev_id - https://phabricator.wikimedia.org/T405944#11539582 (10xcollazo) >>! In T405944#11272322, @xcollazo wrote: > Ope... [21:46:38] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: Wait till november wmf_raw.mediawiki_slots sqoop table is available, and apply origin_rev_id fix to mw_content tables - https://phabricator.wikimedia.org/T407237#11539583 (10xcollazo) 05Open→03In progress p:05Triage→03Med...