[00:41:12] 10Data-Engineering (Q3 2024 January 1st - March 31th), 07Essential-Work, 13Patch-For-Review: Migrate analytics Airflow DAGs to k8s Airflow deployment - https://phabricator.wikimedia.org/T386282#10581192 (10amastilovic) I'm documenting an updated list of steps that should be taken for each DAG being migrated.... [01:38:40] 06Data-Engineering, 10Wmfdata-Python: Deprecate the Hive module - https://phabricator.wikimedia.org/T384541#10581343 (10nshahquinn-wmf) >>! In T384541#10486440, @nshahquinn-wmf wrote: > The main question I have is whether all or virtually all existing queries built for Hive will run without changes on Spark.... [10:35:28] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 06MediaWiki-Platform-Team, and 3 others: Drop module_deps table in WMF prod - https://phabricator.wikimedia.org/T385997#10582021 (10Ladsgroup) 05Open→03Resolved [10:52:19] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop event_variant column from echo_event - https://phabricator.wikimedia.org/T385645#10582080 (10Ladsgroup) Three wikis (metawiki, mediawikiwiki, officewiki, check `wmgEchoCluster`) have their Echo tables in core db instea... [10:54:40] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop event_variant column from echo_event - https://phabricator.wikimedia.org/T385645#10582096 (10Marostegui) >>! In T385645#10582080, @Ladsgroup wrote: > Three wikis (metawiki, mediawikiwiki, officewiki, check `wmgEchoClus... [10:55:49] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop event_variant column from echo_event - https://phabricator.wikimedia.org/T385645#10582103 (10Marostegui) [10:55:52] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop event_variant column from echo_event - https://phabricator.wikimedia.org/T385645#10582106 (10jcrespo) This is T119154. We could reconsider reopening if it makes things better in the long term, if the effort is worth th... [11:01:06] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop event_variant column from echo_event - https://phabricator.wikimedia.org/T385645#10582143 (10Marostegui) [11:04:34] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop event_variant column from echo_event - https://phabricator.wikimedia.org/T385645#10582161 (10Marostegui) [11:06:40] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop event_variant column from echo_event - https://phabricator.wikimedia.org/T385645#10582165 (10Ladsgroup) >>! In T385645#10582106, @jcrespo wrote: > This is T119154. We could reconsider reopening if it makes things bette... [13:14:38] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 7 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10582713 (10xcollazo) I suspected that, due to th... [13:44:05] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 7 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10582875 (10Lucas_Werkmeister_WMDE) FWIW, I would... [15:45:30] 10Data-Engineering (Q3 2024 January 1st - March 31th), 07Essential-Work, 13Patch-For-Review: Migrate analytics Airflow DAGs to k8s Airflow deployment - https://phabricator.wikimedia.org/T386282#10583530 (10Ottomata) Nice! @amastilovic could you update the task description with the chosen 'main' instance sol... [15:57:07] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Abstract Wikipedia team, 10function-evaluator, 10function-orchestrator, 07Essential-Work: WF service logging seems to be partially missing - https://phabricator.wikimedia.org/T386972#10583585 (10Jdforrester-WMF) Huge thanks to @tchin for the fixes,... [16:08:21] 10Data-Engineering (Q3 2024 January 1st - March 31th): Update wikifunctions Grafana dashboard for service-utils - https://phabricator.wikimedia.org/T387360 (10tchin) 03NEW [17:33:16] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10function-evaluator, 10function-orchestrator, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 07Essential-Work: WF service logging seems to be partially missing - https://phabricator.wikimedia.org/T386972#10584128 (10Jdforrester-WMF) p:05Triage→03High [17:33:19] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10function-evaluator, 10function-orchestrator, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 07Essential-Work: WF service logging seems to be partially missing - https://phabricator.wikimedia.org/T386972#10584130 (10Jdforrester-WMF) [17:46:48] 10Data-Engineering (Q3 2024 January 1st - March 31th), 03Abstract Wikipedia Fix-It tasks, 10Abstract Wikipedia team (25Q3 (Jan–Mar)): Update wikifunctions Grafana dashboard for service-utils - https://phabricator.wikimedia.org/T387360#10584270 (10Jdforrester-WMF) [17:57:15] 10Data-Engineering (Q3 2024 January 1st - March 31th), 07Essential-Work: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10584316 (10HCoplin-WMF) Thanks for the detailed update, @Ahoelzl ! As a follow up, I'm still unable to access the dashboard due to permissions issues. Co... [18:12:18] 10Data-Engineering (Q3 2024 January 1st - March 31th), 07Essential-Work: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10584357 (10Ottomata) @HCoplin-WMF you need more access! https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#Dashboards_in_Superset_/_Hive_int... [18:30:48] 10Data-Engineering (Q3 2024 January 1st - March 31th), 07Essential-Work, 13Patch-For-Review: Migrate analytics Airflow DAGs to k8s Airflow deployment - https://phabricator.wikimedia.org/T386282#10584390 (10amastilovic) [18:31:22] 10Data-Engineering (Q3 2024 January 1st - March 31th), 07Essential-Work, 13Patch-For-Review: Migrate analytics Airflow DAGs to k8s Airflow deployment - https://phabricator.wikimedia.org/T386282#10584398 (10amastilovic) @Ottomata done! [18:36:13] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10DPE-Mediawiki-Content, 06Research, 10Research-engineering: Update knowledge gaps pipeline to use article quality features V2 - https://phabricator.wikimedia.org/T386003#10584439 (10Ahoelzl) [19:07:20] 10Data-Engineering (Q3 2024 January 1st - March 31th): Provide Data Engineering Q4 draft - https://phabricator.wikimedia.org/T387385 (10Ahoelzl) 03NEW [19:16:43] 10Data-Engineering (Q3 2024 January 1st - March 31th), 07Essential-Work, 13Patch-For-Review: Migrate analytics Airflow DAGs to k8s Airflow deployment - https://phabricator.wikimedia.org/T386282#10584524 (10Ottomata) TY! [21:20:21] 10Data-Engineering (Q3 2024 January 1st - March 31th), 03Abstract Wikipedia Fix-It tasks, 10Abstract Wikipedia team (25Q3 (Jan–Mar)): Update wikifunctions Grafana dashboard for service-utils - https://phabricator.wikimedia.org/T387360#10584785 (10tchin) Here are the current metrics I'm seeing for `function-o... [23:36:26] 10Data-Engineering (Q3 2024 January 1st - March 31th), 03Abstract Wikipedia Fix-It tasks, 10Abstract Wikipedia team (25Q3 (Jan–Mar)): Update wikifunctions Grafana dashboard for service-utils - https://phabricator.wikimedia.org/T387360#10585355 (10tchin) ==== NodeJS ===== Split by path `lang=diff - sum by(pa...