[01:15:58] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10GitLab (Pipeline Services MigrationšŸ¤), 13Patch-For-Review: Migrate Data Engineering Pipelinelib repos to GitLab - https://phabricator.wikimedia.org/T344730#9840617 (10Snwachukwu) >>! In T344730#9838248, @Ottomata wrote: > Nit: > > Can... [03:27:41] 10Data-Engineering (Q4 2024 April 1st - June 30th): Fix DPE alerts dashboard to work with Google Groups - https://phabricator.wikimedia.org/T365829#9840684 (10tchin) This might be harder than I thought. Creating a dummy google account to act as the receiver seems off the table. All of Google's APIs require OAuth... [05:55:09] (03PS7) 10David Martin: Create schema for tracking WikiLambda run-function API endpoints [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1019138 (https://phabricator.wikimedia.org/T356228) [07:15:09] 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search, 10Dumps-Generation, 13Patch-For-Review: Some dumps are not available since mid may 2024 - https://phabricator.wikimedia.org/T366043#9840839 (10dcausse) @BTullis thanks! Categories are reloaded via a cronjob on all WDQS machine, the job is about... [07:37:02] 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search, 10Dumps-Generation, 13Patch-For-Review: Some dumps are not available since mid may 2024 - https://phabricator.wikimedia.org/T366043#9840883 (10dcausse) >>! In T366043#9840839, @dcausse wrote: > @BTullis thanks! Categories are reloaded via a cro... [09:17:50] 10Analytics-Canonical-Data, 06Movement-Insights, 06Product-Analytics: Add a column to differentiate various wiki-types on canonical_data.wikis - https://phabricator.wikimedia.org/T361211#9841153 (10KCVelaga_WMF) @nshahquinn-wmf What do you think of the changes? As a next step, should we post on #working-w... [11:57:32] 06Data-Engineering, 10Data-Engineering-Wikistats, 10MediaWiki-Special-pages: Statistics - https://phabricator.wikimedia.org/T365952#9841725 (10Kizule) [12:54:10] 06Data-Engineering, 10Data-Engineering-Wikistats, 06Data Products, 10MediaWiki-Special-pages: Statistics - https://phabricator.wikimedia.org/T365952#9841970 (10lbowmaker) [13:23:05] Pause DAG: metadata_ingest_daily for T361185 [13:23:05] T361185: Move datahub to dse-k8s cluster - https://phabricator.wikimedia.org/T361185 [13:26:13] 06Data-Engineering, 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#9842066 (10Ottomata) I have deployed this! Still todo:... [14:01:33] 06Data-Engineering, 10MediaWiki-extensions-WikimediaEvents, 10Observability-Metrics, 10Event-Platform, and 2 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#9842258 (10Ottomata) [14:04:08] !log getting started on moving datahub to dse-k8s T361185 [14:04:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:04:11] T361185: Move datahub to dse-k8s cluster - https://phabricator.wikimedia.org/T361185 [14:20:44] 10Data-Engineering (Q1 2024 July 1st - September 30th): [Data Quality] Implement wiki completeness check for MediaWiki History - https://phabricator.wikimedia.org/T365203#9842348 (10lbowmaker) Notes from chat with @gmodena: - Should be easy enough for someone in the DE team to implement - Maybe 3 points,... [15:27:03] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10GitLab (Pipeline Services MigrationšŸ¤), 13Patch-For-Review: Migrate Data Engineering Pipelinelib repos to GitLab - https://phabricator.wikimedia.org/T344730#9842669 (10Snwachukwu) In regards to [[ https://gerrit.wikimedia.org/g/node-rd... [15:31:28] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10GitLab (Pipeline Services MigrationšŸ¤): Migrate Data Engineering Pipelinelib repos to GitLab - https://phabricator.wikimedia.org/T344730#9842709 (10Jdforrester-WMF) >>! In T344730#9842669, @Snwachukwu wrote: > In regards to [[ https://ge... [15:53:04] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 14), 10MediaWiki-Platform-Team (Radar): Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#9842860 (10VirginiaPoundstone) > Idea... [15:53:24] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 15), 10MediaWiki-Platform-Team (Radar): Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#9842861 (10VirginiaPoundstone) [16:02:06] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10GitLab (Pipeline Services MigrationšŸ¤): Migrate Data Engineering Pipelinelib repos to GitLab - https://phabricator.wikimedia.org/T344730#9842895 (10Snwachukwu) @Jdforrester-WMF thanks for the response. How do we go about this? DO we submi... [16:14:36] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10GitLab (Pipeline Services MigrationšŸ¤): Migrate Data Engineering Pipelinelib repos to GitLab - https://phabricator.wikimedia.org/T344730#9842969 (10tchin) >>! In T344730#9838311, @Ottomata wrote: >> Yes, that's actually what I do for serv... [16:17:40] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10GitLab (Pipeline Services MigrationšŸ¤): Migrate Data Engineering Pipelinelib repos to GitLab - https://phabricator.wikimedia.org/T344730#9843008 (10tchin) Don't forget that any CI that has a production deployment pipeline needs the repo t... [16:59:09] !log deploy airflow-dags to analytics instance to Change the datahub ingestion url T366135 [16:59:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:59:14] T366135: Change datahub ingestion endpoint for all airflow DAGS - https://phabricator.wikimedia.org/T366135 [17:01:48] Unpause DAG: metadata_ingest_daily for T361185 T366135 [17:01:48] T361185: Move datahub to dse-k8s cluster - https://phabricator.wikimedia.org/T361185 [19:02:39] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 15), 10MediaWiki-Platform-Team (Radar): Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#9843723 (10Htriedman) Reading up on t... [19:23:13] 14Analytics, 06Data-Engineering-Icebox, 10ContentTranslation, 10Language-analytics, and 3 others: Special:ContentTranslationStats is slow and getting crowded - https://phabricator.wikimedia.org/T325790#9843796 (10Pginer-WMF) p:05Triageā†’03Medium [20:16:57] (03PS1) 10Xcollazo: Commons Impact Metrics: Reduce the top calculations from top 1000 to top 100. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1037167 (https://phabricator.wikimedia.org/T364583) [20:18:26] (03CR) 10Xcollazo: [V:03+2 C:03+2] "Minor change, self merging." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1037167 (https://phabricator.wikimedia.org/T364583) (owner: 10Xcollazo) [21:47:58] (03PS1) 10Milimetric: Move to newer hosts [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/1037183 [21:48:08] (03CR) 10Milimetric: [V:03+2 C:03+2] Move to newer hosts [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/1037183 (owner: 10Milimetric) [23:15:21] (03CR) 10Jforrester: "Some quick thoughts about excess data, but looks fine." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1019138 (https://phabricator.wikimedia.org/T356228) (owner: 10David Martin)