[05:12:18] (03PS5) 10David Martin: Create schema for tracking WikiLambda run-function API endpoints [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1019138 (https://phabricator.wikimedia.org/T356228) [05:13:24] (03PS6) 10David Martin: Create schema for tracking WikiLambda run-function API endpoints [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1019138 (https://phabricator.wikimedia.org/T356228) [05:28:54] (03CR) 10David Martin: Create schema for tracking WikiLambda run-function API endpoints (033 comments) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1019138 (https://phabricator.wikimedia.org/T356228) (owner: 10David Martin) [08:41:07] 06Data-Engineering, 10Data Pipelines: Upgrade Airflow to 2.9.1 - https://phabricator.wikimedia.org/T365449#9820138 (10Antoine_Quhen) [08:46:08] 10Data-Engineering (Q4 2024 April 1st - June 30th), 13Patch-For-Review: [Refine refactoring] Extract refine schema management into a dedicated tool - https://phabricator.wikimedia.org/T356762#9820156 (10Antoine_Quhen) [08:48:38] !log deploying AQS device-analytics for T360531 [08:48:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:48:40] T360531: [Commons Impact Metrics] Refactor/create helm config for AQS service accessing both Cassandra and Druid - https://phabricator.wikimedia.org/T360531 [08:53:16] 06Data-Engineering, 10Observability-Logging, 06Traffic: HAProxy should not log information we don't actually need - https://phabricator.wikimedia.org/T365566 (10Fabfur) 03NEW [09:00:32] 06Data-Engineering: Timeout hive-metastore locks - https://phabricator.wikimedia.org/T365563#9820234 (10Aklapper) @Antoine_Quhen: Assuming this task is for #Data-Engineering, hence adding that project tag so other people can also find this task when searching via projects or looking at workboards. Please set ap... [09:06:26] (03PS1) 10Aqu: Add table lock release mechanism for Hive-Metastore [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1034851 (https://phabricator.wikimedia.org/T365563) [09:21:42] (03PS2) 10Aqu: Add table lock release mechanism for Hive-Metastore [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1034851 (https://phabricator.wikimedia.org/T365563) [10:02:30] 06Data-Engineering, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 10 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#9820354 (10Jgiannelos) [13:44:10] 06Data-Engineering: Include geocoded subdivision ISO code in webrequest table - https://phabricator.wikimedia.org/T365594 (10CDanis) 03NEW [13:59:48] 06Data-Engineering, 10probenet: Include geocoded subdivision ISO code in webrequest table - https://phabricator.wikimedia.org/T365594#9821338 (10CDanis) [14:09:50] 14Analytics-Radar, 06Data-Engineering, 06DBA, 10MediaWiki-Categories, and 4 others: {{PAGESINCATEGORY}} returns incorrect value on en-wiki Category:Candidates for speedy deletion - https://phabricator.wikimedia.org/T195397#9821433 (10Krinkle) [14:15:39] 06Data-Engineering, 10Cassandra, 10Data Pipelines, 06Data-Platform-SRE: Create puppet defined type for adding/updating/deleting secrets or other small files on HDFS - https://phabricator.wikimedia.org/T323692#9821484 (10lbowmaker) [14:46:40] 06Data-Engineering, 06Tech-Docs-Team, 05Goal: Define dataset documentation strategy - https://phabricator.wikimedia.org/T349103#9821687 (10TBurmeister) [15:08:18] 06Data-Engineering, 10Cassandra, 10Data Pipelines, 06Data-Platform-SRE: Create puppet defined type for adding/updating/deleting secrets or other small files on HDFS - https://phabricator.wikimedia.org/T323692#9821917 (10xcollazo) @BTullis I believe this is the ticket we need to move forward with not keepin... [15:08:30] 06Data-Engineering, 10Cassandra, 10Data Pipelines, 06Data-Platform-SRE: Create puppet defined type for adding/updating/deleting secrets or other small files on HDFS - https://phabricator.wikimedia.org/T323692#9821921 (10xcollazo) 05Stalled→03Open [15:09:19] 06Data-Engineering, 06Tech-Docs-Team, 05Goal: Define dataset documentation strategy - https://phabricator.wikimedia.org/T349103#9821926 (10TBurmeister) [15:37:52] 06Data-Engineering, 06Java-Scala-Standardization, 07Code-Health: Integrate SonarCloud analysis as part of the analytics refinery builds - https://phabricator.wikimedia.org/T258800#9822096 (10Gehel) It looks like this has been done at some point, analysis is running on each build at this time: https://sonarcl... [15:38:00] 06Data-Engineering, 06Java-Scala-Standardization, 07Code-Health: Integrate SonarCloud analysis as part of the analytics refinery builds - https://phabricator.wikimedia.org/T258800#9822097 (10Gehel) 05Open→03Resolved a:03Gehel [17:28:14] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9822778 (10Ottomata) > state of an application is best done on terms that the application controls, e.g. via the APIs that it exposes to end users. I ag... [17:29:03] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9822779 (10Ottomata) At the risk of being obtuse, let me ask my MW MariaDB question again with different terms: In normal conditions, does MediaWiki a... [18:29:32] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9822935 (10apaskulin) [18:35:13] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 07Spike: [SPIKE] Can we express Event Platform configs in Datasets Config? - https://phabricator.wikimedia.org/T361017#9822955 (10JAllemandou) >>! In T361017#9761303, @Ottomata wrote: >> we had programmatic ways to clean airflow (values?)... [19:24:59] 06Data-Engineering, 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#9823042 (10Ottomata) > Afaik PHP execution is limited fo... [19:43:48] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats, 06Data Products: Pageviews complete dumps have lots of rows with article name = '-' - https://phabricator.wikimedia.org/T365321#9823098 (10JAllemandou) Hi @Audiodude, thank you for reporting your issue. I have double checked the data and here... [20:11:16] (03PS5) 10Xcollazo: SQL queries that format the base Commons Impact Metrics datasets into the expected shape for the 14 Cassandra tables. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1023461 (https://phabricator.wikimedia.org/T358707) [20:19:52] (03CR) 10Xcollazo: [V:03+2] "I have verified all the queries against the Cassandra Staging cluster using COALESCE(3) and year_month='2023-11'." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1023461 (https://phabricator.wikimedia.org/T358707) (owner: 10Xcollazo) [20:34:33] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Datasets Config][Spike] Understand and document the details and conflicts between Datasets Config, Refine refactor, Dynamic EventStreamConfig, and Metrics Platform Instrumentation Configurator - https://phabricator.wikimedia.org/T361853#9823334 (10Ottomata) >... [23:03:10] 10Data-Engineering (Q4 2024 April 1st - June 30th): Update Airflow Developer Guide on WikiTech - https://phabricator.wikimedia.org/T365658 (10amastilovic) 03NEW [23:04:44] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Maintenance] Migrate wmcs to Airflow - https://phabricator.wikimedia.org/T357938#9823782 (10amastilovic) @lbowmaker this task can be closed. [23:05:07] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Maintenance] Migrate pingback to Airflow - https://phabricator.wikimedia.org/T357372#9823783 (10amastilovic) @lbowmaker this task can be closed. [23:05:36] 10Data-Engineering (Q4 2024 April 1st - June 30th): Update converted reportupdater DAG queries to correct CSV options - https://phabricator.wikimedia.org/T362699#9823786 (10amastilovic) @lbowmaker this task can be closed. [23:05:47] 10Data-Engineering (Q4 2024 April 1st - June 30th), 13Patch-For-Review: [Maintenance] Migrate ReportUpdater browser queries to Airflow - https://phabricator.wikimedia.org/T354552#9823788 (10amastilovic) @lbowmaker this task can be closed. [23:10:38] 10Data-Engineering (Q4 2024 April 1st - June 30th): Implement automatic sync of refinery HQL files to HDFS - https://phabricator.wikimedia.org/T365659 (10amastilovic) 03NEW