[00:33:41] 06Data-Engineering, 06Data-Platform: NEW BUG REPORT Cannot add terms in DataHub glossary - https://phabricator.wikimedia.org/T363612#9756026 (10nshahquinn-wmf) 05Open→03Resolved @lbowmaker yes, it works now. Thank you! 😁 [00:37:35] 06Data-Engineering, 06Data-Platform: NEW BUG REPORT Cannot add terms in DataHub glossary - https://phabricator.wikimedia.org/T363612#9756036 (10nshahquinn-wmf) As a side note, why not give this permission to all DataHub users? I don't think we've had any real problems even when things are world-editable on... [02:04:48] 06Data-Engineering: [DQ][NEEDS GROOMING] Add support for deequ's RowLevelSchemaValidator in refinery - https://phabricator.wikimedia.org/T362782#9756145 (10Snwachukwu) Based on the Mediawiki History checker use case, the RowLevelSchemaValidator has some Limitations that may not allow us to use it for our use cas... [11:48:15] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9757336 (10Ottomata) [11:48:38] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9757340 (10Ottomata) ^ changed title to remove the controversial 'source of truth' terminology. [12:04:25] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9757399 (10Ottomata) > highlights some use cases FWIW, this section was gathered as example use cases, not a comprehensive list. I gathered these quotes... [12:09:27] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9757436 (10Ottomata) > starting to measure how consistent the streams are compared to the source of truth? Perhaps {T358373} could be amended to provid... [12:38:38] (03CR) 10Gmodena: Upgrade MediawikiHistory Checker to use AWS Deequ. 1. Update User history checker 2. Update Page history checker 3. Update Denormalized hist (038 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1024423 (https://phabricator.wikimedia.org/T361016) (owner: 10Snwachukwu) [12:42:34] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9757550 (10Ottomata) > simply store a "handled via the job" list of ids for a week in the updater service, and check every day against the list on wikida... [12:46:00] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 07Spike: [SPIKE] Can we express Event Platform configs in Datasets Config? - https://phabricator.wikimedia.org/T361017#9757574 (10Ottomata) > IMHO it should be explicitly stated that the system we are building is the Airflow Dataset Config... [13:45:13] 06Data-Engineering, 06Data-Platform, 06Movement-Insights: Add the global registration date to mediawiki_history - https://phabricator.wikimedia.org/T363775#9757831 (10lbowmaker) [13:46:29] 06Data-Engineering, 10FY2023-24-WE 2.1 Typography and palette customizations, 10Data Products (Data Products Sprint 12), 13Patch-For-Review, 10Web-Team-Backlog (FY2023-24 Q4 Sprint 2): Update Sample Rates for Metrics Platform Events - https://phabricator.wikimedia.org/T361962#9757846 (10phuedx) [13:47:01] 06Data-Engineering, 06Data-Platform, 06Movement-Insights: Add the global registration date to mediawiki_history - https://phabricator.wikimedia.org/T363775#9757850 (10lbowmaker) Moved to discuss with team. @JAllemandou @Milimetric - to be discussed but if this makes sense to implement then maybe we could do... [14:28:16] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117#9758047 (10gmodena) @Fabfur f/up from our chat earlier; these would be the pending config bits that we'll the to finalize whe... [14:34:25] (03CR) 10Aleksandar Mastilovic: [C:03+2] "Verified!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1024530 (owner: 10Cicalese) [14:34:38] (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] Update PHP and version queries [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1024530 (owner: 10Cicalese) [15:14:51] (03CR) 10Xcollazo: [V:03+2 C:03+2] "Synced up with Joseph on 2024-04-26 on these necessary Scoop changes that need to be merged before the 1st of the month." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1023399 (https://phabricator.wikimedia.org/T345771) (owner: 10Joal) [15:39:51] (03CR) 10Xcollazo: [C:03+2] "As discussed with Joseph on 2024-04-26, Patchset 6 and Patchset 7 are the actual changes to cope with pagelinks schema changes as per T355" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1023828 (https://phabricator.wikimedia.org/T355588) (owner: 10Joal) [15:57:58] (03Merged) 10jenkins-bot: Update ClickstreamBuilder [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1023828 (https://phabricator.wikimedia.org/T355588) (owner: 10Joal) [16:16:36] (03PS1) 10Xcollazo: Update changelog for 0.2.38 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1025807 [16:16:59] (03CR) 10Xcollazo: [V:03+2 C:03+2] Update changelog for 0.2.38 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1025807 (owner: 10Xcollazo) [16:20:54] Starting build #4 for job analytics-refinery-maven-release [16:47:09] Project analytics-refinery-maven-release build #4: 09SUCCESS in 26 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/4/ [16:50:18] Starting build #4 for job analytics-refinery-update-jars [16:51:53] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.38 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1024763 [16:51:54] Project analytics-refinery-update-jars build #4: 09SUCCESS in 1 min 35 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/4/ [16:53:05] (03CR) 10Xcollazo: [V:03+2 C:03+2] Add refinery-source jars for v0.2.38 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1024763 (owner: 10Maven-release-user) [16:54:31] !log Deployed refinery-source using jenkins [16:54:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:58:49] !log starting deploy of refinery... [16:58:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:21:29] !log aborting deploy of refinery due to scap global lock held by T358636. Will attempt again in about an hour. [17:21:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:21:32] T358636: etcdmirror does not recover from a cleared waitIndex - https://phabricator.wikimedia.org/T358636 [17:59:10] !log starting deploy of refinery... [17:59:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:33:03] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 07Spike: [SPIKE] Can we express Event Platform configs in Datasets Config? - https://phabricator.wikimedia.org/T361017#9759105 (10gmodena) >>! In T361017#9757573, @Ottomata wrote: >> IMHO it should be explicitly stated that the system we a... [18:58:47] 14Data-Engineering (Sprint 9), 06Data Products: Adapt Sqoop to pagelinks schema change - https://phabricator.wikimedia.org/T345771#9759175 (10xcollazo) Ran the following as the `analytics` user in `an-launcher1002.eqiad.wmnet`: ` USE wmf_raw; DROP TABLE wmf_raw.mediawiki_pagelinks; CREATE EXTERNAL TABLE... [22:24:54] 10Data-Engineering (Q4 2024 April 1st - June 30th), 06Data Products, 13Patch-For-Review: Modify ClickStreamBuilder pipeline to cope with pagelinks schema changes - https://phabricator.wikimedia.org/T355588#9759776 (10xcollazo) ( Could not get to merging the `airflow-dags` MR today, so paused the `clickstream...