[07:10:11] (03CR) 10Santiago Faci: [C:03+2] Update Metrics Platform web base major version bump with common updates: [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1071663 (https://phabricator.wikimedia.org/T366802) (owner: 10Clare Ming) [07:10:35] (03Merged) 10jenkins-bot: Update Metrics Platform web base major version bump with common updates: [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1071663 (https://phabricator.wikimedia.org/T366802) (owner: 10Clare Ming) [07:45:29] (03CR) 10Joal: "I think I would add a one liner before the first comment line saying that this file is a copy from etc." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1071624 (https://phabricator.wikimedia.org/T372677) (owner: 10Milimetric) [10:37:08] 06Data-Engineering, 10Dumps 2.0, 10Event-Platform: [SPIKE] how can we support Spark producer/consumers in Event Platform - https://phabricator.wikimedia.org/T374341#10133229 (10gmodena) @dcausse @pfischer and I had a chat about this phab today. Here are some notes from our conversation, Search has two use c... [10:39:49] 06Data-Engineering, 10Dumps 2.0, 10Event-Platform: [SPIKE] how can we support Spark producer/consumers in Event Platform - https://phabricator.wikimedia.org/T374341#10133236 (10gmodena) [11:00:51] 06Data-Engineering, 10CheckUser, 06Data Products, 06DBA, 07Schema-change-in-production: Remove cuc_actiontext, cuc_only_for_read_old, and cuc_private from cu_changes on WMF wikis - https://phabricator.wikimedia.org/T370903#10133296 (10Ladsgroup) [11:06:06] 06Data-Engineering, 06Data-Platform, 06DBA, 07Schema-change-in-production: Change page.page_links_updated to fixed-length timestamp in wmf wikis - https://phabricator.wikimedia.org/T371742#10133321 (10Ladsgroup) [12:19:27] 06Data-Engineering, 10Data-Platform-SRE (2024.09.06 - 2024.09.27): an-launcher1002 /srv filling up mostly because of logs from dynamic mapped Airflow tasks - https://phabricator.wikimedia.org/T370437#10133585 (10BTullis) 05Open→03Resolved I have applied the change and purged the logs from an-launcher10... [12:24:05] 06Data-Engineering, 10Data Pipelines, 10Data-Platform-SRE (2024.09.06 - 2024.09.27): [Airflow] Add log rotation to scheduler logs - https://phabricator.wikimedia.org/T315326#10133594 (10BTullis) 05Open→03Resolved On reflection, I believe that this ticket can now be called done. We have a mechanism th... [13:03:25] 06Data-Engineering, 10Dumps 2.0, 10Event-Platform: [SPIKE] how can we support Spark producer/consumers in Event Platform - https://phabricator.wikimedia.org/T374341#10133796 (10Ottomata) FYI, CanaryEventsProducer in wikimedia-event-utilities java has [[ https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedi... [13:04:13] 06Data-Engineering, 10Dumps 2.0, 10Event-Platform: [SPIKE] how can we support Spark producer/consumers in Event Platform - https://phabricator.wikimedia.org/T374341#10133799 (10Ottomata) > High volume: Image Suggestions + Weighted Tags. What is high volume in this case? I would love to have a Spark->Event... [13:04:47] 06Data-Engineering, 06Data-Persistence, 10Temporary accounts, 10Event-Platform: Define MediaWiki user types - https://phabricator.wikimedia.org/T336176#10133803 (10kostajh) [13:06:06] 06Data-Engineering, 06Data-Persistence, 10Temporary accounts, 10Event-Platform: Define MediaWiki user types - https://phabricator.wikimedia.org/T336176#10133816 (10kostajh) @Ottomata do you want to do anything else with this task ahead of pilot wiki deployments in October? [13:33:54] !log About to deploy analytics/refinery using scap [13:33:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:43:43] (03CR) 10Xcollazo: "I know there is a mechanism so that we can call your Data Source via its short name, `wmf-jdbc`, instead of the FQDN. I think it is this c" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1071624 (https://phabricator.wikimedia.org/T372677) (owner: 10Milimetric) [14:06:26] 06Data-Engineering: Handle Late-Arrived Events from Gobblin into Airflow triggered Refine - https://phabricator.wikimedia.org/T370665#10134169 (10Ottomata) Today we did a bit of pair coding but ran out of time. Parallelized Antione's snippet here: https://gitlab.wikimedia.org/-/snippets/166 [14:07:55] !log Deployed refinery using scap, then deployed onto hdfs [14:07:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:30:41] ottomata: ok to merge your change? [14:30:56] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/dba4e70dcdff138801fc390efeac31fd2cda8503 [14:33:09] ottomata: ok forgive me then, I will merge since it will block every other merge as well :> [15:22:50] (03PS2) 10Snwachukwu: Edit Repo Config [schemas/event/primary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1071896 [16:45:09] 06Data-Engineering, 06Data Products, 06Traffic: Prepare puppet configuration to send haproxy logs to haproxykafka socket - https://phabricator.wikimedia.org/T374473 (10Fabfur) 03NEW [17:31:25] 06Data-Engineering, 06Data-Persistence, 10Temporary accounts, 10Event-Platform: Define MediaWiki user types - https://phabricator.wikimedia.org/T336176#10135149 (10Ottomata) @kostajh no, this is probably one of those tasks like T20493 that would make MW's internal data model better, and a MW team should ow... [17:34:16] (03CR) 10Ottomata: "Naming nit:" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1071624 (https://phabricator.wikimedia.org/T372677) (owner: 10Milimetric) [17:36:30] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Migrate Event Platform Schema Respositories to Gitlab - https://phabricator.wikimedia.org/T366836#10135165 (10Ottomata) @Snwachukwu nice! I've asked @gmodena to help review. BTW, before we make these changes, we... [18:01:45] 06Data-Engineering, 06Data-Persistence, 06MediaWiki-Engineering, 10Temporary accounts, 10Event-Platform: Define MediaWiki user types - https://phabricator.wikimedia.org/T336176#10135197 (10kostajh) [18:40:52] (03PS7) 10Milimetric: Implement custom jdbc datasource [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1071624 (https://phabricator.wikimedia.org/T372677) [18:50:37] (03PS8) 10Milimetric: Implement custom jdbc datasource [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1071624 (https://phabricator.wikimedia.org/T372677) [18:50:57] (03CR) 10Milimetric: "thanks! the internets said to add a resource manifest file, so I tried it and it works" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1071624 (https://phabricator.wikimedia.org/T372677) (owner: 10Milimetric) [19:11:26] (03CR) 10Xcollazo: [C:03+1] "LGTM!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1071624 (https://phabricator.wikimedia.org/T372677) (owner: 10Milimetric) [19:16:08] (03PS9) 10Milimetric: Implement custom jdbc datasource [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1071624 (https://phabricator.wikimedia.org/T372677) [19:16:17] (03CR) 10Milimetric: "sounds good to me, replacing" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1071624 (https://phabricator.wikimedia.org/T372677) (owner: 10Milimetric)