[00:02:05] 07Analytics-Data-Problem, 06Design-System-Team, 06MW-Interfaces-Team, 06Web-Team-Backlog, 07JavaScript: Send Api-User-Agent header from MediaWiki client-side code - https://phabricator.wikimedia.org/T373874#10209050 (10Krinkle) What's the use case? Where do we want to see it used, and why? I suggest re-t... [03:32:15] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Publish Data Engineering maintained NodeJS packages to GitLab and use them in depender code - https://phabricator.wikimedia.org/T366612#10209228 (10tchin) [03:35:35] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Publish Data Engineering maintained NodeJS packages to GitLab and use them in depender code - https://phabricator.wikimedia.org/T366612#10209233 (10tchin) Also added node-rdkafka-prometheus to the list [04:21:32] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Publish Data Engineering maintained NodeJS packages to GitLab and use them in depender code - https://phabricator.wikimedia.org/T366612#10209285 (10tchin) [04:22:59] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Publish Data Engineering maintained NodeJS packages to GitLab and use them in depender code - https://phabricator.wikimedia.org/T366612#10209286 (10tchin) [[ https://gitlab.wikimedia.org/repos/data-engineering/node-rdkafka-prometheus |... [04:26:51] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Publish Data Engineering maintained NodeJS packages to GitLab and use them in depender code - https://phabricator.wikimedia.org/T366612#10209288 (10tchin) Oh jeez there's also [[ https://github.com/wikimedia/node-url-get/tree/master |... [05:05:49] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Publish Data Engineering maintained NodeJS packages to GitLab and use them in depender code - https://phabricator.wikimedia.org/T366612#10209397 (10tchin) [08:49:52] 10Data-Engineering (Q1 2024 July 1st - September 30th): Some Gobblin folders don't have `_IMPORTED` flags - https://phabricator.wikimedia.org/T376144#10209695 (10JAllemandou) I've decided to implement the solution number 2, updating the Gobblin algorithm flagging folders. This allows to keep the solution interna... [08:50:22] 10Data-Engineering (Q1 2024 July 1st - September 30th): Some Gobblin folders don't have `_IMPORTED` flags - https://phabricator.wikimedia.org/T376144#10209696 (10JAllemandou) I'm also taking this as an opportunity to refactor tests for this algorithm. [09:45:25] (03PS1) 10Joal: Update time-partitioned-flag data publisher [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/1078623 (https://phabricator.wikimedia.org/T376144) [09:47:03] (03CR) 10CI reject: [V:04-1] Update time-partitioned-flag data publisher [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/1078623 (https://phabricator.wikimedia.org/T376144) (owner: 10Joal) [10:06:40] 06Data-Engineering, 06serviceops, 06SRE-OnFire, 10Event-Platform: Incident: 2022-12-09 api appserver worker starvation - https://phabricator.wikimedia.org/T324994#10209907 (10Clement_Goubert) 05Open→03Resolved No more action needed on this incident. [10:08:01] 06Data-Engineering, 06Data-Platform-SRE, 06serviceops-radar, 06SRE-OnFire, and 2 others: Uneven CPU throttling of eventgate-analytics under load - https://phabricator.wikimedia.org/T325068#10209933 (10Clement_Goubert) [10:21:53] (03PS2) 10Joal: Update time-partitioned-flag data publisher [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/1078623 (https://phabricator.wikimedia.org/T376144) [10:25:40] 06Data-Engineering, 10EventStreams, 10Observability-Tracing, 10Prod-Kubernetes, and 3 others: eventstreams regularly uses more than 95% of its memory limit - https://phabricator.wikimedia.org/T357005#10210044 (10Clement_Goubert) Maybe some internal OpenTelemetry instrumentation could help shed some light o... [11:47:48] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for Seanleong-WMDE - https://phabricator.wikimedia.org/T376034#10210273 (10kamila) 05In progress→03Resolved [12:55:28] 10Data-Engineering (Q1 2024 July 1st - September 30th): Migrate and re-deploy eventstreams using service-utils - https://phabricator.wikimedia.org/T361769#10210437 (10tchin) Just saw that `service-utils` passed security review, so the only step right now is just to deploy the patch to production [13:52:32] (03PS4) 10Dbrant: Add is_temp field to iOS schema fragment. [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1077742 (https://phabricator.wikimedia.org/T374058) [14:50:16] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Some Gobblin folders don't have `_IMPORTED` flags - https://phabricator.wikimedia.org/T376144#10211061 (10Ottomata) From the patch: ` * - For low-volume topics (sum of records written below threshold): * Any time-partitio... [15:38:51] (03PS2) 10Snwachukwu: Edit Repo Config [schemas/event/primary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1078704 [15:41:42] (03Abandoned) 10Snwachukwu: Edit Repo Config [schemas/event/primary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1078704 (owner: 10Snwachukwu) [15:41:59] (03Abandoned) 10Snwachukwu: Edit Repo Config [schemas/event/primary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1071896 (owner: 10Snwachukwu) [15:43:09] (03PS2) 10Snwachukwu: Edit Repo Config [schemas/event/primary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1078705 [15:43:15] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Edit Repo Config [schemas/event/primary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1078705 (owner: 10Snwachukwu) [16:27:00] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Some Gobblin folders don't have `_IMPORTED` flags - https://phabricator.wikimedia.org/T376144#10211457 (10JAllemandou) Our algorithm about flagging folders is working at topic-partition-level for high-volume topics, and at topic-level... [16:31:59] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Some Gobblin folders don't have `_IMPORTED` flags - https://phabricator.wikimedia.org/T376144#10211465 (10Ottomata) > keyed-topics is theoretical for us as we don't use them FWIW, we do use them: https://wikitech.wikimedia.org/wiki/Ev... [16:41:10] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Some Gobblin folders don't have `_IMPORTED` flags - https://phabricator.wikimedia.org/T376144#10211490 (10JAllemandou) Ah! I had forgotten :) But this st4ream is not ingested in hadoop. Anyhow, if you agree with the approach, I can imp... [16:54:26] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data-Platform-SRE (2024.09.28 - 2024.10.18), 10Event-Platform: Migrate Event Platform Schema Respositories to Gitlab - https://phabricator.wikimedia.org/T366836#10211527 (10Snwachukwu) The switchover has been done. The gerrit repositories are deprecat... [17:19:14] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Some Gobblin folders don't have `_IMPORTED` flags - https://phabricator.wikimedia.org/T376144#10211643 (10Ottomata) It isn't yet, but I think it will be? There are others that are ingested. mediawiki.page_change.v1, but it only has o... [17:29:11] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data-Platform-SRE (2024.09.28 - 2024.10.18), 10Event-Platform: Migrate Event Platform Schema Respositories to Gitlab - https://phabricator.wikimedia.org/T366836#10211703 (10Ottomata) !!! amazing! [18:51:13] 06Data-Engineering, 10Event-Platform, 10Web Team Essential Work 2024 (Migrate to new Event Platform), 10Web-Team-Backlog (FY2024-25 Q2 Sprint 1): Deprecate use of desktop- and mobilewebuiactions in Event Platform - https://phabricator.wikimedia.org/T368678#10212092 (10Jdrewniak) a:03bwang [18:57:21] (03PS1) 10Milimetric: Shift is_redirect_to_pageview upstream to webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1078733 (https://phabricator.wikimedia.org/T375527) [19:00:17] 06Data-Engineering, 10Data Pipelines: Add support for repository artifacts in Airflow - https://phabricator.wikimedia.org/T322690#10212146 (10Ottomata) @mforns in [[ https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/merge_requests/45#note_109467 | this workflow_utils MR ]], @amastilovic and... [19:04:52] 06Data-Engineering, 06Data-Platform, 10Dumps-Generation, 06SRE, and 3 others: Hide autoblocks from the globalblocks table database dump - https://phabricator.wikimedia.org/T376726#10212150 (10Ottomata) [19:17:39] 06Data-Engineering, 10CheckUser, 06Privacy Engineering: Add cu_log_event and cu_private_event CheckUser tables to data lake - https://phabricator.wikimedia.org/T376752 (10Tgr) 03NEW [19:35:59] 06Data-Engineering, 10CheckUser, 06Privacy Engineering: Add cu_log_event and cu_private_event CheckUser tables to data lake - https://phabricator.wikimedia.org/T376752#10212273 (10Dreamy_Jazz) You may also want to consider including `cu_useragent`, `cu_useragent_clienthints` and `cu_useragent_clienthints_map... [19:36:21] 06Data-Engineering, 10CheckUser, 06Privacy Engineering: Add cu_log_event and cu_private_event CheckUser tables to data lake - https://phabricator.wikimedia.org/T376752#10212263 (10Dreamy_Jazz) As long as the data is properly purged and access is restricted to those which have signed an NDA for accessing the... [23:50:44] 06Data-Engineering, 10Research-engineering, 06Research-Freezer, 10Event-Platform: [Research Engineering Request] Productionized Edit Types - https://phabricator.wikimedia.org/T351225#10212892 (10XiaoXiao-WMF)