[02:18:56] (03PS1) 10Jdlrobson: Mark repo as archived [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1087611 (https://phabricator.wikimedia.org/T366836) [08:47:55] 06Data-Engineering, 06Traffic, 13Patch-For-Review: Rollout haproxykafka on all hosts - https://phabricator.wikimedia.org/T378578#10295269 (10Fabfur) [08:49:31] 06Data-Engineering, 06Traffic, 13Patch-For-Review: Rollout haproxykafka on all hosts - https://phabricator.wikimedia.org/T378578#10295271 (10Fabfur) Ticket description has been updated to reflect the outcome of latest meeting with @gmodena and @Ahoelzl . [09:34:11] (03PS2) 10Gehel: Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 [09:34:17] (03CR) 10CI reject: [V:04-1] Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [09:34:50] (03PS3) 10Gehel: Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 [09:34:56] (03CR) 10CI reject: [V:04-1] Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [10:24:43] (03PS4) 10Gehel: Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 [10:24:49] (03CR) 10CI reject: [V:04-1] Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [10:27:45] (03PS5) 10Gehel: Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 [10:27:51] (03CR) 10CI reject: [V:04-1] Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [10:39:16] (03PS6) 10Gehel: Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 [10:39:22] (03CR) 10CI reject: [V:04-1] Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [10:58:54] 06Data-Engineering, 06Traffic, 13Patch-For-Review: Rollout haproxykafka on all hosts - https://phabricator.wikimedia.org/T378578#10295610 (10Fabfur) [12:57:38] dse-k8s-etcd1002 will briefly switch to DRBD to move it to a new Ganeti node, latencies will rise during that period [13:02:27] (03PS7) 10Gehel: Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 [13:02:33] (03CR) 10CI reject: [V:04-1] Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [13:27:29] (03PS8) 10Gehel: Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 [13:27:35] (03CR) 10CI reject: [V:04-1] Extraction of RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [13:31:21] 10Data-Engineering (Q2 2024 October 1st - December 31th): [Data Quality] Implement wiki completeness check for MediaWiki History - https://phabricator.wikimedia.org/T365203#10296016 (10Snwachukwu) a:03Snwachukwu [13:33:27] (03CR) 10Ottomata: [C:03+2] Clear repository to make it clearer it has moved [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1087573 (https://phabricator.wikimedia.org/T366836) (owner: 10Jdlrobson) [13:33:29] (03CR) 10Ottomata: [C:03+2] Mark repo as archived [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1087611 (https://phabricator.wikimedia.org/T366836) (owner: 10Jdlrobson) [13:33:44] (03Merged) 10jenkins-bot: Clear repository to make it clearer it has moved [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1087573 (https://phabricator.wikimedia.org/T366836) (owner: 10Jdlrobson) [13:33:46] (03Merged) 10jenkins-bot: Mark repo as archived [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1087611 (https://phabricator.wikimedia.org/T366836) (owner: 10Jdlrobson) [13:46:48] dse-k8s-etcd1002 is moved [13:49:00] 06Data-Engineering, 06Data-Platform-SRE, 06SRE Observability: [Data Platform] Install a Prometheus connector for Presto, pointed at thanos-query - https://phabricator.wikimedia.org/T347430#10296086 (10Ottomata) @CDanis I believe people currently want this kind of work to be planned more strategically, and to... [14:14:37] 06Data-Engineering: Monthly pageview stats for Oct 2024 missing - https://phabricator.wikimedia.org/T379158 (10Radim.kubacki) 03NEW [14:27:41] (03CR) 10Ottomata: Extraction of RefineHelper (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [14:39:50] 06Data-Engineering, 10Event-Platform: Refine drops $schema field values - https://phabricator.wikimedia.org/T255818#10296374 (10Ottomata) [14:39:52] 10Data-Engineering (Q2 2024 October 1st - December 31th), 13Patch-For-Review: [Refine refactoring] Refine jobs should be scheduled by Airflow: implementation - https://phabricator.wikimedia.org/T356762#10296375 (10Ottomata) [14:41:33] 06Data-Engineering, 10Event-Platform: Refine drops $schema field values - https://phabricator.wikimedia.org/T255818#10296378 (10Ottomata) This should be fixed as part of the work done for {T356762}. We can resolve this after we are done {T369845} [15:13:40] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10296512 (10Marostegui) We should try to do db2155 and db2219 as soon as we can - we should minimize the chances of gett... [15:42:50] 06Data-Engineering, 10MediaWiki-extensions-WikimediaEvents, 10Observability-Metrics, 10Event-Platform, and 3 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10296627 (10lmata) >>! In T355837#10284492, @Ottomata wrote: > FWIW, with the event platform co... [15:46:08] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform, 06Movement-Insights: Modify the automated traffic detection to be applied at the project family level - https://phabricator.wikimedia.org/T377257#10296637 (10Ahoelzl) [15:51:23] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Traffic, 13Patch-For-Review: Rollout haproxykafka on all hosts - https://phabricator.wikimedia.org/T378578#10296649 (10Ahoelzl) [16:15:10] (03PS3) 10Gmodena: gobblin: add webrequest_frontend pull job. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1082432 (https://phabricator.wikimedia.org/T377931) [16:15:31] (03CR) 10Gmodena: gobblin: add webrequest_frontend pull job. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1082432 (https://phabricator.wikimedia.org/T377931) (owner: 10Gmodena) [16:35:26] (03CR) 10Aqu: [C:03+1] gobblin: add webrequest_frontend pull job. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1082432 (https://phabricator.wikimedia.org/T377931) (owner: 10Gmodena) [16:38:19] Starting build #4 for job wikimedia-event-utilities-maven-release [16:44:21] Project wikimedia-event-utilities-maven-release build #4: 09SUCCESS in 6 min 2 sec: https://integration.wikimedia.org/ci/job/wikimedia-event-utilities-maven-release/4/ [16:57:16] !log Manually rerun pageview monthly dumps to validate memory config change [16:57:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:03:14] (03CR) 10Ottomata: [C:03+1] gobblin: add webrequest_frontend pull job. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1082432 (https://phabricator.wikimedia.org/T377931) (owner: 10Gmodena) [17:24:40] 06Data-Engineering, 10ConfirmEdit (CAPTCHA extension), 10MediaWiki-extensions-EventLogging: Send captcha API response data to event logging - https://phabricator.wikimedia.org/T379179 (10Reedy) 03NEW [17:27:58] 06Data-Engineering, 10ConfirmEdit (CAPTCHA extension), 06Data Products, 10MediaWiki-extensions-EventLogging, 10Metrics Platform: Send captcha API response data to event logging - https://phabricator.wikimedia.org/T379179#10297016 (10Ottomata) [18:38:01] 10Data-Engineering (Q2 2024 October 1st - December 31th), 03Discovery-Search (Current work), 10Dumps 2.0 (Kanban Board): Add support for Spark producers in Event Platform - https://phabricator.wikimedia.org/T374341#10297336 (10Ottomata) [18:38:03] 10Data-Engineering (Q2 2024 October 1st - December 31th), 03Discovery-Search (Current work), 10Dumps 2.0 (Kanban Board): Add support for Spark producers in Event Platform - https://phabricator.wikimedia.org/T374341#10297337 (10Ottomata) > @gmodena @pfischer can you please add a definition of done / deliverab... [18:38:42] 10Data-Engineering (Q2 2024 October 1st - December 31th), 03Discovery-Search (Current work), 10Dumps 2.0 (Kanban Board): Add support for Spark producers in Event Platform - https://phabricator.wikimedia.org/T374341#10297339 (10Ottomata) [18:38:44] 14Analytics, 06Data-Engineering, 10Event-Platform, 05Goal: Event Platform: Stream Connectors - https://phabricator.wikimedia.org/T214430#10297340 (10Ottomata) [18:39:13] 14Analytics, 06Data Products, 07Epic: [Metrics Platform] Event Platform Client Libraries - https://phabricator.wikimedia.org/T228175#10297341 (10Ottomata) [18:40:03] 14Analytics, 06Data Products, 07Epic: [Metrics Platform] Event Platform Client Libraries - https://phabricator.wikimedia.org/T228175#10297344 (10Ottomata) Can we decline / resolve this task? [18:45:42] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Deploy the HDFS synchronizer (blunderbuss) service to the dse-k8s cluster - https://phabricator.wikimedia.org/T371994#10297384 (10bking) [19:24:19] 06Data-Engineering, 06Release-Engineering-Team, 10GitLab (Project Migration): Create new GitLab project: repos/wmf-packages - https://phabricator.wikimedia.org/T379187#10297494 (10Ottomata) [19:30:02] 06Data-Engineering, 06Release-Engineering-Team, 10GitLab (Project Migration): Create new GitLab project: repos/wmf-packages - https://phabricator.wikimedia.org/T379187#10297511 (10Ottomata) [19:32:11] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization, 07Epic: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#10297519 (10Ottomata) [19:32:32] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization, 07Epic: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#10297516 (10Ottomata) [19:44:52] (03PS1) 10Milimetric: Include user_is_temp column in user table sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1087964 [19:47:52] (03CR) 10Mforns: [C:03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1087964 (owner: 10Milimetric) [19:49:38] 06Data-Engineering: Find a way to make spark have an intermediate data materialization step before coalescing - https://phabricator.wikimedia.org/T379194 (10JAllemandou) 03NEW [21:02:27] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform-SRE: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 - https://phabricator.wikimedia.org/T338057#10297690 (10Ahoelzl) [21:05:41] 06Data-Engineering, 06Research: Rate Limited for data science project - https://phabricator.wikimedia.org/T378184#10297697 (10Ottomata) 05Open→03Resolved a:03Ottomata [21:11:57] 06Data-Engineering, 10Data-Engineering-Wikistats, 06Data Products, 10PageViewInfo, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#10297728 (10Ottomata) [21:20:42] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06MW-Interfaces-Team, 13Patch-For-Review: Explore mechanism for publishing domain events - https://phabricator.wikimedia.org/T378933#10297757 (10Ahoelzl) [21:23:30] 10Data-Engineering (Q2 2024 October 1st - December 31th): Find a way to make spark have an intermediate data materialization step before coalescing - https://phabricator.wikimedia.org/T379194#10297764 (10Ahoelzl) [22:13:37] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10CheckUser, 06Privacy Engineering: Add cu_log_event and cu_private_event CheckUser tables to data lake - https://phabricator.wikimedia.org/T376752#10297879 (10Tgr) Probably just decline? Does not make sense to keep the task open if the reason to not...