[03:12:25] 06Data-Engineering, 03Discovery-Search (Current work): Datahub - ingest Hive discovery database - https://phabricator.wikimedia.org/T374118#10190315 (10tchin) > We might need to simply ingest all the tables I can probably take a look at why the table match isn’t working, next thing we could try is providing a... [07:03:05] (03PS1) 10KCVelaga: Add MinT translation provider [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076902 (https://phabricator.wikimedia.org/T357250) [07:11:43] (03CR) 10Nik Gkountas: [C:03+2] Add MinT translation provider [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076902 (https://phabricator.wikimedia.org/T357250) (owner: 10KCVelaga) [07:12:08] (03Merged) 10jenkins-bot: Add MinT translation provider [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076902 (https://phabricator.wikimedia.org/T357250) (owner: 10KCVelaga) [07:26:51] !log Delete unused druid segment for the netflow datasource [07:26:54] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:08:10] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE: Update druid config to automatically drop unused segments - https://phabricator.wikimedia.org/T376118 (10JAllemandou) 03NEW [10:56:26] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Update druid config to automatically drop unused segments - https://phabricator.wikimedia.org/T376118#10191082 (10BTullis) [10:58:09] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Update druid config to automatically drop unused segments - https://phabricator.wikimedia.org/T376118#10191084 (10BTullis) Will the be only the `druid-analytics` cluster that is affected, or should we apply t... [11:05:44] (03PS1) 10KCVelaga: Update translation provider enum values [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076988 (https://phabricator.wikimedia.org/T357250) [11:06:07] (03CR) 10CI reject: [V:04-1] Update translation provider enum values [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076988 (https://phabricator.wikimedia.org/T357250) (owner: 10KCVelaga) [11:11:31] 06Data-Engineering, 06Data Products, 06DBA, 06Trust and Safety Product Team, and 2 others: Update gb_address index on the globalblocks table - https://phabricator.wikimedia.org/T376125 (10Dreamy_Jazz) 03NEW [11:11:51] 06Data-Engineering, 06Data Products, 06DBA, 06Trust and Safety Product Team, and 2 others: Update gb_address index on the globalblocks table - https://phabricator.wikimedia.org/T376125#10191151 (10Dreamy_Jazz) [11:13:49] (03PS2) 10KCVelaga: Update translation provider enum values [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076988 (https://phabricator.wikimedia.org/T357250) [11:14:12] (03CR) 10CI reject: [V:04-1] Update translation provider enum values [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076988 (https://phabricator.wikimedia.org/T357250) (owner: 10KCVelaga) [11:15:40] (03Abandoned) 10KCVelaga: Update translation provider enum values [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076988 (https://phabricator.wikimedia.org/T357250) (owner: 10KCVelaga) [11:23:38] (03PS1) 10KCVelaga: Add elia translation provider [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076996 (https://phabricator.wikimedia.org/T357250) [11:23:50] (03CR) 10CI reject: [V:04-1] Add elia translation provider [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076996 (https://phabricator.wikimedia.org/T357250) (owner: 10KCVelaga) [11:25:03] (03CR) 10KCVelaga: "recheck" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076996 (https://phabricator.wikimedia.org/T357250) (owner: 10KCVelaga) [11:44:52] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Update druid config to automatically drop unused segments - https://phabricator.wikimedia.org/T376118#10191290 (10JAllemandou) We very much can test the setting onto the test-cluster. I think there is small b... [11:50:30] !log roll restart hadoop analytics master to pick up new hosts T353788 [11:50:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:50:34] T353788: Add kafka-stretch100[1-2] to the hadoop cluster - https://phabricator.wikimedia.org/T353788 [12:20:53] !log Delete HDFS webrequest staging data used for the haproxy log migration [12:20:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:56:23] (03CR) 10Nik Gkountas: [C:03+2] Add elia translation provider [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076996 (https://phabricator.wikimedia.org/T357250) (owner: 10KCVelaga) [12:56:52] (03Merged) 10jenkins-bot: Add elia translation provider [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1076996 (https://phabricator.wikimedia.org/T357250) (owner: 10KCVelaga) [13:34:02] 06Data-Engineering, 06Data Products, 06Data-Platform, 06Movement-Insights, and 2 others: Temporary Accounts Initiative (IP Masking) - Add user_is_temp to data tables - https://phabricator.wikimedia.org/T356701#10191751 (10fkaelin) Thanks for the clarifications @Mayakp.wiki, though in my opinion we should s... [13:48:11] 10Data-Engineering (Q1 2024 July 1st - September 30th): Some Gobblin folders don't have `_IMPORTED` flags - https://phabricator.wikimedia.org/T376144 (10JAllemandou) 03NEW [13:52:34] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 10Event-Platform, 13Patch-For-Review: Migrate Event Platform Schema Respositories to Gitlab - https://phabricator.wikimedia.org/T366836#10191808 (10Snwachukwu) [14:17:05] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 10Event-Platform, 13Patch-For-Review: Migrate Event Platform Schema Respositories to Gitlab - https://phabricator.wikimedia.org/T366836#10191906 (10Snwachukwu) Plan for EventPlatform Schema Migration. # [X]**Schema repositori... [14:17:54] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 10Event-Platform, 13Patch-For-Review: Migrate Event Platform Schema Respositories to Gitlab - https://phabricator.wikimedia.org/T366836#10191912 (10Snwachukwu) We plan to do the switch in 1 week time i.e 8th October, 2024. #data-p... [14:22:43] (03PS1) 10Máté Szabó: ipinfo_interaction: Add event_ip_data_sources [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1077035 (https://phabricator.wikimedia.org/T356105) [14:23:59] 06Data-Engineering, 10Dumps 2.0, 10Event-Platform: [SPIKE] how can we support Spark producer/consumers in Event Platform - https://phabricator.wikimedia.org/T374341#10191938 (10pfischer) I started working on a kafka sink for T372912, that leverages `JsonSchemSparkConverter` from `eventutilities-spark`. It e... [15:05:37] (03PS1) 10Gmodena: changelog: update for v0.2.51 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1077045 [15:22:45] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 06Java-Scala-Standardization, 03Discovery-Search (Current work): Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all de... - https://phabricator.wikimedia.org/T367405#10192211 [15:27:25] Hi, I'm wondering how do you get rid of "Old event processed"? We are getting this in change-propagation but also saw "Pending commit is older than 1 minute". Is there anything to worry with this? [15:31:22] https://github.com/wikimedia/mediawiki-services-change-propagation/blob/c0d410a22e05c6efead37e8f275995607fbdda6f/lib/base_executor.js#L485 seems to suggest it's due to jobs being delayed, how exactly do you get it to run before then? [15:34:49] Do I have to set disable_delayed_execution for all jobs. Or how does delayed work and why do some jobs get >1h backlog and some jobs take a while to then execute. [15:49:34] (03CR) 10Gmodena: [C:03+2] changelog: update for v0.2.51 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1077045 (owner: 10Gmodena) [15:50:02] Starting build #18 for job analytics-refinery-maven-release [15:50:55] !log releasing refinery source v0.2.51 [15:50:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:52:11] Project analytics-refinery-maven-release build #18: 15ABORTED in 2 min 9 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/18/ [16:03:49] (03Merged) 10jenkins-bot: changelog: update for v0.2.51 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1077045 (owner: 10Gmodena) [16:07:18] Starting build #19 for job analytics-refinery-maven-release [16:30:50] Project analytics-refinery-maven-release build #19: 09SUCCESS in 23 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/19/ [16:34:45] Starting build #16 for job analytics-refinery-update-jars [16:36:36] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.51 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1077070 [16:36:36] Project analytics-refinery-update-jars build #16: 09SUCCESS in 1 min 51 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/16/ [16:38:01] !log deployed refinery source v0.2.51 [16:38:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:38:57] (03CR) 10Gmodena: [C:03+2] Add refinery-source jars for v0.2.51 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1077070 (owner: 10Maven-release-user) [17:00:41] 14Analytics-Radar, 06Data-Engineering-Icebox, 06SRE, 06Traffic, and 3 others: Requests for /static get an invalid WMF-Last-Access cookie for wikipedia.org on non-Wikipedia requests - https://phabricator.wikimedia.org/T261803#10192971 (10matmarex) This is still a problem today, and it makes for a distractio... [17:59:00] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Deploy the HDFS synchronizer service to the dse-k8s cluster - https://phabricator.wikimedia.org/T371994#10193264 (10bking) [18:35:11] 14Analytics-Radar, 06Data-Engineering-Icebox, 06SRE, 06Traffic, and 3 others: Requests for /static get an invalid WMF-Last-Access cookie for wikipedia.org on non-Wikipedia requests - https://phabricator.wikimedia.org/T261803#10193353 (10Tgr) Yeah, the wider issue here is that setting the cookie on cross-si... [20:29:05] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Update druid config to automatically drop unused segments - https://phabricator.wikimedia.org/T376118#10193763 (10BTullis) Is there some overlap between this ticket and {T296207}? [21:04:20] (03PS1) 10Mforns: Coalesce anomaly detection queries to output just 1 file [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1077098 (https://phabricator.wikimedia.org/T376212) [23:14:09] 06Data-Engineering, 06Data Products, 06DBA, 10wikitech.wikimedia.org, 07Schema-change: Please drop globalblocks table from labswiki - https://phabricator.wikimedia.org/T375783#10194248 (10bd808) 05Stalled→03Open