[02:30:16] 10Analytics-Canonical-Data, 06Movement-Insights: Update the canonical wiki dataset - https://phabricator.wikimedia.org/T363287#9779482 (10nshahquinn-wmf) I've [added the wikis](https://github.com/wikimedia-research/canonical-data/commit/f2492fa17ef71abe09994e3df4753503390f243e) to the TSV file, but at the mome... [08:46:19] Starting build #5 for job analytics-refinery-maven-release [09:15:10] Project analytics-refinery-maven-release build #5: 09SUCCESS in 28 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/5/ [09:22:26] Starting build #5 for job analytics-refinery-update-jars [09:24:22] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.39 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1028930 [09:24:24] Project analytics-refinery-update-jars build #5: 09SUCCESS in 1 min 57 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/5/ [09:41:17] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data-Platform-SRE (2024.05.06 - 2024.05.26): Migrate Dumps Snapshot hosts from Buster to Bullseye - https://phabricator.wikimedia.org/T325228#9780020 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host snapsh... [09:50:15] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data-Platform-SRE (2024.05.06 - 2024.05.26): Migrate Dumps Snapshot hosts from Buster to Bullseye - https://phabricator.wikimedia.org/T325228#9780073 (10BTullis) [10:19:09] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data-Platform-SRE (2024.05.06 - 2024.05.26): Migrate Dumps Snapshot hosts from Buster to Bullseye - https://phabricator.wikimedia.org/T325228#9780147 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host snapshot10... [10:19:53] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data-Platform-SRE (2024.05.06 - 2024.05.26): Migrate Dumps Snapshot hosts from Buster to Bullseye - https://phabricator.wikimedia.org/T325228#9780151 (10BTullis) [12:35:56] so, i have my superset dashboard, which currently queries tables that have individual events in them. Queries such as https://phabricator.wikimedia.org/P62077 [12:36:17] I want to be able to run this over much longer periods of time, such as 2 years, but thats too many rows for this query in super set [12:36:38] 1) can I optimize this any more just in this query, or 2) should I be summarizing these events into some sort of daily table? and if so how? [15:18:53] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117#9781136 (10Ottomata) > adopt topic names that follow EP conventions: . I'm sorry for not thinking about thi... [15:22:31] 06Data-Engineering-Icebox, 06Data-Platform-SRE: Upgrade to Kafka MirrorMaker 2 and revisit Kafka topic prefix convention - https://phabricator.wikimedia.org/T277467#9781147 (10Ottomata) [15:23:07] 06Data-Engineering-Icebox, 06Data-Platform-SRE, 10Event-Platform: Upgrade to Kafka MirrorMaker 2 and revisit Kafka topic prefix convention - https://phabricator.wikimedia.org/T277467#9781161 (10Ottomata) [15:52:51] (03CR) 10Mforns: [C:03+2] Add refinery-source jars for v0.2.39 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1028930 (owner: 10Maven-release-user) [15:52:55] (03CR) 10Mforns: [V:03+2 C:03+2] Add refinery-source jars for v0.2.39 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1028930 (owner: 10Maven-release-user) [16:21:01] !log Deploying refinery [16:21:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:22:24] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data-Platform-SRE (2024.05.06 - 2024.05.26), 13Patch-For-Review: Migrate Dumps Snapshot hosts from Buster to Bullseye - https://phabricator.wikimedia.org/T325228#9781322 (10BTullis) I have created https://gerrit.wikimedia.org/r/c/operations/puppet/+/102922... [16:28:47] 06Data-Engineering: Airflow DAG (hdfs_usage_weekly) failed with no details in the application log - https://phabricator.wikimedia.org/T364487 (10Sfaci) 03NEW [16:57:33] !log Deployed refinery using scap, then deployed onto hdfs [16:57:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:04:34] !log Deployed refinery-source using jenkins [17:04:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:40:27] (03CR) 10Xcollazo: [C:03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1027525 (https://phabricator.wikimedia.org/T362892) (owner: 10Mforns) [21:29:10] (03CR) 10Snwachukwu: [C:03+2] Upgrade MediawikiHistory Checker to use AWS Deequ. 1. Update User history checker 2. Update Page history checker 3. Update Denormalized hist [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1024423 (https://phabricator.wikimedia.org/T361016) (owner: 10Snwachukwu) [21:33:02] 06Data-Engineering: Airflow DAG (hdfs_usage_weekly) failed with no details in the application log - https://phabricator.wikimedia.org/T364487#9782161 (10amastilovic) a:03amastilovic [21:37:19] 06Data-Engineering: Airflow DAG (hdfs_usage_weekly) failed with no details in the application log - https://phabricator.wikimedia.org/T364487#9782167 (10amastilovic) The issue was in the path to the configured log4j.properties file in Airflow UI, `hdfs:///user/aqu/aqu-log4j.properties` was not accessible by the... [21:42:04] 06Data-Engineering, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Benthos loses messages when under high load - https://phabricator.wikimedia.org/T364379#9782170 (10Fabfur) @CDanis helped me a lot in this direction and he found a workaround|solution for this specific issue, optimizing Benthos con... [21:42:06] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Upgrade MediawikiHistory Checker to use AWS Deequ. 1. Update User history checker 2. Update Page history checker 3. Update Denormalized hist (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1024423 (https://phabricator.wikimedia.org/T361016) (owner: 10Snwachukwu) [21:42:09] (03CR) 10Ahoelzl: [C:03+1] Upgrade MediawikiHistory Checker to use AWS Deequ. 1. Update User history checker 2. Update Page history checker 3. Update Denormalized hist [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1024423 (https://phabricator.wikimedia.org/T361016) (owner: 10Snwachukwu) [23:36:49] 10Analytics-Canonical-Data, 06Movement-Insights: Update the canonical wiki dataset - https://phabricator.wikimedia.org/T363287#9782297 (10nshahquinn-wmf) 05Open→03Resolved Data Lake table is now updated.