[09:22:52] (03PS1) 10Joal: Update canary event job expcetion [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1028770 [09:23:40] (03PS2) 10Joal: Update canary event job exception [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1028770 [09:25:01] (03CR) 10Joal: [C:03+2] "Self merging before today's deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1028770 (owner: 10Joal) [09:41:28] (03Merged) 10jenkins-bot: Update canary event job exception [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1028770 (owner: 10Joal) [10:46:01] 06Data-Engineering, 10Observability-Logging, 06Traffic: Benthos loses messages when under high load - https://phabricator.wikimedia.org/T364379 (10Fabfur) 03NEW [12:03:36] 06Data-Engineering, 10Cassandra, 06Data-Persistence, 10Data-Platform-SRE (2024.05.06 - 2024.05.26), 13Patch-For-Review: Encrypt Airflow connections to AQS Cassandra - https://phabricator.wikimedia.org/T362181#9777193 (10CodeReviewBot) btullis merged https://gitlab.wikimedia.org/repos/data-engineering/air... [12:04:50] !log deploying airflow analytics instance for https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/678 and T362181 [12:04:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:04:53] T362181: Encrypt Airflow connections to AQS Cassandra - https://phabricator.wikimedia.org/T362181 [12:19:15] 06Data-Engineering, 10Cassandra, 06Data-Persistence, 10Data-Platform-SRE (2024.05.06 - 2024.05.26), 13Patch-For-Review: Encrypt Airflow connections to AQS Cassandra - https://phabricator.wikimedia.org/T362181#9777202 (10BTullis) The first two DAG runs failed with an error like this: ` Caused by: com.data... [13:09:53] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9777316 (10lbowmaker) [13:11:43] 06Data-Engineering, 06Data-Platform, 06Movement-Insights: Unique devices tables have missing or incorrect data for January and February 2024 - https://phabricator.wikimedia.org/T361242#9777327 (10lbowmaker) 05Open→03Resolved [13:20:14] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9777425 (10lbowmaker) @diego - when we make the change it should be as timely as it was before we started skipping it. I moved this as next up in our c... [13:36:05] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9777447 (10lbowmaker) @diego @nshahquinn-wmf - just chatted with @JAllemandou about this. The job to load this data was originally scheduled for the 19... [13:48:11] 06Data-Engineering, 10Cassandra, 06Data-Persistence, 10Data-Platform-SRE (2024.05.06 - 2024.05.26), 13Patch-For-Review: Encrypt Airflow connections to AQS Cassandra - https://phabricator.wikimedia.org/T362181#9777466 (10CodeReviewBot) btullis opened https://gitlab.wikimedia.org/repos/data-engineering/air... [14:03:05] 06Data-Engineering, 10Cassandra, 06Data-Persistence, 10Data-Platform-SRE (2024.05.06 - 2024.05.26), 13Patch-For-Review: Encrypt Airflow connections to AQS Cassandra - https://phabricator.wikimedia.org/T362181#9777529 (10CodeReviewBot) btullis merged https://gitlab.wikimedia.org/repos/data-engineering/air... [14:03:40] !log deploying airflow analytics instance for T362181 to fix cassandra cipher list [14:03:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:03:43] T362181: Encrypt Airflow connections to AQS Cassandra - https://phabricator.wikimedia.org/T362181 [14:05:42] !log unpaused cassandra_load_pageview_per_project_hourly for T362181 [14:05:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:17:20] 06Data-Engineering, 10Cassandra, 06Data-Persistence, 10Data-Platform-SRE (2024.05.06 - 2024.05.26), 13Patch-For-Review: Encrypt Airflow connections to AQS Cassandra - https://phabricator.wikimedia.org/T362181#9777561 (10BTullis) 05Open→03Resolved This is now working as expected. {F50705664,width... [14:19:54] 06Data-Engineering, 10Cassandra, 06Data-Persistence, 10Data-Platform-SRE (2024.05.06 - 2024.05.26), 13Patch-For-Review: Encrypt Airflow connections to AQS Cassandra - https://phabricator.wikimedia.org/T362181#9777568 (10BTullis) Thanks again @elukey for all your help to get this working. [14:36:13] 06Data-Engineering, 10Cassandra, 06Data-Persistence, 10Data-Platform-SRE (2024.05.06 - 2024.05.26): Encrypt Airflow connections to AQS Cassandra - https://phabricator.wikimedia.org/T362181#9777644 (10elukey) Niceee thanks a lot for all the work! [14:40:03] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data-Platform-SRE (2024.05.06 - 2024.05.26): Migrate Dumps Snapshot hosts from Buster to Bullseye - https://phabricator.wikimedia.org/T325228#9777657 (10xcollazo) Hello there. Due to T364250, the host `snapshot1011` will not be running the typical `wikidata... [14:52:06] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9777704 (10JAllemandou) I'm sorry, what Luke expressed before can't be done, I messed up when looking at wikidata dumps generation: I looked at a subset... [15:20:05] (03PS3) 10Snwachukwu: Upgrade MediawikiHistory Checker to use AWS Deequ. 1. Update User history checker 2. Update Page history checker 3. Update Denormalized history checker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1024423 (https://phabricator.wikimedia.org/T361016) [15:23:53] (03CR) 10Snwachukwu: Upgrade MediawikiHistory Checker to use AWS Deequ. 1. Update User history checker 2. Update Page history checker 3. Update Denormalized hist (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1024423 (https://phabricator.wikimedia.org/T361016) (owner: 10Snwachukwu) [17:01:27] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9778295 (10XiaoXiao-WMF) @JAllemandou what's the impact of this as we can see right now? How long do we expect to have this built? [17:10:37] (03PS1) 10Santiago Faci: Updating changelog to prepare next deployment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1028871 [17:24:39] (03CR) 10Joal: Updating changelog to prepare next deployment (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1028871 (owner: 10Santiago Faci) [17:29:41] (03PS2) 10Santiago Faci: Updating changelog to prepare next deployment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1028871 [17:30:06] (03CR) 10Santiago Faci: "Fixed!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1028871 (owner: 10Santiago Faci) [20:14:34] 06Data-Engineering, 06DC-Ops, 10ops-eqiad: Q4:rack/setup/install an-conf100[4-6] - https://phabricator.wikimedia.org/T364429 (10RobH) 03NEW [20:15:30] 06Data-Engineering, 06DC-Ops, 10ops-eqiad: Q4:rack/setup/install an-conf100[4-6] - https://phabricator.wikimedia.org/T364429#9779047 (10RobH) [22:27:24] (03CR) 10Joal: [C:03+2] Updating changelog to prepare next deployment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1028871 (owner: 10Santiago Faci) [22:45:50] (03Merged) 10jenkins-bot: Updating changelog to prepare next deployment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1028871 (owner: 10Santiago Faci)