[00:25:33] RECOVERY - Check unit status of monitor_refine_eventlogging_analytics on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_eventlogging_analytics https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [02:52:42] 10Analytics, 10Product-Analytics (Kanban), 10Readers-Web-Backlog (Tracking): Add UniversalLanguageSelector to the allowlist - https://phabricator.wikimedia.org/T287256 (10jwang) [03:42:13] (03PS1) 10Jenniferwang: T287256: add UniversalLanguageSelector to sanitized event database Change-Id: I0eee231f95cceb9949a4a2b77f5774760f045b58 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/713570 [06:56:45] \o/ back I am! [08:10:33] Welcome back joal. [08:11:44] 10Analytics-Clusters, 10Analytics-Kanban, 10observability, 10Patch-For-Review: Setup Analytics team in VO/splunk oncall - https://phabricator.wikimedia.org/T273064 (10fgiunchedi) [08:30:25] (03PS6) 10Joal: Load cassandra3 from spark [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/686629 (https://phabricator.wikimedia.org/T280649) (owner: 10Milimetric) [08:38:44] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deployy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/712209 (https://phabricator.wikimedia.org/T255148) (owner: 10Btullis) [08:55:58] !log Deploying refinery with scap [08:56:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:59:03] 10Analytics, 10Analytics-Kanban: Delete HDFS raw *_camus directories 60 days after July 12 (after 2021-09-10) - https://phabricator.wikimedia.org/T287685 (10JAllemandou) a:03JAllemandou [09:00:26] 10Analytics, 10Analytics-Kanban: Delete HDFS raw *_camus directories 60 days after July 12 (after 2021-09-10) - https://phabricator.wikimedia.org/T287685 (10JAllemandou) I deleted the old files with Dan today, assuming that 30 days of waiting was enough (this is the time we keep raw webrequest data). Moving th... [11:11:45] (03CR) 10David Caro: add stop query function (031 comment) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710067 (https://phabricator.wikimedia.org/T71037) (owner: 10Michael DiPietro) [12:13:26] (03CR) 10Michael Große: [C: 03+1] "Looks good, but should this change target the branch master or production? I'm a bit confused as to which of those is actually in use and " [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/713464 (https://phabricator.wikimedia.org/T286903) (owner: 10Lucas Werkmeister (WMDE)) [12:33:56] (03CR) 10Lucas Werkmeister (WMDE): "It should be merged on master, then someone with the right privileges (probably Amir) can cherry-pick it to the production branch and depl" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/713464 (https://phabricator.wikimedia.org/T286903) (owner: 10Lucas Werkmeister (WMDE)) [12:45:20] (03CR) 10Michael Große: [C: 03+2] Track number of active items (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/713464 (https://phabricator.wikimedia.org/T286903) (owner: 10Lucas Werkmeister (WMDE)) [12:46:25] (03Merged) 10jenkins-bot: Track number of active items [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/713464 (https://phabricator.wikimedia.org/T286903) (owner: 10Lucas Werkmeister (WMDE)) [13:01:48] joal: morning, I'm around if you wanna talk [13:16:38] 10Analytics, 10Analytics-Kanban, 10 Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Test Alluxio as cache layer for Presto - https://phabricator.wikimedia.org/T266641 (10BTullis) I have now got to the point where I believe the puppet patch that I've been working on needs to be merged in... [13:37:18] 10Analytics, 10Analytics-Kanban, 10 Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Test Alluxio as cache layer for Presto - https://phabricator.wikimedia.org/T266641 (10BTullis) Looking more closely, I see that the sysvinit files that are provided with alluxio don't actually run the dae... [14:11:13] Good morning milimetric :) [14:11:21] milimetric: let me know when is a good time for you [14:11:43] morning, um, is 10 minutes ok? [14:11:53] for sure, when you wish :) [14:41:18] 10Analytics, 10 Data-Engineering, 10Cassandra, 10Data-Engineering-Kanban, 10Epic: Cassandra3 migration for Analytics AQS - https://phabricator.wikimedia.org/T249755 (10JAllemandou) [14:41:24] 10Analytics-Kanban, 10Patch-For-Review: Add a spark job loading Cassandra 3 - https://phabricator.wikimedia.org/T280649 (10JAllemandou) [14:46:09] 10Analytics, 10 Data-Engineering, 10Cassandra, 10Data-Engineering-Kanban, 10Epic: Update cassandra oozie jobs to laod cassandra3 using Spark job - https://phabricator.wikimedia.org/T289161 (10JAllemandou) [14:46:47] 10Analytics, 10Analytics-Kanban, 10 Data-Engineering, 10Cassandra, and 2 others: Update cassandra oozie jobs to laod cassandra3 using Spark job - https://phabricator.wikimedia.org/T289161 (10JAllemandou) a:03JAllemandou [14:47:03] 10Analytics, 10Analytics-Kanban, 10 Data-Engineering, 10Cassandra, and 2 others: Update cassandra oozie jobs to load cassandra3 using Spark job - https://phabricator.wikimedia.org/T289161 (10JAllemandou) [15:08:18] !log Restart oozie jobs loading druid to use new druid-host [15:08:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:23:47] (03PS1) 10MNeisler: Add the mediawiki_pref_diff event platform stream to the allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/713644 (https://phabricator.wikimedia.org/T287255) [15:28:36] (03PS1) 10Andrew-WMDE: Add the total number of times a template had template data in TemplateWizard [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/713647 (https://phabricator.wikimedia.org/T272589) [15:29:33] 10Analytics, 10Patch-For-Review, 10Product-Analytics (Kanban): Add mediawiki_pref_diff to the allowlist - https://phabricator.wikimedia.org/T287255 (10MNeisler) [15:40:28] ok I confirm oozie with the new druid overlord works :) Will restart jobs one after the other gently [16:05:40] (03PS1) 10Andrew-WMDE: Add the total number of times a template had template data in VE's template dialog [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/713650 (https://phabricator.wikimedia.org/T272589) [16:25:39] (03PS2) 10Andrew-WMDE: Add the total number of times a template had template data in TemplateWizard [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/713647 (https://phabricator.wikimedia.org/T272589) [16:55:20] 10Analytics, 10 Data-Engineering, 10Growth-Team, 10Metrics-Platform, and 3 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10Mholloway) >>! In T288853#7289672, @Tgr wrote: > IMO there is value in client-side and server-side... [17:12:25] \o/ all druid oozie jobs restarted :) [17:13:12] btullis: ready on oozie side to decom druid1001 :) [17:15:40] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10wmfdata-python: wmfdata-python's Hive query output includes logspam - https://phabricator.wikimedia.org/T275233 (10Mayakp.wiki) @BTullis @Milimetric I re-ran the query that was giving me log messages in the output and it seems like the issue still per... [17:20:14] (03PS1) 10Joal: Fix pageview-complete monthly dump format [analytics/refinery] - 10https://gerrit.wikimedia.org/r/713662 [17:23:20] Thanks joal. I'm now certain that turnilo is already using an-druid1001.eqiad.wmnet as its host. I'll be checking on superset later this evening and I'll let product-analytics know either way. [17:23:51] awesome - thanks btullis :) I you feel like it, please send a picture of the beetle tomorrow :) [17:24:16] Will do. :-) [17:30:22] Yeah, superset will need changing. [17:30:26] https://www.irccloud.com/pastebin/WMpZRSTq/ [17:31:53] Hang on. How is superset even working then, that host is off already. Maybe I'm looking in the wrong place. [17:32:06] I was asking myself the same question btullis :) [17:32:11] triple checking now on superset [17:33:09] btullis: ok druid fails for superset :) [17:33:27] or at least, druid with old way of quering [17:34:03] Oh, right. Maybe I should just try to unbreak now then. [17:35:56] interesting btullis - the new superset interface doesn't let me see the old druid datasource [17:36:22] Are you OK for me to do this?`MariaDB [superset_production]> update clusters set broker_host='an-druid1001.eqiad.wmnet' where cluster_name='analytics-eqiad';` [17:36:43] btullis: I don't have a better option :) [17:37:14] !log on an-coord1001: MariaDB [superset_production]> update clusters set broker_host='an-druid1001.eqiad.wmnet' where cluster_name='analytics-eqiad'; [17:37:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:37:22] Rows matched: 1 Changed: 1 Warnings: 0 [17:37:32] trying my failing dashboard btullis [17:37:41] https://www.irccloud.com/pastebin/b0UBE7Oa/ [17:38:14] Fixed for me btullis - awesome work :) [17:38:31] Phew! Thanks. [17:39:28] btullis: superset was mostly not broken because the problem occured for deprecated datasources only [17:39:42] the new druid-sql datasource was already using the new server [17:40:03] But I have old-fashion stuff using the old datasources :) [17:40:52] Ah, thanks. Will check out that druid-sql datasource too. Thanks. [17:41:13] btullis: I already did, it's ok (changeable on the UI) [17:43:03] Can you show me where please? I can't see any details of how to see it in the UI. [17:44:29] sure btullis - batcave? [17:44:40] 👍 [18:04:19] ok folks - Gone for tonight [19:27:29] (03PS1) 10GoranSMilovanovic: T283571 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/713688 [19:27:42] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] T283571 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/713688 (owner: 10GoranSMilovanovic) [19:54:06] (03PS1) 10GoranSMilovanovic: T283570 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/713691 [19:54:18] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] T283570 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/713691 (owner: 10GoranSMilovanovic) [20:06:24] (03PS1) 10GoranSMilovanovic: T283570 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/713694 [20:06:53] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] T283570 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/713694 (owner: 10GoranSMilovanovic) [20:28:55] 10Analytics, 10 Data-Engineering, 10Growth-Team, 10Metrics-Platform, and 4 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10Milimetric) I'm just making sure I didn't miss anything: it looks to me like the instrumentation's... [21:47:46] 10Analytics, 10 Data-Engineering, 10Growth-Team, 10Metrics-Platform, and 4 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10Tgr) >>! In T288853#7292309, @Mholloway wrote: > Just to be clear, you're advocating here for gett... [23:22:18] (03PS1) 10GoranSMilovanovic: T283570 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/713715 [23:22:29] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] T283570 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/713715 (owner: 10GoranSMilovanovic)