[00:20:37] PROBLEM - Check unit status of monitor_refine_eventlogging_analytics on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_eventlogging_analytics https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [09:17:10] (03CR) 10David Caro: [C: 04-1] "Tested it in a couple ways:" [analytics/quarry/web] (buster) - 10https://gerrit.wikimedia.org/r/711208 (https://phabricator.wikimedia.org/T288528) (owner: 10Michael DiPietro) [09:46:34] (03CR) 10David Caro: [C: 03+2] db: Added a script to generate a DB schema from the models [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/711133 (https://phabricator.wikimedia.org/T288523) (owner: 10David Caro) [10:00:49] (03CR) 10David Caro: [C: 03+1] upgrade quarry to python 3.7 (031 comment) [analytics/quarry/web] (buster) - 10https://gerrit.wikimedia.org/r/711208 (https://phabricator.wikimedia.org/T288528) (owner: 10Michael DiPietro) [10:03:06] (03CR) 10David Caro: [C: 03+1] upgrade quarry to python 3.7 (031 comment) [analytics/quarry/web] (buster) - 10https://gerrit.wikimedia.org/r/711208 (https://phabricator.wikimedia.org/T288528) (owner: 10Michael DiPietro) [10:18:23] (03CR) 10David Caro: [C: 03+2] Add database autocompletion (031 comment) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/711456 (https://phabricator.wikimedia.org/T287471) (owner: 10David Caro) [10:31:01] (03PS1) 10David Caro: Revert "Add database autocompletion" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/712961 [10:31:09] (03CR) 10David Caro: [C: 03+2] Revert "Add database autocompletion" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/712961 (owner: 10David Caro) [10:32:53] (03Merged) 10jenkins-bot: Revert "Add database autocompletion" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/712961 (owner: 10David Caro) [10:50:48] (03PS1) 10David Caro: Add database autocompletion [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/713237 (https://phabricator.wikimedia.org/T287471) [11:15:41] 10Analytics, 10Analytics-Kanban, 10 Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Test Alluxio as cache layer for Presto - https://phabricator.wikimedia.org/T266641 (10BTullis) [12:40:04] (03CR) 10Mholloway: "Yes, please bump the minor version." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/710348 (https://phabricator.wikimedia.org/T270636) (owner: 10DLynch) [12:54:48] I got disconnected over the weekend apparently. Taking a look at failed job and rerunning now [13:01:58] so apparently this failed with some generic HiveSQLException: https://hue.wikimedia.org/hue/jobbrowser#!id=job_1626805905645_137002 [13:02:21] how could that be if subsequent jobs ran fine... [13:02:37] Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask [13:02:54] That sounds like the translation from SQL to MR failed... but that's not data-dependent... [13:06:42] Beats me. [13:12:57] hm... not a lot of leads on the internets... this is a stretch but the best I can come up with: [13:13:30] when trying to load the allow list (https://github.com/wikimedia/analytics-refinery/blob/master/oozie/virtualpageview/hourly/virtualpageview_hourly.hql#L59) into memory for in-memory join with a small table, it didn't have enough memory at that specific time (temporarily too many jobs or something) [13:13:41] rerunning [15:20:17] (03PS2) 10DLynch: EditAttemptStep: Add a new integration [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/710348 (https://phabricator.wikimedia.org/T270636) [15:21:30] (03CR) 10DLynch: EditAttemptStep: Add a new integration (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/710348 (https://phabricator.wikimedia.org/T270636) (owner: 10DLynch) [15:31:22] (03CR) 10Mholloway: [C: 03+1] EditAttemptStep: Add a new integration [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/710348 (https://phabricator.wikimedia.org/T270636) (owner: 10DLynch) [16:33:55] (03CR) 10Bartosz Dziewoński: [C: 03+2] EditAttemptStep: Add a new integration [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/710348 (https://phabricator.wikimedia.org/T270636) (owner: 10DLynch) [16:34:42] (03Merged) 10jenkins-bot: EditAttemptStep: Add a new integration [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/710348 (https://phabricator.wikimedia.org/T270636) (owner: 10DLynch) [16:35:10] (03PS11) 10Michael DiPietro: add stop query function [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710067 (https://phabricator.wikimedia.org/T71037) [16:44:05] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Refresh Druid nodes (druid100[1-3]) - https://phabricator.wikimedia.org/T255148 (10Cmjohnson) [16:49:58] 10Analytics: How often to people try to edit on mobile devices, using the desktop site, at the English Wikipedia? - https://phabricator.wikimedia.org/T288972 (10Whatamidoing-WMF) [16:50:30] 10Analytics: How often do people try to edit on mobile devices, using the desktop site, at the English Wikipedia? - https://phabricator.wikimedia.org/T288972 (10Whatamidoing-WMF) [17:06:29] 10Analytics-Clusters, 10Analytics-Kanban, 10observability, 10Patch-For-Review, and 2 others: Setup Analytics team in VO/splunk oncall - https://phabricator.wikimedia.org/T273064 (10razzi) a:05razzi→03None There's more work to be done here, but it's been sitting in my personal backlog for a while so I'm... [17:07:15] 10Quarry: Add black formatting to quarry linter - https://phabricator.wikimedia.org/T288976 (10mdipietro) [17:13:42] (03PS12) 10Michael DiPietro: add stop query function [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710067 (https://phabricator.wikimedia.org/T71037) [17:14:03] 10Quarry: Add black formatting to quarry linter - https://phabricator.wikimedia.org/T288976 (10Bstorm) We can add a check for formatting with something similar to https://gerrit.wikimedia.org/r/plugins/gitiles/labs/tools/maintain-kubeusers/+/refs/heads/master/tox.ini#30 Since the thing will fail if we add it no... [17:16:29] 10Quarry, 10Patch-For-Review: update python for quarry - https://phabricator.wikimedia.org/T288249 (10Bstorm) [17:16:32] 10Quarry, 10Patch-For-Review: quarry to python 3.7 - https://phabricator.wikimedia.org/T288528 (10Bstorm) [17:48:04] 10Analytics, 10Product-Analytics: Change /user/hive/warehouse/neilpquinn.db/editor_month ownership to iflorez - https://phabricator.wikimedia.org/T288864 (10Iflorez) 05Open→03Resolved [18:20:01] 10Quarry, 10Epic, 10cloud-services-team (Kanban): Productionize quarry a bit - https://phabricator.wikimedia.org/T288982 (10Bstorm) [18:20:04] 10Analytics, 10Product-Analytics: Reconstruct Hive & Hadoop permissions for shared database - https://phabricator.wikimedia.org/T288983 (10Mayakp.wiki) [18:20:25] 10Quarry, 10Epic, 10cloud-services-team (Kanban): Productionize quarry a bit - https://phabricator.wikimedia.org/T288982 (10Bstorm) [18:20:28] 10Quarry: Add black formatting to quarry linter - https://phabricator.wikimedia.org/T288976 (10Bstorm) [18:30:59] 10Quarry, 10Epic, 10cloud-services-team (Kanban): Productionize quarry a bit - https://phabricator.wikimedia.org/T288982 (10Bstorm) [18:31:02] 10Quarry, 10Patch-For-Review: quarry to python 3.7 - https://phabricator.wikimedia.org/T288528 (10Bstorm) [18:32:27] 10Quarry, 10Epic, 10cloud-services-team (Kanban): Productionize quarry a bit - https://phabricator.wikimedia.org/T288982 (10Bstorm) [20:10:04] (03CR) 10Bstorm: [C: 03+2] upgrade quarry to python 3.7 [analytics/quarry/web] (buster) - 10https://gerrit.wikimedia.org/r/711208 (https://phabricator.wikimedia.org/T288528) (owner: 10Michael DiPietro) [20:10:48] (03Merged) 10jenkins-bot: upgrade quarry to python 3.7 [analytics/quarry/web] (buster) - 10https://gerrit.wikimedia.org/r/711208 (https://phabricator.wikimedia.org/T288528) (owner: 10Michael DiPietro) [20:13:12] (03CR) 10Bstorm: "This will make dev easier but merging the final product harder. I guess we can just revert when we are ready to final merge?" [analytics/quarry/web] (buster) - 10https://gerrit.wikimedia.org/r/711198 (owner: 10Andrew Bogott) [20:14:26] (03CR) 10Andrew Bogott: .gitreview: associate local 'buster' branch with gerrit 'buster' branch (031 comment) [analytics/quarry/web] (buster) - 10https://gerrit.wikimedia.org/r/711198 (owner: 10Andrew Bogott) [20:35:47] 10Analytics: Improve Refine bad data handling - https://phabricator.wikimedia.org/T289003 (10Milimetric) [20:35:58] 10Analytics: Improve Refine bad data handling - https://phabricator.wikimedia.org/T289003 (10Milimetric) p:05Triage→03High a:03Milimetric [21:43:01] (03CR) 10Bstorm: [C: 04-1] "The other issue (which could be addressed elsewhere) is that while this works very well to stop the query, you get "failed" as the status." [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710067 (https://phabricator.wikimedia.org/T71037) (owner: 10Michael DiPietro) [21:43:38] (03CR) 10Bstorm: [C: 04-1] add stop query function (031 comment) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710067 (https://phabricator.wikimedia.org/T71037) (owner: 10Michael DiPietro) [22:20:37] 10Analytics, 10Growth-Team, 10Product-Analytics: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10mewoph) @Tgr @Ottomata @kostajh — is there a token that can be used to join client_ip from a client-side event or does the IP need to... [23:00:23] (03PS2) 10Fdans: Refactor graphmodel to allow more than one project [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/712547 (https://phabricator.wikimedia.org/T283254) [23:01:48] (03CR) 10jerkins-bot: [V: 04-1] Refactor graphmodel to allow more than one project [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/712547 (https://phabricator.wikimedia.org/T283254) (owner: 10Fdans) [23:14:21] 10Analytics, 10Growth-Team, 10Product-Analytics: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10Tgr) MediaWiki should have no problem adding the IP to the request. (Due to proxy mechanics, this might differ from the client-side IP...