[13:00:50] <ottomata>	 mornin joal !
[13:15:32] <wikibugs>	 10Analytics: Refactor webrequest_source partitions and oozie jobs - https://phabricator.wikimedia.org/T116387 (10Ottomata) 05Open→03Declined Never worked on, and we are doing a small change to how webrequest ingestion works as part of T271232, althought it probably won't do what this ticket desires.  Moving...
[13:19:27] <joal>	 Hi ottomata :)
[13:20:21] <ottomata>	 hello!
[13:25:46] <ottomata>	 ok so!
[13:25:54] <ottomata>	 yes joal empty run is good!
[13:26:04] <ottomata>	 i can merge the systemd timer for webrequest_test anytime
[13:26:09] <joal>	 ack
[13:26:59] <joal>	 so, empty run now, more-or-less regular manual runs while I have the kids, then timer?
[13:27:05] <joal>	 or timer right now?
[13:27:11] <joal>	 Oh an
[13:27:20] <joal>	 There is another thing we need to do ottomata 
[13:27:27] <ottomata>	 joal: i guess we can just do 1 manual run
[13:27:29] <ottomata>	 and then do the timer
[13:27:32] <ottomata>	 no difference really, right?
[13:27:34] <joal>	 works for me ottomata 
[13:27:45] <joal>	 I should have the time to actually do that before the kids :)
[13:28:31] <ottomata>	 ok great!
[13:29:37] <joal>	 !log Clean gobblin state_store and data before starting webrequest_test on analytics-test-hadoop
[13:29:40] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:29:57] <joal>	 !log Run first manual empty job for webrequest_test on analytics-test-hadoop
[13:29:59] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:31:03] <joal>	 ottomata: ready for timer test :)
[13:31:18] <ottomata>	 joal:  first run done?  or you want to do first run with timer?
[13:31:22] <ottomata>	 i guess we could eh?
[13:31:28] <joal>	 ottomata: first run done
[13:31:32] <ottomata>	 oh wow
[13:31:34] <ottomata>	 ok merging timer then
[13:31:38] <joal>	 ottomata: there was no data :)
[13:31:48] <ottomata>	 oh because of latest?
[13:31:53] <joal>	 correct ottomata 
[13:31:57] <joal>	 ottomata: in case: sudo -u analytics PYTHONPATH=/srv/deployment/analytics/refinery/python:$PYTHONPATH kerberos-run-command analytics /srv/deployment/analytics/refinery/bin/gobblin   --sysconfig /srv/deployment/analytics/refinery/gobblin/common/analytics-test-hadoop.sysconfig.properties   /srv/deployment/analytics/refinery/gobblin/jobs/webrequest_test.pull
[13:32:02] <joal>	 the command I manually ran
[13:32:18] <ottomata>	 great
[13:32:38] <ottomata>	 i'll check that that looks like what is in the systemd timer unit
[13:33:06] <joal>	 ack
[13:34:29] <ottomata>	  /usr/local/bin/kerberos-run-command analytics /srv/deployment/analytics/refinery/bin/gobblin --sysconfig=/srv/deployment/analytics/refinery/gobblin/common/analytics-test-hadoop.sysconfig.properties --jar=/srv/deployment/analytics/refinery/artifacts/gobblin-wmf.jar /srv/deployment/analytics/refinery/jobs/webrequest_test.pull
[13:35:10] <ottomata>	 hmm jobconfig file not correct
[13:35:14] <ottomata>	 fixing
[13:35:22] <joal>	 ottomata: typo - /srv/deployment/analytics/refinery/jobs/webrequest_test.pull -->  /srv/deployment/analytics/refinery/gobblin/jobs/webrequest_test.pull
[13:37:00] <joal>	 ottomata: I created a CR to bump the AQS druid snapshot
[13:37:21] <joal>	 ottomata: The procedure Luca wrote is here: /srv/deployment/analytics/refinery/jobs/webrequest_test.pull
[13:37:24] <joal>	 oops
[13:37:32] <joal>	 ottomata: here: https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS#Deploy_new_History_snapshot_for_Wikistats_Backend
[13:37:59] <ottomata>	 oh ok i merged
[13:38:04] <ottomata>	 lemme get the gobbln thing fixed
[13:38:08] <ottomata>	 then can do the other steps
[13:38:36] <joal>	 sure ottomata 
[13:39:43] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Replace Camus by Gobblin - https://phabricator.wikimedia.org/T271232 (10Ottomata)
[13:44:04] <ottomata>	 ok joal
[13:44:11] <ottomata>	 shall i force a run of the timer?
[13:44:20] <joal>	 sure let's try :)
[13:44:55] <ottomata>	 hmm we need PYTHONPATH
[13:44:57] <ottomata>	 in the unit
[13:44:58] <ottomata>	 fixing
[13:47:43] <ottomata>	 joal:  parallelizing
[13:47:59] <ottomata>	 going to restart aqs servers and test on canary
[13:48:30] <joal>	 ottomata: ping me when canary is ready, I can testn
[13:48:34] <ottomata>	 ok great
[13:49:33] <ottomata>	 joal:  canary ready
[13:49:46] <joal>	 testing
[13:49:49] <joal>	 eqi aqs1004
[13:49:53] <joal>	 woppsy
[13:50:45] <joal>	 all good ottomata - you can finish the deploy
[13:50:59] <ottomata>	 ok
[13:53:14] <ottomata>	 ok joal  gobblin job running
[13:53:28] <joal>	 checking on yarn ottomata 
[13:54:03] <ottomata>	 joal:  job just finished
[13:54:12] <ottomata>	 Number of bytes written=232064
[13:54:34] <joal>	 ottomata: there is now data in the correct folder :)
[13:54:38] <ottomata>	 great!
[13:54:53] <ottomata>	 also, FYI aqs restart finished
[13:55:09] <joal>	 ottomata: I just checked stats.wikimedia.org - all looks good
[13:55:21] <joal>	 ottomata: thank you for the fast actions :)
[13:55:33] <ottomata>	 joal:  i think wrong group ownership on webrequest_gobblin?
[13:55:46] <joal>	 hm - I don't think :)
[13:56:03] <ottomata>	 hmmm
[13:56:15] <joal>	 ottomata: we had it to analytics, but you asked for a change
[13:56:16] <ottomata>	 oh its different than webrequest
[13:56:17] <ottomata>	 but i thtink you are right
[13:56:19] <ottomata>	 RIIGHHHTHTH
[13:56:20] <ottomata>	 very good!
[13:56:23] <joal>	 :)
[13:56:32] <joal>	 I changed it that was on purpose
[13:56:36] <ottomata>	 the gids are not synced yet 
[13:56:41] <ottomata>	 i just saw 99 on an-test-coord
[13:56:51] <ottomata>	 we are waiting for all nodes buster to enforce consistent gids
[13:57:03] <ottomata>	 hdfs dfs -ls looks correct
[13:57:23] <joal>	 ottomata: I suggest we let gobblin run while I'm with the kids, then we finalize the migration?
[13:57:29] <ottomata>	 ok!
[13:57:34] <joal>	 perfect :)
[13:57:42] <joal>	 see you in ~1h30
[13:57:49] <ottomata>	 ok!
[13:58:08] <ottomata>	 hm the topic dir got written with 777 perms
[13:58:08] <ottomata>	 drwxrwxrwx 3 analytics analytics-privatedata-users 4096 Jul  6 13:53 webrequest_test_text
[13:59:50] <ottomata>	 i guess it doesn't really matter since parent dir is correct
[14:01:15] <ottomata>	 oh,  FYI i have a dentist appt in 3 hrs
[14:16:22] <ottomata>	 !log restarted aqs for july mw histroy snapshot deploy
[14:16:24] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:24:59] <wikibugs>	 (03PS1) 10Gerrit maintenance bot: Add jv.wikisource to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/703454 (https://phabricator.wikimedia.org/T286241)
[15:34:38] <joal>	 Heya ottomata - Here I am
[15:35:37] <ottomata>	 hello!
[15:35:44] <ottomata>	 runs looking good from here joal
[15:35:47] <joal>	 No gobblin failure so far?
[15:35:50] <joal>	 \o/
[15:36:05] <joal>	 Let's replicate the plan we have for webrequest?
[15:36:11] <joal>	 I'll list here:
[15:36:17] <ottomata>	 ok!
[15:36:20] <joal>	  - Stop gobblin and camus
[15:36:27] <joal>	  - Move camus data
[15:36:34] <joal>	  - Drop table
[15:36:38] <joal>	  - Move gobblin data
[15:36:46] <joal>	 (dropping incomplete first hour)
[15:36:54] <joal>	  - recreate table and partitions
[15:37:15] <joal>	  - Update gobblin job + deploy (mwarf)
[15:37:18] <joal>	  - restart job
[15:37:30] <joal>	 How does that look --^ ? Have I forgotten anything?
[15:37:32] <ottomata>	 to recreate table, do we do msck  repair?
[15:38:04] <ottomata>	 we also need a puppet patch to remove camus job
[15:38:07] <joal>	 nope ottomata - We recreate schema, and manually create partition with explicit paths (we miss the webrequest_source= bit)
[15:38:14] <ottomata>	 ah right
[15:39:06] <joal>	  - And puppet patch to absent the camus job, right
[15:39:23] <joal>	 Happy with the plan ottomata? Action?
[15:39:24] <ottomata>	 ok going to stop pupept and the camus and gobblini jobs
[15:39:26] <ottomata>	 ya!
[15:39:28] <joal>	 ack!
[15:40:21] <ottomata>	 done
[15:40:34] <ottomata>	 joal i'll prepare a puppet patch if you want to proceed with the data moves
[15:40:57] <joal>	 ottomata: currently preparing the refinery patch, will do the data move in a minute
[15:41:17] <ottomata>	 ok
[15:43:14] <wikibugs>	 (03PS1) 10Joal: Update webrequest_test gobblin job to prod folder [analytics/refinery] - 10https://gerrit.wikimedia.org/r/703456 (https://phabricator.wikimedia.org/T271232)
[15:43:18] <joal>	 ottomata: --^
[15:44:33] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] Update webrequest_test gobblin job to prod folder [analytics/refinery] - 10https://gerrit.wikimedia.org/r/703456 (https://phabricator.wikimedia.org/T271232) (owner: 10Joal)
[15:44:35] <wikibugs>	 (03CR) 10Ottomata: [V: 03+2 C: 03+2] Update webrequest_test gobblin job to prod folder [analytics/refinery] - 10https://gerrit.wikimedia.org/r/703456 (https://phabricator.wikimedia.org/T271232) (owner: 10Joal)
[15:44:59] <ottomata>	 joal you want to deploy or shall I?
[15:45:11] <joal>	 please go ottomata if you have bandwidth
[15:45:14] <ottomata>	 k
[15:46:04] <joal>	 ottomata: please wait
[15:46:12] <joal>	 ottomata: we also need to merge the patch for webrequest
[15:46:16] <ottomata>	 ok
[15:46:18] <ottomata>	 ?
[15:46:19] <joal>	 I thought it was
[15:46:24] <ottomata>	 oh the table?
[15:46:26] <joal>	 correct
[15:46:34] <ottomata>	 i guess that doesn't really matter whern we merge it since it is manual
[15:46:35] <ottomata>	 but ya ok
[15:46:41] <joal>	 not only the table definition, also the dataset.xml file (path changes)
[15:46:53] <ottomata>	 OH
[15:46:54] <ottomata>	 ok
[15:46:56] <ottomata>	 hmmm
[15:47:01] <ottomata>	 won't this break regular?
[15:47:03] <ottomata>	 non test?
[15:47:23] <joal>	 we'll deploy to test only, and breaking regular would require a job restart - we're safe
[15:47:31] <ottomata>	 ok
[15:47:36] <wikibugs>	 (03CR) 10Ottomata: [V: 03+2 C: 03+2] Update webrequest for gobblin [analytics/refinery] - 10https://gerrit.wikimedia.org/r/702073 (https://phabricator.wikimedia.org/T271232) (owner: 10Joal)
[15:47:54] <joal>	 Thank you ottomata :)
[15:48:01] <ottomata>	 !log deploying refinery to test cluster for webrequest_test gobblin job
[15:48:04] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:51:50] <joal>	 ! log Moved camus and gobblin data
[15:52:02] <joal>	 !log Moved camus and gobblin data for webrequest on analytics-test-hadoop
[15:52:04] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:54:45] <ottomata>	 ok joal  deploy finished
[15:55:00] <joal>	 !log Drop and recreate wmf_raw.webrequest table
[15:55:04] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:55:18] <joal>	 ack ottomata - doing some checks on data movement
[16:01:55] <joal>	 ottomata: all good on my side - Have you deployed refinery onto HDFS?
[16:01:59] <ottomata>	 oh no
[16:02:00] <ottomata>	 doing
[16:02:06] <joal>	 ottomata: in any case, ou can restart gobblin :)
[16:02:30] <joal>	 ottomata: I can care the HDFS-deploy if you wish
[16:02:40] <ottomata>	 am doing now
[16:02:43] <joal>	 ack
[16:02:58] <ottomata>	 ok going to merge the absent camus patch so I can start pupppet
[16:03:21] <joal>	 !log Kill webrequest_test oozie job
[16:03:24] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[16:06:16] <ottomata>	 ok joal camus webrequest gone
[16:06:21] <ottomata>	 gobblin job should be restart
[16:06:31] <joal>	 ok ottomata - let's wait and see :)
[16:06:37] <joal>	 Is refinery deployed onto HDFS?
[16:06:52] <ottomata>	 its doing....
[16:07:00] <joal>	 ack - It's always long:)
[16:07:59] <ottomata>	 ok its done joal
[16:08:18] <joal>	 ack ottomata - will restart oozie job with an hour to be redone
[16:08:22] <joal>	 use wmf;
[16:08:24] <joal>	 oops
[16:10:04] <joal>	 :q
[16:11:42] <ottomata>	 k!
[16:21:03] <joal>	 ottomata: soving test-related issues with oozie - hopefully done soon
[16:21:06] <ottomata>	 ok
[16:21:12] <ottomata>	 i gotta leave for dentist appt in 20 mins
[16:22:49] <joal>	 ack ottomata - will continue and let you know when you're back
[16:27:53] <joal>	 ok - problem found and solved
[16:28:47] <ottomata>	 what was it?
[16:29:20] <joal>	 ottomata: month=${"$"}{MONTH + 0}/day=${"$"}{DAY + 0}/hour=${"$"}{HOUR + 0} doesn't work, as gobblin imports padded hours - we need month=${MONTH}/day=${DAY}/hour=${HOUR}
[16:29:25] <joal>	 Sending a patch now
[16:29:53] <ottomata>	 oh nice
[16:30:00] <ottomata>	 even better
[16:31:37] <joal>	 ottomata: I had copied the pattern from webrequest without thinking (it was done the correct way for camus originally )
[16:32:17] <wikibugs>	 (03PS1) 10Joal: Fix webrequest datasets_raw.xml for gobblin [analytics/refinery] - 10https://gerrit.wikimedia.org/r/703465 (https://phabricator.wikimedia.org/T271232)
[16:32:32] <joal>	 ottomata: If you want, we can do webrequest and netflow in prod when you're back :)
[16:34:09] <ottomata>	 joal:  i'm down!
[16:34:29] <ottomata>	 my dentist is very fast, i betcha i'll be back by :40 after the next hour
[16:38:00] <joal>	 ack - I'm gonna merge/deploy as need, and starting importing data
[16:38:24] <ottomata>	 +1 k!
[16:38:32] <ottomata>	 ok back shortly!
[16:38:36] <joal>	 later!
[16:39:17] <wikibugs>	 (03CR) 10Joal: [V: 03+2 C: 03+2] "Merge for deploy of gobblin" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/703465 (https://phabricator.wikimedia.org/T271232) (owner: 10Joal)
[16:39:57] <wikibugs>	 (03CR) 10Joal: [V: 03+2 C: 03+2] "Merge for gobblin deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/702075 (https://phabricator.wikimedia.org/T271232) (owner: 10Joal)
[16:41:18] <joal>	 !log Deploy refinery for gobblin
[16:41:20] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:07:33] <joal>	 ottomata: I have an impromptu visit - Deploy is taking ages, will continue to make it work 
[17:31:43] <ottomata>	 joal:  back!
[17:31:54] <joal>	 wow - even faster than expected :)
[17:32:22] <ottomata>	 i know!
[17:32:44] <ottomata>	 i even got a couple of superficial fillings (not the kind you have to drill for, just a weird paste they put on top and then shine a UV light on)
[17:32:45] <ottomata>	 AND!
[17:32:54] <ottomata>	 my dentist gave me a cutting of a plant!
[17:33:03] <joal>	 :)
[17:33:14] <joal>	 Impressive dentist!
[17:33:21] <joal>	 !log Deploy refinery onto HDFS
[17:33:23] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:33:52] <joal>	 ok ottomata - test-cluster ok (I had some issues related to restarting the job, settings changes etc - nothing gobblin related)
[17:35:41] <joal>	 Currently deploying refinery onto hdfs, and also just launched a manual run of webrequest, will do netflow just after
[17:36:12] <joal>	 ottomata: if ok for you, you can deploy timers for both, we leave a couple hours and then switch
[17:38:02] <ottomata>	 on prod cluster?
[17:38:07] <joal>	 eah!
[17:38:08] <ottomata>	 ok making patch gimme a fe mins
[17:38:09] <ottomata>	 k
[17:38:23] <joal>	 ottomata: will be on/off, I have friends at home :)
[17:38:36] <ottomata>	 k!
[17:38:43] <joal>	 ottomata: initial jobs have run for both webrequest and netflow (sudo -u analytics PYTHONPATH=/srv/deployment/analytics/refinery/python:$PYTHONPATH kerberos-run-command analytics /srv/deployment/analytics/refinery/bin/gobblin   /srv/deployment/analytics/refinery/gobblin/jobs/netflow.pull
[17:39:03] <ottomata>	 niiiice
[17:39:18] <joal>	  and sudo -u analytics PYTHONPATH=/srv/deployment/analytics/refinery/python:$PYTHONPATH kerberos-run-command analytics /srv/deployment/analytics/refinery/bin/gobblin \
[17:39:21] <joal>	 >   /srv/deployment/analytics/refinery/gobblin/jobs/webrequest.pull
[17:52:31] <ottomata>	 ok joal  jobs declared
[17:52:36] <joal>	 \o/
[17:52:51] <ottomata>	 i guess we have to wait a bit now right?
[17:53:16] <joal>	 ottomata: we have a perms issue, but as you noticed, having the parent folder perm setup correctly mitigates for now - I'll investigate soon
[17:53:20] <ottomata>	 ok
[20:23:30] <joal>	 ottomata: Hi! Is now the time to switch data and oozie, or do you prefer tomorrow?
[20:40:07] <joal>	 ok - let's make it tomorrow :)
[21:17:05] <ottomata>	 ah sorry joal!
[21:17:12] <ottomata>	 lets do tomrrow!
[21:17:17] <ottomata>	 i'll be on by 9 my time, maybe before?