[08:29:48] <dcausse>	 gehel: the wcqs cookbook seems to have failed do you have access to the logs? 
[08:30:02] <gehel>	 dcausse: looking
[08:31:27] <gehel>	 looks like it failed to stop the updater
[08:31:43] <gehel>	 https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-reload.py#L151
[08:32:05] <dcausse>	 ah indeed...
[08:32:06] <gehel>	 yep, to the data load has not started
[08:32:12] <dcausse>	 hm...
[08:32:39] <gehel>	 we could make the cookbook smarter, if the updater is already stopped this should not be an issue
[08:32:52] <dcausse>	 the systemunit is not even there
[08:33:16] <gehel>	 ah yes, that's a bigger issue, the cookbook should probably not be robust to that
[08:34:11] <zpapierski>	 I wonder should we add a parameter for this - doesn't sound terribly useful in the future and adds to code complexity
[08:35:07] <gehel>	 I don't think we should, this is most probably the only time we'll have this issue
[08:36:12] <dcausse>	 systemctl list-unit-files | grep -E ^wcqs-updater > /dev/null && echo systemctl stop wcqs-updater ?
[08:36:29] <dcausse>	 or we enable the updater...
[08:37:33] <zpapierski>	 can't we just modify the script to disable updater changes now ?
[08:37:43] <zpapierski>	 (I know it's a hack)
[08:38:12] <gehel>	 or we finish this run manually
[08:39:08] <gehel>	 if we restart the cookbook (after fixing), we can skip the download, but I don't think we have an option to skip the munging
[08:39:58] <dcausse>	 if the munging is done then we can prep a small set of command lines to continue the import
[08:40:27] <gehel>	 the data load is trivial
[08:41:02] <gehel>	 not sure about the kafka magic
[08:41:20] <dcausse>	 there should be no kafka magic
[08:41:30] <zpapierski>	 there is, actually
[08:41:48] <zpapierski>	 but it's weird, I don't remember that too
[08:43:02] <gehel>	 that cookbook should be simplified, too many level of indirection
[08:43:24] <gehel>	 we SRE are simple people, all those function handles will confuse us :)
[08:44:11] <dcausse>	 should be something around those lines: https://etherpad.wikimedia.org/p/import_wcqs
[08:44:35] <zpapierski>	 looks ok
[08:44:51] <zpapierski>	 that Kafka thing is bothering me, I'm checking it out
[08:44:56] <gehel>	 agreed
[08:44:59] <gehel>	 I can start on those
[08:45:13] <gehel>	 while zpapierski figures out the kafka thingy
[08:45:30] <dcausse>	 gehel: wait the last touch is wrong
[08:45:53] <dcausse>	 btw it seems to be missing on the commons reload in the cookbook
[08:46:12] <gehel>	 I'm not going to script, just send the commands one after the other, you have time 
[08:46:34] <dcausse>	 ok
[08:47:20] <zpapierski>	 huh, interesting
[08:47:21] <dcausse>	 the cookbook is missing some stuff post-reload
[08:48:01] <zpapierski>	 obviously, I was the one to throw in that kafka stuff, since we discussed that we should set the kafka offset after being done, otherwise updater might fail
[08:48:53] <dcausse>	 yes or it might reset to earliest, checking
[08:48:55] <zpapierski>	 but we didn't use the --kafka-timestamp parameter, which I don't what means - default value? would fail with, because it's none
[08:48:56] <zpapierski>	 ?
[08:49:17] <gehel>	 com.bigdata.rdf.sail.webapp.DatasetNotFoundException: namespace=wcqs
[08:49:37] <gehel>	 where do we create that namespace...
[08:50:02] <zpapierski>	 those were adapted from wdqs, maybe we don't
[08:50:16] <zpapierski>	 but there should be data already, probably namespace name is wrong
[08:50:27] <dcausse>	 gehel: "wcq" perhaps
[08:50:52] <gehel>	 my bad :/
[08:51:00] <zpapierski>	 yep
[08:51:03] <zpapierski>	 that's the one
[08:51:34] <gehel>	 data load in progress
[08:52:12] <zpapierski>	 anyway, we don't need to care about that kafka thing, we wanted to take care of that ourselves after initiating updater consumer on those
[08:52:46] <dcausse>	 this timestamp param should be mandatory in case of reloads now
[08:54:04] <zpapierski>	 yeah, we only could get away from that because producer hasn't started yet
[09:01:41] <zpapierski>	 it seems that import_commons_ttl was completed
[09:07:07] <dcausse>	 nice
[09:07:18] <zpapierski>	 I'm downloading the rev map
[09:08:02] <zpapierski>	 where are we deploying the jobs from? deploy hosts?
[09:08:31] <zpapierski>	 https://www.irccloud.com/pastebin/JF248v9X/
[09:08:46] <zpapierski>	 I see that the fix worked, no need for sed/awk the output :)
[09:09:15] <dcausse>	 cool
[09:27:25] <zpapierski>	 ok, it seems I can create a savepoint now, I'll do so
[09:33:39] <zpapierski>	 dcausse: do I need to bootstrap that state into any specific path on swift?
[09:33:50] <dcausse>	 zpapierski: yes
[09:34:08] <dcausse>	 lemme check
[09:35:24] <dcausse>	 something like swift://rdf-streaming-updater-eqiad.thanos-swift/commons/savepoints/bootstrap_$DUMP_DATE
[09:35:40] <dcausse>	 and then same for rdf-streaming-updater-codfw.thanos-swift
[09:36:10] <zpapierski>	 got it, starting it now
[09:40:36] <zpapierski>	 ok, submitted, let's see if it fails
[09:56:58] <zpapierski>	 huh, weird - I can't see that savestate on swift, even though the job finished without exceptions
[09:57:36] <zpapierski>	 there's some from a week ago, strangely
[09:58:39] <dcausse>	 hm.. strange
[09:58:47] <zpapierski>	 let's see if I get the same thing on codfw
[10:02:32] <zpapierski>	 I probably messed up some parameters
[10:02:52] <dcausse>	 wrong swift container perhaps?
[10:03:14] <zpapierski>	 I'm checking
[10:03:24] <dcausse>	 should be rdf-streaming-updater-eqiad or rdf-streaming-updater-codfw
[10:04:46] <zpapierski>	 arghh
[10:04:54] <zpapierski>	 SAVEPOINT_DIR= swift://rdf-streaming-updater-codfw.thanos-swift/commons/savepoints/bootstrap_20220109
[10:05:01] <zpapierski>	 you see what's wrong :) ?
[10:05:35] <dcausse>	 hm the space?
[10:05:39] <zpapierski>	 yep
[10:05:46] <zpapierski>	 it's bash, after all
[10:05:53] <dcausse>	 I wonder where it went then
[10:06:36] <zpapierski>	 no idea, but it definitely did something
[10:06:40] <zpapierski>	 nvm, I'm running again
[10:17:59] <zpapierski>	 and now it's a different story, savepoint is there
[10:18:05] <zpapierski>	 now codfw
[10:47:22] <zpapierski>	 ok, got both
[10:49:49] <zpapierski>	 dcausse: will the query from the bootstrap instructions work for commons? 
[10:49:52] <zpapierski>	 https://www.irccloud.com/pastebin/qd8BeXqj/
[10:50:00] <zpapierski>	 ah, wiki=commons?
[10:50:14] <dcausse>	 yes and the date
[10:50:22] <zpapierski>	 yep
[10:50:23] <zpapierski>	 ok ,thx
[10:53:40] <gehel>	 lunch
[10:54:30] <gehel>	 WCQS data load progress: 84/724
[10:54:56] <zpapierski>	 and I got the start date for consumers
[11:01:37] <ejoseph>	 zpapierski: can we continue
[11:01:48] <ejoseph>	 The build failed yesterday 
[11:03:11] <zpapierski>	 ejoseph: sure, there's no rush with streaming updater, I'll get back to that later on
[11:04:26] <zpapierski>	 https://meet.google.com/tzp-bave-spz
[11:13:39] <dcausse>	 errand+lunch
[11:15:08] <zpapierski>	 ejoseph: I've lost you, unfortunately
[11:15:22] <zpapierski>	 I need to consume, but I'll be back in few minures
[11:15:25] <zpapierski>	 s/minures/minutes
[11:19:22] <ejoseph>	 Sorry my network
[11:35:55] <zpapierski>	 no worries
[11:35:57] <zpapierski>	 I'm back
[11:36:58] <ejoseph>	 5minutes pls
[11:37:03] <zpapierski>	 ok
[12:05:17] <zpapierski>	 ltr plugin is proving difficult to build - dcausse: did you have any issues in the past? I see that you build (or at least stored) the last version
[12:56:10] <zpapierski>	 apparently it requires java version precisely 12 to compile
[13:00:59] <zpapierski>	 lunch
[13:06:49] <gehel>	 WCQS data load progress: 144/724
[13:45:53] <dcausse>	 zpapierski: yes the elastic gradle build system requires specific java version for building
[14:05:39] <gehel>	 ryankemper: for when around, can you confirm (here or email) your availability for the ES training (https://www.elastic.co/training/elasticsearch-engineer/6342)
[14:08:19] <inflatador>	 greetings!
[14:08:26] <gehel>	 o/
[14:08:41] <inflatador>	 gehel you'll have to show me how you calculated progress on that data load!
[14:11:25] <gehel>	 WCQS data load progress: 171/724
[14:11:35] <gehel>	 I'll show you in 20' in our 1-on-1
[14:14:57] <dcausse>	 o/
[14:33:23] <zpapierski>	 o/
[14:37:38] <zpapierski>	 how do I upload to that people.wikimedia.org?
[14:44:18] <dcausse>	 zpapierski: scp file people.eqiad.wmnet:~/public_html
[14:44:25] <zpapierski>	 thanks!
[14:44:51] <dcausse>	 well create the public_html folder first :)
[14:55:16] <zpapierski>	 that's a sound advice
[14:55:50] <zpapierski>	 apparently I already had one
[15:00:47] <zpapierski>	 dcausse: last we need to check is analysis hebrew and it's shared from you dir - is this the plugin - https://github.com/synhershko/elasticsearch-analysis-hebrew?
[15:02:52] <dcausse>	 zpapierski: yes, we need to check if there exists an official version or we still have to build it on our own
[15:03:12] <zpapierski>	 yeah, we've been following that process so far
[15:03:27] <zpapierski>	 (only needed for two, and one was ours, fortunately)
[16:00:27] <Trey314159>	 It's time for office hours
[16:04:24] <Trey314159>	 zpapierski dcausse office hours?
[16:04:36] <zpapierski>	 we are presenting on all tech today
[16:04:42] <ryankemper>	 gehel: confirming avail for https://www.elastic.co/training/elasticsearch-engineer/6342
[16:41:07] <gehel>	 WCQS data load progress: ~227/724
[16:51:06] <zpapierski>	 speed demon, that one is
[17:08:44] <inflatador>	 ┗(-_- )┓┗(-_-)┛┏( -_-)┛ ┗(-_- )┓┗(-_-)┛┏( -_-)┛
[17:33:19] <inflatador>	 family emergency, back as soon as I can
[17:40:46] <ejoseph>	 zpapierski: i have managed to ruin my java setup
[17:40:56] <zpapierski>	 that happens :)
[17:41:10] <ejoseph>	 I'm working on fixing it
[17:41:18] <zpapierski>	 I'm guessing installing JDK 12 didn't work out well?
[17:41:31] <ejoseph>	 Yh
[17:41:43] <zpapierski>	 btw - in my experience, sdkman is a better tool than jenv
[17:41:54] <zpapierski>	 and it handles other sdks than javas as well
[18:02:45] <zpapierski>	 unmeeting anyone?
[18:09:31] <inflatador>	 Got some bad news, taking rest of the day off. See you tomorrow!
[21:41:58] * ebernhardson fights with cindy/mwv to turn display_errors off and keep it off
[22:13:30] <ebernhardson>	 meh, also fun: grunt on cirrus-integ.eqiad.wmflabs (runs cindy) gets stuck at the end of a run, but doesn't exit. poked a bit but it's non-obvious, and while poking built a new env locally and it seems to work. Going to bring up a new instance instead of trying to fix old one
[22:14:22] <ebernhardson>	 also, lunch
[22:46:21] <ebernhardson>	 back