[08:22:00] <dcausse>	 ejoseph, zpapierski: Erik merged the patch for extra@6.8 yesterday
[08:24:05] <dcausse>	 I think this means that we're good for es 6.8
[08:37:48] <ejoseph>	 Oh great
[08:38:08] <ejoseph>	 Good morning all
[08:38:38] <dcausse>	 ejoseph: added a small comment on MW core Special:Search UI fix
[08:38:50] <dcausse>	 I think there should be two patches
[08:39:09] <dcausse>	 this makes the UI fix harder to review
[08:42:45] <ejoseph>	 Yh
[08:42:55] <ejoseph>	 DO you have a minute
[08:43:43] <dcausse>	 sure
[09:08:21] <zpapierski>	 gehel: awesome, I missed David fixes to our latest work - and yeah, it looks like this part is good to go
[09:11:48] <gehel>	 we're still missing the other plugins (the one we don't maintain
[10:25:18] <zpapierski>	 errand
[12:15:52] <zpapierski>	 ejoseph: I have some time before lunch, if you want to work on something
[12:17:12] <ejoseph>	 I need to run an errand 
[12:17:26] <ejoseph>	 I would be back by 2:30
[12:17:36] <zpapierski>	 cool, I'll be done with lunch by then
[12:17:42] <ejoseph>	 Would you be available by then
[12:18:43] <zpapierski>	 I should
[12:18:55] <zpapierski>	 I'll take my break now in that case
[13:36:51] <zpapierski>	 ejoseph: let me know when you're back
[13:37:56] <zpapierski>	 gehel: I probably asked in the past, but where can I find the list of external plugins to verify?
[13:38:30] <gehel>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/elasticsearch/plugins/+/refs/heads/master/debian/plugin_urls.lst
[13:38:40] <zpapierski>	 thanks!
[14:00:40] <inflatador>	 Greetings
[14:00:59] <dcausse>	 o/
[14:01:21] <zpapierski>	 o/
[14:07:32] <inflatador>	 Where do y'all prefer to keep your work notes? I know Trey314159 mentioned using one of the wikis. I was thinking there, or maybe if there is a shared Google notes kinda thing for our team?
[14:08:55] <gehel>	 I have a text file named notes.txt in my home folder
[14:09:43] <dcausse>	 inflatador: there's a shared google drive but not sure it's well designed for notes, I find wiki pages under your username a nice way of storing notes (and it's searcheable!)
[14:10:13] <zpapierski>	 I use Simplenote - seemed like a closest approx. to Apple Notes, available everywhere
[14:10:19] <inflatador>	 dcausse sounds good. Is that on officewiki or which one?
[14:10:49] <dcausse>	 inflatador: office wiki would be for private (private here is visible to WMF employees only) notes
[14:11:21] <dcausse>	 wikitech for very technical notes (cluster setup and the like)
[14:11:46] <dcausse>	 mediawiki.org for more generic with a wider public audience
[14:12:40] <dcausse>	 I generally use wikitech for notes
[14:14:26] <inflatador>	 yeah, I would like to have the notes visible at least to other employees. Not sure if I'd like everything to be public. I guess my main goal is to have everything in one place, but I'm pretty bad at that ;(
[14:16:37] <inflatador>	 zpapierski with Simplenote, can you share notes with people that don't have the app?
[14:17:28] <zpapierski>	 I wonder - I picked up the experimental highlighter code and it skips 6.8.x versions. I created a branch of the 6.5.4 commit and made it work on 6.8.20 - I wonder how to proceed with this now. Would we rather leave it in a branch? I don't see much point to integrate it with the master tree, but maybe that's what I should do somehow. dcausse, gehel - thoughts?
[14:18:12] <zpapierski>	 (I need to note that I have no idea how to make gerrit rewrite a parent of the next in line commit)
[14:18:27] <gehel>	 zpapierski: you've lost me already
[14:19:17] <dcausse>	 zpapierski: did you create the branch via gerrit?
[14:19:26] <gehel>	 Oh, I see
[14:19:28] <zpapierski>	 nope, didn't know you can do that
[14:19:59] <dcausse>	 so in https://gerrit.wikimedia.org/r/admin/repos/search/highlighter,branches
[14:19:59] <zpapierski>	 2 years+ and gerrit still manages to surprise me
[14:20:12] <dcausse>	 create a branch from the commit you want to branch from
[14:20:48] <gehel>	 most of the commits > release 6.5.4 seem to be build stuff, I would actually create the 6.8.20 branch from the current tip of master
[14:21:20] <zpapierski>	 ok, that would make things easier, because there's no 6.5.4 branch
[14:21:41] <dcausse>	 not sure I follow
[14:22:21] <zpapierski>	 don't worry, it's just me getting lost in the interface, I see how would I do that now
[14:22:37] <dcausse>	 I think it's easier to create a 6.8 branch starting from the commit just before the 7 upgrade
[14:23:12] <zpapierski>	 will do
[14:24:18] <zpapierski>	 https://gerrit.wikimedia.org/g/search/highlighter/+/refs/heads/6.8.20
[14:24:27] <gehel>	 that would work as well, all the later commits seem to be minor and can be ignored for 6.8.20
[14:25:10] <dcausse>	 https://gerrit.wikimedia.org/r/c/search/highlighter/+/652555 has some stuff in it
[14:26:27] <zpapierski>	 I don't follow - I was suppose to skip 7 upgrade?
[14:27:00] <dcausse>	 zpapierski: I don't think 17223e284f38c35763cb009f93e551f378d8b70f is correct
[14:27:25] <dcausse>	 should be afe958a6f31bb2be17161b1d484470061c6530a7 I think
[14:27:40] <zpapierski>	 ok, I'll use this one
[14:28:59] <zpapierski>	 ah, you meant the latest version upgrade in 7, not the 7 in general
[14:30:17] <dcausse>	 hm wait
[14:31:20] <dcausse>	 wow scratch everything I said
[14:32:09] <dcausse>	 completely forgot there was so many patches with es 7 compatibility, sorry for the confusion
[14:32:25] <dcausse>	 sigh...
[14:33:11] <zpapierski>	 yeah, this won't do - a lot of isses with compilation for previous version
[14:34:24] <dcausse>	 17223e284f38c35763cb009f93e551f378d8b70f is actually correct my bad, sorry about that
[14:34:38] <zpapierski>	 no worries, I kept the changes in stash
[14:35:41] <dcausse>	 we could cherry-pick later patches from Guillaume if we want to tho
[14:36:27] <dcausse>	 when creating a branch in gerrit the first patch you make is to switch .gitreview to use that branch by default
[14:36:50] <dcausse>	 and the scm urls in the poms
[14:41:20] <dcausse>	 hm.. maybe not the scm url in the poms... can't remember if the release plugin is going to be confused by that not pointing to the right branch
[14:43:55] <zpapierski>	 dcausse: can I change that .gitreview with my changes, or does that patch needs to only contain a change to that file?
[14:45:30] <dcausse>	 zpapierski: hm... I think the only advantage to separate them is to make your change cherry-pickable elsewhere
[14:46:17] <zpapierski>	 which I guess we won't need? the only changes not already made in future versions are poms
[14:46:44] <dcausse>	 probably not
[14:51:44] <zpapierski>	 inflatador: Simplenote has a collaboration feature, haven't tried it yet though
[14:52:08] <zpapierski>	 apparently it's only between simplenote accounts (which makes sense)
[14:52:52] <zpapierski>	 huh, you can publish your notes, so that they are available on the web, though, but only read only
[14:53:37] <zpapierski>	 it doesn
[14:54:01] <zpapierski>	 it doesn't have much more than that,though - it's quite a simple app, like the name suggests 
[14:54:05] <zpapierski>	 that's why I like it
[14:59:10] <zpapierski>	 dcausse: from what I see, scm seems the same on branches - https://gerrit.wikimedia.org/r/plugins/gitiles/search/highlighter/+/refs/heads/5.6/pom.xml
[14:59:57] <dcausse>	 zpapierski: cool! then I guess it'll just work
[15:00:04] <zpapierski>	 famous last words :D
[15:00:16] <dcausse>	 :)
[15:04:58] <zpapierski>	 I pushed the review, if it's suppose to be picked up by jenkins there - https://gerrit.wikimedia.org/r/c/search/highlighter/+/753056 
[15:05:26] <dcausse>	 nice, yes it should be
[15:07:41] <zpapierski>	 yep, it was 
[15:34:06] <dcausse>	 elastic1049@psi is acting up again with CirrusSearchJVMGCOldPoolFlatlined
[15:43:50] <ebernhardson>	 \o
[15:45:22] <dcausse>	 o/
[15:46:44] <ebernhardson>	 1049@psi does seem fairly full, although i wonder if it's truly struggling.  It only has a an 8G heap and i see mins around 6G in the last hour or two
[15:47:08] <ebernhardson>	 maybe i didn't consider the smaller instances well enough when setting the threshold, it's a static value rather than a % of heap
[15:47:19] <dcausse>	 yes it seems to survive and be able to reclaim some space over time
[15:47:46] <dcausse>	 but looking at greater time window something is diminishing
[15:47:48] <dcausse>	 https://grafana-rw.wikimedia.org/d/000000462/elasticsearch-memory?viewPanel=56&orgId=1&from=now-30d&to=now&var-datasource=eqiad%20prometheus%2Fops&var-exported_cluster=production-search-psi-eqiad&var-instance=elastic1049&var-top5_fielddata=All&var-top10_terms=All&var-top5_completion=All
[15:48:43] <ebernhardson>	 oh! thats much more telling than the node compariosn graph i was looking at
[15:48:51] <ebernhardson>	 yea, thats having trouble :)
[15:49:38] <ebernhardson>	 i think the only available solution is to restart the instance, looks like it only lasted a month since the last one
[15:49:46] * ebernhardson does that
[15:49:50] <dcausse>	 thanks!
[15:50:47] * dcausse still struggles with retries & timeout in airflow sensors...
[15:52:02] <ebernhardson>	 they are odd :( we also need to upgrade to airflow 2 at some point, and unify with the other teams airflow work
[15:52:12] <dcausse>	 I'm tempted to not set them
[15:52:21] <dcausse>	 and use sla as a warning
[15:52:29] <ebernhardson>	 seems reasonable
[15:54:35] <ebernhardson>	 inflatador: i need to shift our meeting later today, i wasn't expecting to but something changed and at 2:45 (in the middle of meeting) i have to go on a school run
[15:59:00] <ebernhardson>	 dcausse: i also started updating streaming updater docs re: flink-jobs.py changes, could probably use review: https://wikitech.wikimedia.org/w/index.php?title=Wikidata_Query_Service/Streaming_Updater&oldid=1939662
[15:59:22] <ebernhardson>	 i was intending to go through the rest of it, i only managed one section so far :P
[15:59:32] <inflatador>	 ebernhardson that's fine, feel free to change, I'm generally available until 3:30 PM PST
[16:00:47] <dcausse>	 ebernhardson: thanks!
[16:50:01] <zpapierski>	 gehel: can you provide a quick help with cookbook params?
[16:50:40] <gehel>	 zpapierski: what's the issue?
[16:50:49] <zpapierski>	 might join us back? 
[16:50:55] <zpapierski>	 s/might/mind
[16:51:02] <gehel>	 3' 
[16:51:35] <zpapierski>	 great/ thx
[17:17:33] <inflatador>	 may be 5m late to ES kickoff mtg
[17:33:31] <gehel>	 ejoseph: ES7 kick off: https://meet.google.com/hjp-ngzi-czf
[18:21:07] <dcausse>	 dinner
[18:23:01] <inflatador>	 exercise/lunch
[19:02:34] <ejoseph>	 gehel: my calendar did not notify me of the meeting and i was distracted
[19:07:59] <ebernhardson>	 was double checking and we skipped the clear journal step for wcqs earlier. The cookbooks were all munging (haven't started uploading yet) so i cleared them now along with the old aliases.map
[19:12:48] * ebernhardson is oddly just realizing that hexadecimal is backwards, 6-a-1-imal
[19:18:29] <ryankemper>	 :P
[19:18:46] <ryankemper>	 so it'd more accurately be decihexamal?
[19:19:30] <ebernhardson>	 it sounds odd, but i suppose my general expectation is most significant to the left. 
[19:19:56] <ebernhardson>	 there is probably a reason for that order, i might look it up later :)
[19:37:27] <inflatador>	 and back
[19:38:20] <inflatador>	 ebernhardson does that mean you started the process over? Let me know if I need to start it again
[19:40:47] <gehel>	 ebernhardson: btw, the cookbook takes care of cleaning up the journal before reimporting: https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-reload.py#L153
[19:41:03] <ebernhardson>	 gehel: ahh, i wonder why we called it out separately in the checklist then
[19:41:22] <gehel>	 abundance of caution
[19:41:49] <ebernhardson>	 inflatador: it shouldn't need to, the dump is loaded in the third stage and this was still in the second stage (so turns out, it will delete the journal again)
[19:45:01] <inflatador>	 ACK
[20:28:12] <ebernhardson>	 lunch
[20:46:12] <inflatador>	 Is there a phab ticket for the WCQS data load we did earlier? Can't seem to find it
[20:52:26] <ebernhardson>	 back
[20:52:28] <ebernhardson>	 inflatador: https://phabricator.wikimedia.org/T296470
[20:52:43] <ebernhardson>	 i guess not it's own ticket, but its a checklist item
[20:54:37] <inflatador>	 Thanks! I see it's on there on all the data load processes too
[22:29:00] <inflatador>	 ebernhardson we're getting puppet failure alerts for wcqs-beta-01 and I see a note from you that says " Puppet is disabled." - Do we want to keep puppet disabled on this one?
[23:02:11] <ebernhardson>	 inflatador: hmm, i suppose the only real fix we have there is to adjust the prod puppet. The underlying issue is wcqs-beta needs to have no auth, but the prod puppet was written to only support auth (since prod shouldn't be able to be deployed without it)
[23:02:50] <ebernhardson>	 shouldn't be too hard to write, i wrote it before but removed during CR
[23:03:47] <inflatador>	 We could just disable the puppet check if that's easier
[23:04:18] <ebernhardson>	 in general i think we would be safer with puppet off. We are going to continue making small modifications, and wcqs-beta doesn't have prod-like monitoring (we don't know when it breaks)
[23:04:53] <ebernhardson>	 (i mean small modifications to the wcqs puppet code, which will likely break wcqs-beta if applied)
[23:05:27] <inflatador>	 OK, let's leave puppet disabled on beta, and disable the puppet check alert on beta if that works for you
[23:05:54] <ebernhardson>	 i could be mistaken, but i was under the impression the check was hard to disable since it's a global check against all hosts. If it's easy then go for it :)