[10:48:22] <dcausse>	 lunch+errand
[12:57:37] <inflatador>	 greetings
[13:01:43] <dcausse>	 o/
[13:08:29] <inflatador>	 <o/
[14:10:59] <i51>	 hi
[14:11:07] <i51>	 hi
[14:16:22] <inflatador>	 brb
[16:46:19] <inflatador>	 not going to make unMeeting today, got sucked into car repair
[17:30:56] <inflatador>	 sorry, been back
[17:57:53] <ebernhardson>	 previously referenced torvalds rent where he never actually says to use pickaxe, but simply describes using --follow with git log is an "svn noob" pleaser and not "real git functionality". https://marc.info/?l=git&m=123333626615117&w=2
[17:57:58] <ebernhardson>	 s/rent/rant/
[17:58:31] <ebernhardson>	 i suppose i would have seen the pickaxe reference in a discussion about this posting
[17:59:59] <ebernhardson>	 probably something like https://stackoverflow.com/a/5743887/5596181
[18:07:46] <inflatador>	 Oh well, I'm OK with a noob pleasing feature
[18:16:05] <inflatador>	 Lunch, back in ~30
[18:16:24] <ebernhardson>	 i've used it plenty of times, also used to use svn :)
[18:46:35] <inflatador>	 back
[19:10:59] <ebernhardson>	 noticed we didn't re-enable deprecation logging in cloudelastic cluster state. Going to re-apply to all clusters. Wonder if we should have some more automated way to apply the same cluster state update to everyhting
[19:36:38] <inflatador>	 persistent config should "theoretically" live in puppet, I'm guessing most API settings can also live in elasticsearch.yml but if y'all know better let me know
[19:37:05] <ebernhardson>	 the problem with elasticsearch.yml is you have to restart the cluster to apply the settings, thats why they don't end up living in puppet
[19:42:44] <inflatador>	 We do have some cluster settings in elasticsearch.yml
[19:43:27] <inflatador>	 but I dunno if others could be dangerous, like if there's different settings between master eligibles, what happens there
[19:43:59] <ebernhardson>	 elasticsearch has a difference between what it considers static and dynamic settings, things like master eligible are probably not dynamic and settable at runtime
[19:44:24] <ebernhardson>	 in this particular case i'm looking at logging levels, which are set in /_cluster/settings but could be set elsewhere. 
[19:45:03] <ebernhardson>	 i suppose i was thinking about it because while we try to keep elasticsearch.yml in sync with reality, it's not the definative source. The cluster settings provide the final overrides, so perhaps that is what needs to be managed
[19:45:55] <ebernhardson>	 i suppose i mix these terms up, but by cluster settings i mostly mean the live cluster state of the individual clusters, modified by the /_cluster/settings api
[19:47:33] <inflatador>	 yeah, and I believe they are deprecating the transient route https://www.elastic.co/guide/en/elasticsearch/reference/current/transient-settings-migration-guide.html
[19:49:37] <inflatador>	 maybe not deprecating based on that article, just discouraging
[19:59:48] <ebernhardson>	 yea we need to do that at some point too
[21:15:46] <ebernhardson>	 meh, turned on DeprecationLoggedHttps but it's a bit too spammy right now. Need to wait for the disable_coord fixes to deploy next week
[21:16:59] <ebernhardson>	 i guess not the end of the world, but 5-10 per sec (at the low point of the day) seems a bit much
[21:44:14] <inflatador>	 ebernhardson we are getting "Unrecognized VM option 'PrintGCDateStamps'" when trying to start ES6.8 on Bullseye (JDK 11). Web search suggests this option may have changed, does that ring a bell?
[21:46:11] <inflatador>	 ref https://issues.redhat.com/browse/CLOUD-3040
[21:48:06] <ebernhardson>	 inflatador: hmm
[21:48:39] <ebernhardson>	 inflatador: so, my guess before looking closely is we have a jvm.options file that was probably specialized to use case
[21:48:54] <ebernhardson>	 inflatador: and we don't update that directly with version upgrades most of the time. checking
[21:50:07] <inflatador>	 ebernhardson yeah, ryankemper sent me the link, looks like jvm options are coupled to ES version ATM https://github.com/wikimedia/puppet/blob/e1787fadbb2764a4036f2c4fcab79000861696c7/modules/elasticsearch/manifests/instance.pp#L173-L195
[21:51:57] <ebernhardson>	 inflatador: i suppose the default answer would be we have to vary it on the java version. How are we deciding in puppet to use a particular java version?
[21:53:44] <inflatador>	 ebernhardson  At the moment, we aren't ;(  . So buster and later automatically pull in openjdk 11 as the default java. We're in meet.google.com/vbs-enbz-swe  if you wanna talk about it some more, looking at different ways to do this
[22:06:25] <inflatador>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/787106