[09:11:21] <ejoseph>	 dcausse: are you around
[12:54:45] <inflatador>	 greetings!
[13:03:42] <inflatador>	 Just pinged data persistence in their chat room, hopefully we can get https://phabricator.wikimedia.org/T309715 taken care of soon
[13:46:00] <inflatador>	 dropping off my sons, back in ~15
[14:02:57] <inflatador>	 and back
[14:03:28] <inflatador>	 looks like we have creds to the thanos-swift cluster now, gonna give those a shot
[15:03:01] <inflatador>	 If anyone has time to look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/803300/ LMK, I don't understand why jenkins doesn't like it
[15:03:40] <inflatador>	 it warns "double quoted string containing no variables (double_quoted_strings)" but I don't get why that's a failure
[15:05:13] <inflatador>	 looks the same as other file_line invocations, at least as far as I can tell
[15:05:31] <volans>	 inflatador: if there are no variables (like f-strings in python) you need to use single quotes
[15:05:41] <volans>	 it's kinda like you had an f-string without any variable replacement inside
[15:07:06] <ryankemper>	 inflatador: triage? https://meet.google.com/eki-rafx-cxi?authuser=1
[15:07:27] <inflatador>	 OMW
[15:07:29] <volans>	 https://puppet.com/docs/puppet/7/lang_data_string.html#lang_data_string_single_quoted_strings
[15:07:57] <volans>	 actually, to be precise
[15:07:57] <volans>	 https://puppet.com/docs/puppet/5.5/lang_data_string.html#single-quoted-strings
[15:08:57] <volans>	 and to be clear, it's a style thing
[15:12:40] <inflatador>	 volans  Ah OK, got it...it's interpreting the line as a puppet var, I guess I need to escape it
[15:14:11] <volans>	 inflatador: no just use single quotes
[15:14:19] <volans>	 when not using any variable for interpolation
[15:14:45] <inflatador>	 Will give it a try, thanks volans 
[15:20:51] <inflatador>	 LOL, now it thinks it's a variable and I should be using double quotes
[15:21:44] <volans>	 which one?
[15:22:11] <volans>	 ah ${PATH}
[15:22:43] <inflatador>	 line 145, yeah
[15:22:58] <inflatador>	 I guess I can just go with "$PATH"
[15:23:21] <volans>	 so for that use "PATH=\${PATH}...."
[15:25:38] <inflatador>	 ACK
[15:25:38] <volans>	 see https://puppet.com/docs/puppet/5.5/lang_data_string.html#escape-sequences
[16:10:12] <inflatador>	 OK, Jenkins is happy, anyone have time for review? https://gerrit.wikimedia.org/r/c/operations/puppet/+/803300
[16:11:08] <ebernhardson>	 inflatador: seems easy enough, +1
[16:11:27] <inflatador>	 muchas gracias ebernhardson 
[17:08:27] <inflatador>	 ebernhardson or anyone else, do you know if I can find the thanos-swift creds for flink on the wdqs servers?  I'm not having any luck logging in with our new swift thanos user, and I want to compare how it's set up for flink, specifically the api key value at link 28 here https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/769699/2/charts/flink-session-cluster/templates/_flink-conf.tpl#b27
[17:10:18] <ebernhardson>	 hmm, not sure but lets look :) I suppose i would guess somehow those come from helm and get injected into perhaps a mounted volume of the container?
[17:10:46] <inflatador>	 yeah, I think you're right. Was hoping to get lucky but I don't see it on wdqs 
[17:11:21] <ebernhardson>	 if you can login to the host running the container you should be able to run a new container that mounts the volume and reports its contents (would have to check how exactly), otherwise can figure out where helm is getting its data
[17:12:05] <inflatador>	 yeah, I think I can do that...also the helm secrets might be on puppet, I think I did a patch for that recently
[17:12:45] <inflatador>	 openstack (or at least the weird one we used to run at Rackspace) had separate concepts of API key and password. I don't think there is a distinction in vanilla openstack, but I want to make sure I'm not missing anything
[17:13:55] <ebernhardson>	 looking at the volume should be something like `docker run --rm -i -v=<container name>:/mnt --entrypoint bash <some valid image> /bin/bash find /mnt`
[17:13:57] <ebernhardson>	 (untested)
[17:14:15] <ebernhardson>	 err, probably only invoke bash once
[17:18:00] <ebernhardson>	 this works locally at least: docker run --rm -it -v=mwcli-mwdd-default_elasticsearch-data:/mnt --entrypoint /bin/bash docker-registry.wikimedia.org/releng/ci-stretch:latest -c "find /mnt"
[17:20:52] <inflatador>	 I'll dig around a bit, I was hoping to find the info on the deploy server, but it's just templates so far. I guess the secrets are probably in etcd. Maybe peeking in the container is easier
[17:23:24] <ebernhardson>	 i see reference to some /etc/helmfile-defualts/private, but sadly i know nothing about where it comes from or what it is
[17:40:07] <inflatador>	 I notice the swift usernames are $account:$user as opposed to $user in some places, like hieradata/common/profile/swift.yaml
[17:40:13] <inflatador>	 maybe that's the problem
[17:47:21] <inflatador>	 yeah, that was it. Needed to add a colon as in the PR https://gerrit.wikimedia.org/r/c/operations/puppet/+/803284/2/hieradata/common/profile/thanos/swift.yaml
[17:48:01] <inflatador>	 OK, so now we can presumably get uploading! (After I eat lunch, that is!)...back in ~45
[18:25:09] <inflatador>	 back
[18:52:31] <inflatador>	 ryankemper ebernhardson this is the tentative plan for Ryan and I's pairing session today at 2 PM PDT . Feel free to add/change/respond here or in the ticket https://phabricator.wikimedia.org/T309648#7983820
[18:54:01] <inflatador>	 I imagine we'll see some weirdness with elasticsearch-keystore and our multiple instances, but I'm guessing we can work around with a symlink
[18:56:24] <inflatador>	 dr appointment, back in ~1h
[19:02:07] <ebernhardson>	 meh, mwdd detects --port=4444 as "Please set environment var --port to 4444
[19:02:29] <ebernhardson>	 (instead of passing along as a command arg)
[19:58:29] <inflatador>	 back
[19:58:51] <inflatador>	 Also that's gross CLI behavior ;P
[20:01:11] <ryankemper>	 yeah, I think I'd prefer it accepting either and going with the actual arg if both are present
[20:07:48] <ebernhardson>	 i suppose i should patch it, it doesn't seem intentional. They take any arg matching '\w+=\w+' (without anchors, afaict)
[21:08:16] <inflatador>	 ryankemper ebernhardson up at meet.google.com/yez-bmya-hge if you wanna join, although strugglling with swift ATM
[21:29:58] <ebernhardson>	 off a bit early, taking liam to ortho