[08:25:46] <Emperor>	 Should I expect warnings from puppet lookup, or am I Doing It Rong? https://phabricator.wikimedia.org/P21769
[08:31:05] <_joe_>	 Emperor: no you're doing it right, but puppet lookup doesn't source manifests/realm.pp
[08:31:13] <_joe_>	 before evaluating
[08:31:28] <_joe_>	 so it won't be able to look up our hiera hierarchy properly
[08:32:14] <_joe_>	 there is a "hiera_lookup" command under utils/ no idea if it still works
[08:32:54] <_joe_>	 should work on your computer IIRC
[08:45:14] <Emperor>	 does Debian really not package the "puppet" gem?
[08:45:46] <Emperor>	 (I have puppet-lint, puppet-strings, puppet-syntax, puppetlabs_spec_helper)
[08:51:33] <_joe_>	 Emperor: i always use bundler anyways, but puppet is complex to package
[08:51:52] <_joe_>	 esp with puppet 6 and onwards
[09:07:51] <Emperor>	 just seems a bit weird to have a bunch of puppet-* gems, but not puppet
[09:08:08] <Emperor>	 (so I assumed I was missing something obvious)
[09:12:41] <Emperor>	 some of the problem seems to be that bundle isn't finding some of the gems I have installed
[09:15:21] <Emperor>	 le sigh, after shaving those yaks:
[09:15:33] <Emperor>	 "puppet now has this functionality built in.  try the following on the puppet master"
[09:17:10] <Emperor>	 https://phabricator.wikimedia.org/P21776 <-- eliminates one but not both of the warnings. Is this still not going to look up our hiera properly _joe_?
[09:17:31] <Emperor>	 [maybe I should stop trying to get a lookup CLI to work]
[09:17:56] <_joe_>	 Emperor: yes I think it is
[09:18:24] <_joe_>	 you can try with something that's specific to a role
[09:20:35] <_joe_>	 yeah I can confirm it works
[09:20:46] <_joe_>	 you will see warnings that are part of compiling the catalog
[09:25:54] <Emperor>	 The underlying question I was trying to get to is - we have 3 swift clusters, each of which has a cluster name (looked up in the swift roles as profile::swift::cluster or profiles::thanos::swift::cluster) and a stats reporter host (looked up as profile::swift::stats_reporter_host). I want from a puppetmaster host to be able to do something like "for each of [clusters] copy a file from [stats_reporter_host] to
[09:25:54] <Emperor>	 volatile/swift_cluster/path"
[09:26:31] <Emperor>	 ideally in a DRY way (i.e. without repeating the cluster name / stats_reporter_host mapping somewhere else in hiera)
[09:28:49] <Emperor>	 I'm not actually sure this is possible, but...
[09:31:21] <_joe_>	 sorry I'm not sure I understood correctly
[09:31:24] <btullis>	 Emperor: I don't know if it helps, but maybe a combination of cumin and transfer.py would work for you. https://doc.wikimedia.org/transferpy/master/usage.html
[09:31:24] <_joe_>	 lemme reread
[09:31:57] <btullis>	 ...but then you'd have to run it from a cumin host, not a puppetmaster.
[09:32:37] <_joe_>	 Emperor: so this file that is on the stats reporter host
[09:32:42] <_joe_>	 is that generated from puppet?
[09:32:47] <Emperor>	 btullis: I want to run from a puppetmastet [let's beg this question for now]
[09:32:52] <_joe_>	 what is it that you're actually trying to do?
[09:32:59] <_joe_>	 one step back
[09:33:35] <_joe_>	 puppet doesn't have any native facility to copy files between hosts
[09:33:50] <_joe_>	 it does to copy files from the puppetmaster to any other host
[09:34:05] <_joe_>	 hence my question
[09:34:14] <Emperor>	 I lie a bit. I'm working on automating swift ring management. So I have code that runs on 1 swift front-end (which is going to be ring_manager_host ; similarly to stats_reporter_host) per cluster that makes a tarball of ring files. I have adjusted swift::ring to copy that tarball into place rather than individual ring files.
[09:34:35] <Emperor>	 So I have all of those pieces in branches.
[09:34:46] <_joe_>	 ok so if you're automating swift ring management the puppetmaster seems like the wrong place to start tbh
[09:34:54] <_joe_>	 if what you want is automatic rebalancing
[09:35:25] <Emperor>	 I spoke to vo*ans a while back about the best way to get the new rings back from the ring manager node to puppetmaster's volatile, and we concluded that rsync (run from the puppetmaster) was the way to go
[09:35:55] <_joe_>	 ok yes that's an option if you still want to delegate to puppet managing the ring files
[09:36:53] <Emperor>	 we thought that was the best answer; both because puppet currently manages ring deployment (they're currently adjusted and uploaded to puppet by hand
[09:36:55] <_joe_>	 but if you're just trying to copy the files from the stats reporter to the puppetmaster volatile, then it's easy to save the files with names like
[09:37:18] <_joe_>	 ring-$swift-cluster-name
[09:37:27] <_joe_>	 and back to your original question then
[09:37:51] <_joe_>	 you need to keep in hiera (probably in common.yaml) a data structure with the swift clusters metadata
[09:38:06] <_joe_>	 the alternative is a puppetdb query
[09:38:13] <_joe_>	 which I frown upon a lot
[09:38:30] <_joe_>	 and, having a metadata structure will allow you to accomodate other needs in the future
[09:38:52] <_joe_>	 see what we've done with kubernetes_cluster_groups
[09:38:58] <_joe_>	 or with service::catalog even
[09:39:11] <Emperor>	 So I'll need something in hieradata/common.yaml that contains the metadata I need and will just duplicate what's elsewhere in hiera? I can live with that :)
[09:39:24] <_joe_>	 you don't even need to duplicate a lot of it
[09:39:39] <_joe_>	 you can have the swift classes get their metadata from accessing the metadata data structure
[09:39:48] <_joe_>	 and extracting the section relevant to that cluster
[09:41:02] <_joe_>	 Emperor: sorry for the confusion earlier, I thought you wanted to fully automate ring rebalancing in software
[09:41:16] <_joe_>	 and I was suggesting puppet is horrible for orchestration
[09:41:16] <Emperor>	 I'm not immediately sure I see how to make that work (not least because codfw / eqiad swift and thanos swift are structured a bit differently), but I'll have a look. I guess if I write the repetitive version, you can tell me how I should have done it :)
[09:41:42] <_joe_>	 happy to help if I can
[09:41:52] <Emperor>	 NP, I was trying to avoid explaining the entire yak-heap, but probably made things more confusing in the process
[09:43:08] <Emperor>	 Thanks for all the pointers :)
[09:43:24] <Emperor>	 <-- still a bit "well, this is how I'd do it in Ansible, how does puppet..."
[09:46:26] <_joe_>	 Emperor: no it's just that I'm always worried about the XY problem
[09:47:30] <_joe_>	 so I usually ask people to explain the original issue they're trying to solve, then dig in the specific issue they're having
[09:47:38] <Emperor>	 Mmm
[09:47:46] <_joe_>	 so expect me to always ask such questions :)
[09:48:16] <_joe_>	 it's very easy for *everyone* to end up down the wrong rabbithole when trying to create an engineering solution
[09:48:20] <_joe_>	 I do that all the time :)
[09:50:27] <_joe_>	 also yes, puppet is kinda-weird
[09:51:49] <volans>	 to avoid duplication you can also check puppet's aliasiang function for hiera stuff, like %{lookup('foo')}
[09:52:52] <Emperor>	 (I notice that swift cluster name is defined twice already)
[09:53:17] <Emperor>	 e.g. in both hieradata/role/codfw/swift/proxy.yaml and hieradata/role/codfw/swift/storage.yaml
[09:53:51] <Emperor>	 Anyhow, I have a meeting shortly, and will try and put something together thereafter :)
[10:50:10] <kormat>	 _joe_: that incident replay session was excellent 👌
[10:53:36] <jayme>	 did you record by chance?
[10:55:31] <_joe_>	 jayme: no
[10:55:38] <_joe_>	 but I have an americas remix tonight
[10:55:44] <jayme>	 hjrhr
[10:55:53] <_joe_>	 kormat: I forgot to link y'all the etherpad summary
[10:55:55] <_joe_>	 https://etherpad.wikimedia.org/p/Incident-Replay-Sessions
[10:56:16] <jayme>	 _joe_: what time is the US RMX?
[10:56:18] <_joe_>	 I explicitly didn't want you all to end up reading that and skipping ahead
[10:56:48] <_joe_>	 jayme: 6:30-7:15 pm
[10:57:18] <jayme>	 damn...would you do a recording? :D
[10:57:45] <jayme>	 (assuming thats CEST)
[11:00:05] <_joe_>	 sure
[11:00:07] <_joe_>	 and yes
[11:00:08] <jbond>	 _joe_: can you add me to the invite for that
[11:00:21] <_joe_>	 sure
[11:00:24] <jbond>	 thx
[11:02:18] <_joe_>	 jayme: still sent you an invite anyways
[11:02:32] <jayme>	 thanks
[11:04:16] <_joe_>	 but i will record it, although i'm trying to be a bit more interactive now
[11:11:41] <kormat>	 _joe_: btw, db1126 is a single-instance host; it's only been in s8 for it's entire lifetime of service
[11:27:58] <_joe_>	 kormat: uh I did remember incorrectly then
[11:28:38] <_joe_>	 possible, I remember looking at the numbers and they made sense with the state of that slow log page
[11:28:44] <_joe_>	 err that db
[13:46:15] <dcaro>	 any reviewers for https://gerrit.wikimedia.org/r/c/operations/puppet/+/767467 ? it's small I swear ;), just adding an extra param to systemd::timerjob to allow passing the runbook url
[13:46:25] <dcaro>	 jbond: maybe? ^
[13:52:33] <dcaro>	 thanks!
[13:52:58] <cdanis>	 dcaro: +1 from me :) do note that a +1 from a reviewer with just a nit that you since fixed is almost always enough to just self-+2 imo
[13:53:36] <dcaro>	 thanks! I try to do that yep :), I'll keep it in mind
[13:53:51] <dcaro>	 as long as only nits are fixed (and the nits don't mean refactoring/changing a lot)
[13:53:56] <cdanis>	 indeed yeah
[14:22:36] <jbond>	 ^^^ +1 agree with this