[07:11:33] <elukey>	 inflatador: o/ sorry just seen the msg, I was already afk! Thanks btullis :)
[08:05:05] <jynus>	 I will restart db1117:m1 replication, as I saw no issue on librenms
[08:08:50] <XioNoX>	 jynus: +1
[08:12:40] <jynus>	 thanks for the +1, XioNoX!
[09:16:56] <jelto>	 fyi: We are going to depool services in codfw soon for the upcoming pdu replacement later today
[10:55:04] <dcaro>	 I'm getting a bunch of errors for puppet runs about confd, anyone changed anything there lately? (/me going to the git logs)
[10:55:28] <btullis>	 dcaro: o/ Probably me.
[10:55:40] <btullis>	 Sorry.
[10:55:52] <dcaro>	 no problem at all, can I help in any way?
[10:56:35] <dcaro>	 'E: Unable to locate package confd' that's what I'm getting
[10:56:47] <btullis>	 Where are you seeing it?
[10:56:59] <dcaro>	 clouddb1016.eqiad.wmnet (and several others)
[10:57:08] <dcaro>	 https://alerts.wikimedia.org/?q=team%3Dwmcs&q=alertname%3Dpuppet%20last%20run
[10:57:28] <dcaro>	 not only cloud stuff though (just unfilter the team)
[10:58:04] <dcaro>	 maybe a reprepro issue?
[10:58:22] <jynus>	 either a puppet or a repo issue probably
[10:58:43] <jynus>	 let me scan puppet for recent confd related patches
[10:59:19] <btullis>	 OK, possibly not me. Sorry, I've been working more on etcd, bootstrapping a new cluster.
[10:59:39] <dcaro>	 okok
[11:00:53] <dcaro>	 apt policy shows no package available on clouddb1016, but on coludcephosd1025 it comes from http://apt.wikimedia.org/wikimedia buster-wikimedia/main amd64 Packages
[11:01:41] <jynus>	 is clouddb1016 buster or bullseye?
[11:01:50] <dcaro>	 bullseye
[11:01:56] <dcaro>	 (seems to be the main difference)
[11:02:01] <jynus>	 maybe a data point to pull from
[11:02:23] <jynus>	 pointint to something wrong on the repo, probably
[11:03:56] <dcaro>	 yep, confd is not there https://apt-browser.toolforge.org/bullseye-wikimedia/main/
[11:04:33] <dcaro>	 it's a bit shorted than the buster too, so maybe others are missing
[11:06:08] <jynus>	 I wonder if it was ever there, or it just didn't get upgraded yet?
[11:06:33] <dcaro>	 hmm, so that would mean that puppet did not want to install it before?
[11:07:29] <jynus>	 I see no references to confd + bullseye
[11:07:57] <jynus>	 so my guess is you are the lucky first maintainer to upgade a host to bullseye ?
[11:08:12] <jynus>	 with that role / package depenency
[11:09:12] <jynus>	 that part I cannot say
[11:09:14] <btullis>	 I think I remember someone mentioning that we don't yet have confd on bullseye, so if I wanted it I would have to package it.
[11:09:14] <dcaro>	 there's many, there's 448 hosts failing to run puppet: https://alerts.wikimedia.org/?q=alertname%3Dpuppet%20last%20run
[11:09:40] <jynus>	 maybe it is a puppet regression then?
[11:10:05] <dcaro>	 maybe, checked the logs for 'confd', the last diff with that  word is from july 26th
[11:10:28] <jynus>	 backup1002 should absolutely not have a confd
[11:11:15] <jynus>	 so I am close to shutdown puppet master now
[11:12:03] <RhinosF1>	 jbond is fixing the confd stuff I believe
[11:12:13] <dcaro>	 might be yes
[11:12:18] <RhinosF1>	 Amir1: seemed to be talking to him in -operations
[11:12:35] <dcaro>	 hmpf, I think i got the date wrong from the log
[11:12:38] <dcaro>	 ack, will go there
[11:12:38] <Amir1>	 yeah John is on it
[11:12:51] <RhinosF1>	 -operations gets far too noisy
[11:12:59] <RhinosF1>	 I wonder if confd is on the task to fix that
[11:13:31] <jbond>	 sorry all this was a change that should have just made it to sretest1001 but a missing `$` ment it went out everywhere
[11:13:53] <jynus>	 what's the patch?
[11:13:56] <dcaro>	 ack, np
[11:14:01] <dcaro>	 let us know if we can help
[11:14:11] <RhinosF1>	 I added https://phabricator.wikimedia.org/T314118#8142383 for future as a follow up
[11:14:20] <jbond>	 jynus: let me fix the issue first
[11:14:29] <jbond>	 dcaro: i think im fine but thanks
[11:15:21] <jynus>	 dcaro: sorry, I assumed you *needed* confd for some reason there
[11:16:03] <jynus>	 I didn't realize you were confused why it was needed, not why it didn't install, which is what I was trying to debug
[11:17:56] <dcaro>	 I wasn't sure if it was needed either, but I did not notice a clear addition of the package in the puppet logs (was looking for confd as package, not confd::file def)
[11:18:51] <dcaro>	 so I guessed it was there already (actually saw confd running on some other hosts right before the alerts triggered, and was wondering why it was needed xd)
[11:19:45] <dcaro>	 that reminds me, I should clean those up once the fix is in :)
[11:20:55] <jynus>	 I am going to open an incident doc
[11:21:01] <jynus>	 I think this is serious enough to require it
[11:21:22] <jynus>	 I take IC
[11:22:12] <dcaro>	 👍
[11:24:33] <dcaro>	 oh, and I sholud change the way the commits are displayed on my cli, as the date I got was the author date not commit date, that threw me off a bit too 
[11:25:03] <jynus>	 dcaro: please add that to doc, that will be useful
[11:25:09] <dcaro>	 okok
[11:25:16] <dcaro>	 can you share the lin
[11:25:19] <dcaro>	 *link?
[11:25:33] <dcaro>	 (when you have it xd)
[11:26:34] <jynus>	 see ping
[11:26:44] <dcaro>	 👍
[14:41:14] <RhinosF1>	 kart_, urbanecm: andrewbogo.tt has dropped the old labweb hosts from dsh so the warnings about them in scap seen this morning should have gone.
[14:41:24] <urbanecm>	 amazing, thanks!
[14:42:25] <RhinosF1>	 np!
[14:51:00] <kart_>	 RhinosF1: cool!
[16:37:22] <andrewbogott>	 dhinus: likely related to https://gerrit.wikimedia.org/r/c/operations/puppet/+/820248
[16:37:26] <dhinus>	 I'm running the decom for labweb1001 (with help from andrewbogott). The cookbook is showing a seemingly unrelated diff trying to remove gerrit2001
[16:37:57] <andrewbogott>	 Maybe mutante is running the decom script at the same time?
[16:42:56] <dhinus>	 the cookbook is trying to update dns.git I think, and generated a commit with message "fnegri@cumin1001: labweb1001.wikimedia.org decommissioned, removing all IPs except the asset tag one"
[16:43:58] <dhinus>	 but that commit seems to include a bunch of lines removed that are related to gerrit2001
[16:44:27] <andrewbogott>	 dhinus: that above patch (which is merged) means that I'm confident that it's not destructive for you to proceed. I'm definitely puzzled by how the cookbook is acting though. volans would be interested if he were here :)
[16:45:04] <andrewbogott>	 dhinus: does the diff /also/ include the actual host you're decom'ing?
[16:45:09] <dhinus>	 it does
[16:45:30] <andrewbogott>	 ok, I think you should copy/paste all that into a bug and then 'go'
[16:45:56] <dhinus>	 let's see if my screen-inside-tmux window lets me copy it :D
[16:45:59] <andrewbogott>	 I predict that somewhere in california a terminal is waiting for daniel to type 'go' on the gerrit decom :)
[16:50:45] <mutante>	 jbond: I tried to run the decom cookbook but there were unexpected DNS changes 
[16:51:04] <mutante>	 dhinus: ACK, same conflict for me the other way around
[16:51:07] <mutante>	 made me cancel 
[16:51:15] <mutante>	 was trying to do a live demo of decom
[16:51:47] <mutante>	 I am unsure now if I should repeat my cookbook run.. but trying it
[16:51:56] <dhinus>	 haha, what's the best way to untangle this?
[16:52:38] <mutante>	 I don't really know it either. so did you continue your run?
[16:52:41] <mutante>	 at the DNS step?
[16:52:42] <dhinus>	 not yet
[16:52:53] <andrewbogott>	 mutante: I suspect that if dhinus hits 'go' that it'll just clean up both?  Unless that breaks the next run for you...
[16:53:00] <mutante>	 ok, so ... I _do_ want to remove gerrit2001
[16:53:41] <mutante>	 andrewbogott: I already canceled mine. I guess it depends if there are other steps after the DNS change
[16:54:02] <andrewbogott>	 I think dhinus shoudl continue, then mutante should re-run
[16:54:04] <mutante>	 spicerack.remote.RemoteError: No hosts provided
[16:54:08] <mutante>	 During handling of the above exception, another exception occurred:
[16:54:13] <mutante>	 wmflib.interactive.AbortError: Confirmation manually aborted
[16:54:19] <mutante>	 andrewbogott: agreed
[16:54:28] <dhinus>	 I'll hit "go" then
[16:54:33] <andrewbogott>	 and then mutante's run may or may not work, depending on how pedantic the cookbook is :(
[16:54:36] <dhinus>	 let's see what happens
[16:54:37] <mutante>	 dhinus: if the diff shows only labweb1001 and gerrit2001.. then yes, do it
[16:54:42] <dhinus>	 yes, confirmed
[16:55:51] <dhinus>	 my cookbook is proceeding with the following steps
[16:56:00] <dhinus>	 where does that dns.git repo live by the way?
[16:56:18] <dhinus>	 ha, ERROR: some step failed, check the task updates.
[16:56:24] <andrewbogott>	 https://gerrit.wikimedia.org/r/admin/repos/operations/dns
[16:56:26] <andrewbogott>	 dang :(
[16:56:28] <jynus>	 dns is operations/dns
[16:57:04] <andrewbogott>	 dhinus: which step failed?
[16:57:13] <dhinus>	 I'm trying to find it...
[16:57:52] <dhinus>	 I don't see any obvious error lines, but it's quite a long output, I'm going through it
[16:58:07] <andrewbogott>	 I
[16:58:12] <andrewbogott>	 I'd expect it to be at the end
[16:58:16] <RhinosF1>	 Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
[16:58:25] <RhinosF1>	 See https://phabricator.wikimedia.org/T313861#8143291
[16:58:41] <taavi>	 the netbox-generated parts live in https://phabricator.wikimedia.org/source/netbox-exported-dns/browse/master/
[16:58:42] <dhinus>	 right, just found the same comment
[16:59:10] <jbond>	 mutante: still needed?
[16:59:19] <mutante>	 jbond: yes, I think so
[16:59:20] <taavi>	 that failure sounds like it happened because the labweb* hosts have been with puppet disabled for too long
[16:59:21] <RhinosF1>	 I think you can just leave a note for DC-Ops to wipe the disk when they do on-site steps
[16:59:36] <andrewbogott>	 dhinus: that's weird but looks like victory to me.
[16:59:42] <dhinus>	 thanks taavi, there's the commit with the "wrong" diff https://phabricator.wikimedia.org/rONED15cd621f2e045203888b4a0d93a7824a270ddf6f
[16:59:46] <andrewbogott>	 but I'll start puppet on the other host so you don't hit this again
[17:00:19] <andrewbogott>	 mutante: your turn :)
[17:01:07] <Amir1>	 RhinosF1: I'd be hesitant to make such recommendations 
[17:01:29] <RhinosF1>	 Amir1: I'm pretty sure I remember it being done very recently 
[17:01:54] <mutante>	 andrewbogott: trying to rerun cookbook, ACK!
[17:02:41] <Amir1>	 RhinosF1: each case is different and there is the point of dcops being understaffed and having too many things to do while core SREs can take care of it
[17:04:24] <mutante>	 ==> ATTENTION: the query does not match any host in PuppetDB or failed
[17:04:37] <mutante>	 proceeds anyways
[17:05:06] <mutante>	 Found physical host
[17:05:17] <mutante>	 Host not found on Icinga, unable to downtime it//
[17:06:09] <jbond>	 mutante: if thehost has had puppet disabled for more then 2 weeks it wont be in puppetdb or icinaga
[17:06:13] <mutante>	 it's asking me for mgmt pass now..
[17:06:17] <mutante>	 powered off
[17:07:01] <mutante>	 Host gerrit2001.wikimedia.org already missing on Debmonitor
[17:07:01] <jbond>	 rephrase: if puppet hasn't ran for more then two weeks it wont be in puppetdb/icinga
[17:07:02] <mutante>	 Removed from DebMonitor
[17:07:25] <mutante>	 it was removed from puppet in the previous run that I aborted because of the DNS conflict
[17:07:32] <mutante>	 Removed from Puppet master and PuppetDB
[17:07:33] <mutante>	 Sleeping for 3 minutes to get netbox caches in sync
[17:08:48] <jbond>	 ahh well yes tat will do it as well :)
[17:09:43] <mutante>	 Generating the DNS records from Netbox data. It will take a couple of minutes.
[17:10:14] <RhinosF1>	 Should the cookbook have a lock so 2 people don't conflict
[17:10:19] <mutante>	 yes
[17:10:34] <mutante>	 and it should ignore the file site.pp when it checks for the hostname in repos
[17:10:44] <mutante>	 it should alert on all others, but not that one
[17:10:54] <andrewbogott>	 Amir1: now that that server is powered down and purged from e.g. puppetdb, do you know how we can wipe the drives?  Is there a mgmt> way to do it?
[17:11:01] <mutante>	 since you need to have it until after the cookbook run
[17:11:13] <mutante>	 andrewbogott: the decom cookbook wiped the drives.. in my case
[17:11:38] <andrewbogott>	 mutante: right, but it didn't for labweb1001, hence my question.
[17:11:46] <andrewbogott>	 It did everything else but left a warning about failing that part
[17:11:55] <Amir1>	 the cookbook should take care of it
[17:11:55] <jbond>	 re locking, i know its on vola.ns list not sure if there is a task
[17:12:11] <mutante>	 I see, Andrew. gotcha
[17:12:22] <RhinosF1>	 Amir1: it failed for andrewbogott and dhinus
[17:12:29] <jbond>	 mutante: can you raise a task for the site.pp bit
[17:12:40] <Amir1>	 RhinosF1: I know but we are running it again
[17:12:44] <andrewbogott>	 Amir1: but it didn't, which is why RhinosF1 was suggesting that we ask dc-ops to do it, which is where you entered the conversation
[17:12:49] <mutante>	 jbond: ok
[17:12:58] <jbond>	 thx
[17:12:59] <mutante>	 so.. cookbook run finished as failed
[17:13:04] <andrewbogott>	 I don't think it's idempotent, once it's finished it will say the server doesn't exist on a second run
[17:13:07] <mutante>	 DNS was ok: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
[17:13:12] <mutante>	 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts gerrit2001.wikimedia.org
[17:13:13] <jbond>	 andrewbogott: yu can check what the cookbook dose, it likley uses ipmi or redfish
[17:13:43] <RhinosF1>	 Amir1: mutante is running theirs again for another host
[17:13:54] <Amir1>	 oh I confused these two
[17:13:55] <andrewbogott>	 jbond: ok  -- I was assuming that Amir1 had done it by hand since that's what he was advocating :)
[17:14:12] <Amir1>	 okay, let's try seeing what the cookbook does
[17:15:06] <Amir1>	 https://github.com/wikimedia/operations-cookbooks/blob/82367ef847bee8ab67c988229865926c994e0f2d/cookbooks/sre/hosts/decommission.py#L311?
[17:15:41] <mutante>	 yes, /sbin/wipefs --all --force   
[17:15:43] <mutante>	 that looks like it
[17:16:03] <Amir1>	 I wonder if we can power it up and then simply run this
[17:16:18] <andrewbogott>	 won't powering it up cause it to reappear in a bunch of places? netbox, icinga, ?
[17:16:22] <mutante>	 I think the cookbook overwrites the boot sector
[17:16:29] <mutante>	 to prevent that 
[17:17:42] <Amir1>	 if it's powered off, it means it passed that point and wiped it off, right?
[17:17:48] <mutante>	 'Wipe bootloaders to prevent it from booting again'
[17:17:51] <mutante>	 if that step happened
[17:17:57] <Amir1>	 https://github.com/wikimedia/operations-cookbooks/blob/82367ef847bee8ab67c988229865926c994e0f2d/cookbooks/sre/hosts/decommission.py#L330
[17:18:14] <jbond>	 mutante: im a bit confused on the issue with gerrite2001.  from https://phabricator.wikimedia.org/T243027#8143239 it looks like the deomc worked fine, which explains why the next attempt failed
[17:19:16] <mutante>	 jbond: I assume running it twice means this is normal behaviour
[17:19:22] <mutante>	 first time it does some things
[17:19:28] <mutante>	 then we get to the DNS part..it asks me to check
[17:19:30] <mutante>	 I say NO
[17:19:34] <RhinosF1>	 jbond: it got aborted at the remove from dns stage because dhinus and mutante stepped on each other's toes
[17:19:42] <mutante>	 (because of the conflict with another user running it at the same time)
[17:19:51] <mutante>	 I run it again.. it does some remaining steps
[17:20:02] <mutante>	 but only after other stuff "failed" because it was already done
[17:20:37] <mutante>	 I still think it was right to abort though because that is the point of checking the DNS diff
[17:22:01] <jbond>	 mutante: i think you where right to abort as well
[17:22:23] <jbond>	 and thanks for the explanation
[17:22:25] <dhinus>	 maybe we could've aborted _both_ runs, but I'm not sure that would have made things easier?
[17:22:33] <andrewbogott>	 If chaos ensues any time two people decom at the same time we need a lockfile
[17:22:50] <jbond>	 i think it would be nice if https://phabricator.wikimedia.org/T243027#8143239 somehow showed you aborted and added the steps that where skipped
[17:22:55] <mutante>	 jbond: so.. the host is gone from DNS, I can't ssh to it anymore.. I am removing it from site.pp now.. in netbox it's in status "decommissioning" .and mgmt still exists
[17:22:59] <mutante>	 does sound right?
[17:23:24] <jbond>	 mutante: yes that all sounds right to me
[17:23:35] <mutante>	 ok, then I will consider it resolved now. thank you
[17:23:46] <jbond>	 also looking at the output from both tasks i think everything ended up getting completed withthe two runs compined
[17:23:54] <jbond>	 sgtm, no problem
[17:24:35] <mutante>	 andrewbogott: dhinus: yea, so from my side it's over. how about labweb1001? similar to that above ^?
[17:25:01] <andrewbogott>	 I think the script worked fine for labweb1001 except for wiping the disk
[17:25:23] <andrewbogott>	 I mean, /not/ wiping the disk
[17:25:32] <mutante>	 ACK, I think this means we have to ask dcops to wipe it
[17:26:07] <mutante>	 you could try booting it via mgmt but if we can see in logs it wiped the boot sector ..there is little point I suppose
[17:26:22] <dhinus>	 can I proceed with labweb1002 in the meantime?
[17:26:42] <andrewbogott>	 yes
[17:26:46] <mutante>	 I am not going to run another cookbook. go ahead
[17:42:52] <dhinus>	 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts labweb1002.wikimedia.org
[17:43:34] <mutante>	 cool:)
[17:49:43] <dhinus>	 andrewbogott: I'm going to merge the cleanup patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/817386/1
[17:50:02] <andrewbogott>	 ok!
[17:56:13] <dhinus>	 done. can you take care of reassigning to dc-ops and mentioning they will need to wipe the disk?
[17:56:40] <andrewbogott>	 yep!
[18:00:21] <dhinus>	 thanks, and thanks everybody for the help :)
[20:46:11] <denisse|m>	 Hello team, do you know if there's a way to run puppet in an instance and have it 'reset' to it's original state? For ex. If folder permissions were changed I'd like puppet to revert everything as described in the manifest.
[20:46:28] <denisse|m>	 '# run-puppet-agent' does not seem to implement that behavior...
[20:51:22] <rzl>	 run-puppet-agent ought to do that, but the logic around what's actually enforced by puppet, and what isn't, can be... surprising :)
[20:51:30] <rzl>	 can you point me at what directory you're looking at?
[20:52:31] <denisse|m>	 rzl: Yes, it's the '/var/lib/rancid/' directory in the 'netmon1003' instance.
[20:52:52] <rzl>	 looking -- where's it set up in the puppet repo?
[20:54:20] <denisse|m>	 It's in 'modules/rancid/manifests/init.pp'. Thanks in advance. :D
[20:56:51] <rzl>	 this is one of those things where there's a trick to it, but I forget what it is exactly -- one sec while I try to find it again
[20:58:10] <rzl>	 (often the problem is that you didn't say "recurse => true" when you meant to, but that doesn't look like what you want here)
[21:01:17] <rzl>	 hm, I see that you have managehome => true on the user and then you also have the directory as a separate file resource, I wonder if that's getting tangled
[21:02:49] <mutante>	 I see files within /var/lib/rancid are defined in puppet but not the /var/lib/rancid itself?
[21:03:02] <mutante>	 /var/lib/rancid/core is 
[21:04:11] <rzl>	 doh thanks you're right, I was seeing /var/log/rancid
[21:04:28] <rzl>	 okay then yes, mutante's much simpler explanation is correct, thanks :)
[21:04:58] <mutante>	 denisse|m: looks like you would have to add that directory.. but since it isn't now.. maybe the easiest fix is to compare netmon1002 and netmon2001 and manually fix it
[21:05:28] <mutante>	 it does look the same though..
[21:05:37] <RhinosF1>	 mutante: it's listed as the rancid user's homedir
[21:05:47] <RhinosF1>	 So maybe that was auto creating and never managing after
[21:06:23] <mutante>	 found: "This parameter has no effect unless Puppet is also creating or removing the user in the resource at the same time. "
[21:06:29] <mutante>	 https://puppet.com/docs/puppet/7/types/user.html#user-attribute-managehome
[21:06:35] <mutante>	 hmmm
[21:08:04] <mutante>	 the way I read that is.. if the user already existed when somebody added the "managehome" line then it might not do anything to add it
[21:08:38] <mutante>	 and it only says it will "create" and "delete" but does not say if it will enforce any permissions
[21:08:40] <denisse|m>	 Another question team, I see the following alert:... (full message at https://libera.ems.host/_matrix/media/r0/download/libera.chat/881fa59e06d41462c932e04abdb8434ea9c45299)
[21:09:00] <rzl>	 yeah I think that's correct, I saw somewhere that it just means it runs useradd -m 
[21:09:25] <rzl>	 so if you actually care about the directory permissions or other details, you'd want to manage it with a file resource
[21:09:41] <denisse|m>	 mutante: re /var/lib/rancid: That makes sense. So it doesn't change the permissions because they're not explicitly defined in the puppet manifest, right?
[21:10:04] <mutante>	 denisse|m: yea, because there is no  file{} section for the dir itself, only for things inside it
[21:10:48] <denisse|m>	 Thanks for the help team! :D
[21:10:53] <mutante>	 re: keyholder.. "sudo keyholder status" on netmon1003 says "no idenities"
[21:10:59] <mutante>	 so something there failed
[21:11:08] <mutante>	 event though all you say looks right to me in that paste
[21:12:44] <mutante>	 denisse|m: where did you get the passphrase from?
[21:13:29] <denisse|m>	 From the pw repository: ruby pws.rb ed ~/Wikimedia/pw/network-monitoring-keys-passphrase
[21:13:53] <mutante>	 thanks, I was looking for rancid* or keyholder* in there
[21:13:55] <denisse|m>	 'sudo keyholder status' shows it as active now.
[21:13:56] <mutante>	 let me try to arm it
[21:14:08] <mutante>	 wow, indeed
[21:14:16] <mutante>	 something loads and unloads it?!
[21:14:26] <denisse|m>	 The thing is that the changes do not seem to be persistent.
[21:14:33] <mutante>	 I could still make a screenshot of it both not being loaded and being loaded
[21:15:02] <denisse|m>	 Oh, I just ran the '# keyholder arm' process, that's why it shows it's active now.
[21:15:16] <denisse|m>	 But I ran the same process several times hehe.
[21:15:19] <denisse|m>	 I'm not sure how to make the changes persistent.
[21:15:20] <mutante>	 ok, but you already did this a couple times before, right?
[21:15:33] <mutante>	 when does it disappear again?
[21:15:34] <denisse|m>	 Yes, I did it about 3 times.
[21:15:38] <mutante>	 after the next puppet run?
[21:15:57] <mutante>	 this definitely counts as "weird" already
[21:16:13] <mutante>	 I haven't see this with deploy1002 afair
[21:16:17] <denisse|m>	 That's an excellent question, I'm not sure yet. I'll be paying attetion as to what triggers it as it's unexpected behavior.
[21:16:47] <denisse|m>	 I also think it may be related to this issue which topranks is helping me to debug. :D https://phabricator.wikimedia.org/T314936
[21:17:12] <topranks>	 I think it looks like that issue is solved - just dm'd you
[21:18:04] <mutante>	 so you originally asked this because of the "Could not create directory '/var/lib/rancid/.ssh' (Permission denied)."
[21:18:08] <topranks>	 When systemd runs the command it exports an env variable for SSH_AUTH_SOCK, after which passwordless ssh works
[21:18:23] <topranks>	 So I think you may have fixed it changing the permissions for /var/lib/rancid dir?
[21:18:53] <mutante>	 when I tried that command "sudo -u rancid jlogin -c "show version" cr1-eqiad.wikimedia.org" I get a Password: prompt
[21:18:58] <mutante>	 keyholder is armed
[21:19:08] <topranks>	 mutante: yeah, same for us
[21:20:03] <denisse|m>	 Could it be that the password belongs to the 'rancid' user and not to the servers?
[21:22:06] <topranks>	 If you run a shell as the rancid user, export that env var, then the jlogin command works
[21:22:11] <topranks>	 https://www.irccloud.com/pastebin/vkJ6HEKF/
[21:22:27] <topranks>	 The systemd unit that runs rancid sets this before it executes.  So all is working
[21:22:48] <mutante>	 sudo -u rancid SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -oIdentitiesOnly=yes -oIdentityFile=/etc/keyholder.d/rancid  cr1-eqiad.wikimedia.org
[21:22:51] <mutante>	 this works too
[21:23:15] <topranks>	 I believe the fix is that /var/lib/rancid is now owned by "rancid" user not root, so the ssh is not bombing out trying to write to that dir
[21:23:49] <topranks>	 mutante: cool, I was trying to do that myself but the syntax proved elusive :)
[21:24:19] <mutante>	 on netmon1002 it is root:root    but on meton1003 it is rancid:rancid
[21:24:21] <mutante>	 for the rancid home
[21:24:44] <denisse|m>	 topranks: Cool! I changed the directory's owner to test if it worked, glad it did. I'll add it to the puppet repository.
[21:25:01] <mutante>	 topranks: lucky to find it at https://wikitech.wikimedia.org/wiki/Keyholder#Hints
[21:25:08] <topranks>	 yeah I think that is the fix
[21:25:13] <mutante>	 agreed
[21:25:25] <topranks>	 I assume we are just installing from upstream debian package ?
[21:25:37] <topranks>	 Which is creating the systemd service and timer?
[21:25:46] <topranks>	 Or are we adding those ourselves?
[21:25:50] <denisse|m>	 mutante: That's correct however, topranks made a very good observation in that in the 'netmon1002' instance 'rancid' runs as a cron job which is ran as the 'root' user and in the 'netmon1003' instance 'rancid' runs as a systemd service that runs as the 'rancid' user.
[21:25:59] <topranks>	 Seems the difference from netmon1002 is that it was a cronjob, running as root, before.
[21:26:08] <topranks>	 It's not a systemd unit, but set to run as user 'rancid'
[21:26:22] <topranks>	 haha... yep that :)
[21:26:23] <mutante>	 we are creating systemd timer jobs "rancid-differ"
[21:26:30] <topranks>	 ok cool
[21:26:48] <topranks>	 well no need to upstream any fix, we just need to set the owner of the dir correctly to match the systemd unit
[21:26:52] <mutante>	 it all makes sense now
[21:26:57] <mutante>	 because crons were migrated to timers
[21:27:47] <topranks>	 cool cool.  yep all makes sense in the end :)
[21:27:51] <mutante>	 yes, it has "user => 'rancid' for the timer and before it was root then
[21:27:54] <mutante>	 denisse|m: ACK :)
[21:28:14] <denisse|m>	 I'll send a small puppet patch that fixes the directories owner for when it's a Debian Bullseye instance so we won't have this issue in the future. ^^
[21:28:14] <denisse|m>	 Thanks a lot for your help and for sharing your insights!! <3
[21:28:41] <mutante>	 blame https://phabricator.wikimedia.org/T273673 :)
[21:29:14] <RhinosF1>	 Timers are much nicer because monitoring than crons
[21:29:17] <mutante>	 denisse|m: it's actually my fault
[21:29:18] <mutante>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/721854/3/modules/rancid/manifests/init.pp
[21:29:23] <RhinosF1>	 And it's easier to see last run / next run
[21:29:42] <mutante>	 wait.. it said "rancid" before too...but yea
[21:30:20] <topranks>	 RhonosF1: Agreed, I was skeptical at first but I prefer systemd timers.
[21:30:32] <mutante>	 there must have been another change before that that change it from root to rancid I guess
[21:30:49] <RhinosF1>	 topranks: the one thing I never liked about crons was how invisible they were
[21:31:03] <RhinosF1>	 Until someone moaned a while later that something wasn't working
[21:31:16] <topranks>	 exactly 
[21:31:26] <topranks>	 mutante: yeah that is odd
[21:31:39] <mutante>	 at least not visible to the person creating them.. then very visible for root@
[21:32:04] <RhinosF1>	 mutante: well ye if you allow root@ spam then they are
[21:32:09] <mutante>	 topranks: it is. But I think we can agree "root:root" is not intended
[21:32:29] <topranks>	 mutante: checking again the cronjob was actually configured to run as user rancid
[21:32:35] <RhinosF1>	 Native systemd is much cleaner imo
[21:32:51] <topranks>	 But somehow on netmon1002 that didn't cause a problem with /var/lib/rancid owned by root:root
[21:32:55] <mutante>	 but we can probably still blame the cron->timer change
[21:33:14] <mutante>	 not 100% satisfied though :)
[21:33:41] <topranks>	 Yeah.  On netmon1002 there is no directory "/var/lib/rancid/.ssh"
[21:33:58] <topranks>	 It was netmon1003 trying to create that that was failing and causing the problem
[21:34:00] <mutante>	 because only the "active_server" gets it?
[21:34:20] <topranks>	 I checked /etc/ssh/ssh_config to see if host key saving/checking was disabled or anything but they are both the same.
[21:34:34] <topranks>	 I do expect it's some quirk in how SSH is being invoked due to change from cron to timer
[21:39:18] <topranks>	 Ok, think I see the difference in behaviour anyway.
[21:39:24] <topranks>	 https://phabricator.wikimedia.org/T314936#8144068
[21:40:04] <topranks>	 Seems it was previously failing to create the directory, but jlogin / ssh proceeded afterwards, and on newer box it bombed out when it couldn't create the ".ssh" dir
[21:41:26] <topranks>	 I'll have a chat with Arzhel tomorrow on this.  
[21:41:31] <mutante>	 ok, thanks topranks. then it should be the ssh version changes in bullseye
[21:41:48] <topranks>	 Perhaps we don't want to save the host keys, as it might fail if we replace a network device.
[21:42:09] <topranks>	 But obviously it's probably a good thing to store then, in case something nefarious occurs
[21:42:46] <topranks>	 mutante: yes quite likely
[22:01:21] <mutante>	 ssh 7.9:
[22:01:38] <mutante>	   if (mkdir(buf, 0700) < 0)
[22:01:38] <mutante>	                                 error("Could not create directory '%.200s'.",
[22:01:41] <mutante>	                                     buf);
[22:01:48] <mutante>	 ssh 8.4:
[22:02:13] <mutante>	                 if (mkdir(dotsshdir, 0700) == -1)
[22:02:13] <mutante>	                         error("Could not create directory '%.200s' (%s).",
[22:02:16] <mutante>	                             dotsshdir, strerror(errno));
[22:02:54] <mutante>	 (git clone https://salsa.debian.org/ssh-team/openssh ; git checkout bullseye    vs   git checkout buster)
[22:03:33] <mutante>	 previously this was in ssh.c  and now it's in hostfile.c
[22:05:04] <topranks>	 hmm interesting.  I note we only got that precise message on the bullseye host
[22:05:39] <topranks>	 Error logged, but didn't stop process, on buster was "Failed to add the host to the list of known hosts"
[22:05:40] <mutante>	 < 0    vs   == 1
[22:06:09] <topranks>	 == -1
[22:06:35] <mutante>	 right.. 
[22:06:57] <topranks>	 but yeah, if it's equal to -1 now should match <0 before, but I'm way out of my depth here :)
[22:07:13] <mutante>	 the pre-bullseye version does not have the "dotsshdir, strerror(errno));" part
[22:07:26] <mutante>	 I am satisfied enough now though :p
[22:07:39] <mutante>	 the whole code about that is in a different place now
[22:07:57] <topranks>	 haha yeah we're quite far down the rabbit hole.
[22:08:13] <topranks>	 I think we can be confident that a change in there has caused the different behaviour
[22:08:14] <mutante>	 confident enough to blame ssh and move on now :)
[22:08:36] <topranks>	 And likely we always should have had the ownership of that dir set correctly