[00:02:54] <jhathaway>	 there is also a debian package with some scripts to update your local list
[08:46:35] <jbond>	 brett as jhathaway mentioned there is https://wikitech.wikimedia.org/wiki/Wmf-sre-laptop which among other things has tooling for syncing all the fingerprints.  they are synced from https://config-master.wikimedia.org/ if yuo need to grab them manually
[08:47:28] <jbond>	 https://wikitech.wikimedia.org/wiki/Managing_multiple_SSH_agents and https://wikitech.wikimedia.org/wiki/SRE/Production_access#SSH_configuration also usefull
[10:00:29] <vgutierrez>	 wmf-update-prod-known-hosts is your friend to fetch SSH keys :)
[12:41:53] <marostegui>	 wikibugs is down?
[12:50:13] <vgutierrez>	 it's definitely offline
[13:00:14] <marostegui>	 mmm the restarting wikibugs section is gone from wikitech or I cannot find it
[13:01:03] <marostegui>	 https://www.mediawiki.org/w/index.php?title=Wikibugs&type=revision&diff=5171267&oldid=5053974 I am not sure what's the way to do it anymore, legoktm can you check when around?
[13:04:46] <slyngs>	 If it's running in Kubernetes, deleting the pod will restart it
[13:10:19] <marostegui>	 issued toolforge-jobs load libera/k8s-jobs.yaml and it is now back
[14:13:52] <hashar>	 thx for the doc update :)
[15:40:25] <brett>	 I had a lapse in good judgement and made my shell account named "brett" instead of following every other account name I had: bcornwall. Would it reduce confusion for everyone if I requested that username to be changed?
[15:40:32] <brett>	 Or am I overthinking it? :)
[15:41:31] <_joe_>	 brett: IMHO you are :P
[15:41:35] <jynus>	 brett: my shell account is jynus, not jcrespo
[15:41:49] <kormat>	 brett: Definitely overthinking it.
[15:43:43] <brett>	 ahahaha, love the quick response. Thanks :D
[15:59:01] <vgutierrez>	 brett: just ask e.ffie about her summer and winter nicknames when you get the chance :)
[16:05:07] <jbond>	 lol
[17:01:40] <Krinkle>	 _joe_: if you feel like chopping some dead wood, these's a few patches ready at https://phabricator.wikimedia.org/T302465 (about removing /static/current)
[17:43:48] <razzi>	 Just about to do some reboots for kernel updates for superset.wikimedia.org and turnilo.wikimedia.org, sites will be unavailable for a few minutes each
[19:02:26] <jhathaway>	 anyone care to walk me through depooling thanos-fe1001?
[19:03:03] <jhathaway>	 or point me to some docs
[19:03:24] <mutante>	 jhathaway: sure, ssh to a cumin host, f.e. cumin2002.codfw.wmnet and then
[19:03:25] <mutante>	 [cumin2002:~] $ sudo -i confctl select dc=eqiad,name=thanos-fe1001.eqiad.wmnet get
[19:03:31] <mutante>	 to get the current pool status
[19:03:41] <mutante>	 [cumin2002:~] $ sudo -i confctl select dc=eqiad,name=thanos-fe1001.eqiad.wmnet set/pooled=no
[19:03:46] <mutante>	 to depool it
[19:03:59] <mutante>	 there are 3 possible values:  yes, no and inactive
[19:04:13] <jhathaway>	 can I query the current state?
[19:04:15] <mutante>	 no = no traffic but still in config.  inactive = actually removed from config
[19:04:48] <cdanis>	 jhathaway: yeah, 'get' will do that, you can also look at https://config-master.wikimedia.org/pybal/eqiad/thanos-query
[19:04:54] <mutante>	 jhathaway: yes, with "get" instead of "set" and in your browser by looking at https://config-master.wikimedia.org/pybal/eqiad/thanos-swift
[19:05:16] <jhathaway>	 this scared me, but was evidently harmless?
[19:05:18] <jhathaway>	 root@thanos-fe1001:~# depool --help
[19:05:19] <cdanis>	 and, another option is, on most service hosts themselves, there are `pool` and `depool` scripts installed
[19:05:20] <jhathaway>	 Depooling --help on thanos-fe1001.eqiad.wmnet
[19:05:20] <mutante>	 jhathaway: it depends what the reason is whether you want "no" or "inactive"
[19:05:38] <mutante>	 alternatively you can type "depool" on the host itself
[19:05:40] <cdanis>	 you simply run them :D
[19:05:46] <cdanis>	 the scripts could be improved
[19:06:13] <jhathaway>	 if there was not room for improvement, I wouldn't be needed around here ;)
[19:06:16] <jhathaway>	 *no
[19:07:18] <jhathaway>	 thanks for the help cdanis & mutante 
[19:07:45] <herron>	 when I was younger so much younger than today, I never needed depool --help in any way
[20:50:02] <mutante>	 jhathaway: fwiw, re that thanos/swift cron spam. I had that before and I just tried again to run it manually..and it ..works. It gets some numbers. 
[20:50:39] <mutante>	 I think what's happening is that it just sometimes rate limits it
[20:50:54] <jhathaway>	 yeah, but when I run it in a loop I occasionaly get 401s
[20:51:11] <mutante>	 because that cron runs every single minute, along all the other swift-account-stats crons..ALSO running every minute
[20:51:35] <mutante>	 so since it's not an obvious issue with missing or non-working credentials...
[20:52:12] <mutante>	 I would say that deserves a ticket where we can first ask.. what is using these numbers
[20:52:18] <mutante>	 because it sends them to /dev/null
[20:52:46] <mutante>	 like.. is there even a point to running this every minute if it just gets sent to /dev/null and doesn't write to some file
[20:53:00] <mutante>	 would be my first question what I am missing there
[20:55:11] <jhathaway>	 well it also has an arg to send statsd to localhost, so I assume it is sending to both
[20:55:24] <mutante>	 jhathaway: can't even reproduce the issue when running in  "while true ..do .. sleep 5"
[20:55:40] <jhathaway>	 try sleep 1!
[20:55:50] <mutante>	 are the other crons writing to the same place?
[20:55:57] <mutante>	 and sometimes block each other somehow
[20:56:06] <jhathaway>	 it took 10 or so minutes for me to catch one
[20:56:12] <mutante>	 a lot of them..and all of them *  *  *  *  *
[20:56:20] <jhathaway>	 perhaps, I'm not familar with the stack
[20:56:29] <jhathaway>	 what team owns the service?
[20:56:46] <mutante>	 all of this doesn't explain why only this one host
[20:56:49] <mutante>	 and not the others
[20:56:53] <jhathaway>	 true
[20:57:15] <jhathaway>	 I only bounced swift-proxy-service, pehaps somethis else is broken?
[20:57:23] <mutante>	 analyzing owner from https://wikitech.wikimedia.org/w/index.php?title=Thanos&action=history :)
[20:57:42] <mutante>	 observability
[20:59:45] <RhinosF1>	 go.dog is probably best
[21:00:08] <TheresNoTime>	 (aw `.dog` isn't a tld)
[21:00:40] <RhinosF1>	 TheresNoTime: I don't want to ping because 10pm on Friday for me
[21:00:48] <mutante>	 it is https://www.namecheap.com/domains/registration/gtld/dog/ :p
[21:00:49] <RhinosF1>	 And I have no idea what time it is for others
[21:00:57] <TheresNoTime>	 mutante: !!!
[21:01:54] <TheresNoTime>	 and ah RhinosF1 okay that makes sense :P
[21:02:31] <mutante>	 TheresNoTime: they have .dog but not .cat  outreageous
[21:07:47] <TheresNoTime>	 oh even worse, `.fox` is allocated to FOX and isn't for public registration :(((
[21:10:50] <dwisehaupt>	 i'm sad that someone registered up.dog but have nothing answering at whats.up.dog. it has an ip at a university also, which means it really should have something there. :)
[21:11:19] <TheresNoTime>	 oh that is a crime
[22:38:14] <brett>	 Hm, pwstore looks like an abandoned project. Any plans going forward or are we just riding it out for now?
[22:41:18] <brett>	 git-crypt would likely serve the same purpose using the same PGP keys but my experience has shown that rotation is quite an involved process (the project doesn't have a helpful interface, it has to be done manually).
[22:42:28] <brett>	 (just some braindumping, not under the illusion that it'd be a simple switch)
[23:41:17] <bd808>	 brett: that one you found is not the same pwstore. https://people.wikimedia.org/~jmm/pws is almost but not quite https://github.com/formorer/pwstore/. It would be nice to know which upstream is the "real" one.
[23:42:10] <bd808>	 At this point, https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/wmf-sre-laptop/+/refs/heads/master/scripts/pws is our canonical version I suppose.