[09:19:25] <dcaro>	 volans: cookbooks fail trying to post to alertmanager to set downtimes, I'm guessing that the authentication is done by ip?
[09:19:51] <dcaro>	 (so only requests from cumin/internal are allowed)
[09:20:34] <volans>	 dcaro: you mean when running from your latop?
[09:21:27] <volans>	 yes, Alertmanager doesn't have ACL support at all AFAIK and IIRC it's authorized via IPs. I can check but you might want to ask o11y
[09:22:25] <volans>	 see hieradata/common/profile/alertmanager/api.yaml
[09:33:05] <dcaro>	 that broke all our downtiming cookbooks :/
[09:45:54] <dcaro>	 will have to revert to not using alertmanager yet, and only icinga
[09:47:10] <volans>	 FYI o11y is actively migrating alerts from icinga to alertmanager, not downtiming there would most likely mean unwanted IRC spam and potentially also some page
[09:47:24] <dcaro>	 yep
[09:50:49] <dcaro>	 the alternative is that we don't downtime at all, that means more unwanted irc spam, and more pages :), so taking the lesser of two evils until a better solution is possible
[14:10:37] <klausman>	 Any objections to keeping the second section in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/conftool-data/discovery/services.yaml sorted? I'm about to add a service and could fold it into that change
[14:11:13] <klausman>	 It _looks_ like the section used to be sorted alphabetically, but it is not anymore.
[16:10:15] <robh>	 !log ganeti3001 rebooting and reimaging for firmware updates via T308238
[16:10:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:10:19] <stashbot>	 T308238: Upgrade ganeti/esams to Bullseye - https://phabricator.wikimedia.org/T308238
[16:48:28] <brett>	 It looks like CI might be down. Jenkins isn't spawning a new job despite changes on https://gerrit.wikimedia.org/r/c/operations/alerts/+/804450
[16:49:11] <volans>	 brett: just overloaded (it does almost every day around this time), see https://integration.wikimedia.org/zuul/
[16:50:25] <brett>	 volans: oho. Thanks for the info
[16:51:16] <brett>	 that's a lot of jobs all at once O.o
[16:55:04] <RhinosF1>	 A lot of them are chained
[16:56:33] <brett>	 like a spiked flail to the face
[17:10:37] <hnowlan>	 anyone know why thumbor2006 has had puppet disabled since May 31st? 
[17:10:47] <hnowlan>	 reason is "foo" 
[17:10:52] <cdanis>	 😬
[17:11:37] <volans>	 it means has not been disabled properly with the disable-puppet script :/
[17:12:31] <volans>	 hnowlan: 301 mutante 
[17:12:48] <mutante>	 I have not disabled it. I only noticed it was disabled for reasons unknown to me.
[17:13:13] <volans>	 mutante: your bash history says it differently :)
[17:13:15] <mutante>	 then I rebooted both 2006 and 2004. 2004 is the one that is broken physically
[17:13:43] <volans>	 maybe was intented for anothe rhost? anyway please never use 'puppet agent', ever
[17:14:02] <volans>	 but the bash wrapper scripts instead
[17:14:05] <mutante>	 checking SAL
[17:14:21] <volans>	 disable-puppet, enable-puppet, run-puppet-agent
[17:15:04] <mutante>	 ok
[17:15:13] <mutante>	 would it have changed anything about not having a reason?
[17:15:20] <mutante>	 looking for one
[17:15:28] <volans>	 would have added the $USER
[17:15:40] <volans>	 that in this case was you AFAICT from bash history
[17:15:42] <XioNoX>	 why is it not named run-puppet btw? to be consistent :)
[17:15:54] <volans>	 XioNoX: ask j.oe
[17:15:56] <volans>	 :)
[17:16:19] <mutante>	 ok
[17:16:19] <volans>	 also puppet has a lot of subcommands, for example on the puppetmasters
[17:19:11] <mutante>	 hnowlan: it's enabled and puppet ran now, sorry about that. I think it was an alert related to rsyslog
[17:19:41] <hnowlan>	 mutante: ah, cool. Thanks! 
[17:20:50] <mutante>	 hnowlan: so..2004 is down and I depooled it hard (inactive)
[17:20:59] <mutante>	 and the purchase date is > 5 years ago
[17:21:08] <mutante>	 so I'm afraid there won't be much repairing
[17:21:15] <mutante>	 that beign said Papaul took the ticket
[17:21:19] <hnowlan>	 mutante: it just came back up 
[17:21:24] <hnowlan>	 but I don't have high hopes 
[17:21:29] <mutante>	 how did you do that?
[17:21:40] <mutante>	 I had tried powercycle but it was like nothing happened
[17:21:52] <hnowlan>	 not sure, Papaul did it :) 
[17:25:05] <hnowlan>	 mutante: it seems healthy, I will repool if that's okay with you
[17:25:42] <mutante>	 hnowlan: sure, please do
[17:26:07] <mutante>	 I won't question how he did the magic
[17:26:13] <mutante>	 maybe DRAC reset
[17:26:21] <hnowlan>	 ooof, didn't realise how much the other instances are suffering in its absence 
[17:26:44] <hnowlan>	 déjà maps
[17:29:03] <mutante>	 yea, that's the issue here. one host down is too many
[17:29:06] <mutante>	 and the hosts are old
[17:29:13] <mutante>	 so we are lucky this is back :)
[17:30:29] <hnowlan>	 the spec doesn't seem to be too nonstandard so if we're really in trouble in future we could scavenge some old decommed hardware for a temporary replacement
[17:30:58] <hnowlan>	 but I Would Prefer Not To 
[17:31:50] <mutante>	 that was basically what I said in a meeting earlier.. I feel like this is going towards reusing other hardware.. like steal from appservers.. which is horrible but "if we have to"
[17:32:13] <mutante>	 if we can make this survive just a little bit longer that is much better though