[07:23:31] <XioNoX>	 mutante: what about `prometheus_nodes`? do you need all the nodes or only the local ones? (and it's defined (empty set) for cloud)
[07:55:28] <godog>	 also the site's prometheus hosts already have access to all ports via standard firewall rules, what problem are you trying to solve mutante ?
[10:16:35] <claime>	 Last reminder that I'll be running the switchover live test at 11:00 UTC
[10:16:42] <chaosmonkey>	 ack
[10:16:51] * chaosmonkey steps back from the keyboard
[10:16:52] <claime>	 If there's any objections, better voice them now :P
[10:17:14] <claime>	 chaosmonkey: "Put the keyboard down SLOWLY"
[10:17:55] <chaosmonkey>	 claime: don't shout at me.. I'm chaos after all
[10:18:09] <claime>	 chaosmonkey: It's not me, it's the sre police
[10:18:13] <claime>	 I'm not responsible
[12:21:57] <vgutierrez>	 volans: what's a script in SRE cookbook terms?
[12:23:26] <vgutierrez>	 volans: SREBatchRunnerBase pre/post_script() should return a list of scripts, 
[12:24:09] <vgutierrez>	 from sre/misc-clusters/example.txt it seems that's expecting a list of python methods?
[12:24:26] <vgutierrez>	 return [self.script_example]
[12:24:37] <vgutierrez>	 being script_example def script_example(self, hosts)
[12:25:11] <volans>	 vgutierrez: pre/post_script() are a list of commands to execute on the target hosts, in instead you want to run some python function you have to use pre/post_action
[12:25:36] <vgutierrez>	 nope, I wanna run some CMDs before restarting haproxy and some after
[12:25:47] <volans>	 than that's the one
[12:26:41] <vgutierrez>	 but I don't know if the example isn't accurate or am I too thick
[12:27:02] <volans>	 vgutierrez: see for example
[12:27:03] <volans>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/cookbooks/sre/gitlab/reboot-runner.py#40
[12:27:29] <vgutierrez>	 volans: right
[12:27:43] <volans>	 yes, the example.txt is outdated apparently (cc jbond )
[12:27:50] <vgutierrez>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/cookbooks/sre/misc-clusters/example.txt#72
[12:27:53] <vgutierrez>	 that's misleading IMHO
[12:27:55] <volans>	 all that part should move to spicerack and have documenttion auto-generated
[12:27:58] <volans>	 yes
[12:28:32] <vgutierrez>	 thanks for  the example <3
[12:29:49] <volans>	 sorry for the confusion
[12:30:46] <vgutierrez>	 I might need an action after all... cause I'm assuming that run-puppet-agent isn't kosher on a cookbook :)
[12:33:13] <volans>	 what do you mean?
[12:33:34] <volans>	 ah you need to run puppet
[12:33:43] <vgutierrez>	 return ['depool', 'apt-get update', 'apt-get install haproxy', 'run-puppet-agent -q'] seems reasonable?
[12:33:55] <vgutierrez>	 or should I trigger a puppet run on a more idiomatic way?
[12:34:09] <volans>	 depool? if you're using a load balanced service use SRELBBatchRunnerBase instead
[12:35:16] <vgutierrez>	 noted
[12:35:47] <vgutierrez>	 but... I don't think I can use that
[12:36:02] * jbond will create a CR to move theses cookbookbase classes to spicerack this week
[12:36:38] <vgutierrez>	 SRELBBatchRunnerBase performs the depool as part of the action
[12:36:56] <vgutierrez>	 but I need it to be done as part of the pre action
[12:36:59] <vgutierrez>	 if that makes sense :)
[12:38:18] <volans>	 ah, yeah, you still could overwriting self._restart_daemons or self._reboot, bot that's a bit hacky
[12:38:21] <volans>	 and feels wrong
[12:38:58] <jbond>	 it might be worth adding an upgrade package action thats probably common enough that its worth adding (not helpfull now though)
[12:55:11] <vgutierrez>	 volans, jbond first naive approach on https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/891267, let's move the discussion there :)
[12:55:48] <jynus>	 from a very high level, I see a parttern and a big question- how to "alert" on normal, but unintended states? That's very hard!?
[14:08:48] <sukhe>	 volans: thanks for resolving T329773! as per your comment, I wanted to let you know that I will be reimaging a dnsrec host in ~30 mins
[14:08:49] <stashbot>	 T329773: spicerack dnsdisc.Discovery attempts to query depooled/disabled dns auth servers - https://phabricator.wikimedia.org/T329773
[14:09:35] <volans>	 sukhe: ack, it should all be good, but because today is our no-no day who know what will happen :D
[14:09:42] <sukhe>	 haha
[14:09:51] * sukhe postpones reimaging for tomorrow 
[14:09:53] <sukhe>	 :)
[14:10:09] <volans>	 lol, up to you :)
[14:10:44] <sukhe>	 what can I say, I am believer in Murphy's Law :P
[14:12:14] <volans>	 ahahaha
[14:16:29] <claime>	 That's a very spicy day
[14:16:31] <claime>	 for sure
[14:19:36] <godog>	 https://www.youtube.com/watch?v=ErgdUhZteqw
[14:22:25] <sukhe>	 godog: not what I was expecting but I will take it :)
[14:22:56] <godog>	 haha sukhe ! you are welcome
[14:30:50] <ottomata>	 volans: re kafka stretch, we've been waiting for them to be setup since they've been received (but also not had time to use them yet, so haven't pushed).  
[14:33:35] <volans>	 it's just that sends an email every day to root@, so someething is clearly up on that one but not fully puppetized
[16:56:51] <mutante>	 XioNoX: yes, I needed prometheus_nodes and not monitoring_servers. that was actually the answer
[16:57:29] <mutante>	 godog: I was fixing "connection refused" for prometheus nodes to contint servers, using port 1443
[16:58:06] <mutante>	  ci::firewall previously only allowed cp* hosts
[16:58:12] <mutante>	 it's fixed for me now
[17:11:24] <taavi>	 mutante: you don't need an explicit firewall rule for that because of https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/base/manifests/firewall.pp#70
[21:04:24] <zabe>	 could someone merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/891360 for me=
[21:04:26] <zabe>	 ?
[22:12:13] <jhathaway>	 zabe: sure, done!
[22:17:46] <zabe>	 thanks