[10:23:25] <kormat>	 do we run puppet agent on boot, or is it strictly on the 30min timer?
[10:25:03] <jynus>	 I remember vaguely it was done on boot
[10:25:08] <jynus>	 but I would have to check
[10:25:48] <kormat>	 [Install]
[10:25:48] <kormat>	 WantedBy=multi-user.target
[10:25:56] <kormat>	 that sounds like it gets started by systemd on boot
[10:26:18] <kormat>	 oh, heh. /etc/cron.d/puppet also has a `@reboot` entry
[10:26:27] <jynus>	 I think the first is the agent
[10:26:35] <jynus>	 the second could be the run?
[10:26:37] <jynus>	 but yeah
[10:27:14] <volans>	 kormat: @boot
[10:27:43] <kormat>	 volans: ok cool, thanks.
[10:27:53] <volans>	 boot + 30m mark
[10:27:58] <kormat>	 i see the puppet.service is disabled
[10:28:01] <volans>	 yes
[10:28:05] <volans>	 we dont' run the agent
[10:28:10] <volans>	 as a daemon
[10:28:16] <jynus>	 so there you have it, with someone more in the know
[10:28:26] <volans>	 only the cron entries
[10:30:29] <volans>	 also we don't run just puppet in the cron entry
[10:30:44] <volans>	 but a custom script that does a couple of other things inlcuding apt get update before running puppet
[10:31:47] <kormat>	 👍
[10:52:12] <topranks>	 volans: Do we introduce any randomness into the boot + 30 min timer?
[10:52:52] <topranks>	 I'm wondering about a major problem (touch wood) where an entire DC loses power, everything boots back up at once, and the puppet timers become synchronized?
[10:54:32] <XioNoX>	 iirc, yep :)
[10:56:02] <joe>	 topranks: that's right, but OTOH if a whole dc loses power, a puppet run failing is going to be quite down in our list of problems 
[10:56:18] <joe>	 it would be a transient issue anyways
[10:57:00] <joe>	 while the amount of stuff failing badly because it started while other stuff wasn't up, that's what's really scary about that scenario
[10:57:24] <joe>	 I'm not even sure we realize the correct order besides 'first the persistency layer, then mediawiki, then everything else'
[10:57:32] <topranks>	 yeah definitely, we've probably got much bigger problems in that scenario.
[10:57:52] <joe>	 that doesn't mean it wouldn't be annoying
[10:57:57] <effie>	 it is all hands and you are discussing the DC loosing power scenario 
[10:58:06] <effie>	 what can possibly go wrong  :p
[10:58:14] <joe>	 effie: you mean we're jinxing it?
[10:58:19] <topranks>	 I did put a "touch wood" in there :)
[10:58:31] <effie>	 nice try :p 
[10:58:40] <effie>	 joe: yes
[11:11:48] <dcaro>	 something that has happened to me, is when all the racks power up again, the create a power spike that brings the DC down again in a loop xd
[11:11:59] <dcaro>	 *they create
[11:14:39] <volans>	 topranks: a bit: SLEEPVAL=$(($RANDOM % 60))
[11:15:13] <volans>	 see /usr/local/sbin/puppet-run for the full comment around it
[11:36:22] <topranks>	 thanks !
[11:37:11] <topranks>	 dcaro: not happened to me but yes I've heard of that alright.
[13:15:39] <mszabo>	 dcaro: https://youtu.be/LeFtwtvy4Wc
[13:23:17] <dcaro>	 xd, fun times
[13:32:57] <topranks>	 wow.  hurts just to watch
[13:51:02] <mszabo>	 "we got problems"
[14:11:01] <jynus>	 joe: one of the things I want to focus in the next (months? years?) alongside security and the rest of technology is a business continuity plan for things like this- and that will go beyond just "having backups"- so I hope everybody can help me slowing adding procedures/plans for all possible emergencies in the short future
[14:16:23] <hasharLunch>	 mszabo: that looks extremely scary to me. The whole room screams "danger run out of there"
[14:16:54] <mszabo>	 yeah, the sound is very very scary
[19:16:18] <XioNoX>	 elukey: https://www.gentlydownthe.stream/