[05:47:47] <_joe_> urandom: we have repeatedly done that (depooling sessionstore in one DC); while it means we're adding some latency to every request, it's within tolerance as long as we don't keep running in that situation for very long [05:48:11] <_joe_> one of the reasons we introduced the service proxy originally was to make such movements manageable [13:37:30] apergos: have a few minutes to talk about dumps? I'm wondering if you would expect both dumps servers to be rsyncing from master.download.kiwix.org, or one rsyncing from kiwix.org and the other rsyncing from the other dumps server? [13:38:01] If the current behavior (both talking to kiwix.org) is expected then I need to stagger the two rsyncs. [13:41:42] _joe_: that helps; thanks [13:44:27] hey andrewbogott [13:45:07] hi! Does my qustion make sense? [13:45:11] *question [13:45:17] I didn't set that up initially, but it seemed that each server was supposed to be independent and download things. I broke that model with the WME dumps (download to one, rsync across) but that's just me being ornery [13:45:50] ok, cool. So I probably just need to make those downloads happen less often and at different times so they don't overlap. [13:45:50] if you want to restructure it so the primary downloads and then shoves across to the other(s), that's fine by me, whatever you think best [13:45:59] * andrewbogott breaks out the modular arithmetic [13:46:07] they probably don't need to go as often as they do, for sure [13:46:45] of course when a big new release comes out, it might take a little while to grab the contents [13:46:47] so there's that [13:47:32] I feel like someplace I've already written a snippet to make staggered timestamps out of hostnames.... [13:51:59] how long a time difference between them would be good, I wonder [13:53:51] note that I have notified Hannah in another channel so she can follow along in the discussion (I know it can feel a bit rude for you to ping two people etc, as co-maintainer I want her to be up to date on everything.) [13:54:44] thanks! [13:56:59] andrewbogott: if that's in puppet, there's fqdn_rand [13:58:17] claime: does that do something consistent or is it re-randomized on each run? I want to make very sure the runs never happen at the same time. [13:58:28] (it is in puppet) [13:58:29] It's deterministic yes [13:58:42] that might do it then! ty [13:58:45] yw [13:59:06] gtk! [14:02:32] for just 2 you might not get what you want with fqdn_rand though [14:03:37] Depends on what stagger they need yeah [14:07:23] I'm lost in the systemd time docs. All I want is for one of the hosts to schedule at 0:00, 6:00, 12:00, 18:00 and the other at 3:00, 9:00, 15:00, 21:00 [14:08:01] Which I'm hoping is just fqdn_rand(1, 1) * 3 and then subbing that into an interval somehow [14:08:36] e.g. *-*-* ${stagger}/6:15:0" [14:08:43] Second arg is a seed [14:09:01] So it's not needed per se [14:09:29] for this case I'd rather set a hiera value [14:09:36] Ah, I thought if I wanted it to be determinate I'd need to specify. [14:09:49] andrewbogott: No, the fqdn_ part takes care of that [14:10:01] fqdn_rand is random, just deterministing between puppet runs [14:10:04] The seed is for when you need different numbers from the same fqdn [14:10:13] claime: ok [14:10:31] https://www.puppet.com/docs/puppet/5.5/function.html#fqdn_rand [14:10:33] volans, docs say "(That is, each node will get a different random number from this function, but a given node’s result will be the same every time unless its hostname changes.)" [14:10:46] andrewbogott: exactly, random, so they can be the same [14:11:18] hm, that's not how I was reading 'different' in that sentence but I bet you're right [14:11:29] So, great, I'm back to modular math instead! Or hiera. [14:11:43] a different dice roll, that can be the same [14:11:56] Once I have $stagger resolving to 0 or 3, does my interval up there look right? [14:12:43] depends how long it takes one of these to complete, honestly [14:13:18] apergos: right now I'm worried about the syntax of it, not whether 3 hours is long enough :) That we can only learn from experience. [14:13:40] oic, sorry, misunderstood the question [14:13:45] andrewbogott: only skimmed above but id recomend cron_splay which is designed for this use case [14:14:07] however perhaps i missed something as volans did not make that recomendation? [14:14:12] argh so many solutions! OK, reading... [14:14:50] what I have now is [14:14:50] jbond: I thought that cron_splay doesn't support multiple runs per day but I might be basing my assumption on outdated info [14:14:54] https://www.irccloud.com/pastebin/DjCSRkUv/ [14:15:05] which is basically just an even/odd checker [14:16:15] volans: ahh yes thats probably why cron_splay has hourly and daily, so if you want something to run e.g. twice or 4 times per day then cron_splay wont fit the use case [14:17:04] orrrr [14:17:29] download from one,, then rsync to the other when the one is done :-P [14:18:47] apergos: my (arbitrary) goal is to do something that doesn't require the hosts to know about each other in this case :) [14:18:59] I will see what my current nonsense compiles to [14:19:07] it's your boxes, not gonna argue of course :-) [14:20:19] the fact that we went over 6 connections makes me suspicious that something else is happening, but maybe it was just a big diff [14:20:58] that the server runs more than one copy of the script at the same time, still? [14:21:00] possible [14:21:42] * andrewbogott types the hostnames into 'GERRIT_PRIVATE_CHANGE_NUMBER' like always [14:22:08] also I'm going to go with 3 times/day rather than 4 [14:22:09] lol [14:22:18] I'm sure that's fine [14:26:56] apergos: now I have one host with interval => {'start': 'OnCalendar', 'interval': '*-*-* 0/8:15:0'} and one with interval => {'start': 'OnCalendar', 'interval': '*-*-* 4/8:15:0'} which is what I was going for but I don't actually trust those intervals. Do you read that as 'every eight hours, starting at 0:15' and 'every eight hours, starting at 4:15' or am I making a naive cron mistake? [14:27:24] cron timestamps are one of the half-dozen sotware topics that absolutely will not stick in my brain no matter how many times I re-learn [14:29:00] in case you haven't come across it already, 'systemd-analyze calendar <...>' can give some insight [14:30:12] don't memorize the format, memorize this [14:30:15] systemd-analyze calendar --iterations=5 '*-*-* 0/8:15:0' [14:30:22] ah I see go dog got there already [14:30:42] TIL --iterations, thank you! [14:30:51] :-) [14:31:13] first one looks right according to the output [14:31:14] ooh, nice! I was trying a web-based analyzed but it failed to parse [14:31:56] yeah, I think it's right. Let's see if my brain can retain "there's a tool for this" for next time :) [14:31:58] second one looks right according to the output [14:32:09] "analyze this!" [14:32:17] ty both for systemd-analyze ! [14:32:39] it has saved my bacon numerous times [16:19:58] talking about mailman... [16:26:48] I restarted it [16:52:55] ./~ talking 'bout my M-T-A now ./~ [16:55:56] LOL