[09:25:27] <claime>	 Amir1: Stupid question but maybe it's worth adding a fourth VM so we can survive 2 down?
[09:48:32] <Amir1>	 claime: adding a fourth VM will distribute the load only, but I can increase the knob of replication factor to three afterwards (so three out of four)
[09:48:55] <Amir1>	 I'd say let's do it if it really becomes a problem
[09:50:05] <moritzm>	 we have four rows with Ganeti servers in eqiad/codfw, so from the Ganeti PoV that would be fine
[23:06:34] <bd808>	 swfrench-wmf: [non urgent] I have a new syntaxhighlight shellbox build to push out next week. I was wondering if it is still best practice to update all of the shellbox deployments at the same time, or if  when I schedule and do this I should only run the shellbox-syntaxhighlight deployment.
[23:20:12] <swfrench-wmf>	 bd808: it's definitely preferable to have them running the same image version at steady state, but IMO it's fine if one of them gets a bit ahead and we catch the others up. which is to say, if you want to go ahead and update syntaxhighlight next week, I can catch up the rest shortly after.
[23:28:23] <bd808>	 swfrench-wmf: I am happy to run the "deploy all the things" loop on Monday when I ship shellbox. I haven't ever actually paused during that in the past to test anything, but maybe nobody does?
[23:30:40] <swfrench-wmf>	 deployment "strategy" probably varies a lot from person to person, heh. if only we had some sort of "scap for services" that would apply a consistent and uniform strategy :)
[23:31:40] <swfrench-wmf>	 joking aside, I usually usually update all of staging, to make sure things at least stick, then the non-primary DC, then wait 10m while checking grafana for errors / latency, then the primary DC
[23:36:48] <bd808>	 swfrench-wmf: I can do that. staging, staging smoke test, codfw, codfw smoke test, watch grafana for a while, eqiad, eqiad smoke test, watch grafana for a while
[23:37:07] * swfrench-wmf thumbs up
[23:37:10] <swfrench-wmf>	 thank you!
[23:53:03] <bd808>	 swfrench-wmf: I'm going to try to use the designated "MediaWiki infrastructure (UTC late)" deploy  window on Monday. If that ends up being a problem for other things I will push it to later in the day.
[23:54:46] <swfrench-wmf>	 bd808: sounds good. that should actually be pretty ideal. if possible, ping me before you get started, since I will likely also have some changes rolling in that window :)