[06:41:45] FYI, I'll be rebooting bast1003 in a few minutes [09:41:16] FYI, stretch-wikimedia has just been retired on apt.wikimedia.org [09:42:10] 🎉 [09:44:24] \o/ [09:48:16] olé [09:50:17] This reminds me to clear out some old custom docker image floating around on one of my machines. Potato... [09:54:44] very nice re: stretch, feelsgood.gif [10:06:20] heads up, I will rollout (sigh again) cfssl on thanos-fe* in codfw in a wee bit [11:20:27] We just got an oom error on cp3074 and it killed varnishkafka - Does anyone know anything about it? I can restart varnishkafka unless anyone says not to. [11:23:46] oh btullis sorry just wrote on -operations the same [11:24:00] AFAIK we're not working on that host [11:24:10] so is "go" for me [11:24:24] OK, thanks. I'll restart varnishkafka on it. 👍 [11:24:45] tnx, it ever happened in the past (the oom error?) [11:25:21] I've not seen it, personally. [11:25:48] ok, I'll keep an eye on that [12:27:08] head-up in advance, I'm about to reboot an-coord1001 - there may be a little noise from Superset, Druid, Hive services etc. [15:46:41] herron: state of things: [15:46:53] cfssl is deployed on codfw, puppet is stopped on eqiad [15:47:10] kartotherian is pooled only on codfw [15:49:30] herron: we will generally, roll everything back. Generally speaking, we added more CPU to the problem, but the amount of CPU we had to add in order to get codfw to handle the whole traffic, cant be justfied [15:51:21] effie: thanks for the heads up, promising and depressing news all at the same time [15:51:28] hehe [15:51:45] I will let codfw run a bit longer and roll back later [15:52:12] if anything happens and I have not rolled back, just pool kartotherian on eqiad, and depool on codfw [15:52:23] I will take a break for the time being [15:52:39] ack ok, sounds good