[09:36:20] Hmm. I am roll-rebooting our codfw workers, and the cookbook has been sitting idle for over 10m now after one host has come back and it drained the next. Only daemonsets are left. The output from the cookbook does not indiciate what it might be waiting for [09:37:01] And of course the *moment* I hit return on that, it makes progress. [09:55:26] lol [10:18:40] Machines in this roll-reboot cycle that have dimms during boot due to training errors: 2/8 [10:22:03] *lost dimms [13:04:33] if anybody has cycles and wants to get their hands dirty with k8s and calico, I think https://phabricator.wikimedia.org/T365687 could be a nice task. It would remove the need to list the k8s cluster_nodes in hiera completely (so less changes required during renaming, adding, removing nodes) [13:12:27] ^^ definitely interested in this one, but probably won't have cycles until we get airflow running on k8s [13:15:49] Does anyone know if it's possible to disable startupProbe in a helm chart? or is that chart specific? Ref https://phabricator.wikimedia.org/P63696$814 [13:16:10] we don't need it and it's causing crashloops [13:17:15] I'd assume thats completely chart specific (if there is a knob to turn it off) [13:30:12] ACK, I was afraid of that. We can work around it though [15:10:48] Is the local-charts repo still relevant ( https://gerrit.wikimedia.org/r/plugins/gitiles/releng/local-charts/ )? Or is there a better way to adapt an existing chart to WMF? [16:44:09] Releng can answer that [17:25:55] ACK...I think I'm far enough along that it doesn't really matter for the chart I'm working on