[13:43:04] <inflatador_>	 smunene and I are in processing of standing up a net-new dse-k8s-codfw cluster ...we're at this step: https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/New#Calico_node/controllers and it's timing out. Docs say to sync the namespaces, but we've already done that. Did we miss something else?
[13:44:35] <inflatador_>	 helmfile config is https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1179654 if that helps
[14:01:17] <akosiaris>	 There should be an error either in events or in the pods logs about what is going on
[14:01:50] <akosiaris>	 kubectl -n kube-system get events, kubectl -n kube-system get pods, kubectl -n kube-systeme logs <failing_pod> 
[14:01:57] <inflatador_>	 yeah, looks like etcd is borked
[14:08:25] <inflatador_>	 we're checking that out now...firewall rules is my first guess? Digging thru puppet and comparing config vs dse-k8s-eqiad
[14:21:02] <inflatador_>	 OK, etcd is actually fine. But the calico pods aren't being generated with the correct bird config
[14:24:46] <inflatador_>	 ` Unable to open configuration file /etc/calico/confd/config/bird.cfg: No such file or directory`
[14:26:21] <inflatador_>	 may have missed a similar step to https://gerrit.wikimedia.org/r/c/operations/homer/public/+/826579 (per Steve)