[14:34:11] * klausman is once again banging his head against k8s permissions [14:34:25] `Error: query: failed to query with labels: secrets is forbidden: User "article-descriptions-deploy" cannot list resource "secrets" in API group "" in the namespace "article-descriptions"` [14:34:41] But as far as I can tell, all the files and bits are there. [14:36:15] klausman: the namespace isn't there though [14:36:17] kubectl get ns article-descriptions [14:36:17] Error from server (NotFound): namespaces "article-descriptions" not found [14:36:27] I assume this is ml-serve-eqiad btw [14:36:35] even setting createNamespace: true doesn't help [14:36:59] (unless diff is always failing in such cases, let me try apply -i [14:37:16] Nope, doesn't work eiother, same error [14:39:15] I see the namespace existing now. age: 10s [14:39:25] I assume this fixes it? [14:39:26] And manually creating it... which I just did, is not working, either [14:39:55] createNamespace: true sounds pretty wrong tbh [14:40:10] Yeah, we have it set to false everywhere. [14:40:37] it might help to know what you are trying to do ;) [14:40:52] Adding a new service to prod, article-descriptions [14:41:20] I think all the files in admin_ng, ml-services and the private repo(s) are in place [14:43:46] klausman: sounds like you're missing the entries in profile::kubernetes::deployment_server::services [14:44:06] https://wikitech.wikimedia.org/wiki/Kubernetes/Add_a_new_service [14:45:02] oh, sorry - me not git pull :) [14:45:13] Phew, I was doubting my sanity there [14:45:20] but did you actually deploy the admin_ng change? [14:45:44] (adding to profile::kubernetes::deployment_server::services basically is a admin_ng change) [14:45:46] Oh god %-) [14:46:18] easy win :-p [14:46:21] The sad thing is: it's not the first time I make this mistake [14:47:11] and now of course the push failed because trhe NS was already there, well at least that's an easy fix [15:10:39] Oh, also, thanks, both of you [15:13:23] yw [15:35:15] yw