[09:40:03] As part of https://phabricator.wikimedia.org/T334647 I'm now promoting helm-state-metrics to be a default component for all clusters (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/922474). If anybody want's to object - last chance :) [10:28:58] jayme: I'm trying to figure out why .Release.Namespace resolves to default in CI (Trying to move forward with https://phabricator.wikimedia.org/T325071), any ideas? Running helmfile build locally (not in the ci container) does have the correct information in releases [10:45:35] ok that does seem like it's not something I introduced [10:45:45] I still don't understand **why** [10:45:48] <_joe_> claime: it resolves to default in the chart tests - that I would expect - but I don't think it should in the deployment tests [10:46:16] _joe_: It does in deployment tests too https://integration.wikimedia.org/ci/job/helm-lint/10662/consoleFull [10:47:23] <_joe_> 11:56:30 - name: mediawiki-main [10:47:25] <_joe_> 11:56:30 + name: main.default.codfw [10:47:31] <_joe_> looks like something you introduced indeed [10:47:52] <_joe_> it was picking up the correct release before [10:48:09] The release is right [10:48:25] <_joe_> ah the *namespace* isn't [10:48:28] yep [10:48:29] <_joe_> sorry I misread [10:48:33] no worries [10:48:35] <_joe_> uhm [10:48:46] One more weirdness layer [10:48:59] <_joe_> Ah I think I know why, but let me check [10:49:59] <_joe_> no, I stand corrected. I am ready to guess it's some helmfile masterwork [10:51:43] Grr I don't like it. [10:54:44] That means I can't test that what I'm doing will work correctly in prod [10:57:52] <_joe_> claime: I'm not sure tbh. It seems to work with helmfile template so I'm sure it's something we do in CI [10:58:34] <_joe_> can you show me the patch please? [10:59:02] _joe_: gimme a sec [10:59:10] moving some things aroung [10:59:12] around [10:59:24] <_joe_> I am just hoping it's a go text/template scoping problem [10:59:32] <_joe_> because that would be the easier fix [11:01:57] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/922480 [11:05:25] (yes the naming is bad since it's not part of base.name anymore, I'll fix it) [11:20:56] <_joe_> claime: yeah but there is no clear scoping error in the patch, so it's something that has to do with CI. Not sure what [11:21:11] <_joe_> I'll find out in the afternoon :) [11:21:44] It's not especially urgent anyways [11:40:42] that's def. not something new. For chart diff it is because helm assumes "default" as the namespace when rendering templates. For deployment diffs I would assume a bug/feature in helmfile preventing helm to be called with the correct namespace [11:55:54] For chart diff I wasn't too worried, for deployment diffs it's strange since the namespace is specified in the helmfile itself [12:46:02] yeah, I agree and I think it's some kind of limitation of helmfile [13:13:40] <_joe_> jayme: sadly not. I verified running helmfile template on e.g. developer-portal where we use ingress [13:13:47] <_joe_> there the namespace is correctly called [13:14:13] <_joe_> the problem is I printed out the patched helmfile we create, and AFAICS it correctly states the namespace [13:14:47] <_joe_> I have one lingering doubt, that it's some specific flag we modify [13:21:11] curious :) [13:50:06] <_joe_> damn it, I basically excluded most things that could cause this behaviour, and still happens. I am starting to think the problem is actually the --kubeconfig not pointing to an actual kubernetes config in CI [13:51:38] <_joe_> yes, that is definitely it. [13:53:07] <_joe_> claime: I think I have an incoming patch [13:59:59] 🤦 [14:01:50] _joe_: gg! I was tending towards that cause too [14:04:12] <_joe_> oh and ofc it's regexes all the way down [14:04:17] <_joe_> :pain: [14:12:42] I'm sorry [15:02:59] <_joe_> claime: sorry doesn't cover my current misery lol [15:10:50] <_joe_> claime: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/922563 [15:11:00] <_joe_> but I'll comment on your patch, I think you overdid it :) [15:52:00] So I kinda know I overdid it lol. I went with the nuke recursive sed approach first to see if it was worth changing all resource names or not [15:52:09] And I think it isn't for quite a few of them [15:53:21] Ah, I linked the **wrong ticket** [15:53:40] Wondered why o.ttomata had commented, so yeah, makes sense