[11:22:06] have not seen this before: https://helm-playground.com [16:02:49] oh this is nifty [16:03:33] I will never get back the hours of my life that were spent in debugging helm templates, but this might helm me save some [17:20:11] is there also a helm-playground for debugging helmfile weirdness that behaves differently in prod and CI? [17:20:17] or is that one called "production"? [17:33:43] that's called staging usually :) [17:33:57] kamila_: what do you see? [17:34:47] jayme: in CI I have the correct image version: https://integration.wikimedia.org/ci/job/helm-lint/14024/console @ 17:49:00 [17:35:37] but when I try to deploy the diff says `image: docker-registry.discovery.wmnet/kube-state-metrics:v2.9.2` [17:36:10] it's as if it were using the upstream chart without my changes, but the commit with my changes is there [17:36:38] (and yes, it's called staging, you're correct :D) [17:37:08] could be that the new chart version has not yet made it's way to the helm chart repo [17:37:13] it takes a couple of minutes [17:37:21] ah, dammit [17:37:39] kamila_: my bad. In https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/972400 you did change the chart but you did not bump it's version [17:37:48] oh crap -.- [17:37:54] sorry [17:38:06] np, could have spotted that during review :) [17:38:13] and thank you [17:38:25] I could have not forgotten :D [17:39:16] but... how do we actually deal with versioning of upstream-imported charts anyway? [17:39:32] yes! :D [17:40:28] :D [17:40:50] kamila_: there is some discussion about that at https://wikitech.wikimedia.org/wiki/Talk:Kubernetes/Upstream_Helm_charts_policy [17:42:14] seems like my last recent idea was to add a annotation which carries the latest upstream version imported and just bump the actual version like you would with in house charts [17:42:31] yeah [17:43:15] ah, because of the "Helm considers these 'development' versions` thing? makes sense [17:43:24] ok, thanks! [18:21:39] (huh, it actually works! that's weird :D) [18:26:19] question: how do we handle port allocation for system stuff? I know about https://wikitech.wikimedia.org/wiki/Kubernetes/Service_ports , but I'm asking about things in admin_ng, specifically which port to use for kube-state-metrics [18:42:51] (the default is 8080, which seems like a bad idea, so sticking with the default isn't an option here)