[07:44:41] Re sha256 digests. The reason why we had to do it at $prev_job is a bit fuzzy as it was ~5 years ago, but we were trying (and succeeded) to be FedRAMP certified, and one of the requirements were fully hermetic builds (à la bazel), with fully immutable artifacts. _At the time_ (and at least to my recollection), the registry we were using (AWS ECR) [07:44:41] didn't support fully immutable tags, so you could in theory rebuild a certain tag with a different image content and release it to the registry. Relying on image:tag@sha256:checksum was a way to introduce some extra safety in the pull process. We'd always be guaranteed to have the expected image. [07:46:18] Now, maybe this is moot at WMF because a) we support immutable tags or b) we have enough trust in our CI that we are ok with assuming the blob behind a tag will always be the same. At the time I introduced using the checksum for dse-k8s-eqiad workloads, it was a bit of cargo cult from $prev_job as well as a "it's not a hard thing to do and it [07:46:18] brings interesting safety guarantees" attitude. [07:47:40] For example, this is the job that builds and publishes the airflow image: https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/jobs/543963 [07:48:30] that job relies on a set of package constraints, but no actual lockfile, so rebuilding the job a day after might produce a different output, as some packages might have been released to PyPI in between, etc. [07:49:21] so we could a) introduce some proper dependency locking to produce fully reproducible builds, or b) use the checksum to ensure full reproducible pulls. At the time I chose the latter for lack of bandwidth [07:54:03] brouberol: o/ thanks for the explanation :) [07:54:45] np! [07:55:16] the TL;DR is that we are going to workaround it, and allow both tags and checksum. It is a bit more painful, but it should be fine. Chris found a way to reverse map a sha checksum to a tag, it requires a bit of effort and calls to the registry but we may use it. Or we could simply store the sha, instead of the tag, in debmonitor. [07:55:25] (something that we would not have to care about if we collectively decided to stick with punchcards years ago, but that ship has sailed) [07:55:42] At the end we'll have a breakdown of cluster/namespace/image-name that require updates, if there is a tag or a sha it shouldn't matter much [07:56:19] the main point though is that decisions like these may require a broader consensus/discussion, since we decided a long time ago to use tags only [07:56:38] not being aware of the various usages etc.. makes the creation of tools way more difficult and error prone [07:56:47] in this case it is fine, I completely got your point [08:05:55] > since we decided a long time ago to use tags only [08:05:55] This is something that I completely missed at the time, that's on me [08:12:11] nah it is fine, I didn't mean to blame you, on the contrary [08:12:32] I am just thinking out loud to make sure we talk more between each other, it is always valuable [08:12:38] (and this group is the proof of that) [08:17:31] <3<3 [08:58:45] <_joe_> brouberol: I think it's ok to only use sha256s if you take a radically different approach than us to building images [08:59:04] <_joe_> but also, why hate yourself so much :D [09:00:00] haha, TBH I have tooling that automate the fetching of said hash and I only have to copy it into a YAML value file, so I don't think about it that much anymore [09:00:44] re radically different approach: I wouldn't say that's the case here though. We're using kokkuri/blubber, so I'd say we're pretty much walking the walk [09:01:19] so maybe the right thing to do for us is making sure our build job is fully reproducible by introducing dependency locking and go back to using tags? [09:10:55] <_joe_> the problem for fully repro builds is you need to only run apt-get update in the base image [09:11:05] <_joe_> and never again [09:21:50] that's true, and that gets us in nix/bazel territory, and TBH, I'm happy with not having to use any of these [09:22:20] and focus on making our *python* dependency resolution reproducible, with dependency locking [09:22:45] In my mind, that'd be a good enough tradeoff, for a python app [10:00:58] <_joe_> +1