[07:56:20] hey folks! [07:56:49] I was reviewing the docker registry's info to create a task for apus, and https://grafana.wikimedia.org/d/StcefURWz/docker-registry?orgId=1&from=now-90d&to=now&timezone=utc&var-datasource=000000005&var-instance=$__all&viewPanel=panel-48 caught my eye [07:57:07] we added almost 1TB of storage in the past 3 months [07:57:27] why is eqiad and codfw not in sync ? [07:57:35] or am I misunderstanding the graph? [07:57:40] not sure about it, it is weird indeed [07:58:05] the only thing that I can think of is some sort of swift-replication problem [07:58:27] yeah, same here [07:59:22] I recall that Matthew say that it wasn't a big concern to get ~6TB of data on apus, but we may have a problem if we add 1TB every 3 months [07:59:55] there are things to drop for sure [08:00:35] but I suspect that maybe the new MW images take a toll on the Swift's container size, and I don't think we ever clean them up (or do we?) [08:00:41] (assuming that cleaning up worked) [08:02:25] I don't think we do [08:04:07] no, we don't [08:08:27] one thing that we could think about is either 1) re-evaluate https://phabricator.wikimedia.org/T375645 to expand our dockerctl command or 2) send a patch to upstream to expand their DELETE functionality to actually drop what's not needed [08:08:42] they released 3.0 recently so they may be open to a pull request [08:09:02] not sure how long it would take though, and there is the ? if we want to use docker distribution in the future or not [08:10:02] IIRC all/most of the self hosted registries use docker distribution under the hood - so might be worth it [08:16:23] it is not clear to what extent though, maybe they have their own way of cleaning up etc.. we could look at harbor, if it suffers from the same problem [08:19:41] harbor used to suffer from a different problem, that is being a deployment with like 6 or so containers that we couldn't reason easily about back then [08:19:49] Since that back then was 2017 IIRC [08:19:55] this probably doesn't apply anymore [08:23:07] ok so opened https://phabricator.wikimedia.org/T394476 [08:23:13] for the apus onboarding [08:23:22] check if you want to add more info etc.. [08:34:57] https://github.com/distribution/distribution/issues/4282#issuecomment-2225493311 is of course the answer that I expected [08:35:16] they are asking to test the latest upstream now, that is 3.x [08:53:01] left a comment in the task as well [08:53:10] but yeah, upstream is pretty clear [08:54:40] on a side note, podman ecosystem added support for zstd-chunked in theor stuff. See https://github.com/containers/storage/blob/main/docs/containers-storage-zstd-chunked.md [08:58:47] the skippable frames one is pretty interesting. Doubtful though our ecosystem will have the capability to use it anytime soon. [09:29:33] now there is also a stateless (based on kademila dht) alternative to dragonfly that looks less involved/complicated. It only works with containerd though [09:29:45] https://github.com/spegel-org/spegel [10:05:13] while navigating the registry's code I found [10:05:14] "/v2/{name:" + reference.NameRegexp.String() + "}/blobs/{digest:" + digest.DigestRegexp.String() + "}", [10:05:23] listed as available for DELETE [10:05:45] "Operations on blobs identified by `name` and `digest`. Used to fetch or delete layers by digest.", [10:06:25] I am wondering if we could use it to drop the blobs belonging to repositories that have been removed [10:06:41] (so not listed in the catalog but still using swift's space, and breaking the GC) [13:44:26] DELETE by regex? [13:44:33] hmm [13:48:57] no nevermind that was probably to construct the final path [13:51:25] I am rechecking some github issues, to refresh my memory, and I am looking into the garbagecollect.go code (I never done it before) [13:52:09] if we run the GC with --delete-untagged it should, in theory, clean up what we are complaining about [13:52:26] namely images without tags (after we deleted them via registryctl) [13:53:03] now the main issue is that the GC uses the storage interface (in our case, swift) to List for tags, and Swift reports no-one with an error [13:53:18] and that stops the GC completely [13:53:36] the super funny part is that right after the error, I see [13:53:36] if len(tags) == 0 { [13:53:37] emit("manifest eligible for deletion: %s", dgst) [13:53:49] hilarious right? [13:54:12] 🤣 [13:54:55] https://github.com/distribution/distribution/blob/release/2.8/registry/storage/garbagecollect.go#L70 [13:56:16] ah no I misread the code [13:56:19] my bad [13:56:29] the error should come from https://github.com/distribution/distribution/blob/release/2.8/registry/storage/garbagecollect.go#L77 [13:57:07] so indeed it should enter there, but it fails to retrieve the tags [13:57:11] that is expected [13:57:21] wondering if in 3.x they changed it [13:58:38] https://github.com/distribution/distribution/commit/112156321fceeb4a6c845970d971cec84a3f0794 [13:59:31] uff but we cannot really test it since in 3.x there is not swift support [14:02:52] it seems an easy patch to backport though [14:04:41] tbh, we probably would be better off seeing if apus+S3 works than try to GC the current solution [14:04:44] https://github.com/distribution/distribution/commit/601b37d98b9330bd66137cdc89a01896cf7a481d is the parent one [14:05:01] I 've given up on trying to GC it tbh [14:05:24] starting from scratch and making sure we GC from day 1 sounds a more plausible scenario [14:05:26] I think we may have the same problem with S3 though, repositories are not deleted via API. [14:05:36] ah okok yes, assuming it works [14:05:55] well, with a very small registry size we should be able to reason about these things working quickly [14:06:14] S3 is also easier as a thing to work with [14:06:20] do you think that we should try to package 3.0 though with apus? [14:06:30] otherwise I fear we'd get the same issue with S3 [14:06:34] GC-wise I mean [14:06:35] as in, there are >1 compatible clients etc [14:06:37] we can test the use case [14:07:34] they released on Apr 3? hmm [14:09:14] brand new yes.. Although reading the code, I think that the return nil fix in https://github.com/distribution/distribution/commit/601b37d98b9330bd66137cdc89a01896cf7a481d would not stop the GC to run, but it wouldn't clean up a repository without tags [14:09:22] it would simply skip it [14:09:24] that is ok [14:09:40] but we'd keep garbage anyway [14:10:38] https://salsa.debian.org/go-team/packages/golang-github-docker-distribution/-/branches/all [14:10:46] ^ isn't very promising [14:10:50] last patch, 8 months ago [14:11:00] the 3.0.0 branches are older, 1 year ago [14:11:22] I was hoping debian maintainers would have already started the work, nothing yet [14:11:27] they started the RC releases a long time ago, maybe the maintainer didn't catch up with the final release [14:11:59] yeah there is a branch for rc1 from 1 year ago by Milos [14:12:23] ehm tag, not branch, sorry, mispoke [14:12:45] this one https://salsa.debian.org/go-team/packages/golang-github-docker-distribution/-/tags/v3.0.0-beta.1 [14:13:11] this isn't going on Trixie for sure [14:13:30] I am also not very confident on being the testbed of 3.x [14:14:01] makes sense yes [14:14:28] maybe the s3 storage driver is different and fails more gracefully when no tags are there [14:14:41] that one we can try right now in fact [14:14:41] we'll see, we can quickly test it when/if apus is ready [14:15:02] wanna pair up on Monday and work on the puppetization to see if we can expose /foobar using APUs ? [14:16:02] definitely, I am wondering if it is ok to use apus for a quick test like that, maybe we can ping Matthew on Monday to get a basic/test user on it [14:16:36] cool