[10:28:11] hello folks, I am a bit ignorant about emptyDirs, trying to understand if we have default limits on our clusters [10:28:44] so I have a container that pulls data from swift, and writes it to an emptyDir volume mounted as /mnt/models [10:29:11] for some big llm models (~14Gbs of binaries) I get "no space left on device" [10:29:29] but inspecting the pod I see that the emptyDir doesn't have a fixed limit [10:31:53] since I am debugging some issues while loading a model to a gpu on a pod, I am seeing a variety of errors and I am wondering if emptyDir's max size could be a problem [10:35:15] I think by default it uses local ephemeral storage [10:35:22] Is /var/lib/kubelet full ? [10:36:27] That's the default location, idk if we change it [10:40:30] claime: ahhh wait I was convinced that it would end up under /var/lib/docker, now I see in docker inspect [10:40:33] /var/lib/kubelet/pods/2d466283-e123-4210-bce8-02393ccc14ba/volumes/kubernetes.io~empty-dir/kserve-provision-location [10:40:41] and indeed I have only 14G left [10:44:20] is there a preferred suggestion for my use case? Besides expanding the /var/lib/kubelet partition [10:50:17] Good question, I don't think we have a current policy/best practice in place, do we akosiaris ? [10:56:56] in our case it is kind of mandatory to be able to do it [10:57:19] but it may affect wikikube as well, for example if content translation starts to pull binaries from swift for mint [10:57:34] they'll likely end up in the same problem [10:57:38] yep [10:57:40] (lunch back in a bit) [11:19:05] claime: no, we don't have something as a best practice for emptyDirs. It's "use, don't abuse" more or less as it is local. Really big volumes are in reality destined to be used via some abstraction layer (e.g. ceph/rdb) and kubernetes CSI. [11:19:41] which we don't have yet here and the only one experimenting with some ceph for k8s being Data Engineering [11:20:15] elukey: 14GBs per pod btw is asking for problems [11:20:48] one small thing to alleviate your issue is to move to hostPath, and not emptyDir, to at least avoid the duplication on the same node [11:21:17] it's not a silver bullet, but it will at least allow you to avoid the N*14GB issue [11:23:26] elukey: seems like you just got promoted to more than just "interested" in persistent storage (re: the session we had a few months ago) [11:24:09] anyway, for now, expanding /var/lib/kubelet is your way out. [11:24:36] which is, already an LVM volume, so you should be able to do it without too much pain to solve your immediate problem [12:55:59] akosiaris: definitely I agree, we don't directly control the emptyDir, it is managed by kserve, I can try to ask if there is another way