k3s - images keep getting deleted

[cwrx777]

k3s v1.29+1

Hi all,

Why does my image keep getting deleted after it is imported

# after ctr image import
time="2024-03-02T09:06:28.107014894Z" level=info msg="ImageCreate event name:\"localhost/fakerepo:tag\""
time="2024-03-02T09:06:28.125476965Z" level=info msg="ImageCreate event name:\"sha256:70b8b0b10e0e9f93fb547f73f2d98c4e3b28b1e1d2ea0b7f91f5410ccd533df2\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}"
time="2024-03-02T09:06:28.127199394Z" level=info msg="ImageUpdate event name:\"localhost/fakerepo:tag\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}"
# automatically removed
time="2024-03-02T09:15:10.001322800Z" level=info msg="RemoveImage \"sha256:70b8b0b10e0e9f93fb547f73f2d98c4e3b28b1e1d2ea0b7f91f5410ccd533df2\""
time="2024-03-02T09:15:10.003452537Z" level=info msg="ImageDelete event name:\"localhost/fakerepo:tag\""
time="2024-03-02T09:15:10.004975632Z" level=info msg="ImageDelete event name:\"sha256:70b8b0b10e0e9f93fb547f73f2d98c4e3b28b1e1d2ea0b7f91f5410ccd533df2\""
time="2024-03-02T09:15:14.312586076Z" level=info msg="RemoveImage \"sha256:70b8b0b10e0e9f93fb547f73f2d98c4e3b28b1e1d2ea0b7f91f5410ccd533df2\" returns successfully"

[brandond]

Are you low on disk space?

https://kubernetes.io/docs/concepts/architecture/garbage-collection/#containers-images

See the GC threshold options at:
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

[ANBUZHIDAO]

Hi, I use v1.22.9+k3s1. I config image-gc-high-threshold to 100 or 99, then i copy large file to make disk usage exceed 95% and not exceed 97%, it still delete unused image, why ?

/etc/rancher/k3s/config.yaml
`
kubelet-arg:

• image-gc-high-threshold=100
• image-gc-low-threshold=99
  `

log in /var/log/messages:
Apr 28 21:01:05 node1 k3s: time="2024-04-28T21:01:05+08:00" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/data/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/995f5a281daabc1838b33f2346f7c4976b95f449c703b6f1f55b981966eba456/bin --cni-conf-dir=/data/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=datameter --image-gc-high-threshold=100 --image-gc-low-threshold=99 --kubeconfig=/data/k3s/agent/kubelet.kubeconfig --node-labels= --pod-manifest-path=/data/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --tls-cert-file=/data/k3s/agent/serving-kubelet.crt --tls-private-key-file=/data/k3s/agent/serving-kubelet.key"

[ANBUZHIDAO]

I change the log level to 5, and restart k3s, and check the source code.
The image get deleted because DeleteUnusedImages function. DeleteUnusedImages was invoked because delete unused containers.
So the image get deleted because evict not gc.
How can i change config to keep image not deleted ?

[brandond]

These questions are all just about core Kubernetes functionality, not anything that is custom to K3s. I'm confused why you're asking here.

[ANBUZHIDAO]

Very Sorry. Because I used k3s. At first, I thought it was because the GC settings of K3s were not effective, but after research, I found out that it was core Kubernetes functionality. Please just ignore my question。