5

I have a single-node kubernetes cluster and I got a notification that disk usage was over 70%. When I get a shell on the node and run df I see that / is using 75% of the available blocks. I also see a number of entries like this:

overlay 51432064 36953456 12350764 75% /run/containerd/io.containerd.runtime.v2.task/k8s.io/47d405a81214ec9b81ccdffae3674260fa2946d30bdd3b398aa1c2d283c8577e/rootfs overlay 51432064 36953456 12350764 75% /run/containerd/io.containerd.runtime.v2.task/k8s.io/569d1fe262f6bf1a59ca99163f231a426d2251ac1beabc47a0393066f48b3e0e/rootfs overlay 51432064 36953456 12350764 75% /run/containerd/io.containerd.runtime.v2.task/k8s.io/32d14863f46e922b7e0b2bbee8a5321c5fabb5811352449d90e486b94161a8cb/rootfs overlay 51432064 36953456 12350764 75% /run/containerd/io.containerd.runtime.v2.task/k8s.io/8c3fa2f9467e792036782e9587f748f846d61fdc23d9afaecc7e7835e6db4f2b/rootfs overlay 51432064 36953456 12350764 75% /run/containerd/io.containerd.runtime.v2.task/k8s.io/626b21093218edcda466ce6d4f31e7b975a0c0473d233fc062ed657499541fc9/rootfs overlay 51432064 36953456 12350764 75% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c1a7484e1beca4ec1299fc5017002d6fecc9890c86e3246a8bdf53fcc45cf576/rootfs

I have two services that use persistent volumes (not host volumes) and can't think of anything other than docker images that could be using up all of this storage.

How can I figure out what is using this much storage on my kubernetes cluster?

Improve this question

1 Answer 1

Reset to default
8

To find out what is using your Node's disk in Kubernetes, you can follow these steps:

  1. SSH into your Node.

  2. Install the ncdu package on your Node using the following command:

    sudo apt-get install ncdu

  3. Run ncdu on the root directory of your Node using the command:

    sudo ncdu /

  4. Wait for ncdu to finish scanning your Node's disk. It will display a list of directories and their respective sizes.

  5. Use the arrow keys to navigate through the directories and subdirectories to find out which one is taking up the most disk space.

  6. Once you have identified the directory that is taking up the most space, you can use kubectl to find out which pod is using it by running the following command:

    kubectl describe pod <pod-name>

    Replace <pod-name> with the name of the pod you want to check.

  7. Look for the Volume section in the output of the kubectl describe pod command. It will show you the PVC (Persistent Volume Claim) that is mounted on the pod.

  8. Use kubectl to describe the PVC using the following command:

    kubectl describe pvc <pvc-name>

    Replace <pvc-name> with the name of the PVC that was mounted on the pod.

  9. Look for the StorageClass and Access Modes sections in the output of the kubectl describe pvc command. It will tell you which StorageClass the PVC is using and how it is being accessed.

  10. Check the StorageClass configuration to see if there are any storage limits or quotas set on it that might be causing disk usage issues.

By following these steps, you should be able to identify which pod is using the most disk space on your Node in Kubernetes.

Improve this answer
1
  • It looks like the vast majority of my usage is unused images. Hopefully Kubernetes garbage collection will take care of it. Commented Sep 21, 2023 at 13:51

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.