Resource quotas prevent resource contention issues by rationing available resources (cpu, memory, gpu ..). Without these controls one rogue application can potentially make the cluster crawl.
Resource quotas provide constraints that limit resource consumption.
They can be applied only at the namespace level, which means they can be applied to computing resources and limit the number of objects inside the namespace.
In Kubernetes quotas are implemented with a ResourceQuota object. It can be used to limit/control the following objects:
- ConfigMaps
- Persistent Volume Claims (PVCs)
- Pods
- Secrets
- Services
Create quotas
- Check:
Apply quota to a namespace
❯ kubectl create -f compute-resources.yaml -n test-ns resourcequota/compute-resources created ❯ kubectl create -f object-counts.yaml -n test-ns resourcequota/object-counts created Get quota details and status
❯ kubectl get resourcequotas -n test-ns NAME AGE REQUEST LIMIT compute-resources 22s requests.cpu: 0/10, requests.memory: 0/40Gi, requests.nvidia.com/gpu: 0/2 limits.cpu: 0/15, limits.memory: 0/60Gi object-counts 11s configmaps: 1/20, persistentvolumeclaims: 0/20, pods: 0/50, replicationcontrollers: 0/20, secrets: 1/20, services: 0/20, services.loadbalancers: 0/4, services.nodeports: 0/4 Dele quotas
❯ kubectl delete resourcequota/compute-resources resourcequota/object-counts resourcequota "compute-resources" deleted resourcequota "object-counts" deleted Tests
❯ for i in $(seq 1 5); do kubectl create deployment test-dep-${i} --image=alpine --replicas=1 -n test-ns kubectl expose deployment test-dep-${i} --type=NodePort --port=80 -n test-ns done deployment.apps/test-dep-1 created service/test-dep-1 exposed deployment.apps/test-dep-2 created service/test-dep-2 exposed deployment.apps/test-dep-3 created service/test-dep-3 exposed deployment.apps/test-dep-4 created service/test-dep-4 exposed deployment.apps/test-dep-5 created Error from server (Forbidden): services "test-dep-5" is forbidden: exceeded quota: object-counts, requested: services.nodeports=1, used: services.nodeports=4, limited: services.nodeports=4 Side effect:
Even the pods are also not created as there are no memory, cpu requirements given
❯ kubectl get deployments.apps test-dep-1 -n test-ns NAME READY UP-TO-DATE AVAILABLE AGE test-dep-1 0/1 0 0 5m17s ❯ kubectl get rs test-dep-1-85d7f5dcdb -n test-ns NAME DESIRED CURRENT READY AGE test-dep-1-85d7f5dcdb 1 0 0 5m44s ❯ kubectl describe rs test-dep-1-85d7f5dcdb -n test-ns | tail -2 Warning FailedCreate 4m40s replicaset-controller Error creating: pods "test-dep-1-85d7f5dcdb-jh62r" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory Warning FailedCreate 118s (x7 over 4m39s) replicaset-controller (combined from similar events): Error creating: pods "test-dep-1-85d7f5dcdb-6679m" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory We add resource limits:
... spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 resources: limits: memory: '1Gi' cpu: '800m' requests: memory: '256Mi' cpu: '100m' Rerun the deployments (replicaCount=3)
❯ for i in $(seq 1 5); do sed -e "s/nginx-dep/nginx-dep-${i}/" /tmp/dep.yaml | kubectl apply -f - done deployment.apps/nginx-dep-1 created deployment.apps/nginx-dep-2 created deployment.apps/nginx-dep-3 created deployment.apps/nginx-dep-4 created deployment.apps/nginx-dep-5 created ❯ k get deployments.apps -n test-ns NAME READY UP-TO-DATE AVAILABLE AGE nginx-dep-1 3/3 3 3 102s nginx-dep-2 3/3 3 3 102s nginx-dep-3 3/3 3 3 102s nginx-dep-4 3/3 3 3 101s nginx-dep-5 3/3 3 3 101s ❯ k get resourcequotas -n test-ns NAME AGE REQUEST LIMIT compute-resources 22m requests.cpu: 1500m/10, requests.memory: 3840Mi/40Gi, requests.nvidia.com/gpu: 0/2 limits.cpu: 12/15, limits.memory: 15Gi/60Gi object-counts 22m configmaps: 1/20, persistentvolumeclaims: 0/20, pods: 15/50, replicationcontrollers: 0/20, secrets: 1/20, services: 0/20, services.loadbalancers: 0/4, services.nodeports: 0/4 Let us push beyond cpu limits
❯ for i in $(seq 6 10); do sed -e "s/nginx-dep/nginx-dep-${i}/" /tmp/dep.yaml | kubectl apply -f - deployment.apps/nginx-dep-6 created deployment.apps/nginx-dep-7 created deployment.apps/nginx-dep-8 created deployment.apps/nginx-dep-9 created deployment.apps/nginx-dep-10 created done ❯ k get rs -n test-ns NAME DESIRED CURRENT READY AGE ... nginx-dep-6-7fbd56f7d8 3 3 3 7m35s nginx-dep-7-7fbd56f7d8 3 0 0 3m51s nginx-dep-8-7fbd56f7d8 3 0 0 3m39s nginx-dep-9-7fbd56f7d8 3 0 0 3m39s It creates 6 deployments and stalls for deployment-7 .. 10
❯ kubectl describe rs nginx-dep-7-7fbd56f7d8 -n test-ns | tail -2 Warning FailedCreate 4m14s replicaset-controller Error creating: pods "nginx-dep-7-7fbd56f7d8-bv7hb" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=800m, used: limits.cpu=14400m, limited: limits.cpu=15 Warning FailedCreate 92s (x7 over 4m13s) replicaset-controller (combined from similar events): Error creating: pods "nginx-dep-7-7fbd56f7d8-54shl" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=800m, used: limits.cpu=14400m, limited: limits.cpu=15 Setting boundary values for cpu and memory
Use the LimitRange resource
Top comments (1)
Namespaces create virtual clusters within a physical Kubernetes cluster to help users avoid resource naming conflicts and manage capacity, among others. Prayer to separate two people