DEV Community

Hitesh Pattanayak
Hitesh Pattanayak

Posted on

Managing Kubernetes Resources with Resource Quotas

#kubernetes #k8s #resourcequota #deployment

Resource quota

When several users or teams share a cluster with a fixed number of nodes, there is a concern that one team could use more than its fair share of resources.

Resource quotas are a tool for administrators to address this concern.

A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. It can limit the quantity of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that namespace.

Resource quotas work like this:

Different teams work in different namespaces. This can be enforced with RBAC.

The administrator creates one ResourceQuota for each namespace.

Users create resources (pods, services, etc.) in the namespace, and the quota system tracks usage to ensure it does not exceed hard resource limits defined in a ResourceQuota.

If creating or updating a resource violates a quota constraint, the request will fail with HTTP status code 403 FORBIDDEN with a message explaining the constraint that would have been violated.

If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use the LimitRanger admission controller to force defaults for pods that make no compute resource requirements.

Explanation of resource_quota.yaml file

This is a YAML file containing two Kubernetes objects.

apiVersion: v1 kind: Namespace metadata: name: mynamespace --- apiVersion: v1 kind: List items: - apiVersion: v1 kind: ResourceQuota metadata: name: user-compute-quota namespace: mynamespace spec: hard: requests.cpu: "1" requests.memory: 1Gi limits.cpu: "2" limits.memory: 2Gi - apiVersion: v1 kind: ResourceQuota metadata: name: user-object-quota namespace: mynamespace spec: hard: configmaps: "10" persistentvolumeclaims: "4" replicationcontrollers: "20" secrets: "10" services: "10" services.loadbalancers: "2"
Enter fullscreen mode Exit fullscreen mode

The first object is of kind "Namespace" and has a metadata field with the name "mynamespace". This is creating a namespace named "mynamespace" in Kubernetes.

The second object is of kind "List" and contains a list of two "ResourceQuota" objects. The first "ResourceQuota" object has a metadata field with the name "user-compute-quota" and namespace set to "mynamespace". The "spec" field defines resource usage limits for the namespace, such as CPU and memory requests and limits. Specifically, this quota allows for a maximum of 1 CPU and 1Gi of memory for requests and a maximum of 2 CPUs and 2Gi of memory for limits.

The second "ResourceQuota" object has a metadata field with the name "user-object-quota" and namespace set to "mynamespace". The "spec" field defines limits on the number of objects of different types that can be created in the namespace. Specifically, this quota allows for a maximum of 10 configmaps, 4 persistentvolumeclaims, 20 replicationcontrollers, 10 secrets, 10 services, and 2 load balancer services.

Steps to create ‘Resource Quota’ objects and observe their working

  1. Create resource quota objects: 
kubectl apply -f resource_quota.yaml
Enter fullscreen mode Exit fullscreen mode
  1. Verify created resource quota objects: 
 kubectl get quota -n mynamespace NAME AGE REQUEST LIMIT user-compute-quota 79s requests.cpu: 0/1, requests.memory: 0/1Gi limits.cpu: 0/2, limits.memory: 0/2Gi user-object-quota 44s configmaps: 1/10, persistentvolumeclaims: 0/4, replicationcontrollers: 0/20, secrets: 0/10, services: 0/10, services.loadbalancers: 0/2
Enter fullscreen mode Exit fullscreen mode
  1. Create deployment (dep_without_quota.yaml) 
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment namespace: mynamespace labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: k8s-demo image: nginx ports: - name: nginx-port containerPort: 8080
Enter fullscreen mode Exit fullscreen mode 
kubectl create -f dep_without_quota.yaml
Enter fullscreen mode Exit fullscreen mode
  1. Verify deployment 
kubectl get deploy -n mynamespace NAME READY UP-TO-DATE AVAILABLE AGE nginx-deployment 0/3 0 0 31s
Enter fullscreen mode Exit fullscreen mode

nothing in ready state.

  1. Explore reason 
kubectl get rs -n mynamespace NAME DESIRED CURRENT READY AGE nginx-deployment-64ff9dbcdf 3 0 0 2m51s kubectl describe rs nginx-deployment-64ff9dbcdf ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-wkr8s" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-mhn6c" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-77n7m" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-t5m7q" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-j6974" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-ldtx8" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-jzfqg" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo Warning FailedCreate 3m8s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-xrp62" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo Warning FailedCreate 3m8s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-s29ht" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo Warning FailedCreate 25s (x7 over 3m6s) replicaset-controller (combined from similar events): Error creating: pods "nginx-deployment-64ff9dbcdf-9gxvc" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
Enter fullscreen mode Exit fullscreen mode

since quota is not specified in the deployment file, it fails to provision the pods.

  1. Create deployment (dep_with_quota.yaml) 
apiVersion: apps/v1 kind: Deployment metadata: name: helloworld-deployment namespace: mynamespace labels: app: nginx-ssl spec: replicas: 3 selector: matchLabels: app: nginx-ssl template: metadata: labels: app: nginx-ssl spec: containers: - name: k8s-demo image: nginx:1.16 ports: - name: nginxssl-port containerPort: 8081 resources: requests: cpu: 200m memory: 0.5Gi limits: cpu: 400m memory: 1Gi
Enter fullscreen mode Exit fullscreen mode 
kubectl create -f dep_with_quota.yaml
Enter fullscreen mode Exit fullscreen mode
  1. Verify deployment 
kubectl get rs -n mynamespace NAME DESIRED CURRENT READY AGE helloworld-deployment-7986cbf64d 3 2 2 45s nginx-deployment-64ff9dbcdf 3 0 0 8m52s
Enter fullscreen mode Exit fullscreen mode

only 2 out of 3 are in ready state.

  1. Explore reason 
kubectl get rs -n mynamespace NAME DESIRED CURRENT READY AGE helloworld-deployment-7986cbf64d 3 2 2 45s nginx-deployment-64ff9dbcdf 3 0 0 8m52s kubectl describe rs helloworld-deployment-7986cbf64d -n mynamespace Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 59s replicaset-controller Created pod: helloworld-deployment-7986cbf64d-k46hw Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-cxjqx" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi Normal SuccessfulCreate 59s replicaset-controller Created pod: helloworld-deployment-7986cbf64d-r68vf Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-sx2vj" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-zx6pq" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-khchm" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-z8j4s" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-7rqj4" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-sb8ml" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-pxsdv" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi Warning FailedCreate 58s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-jdnfp" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi Warning FailedCreate 39s (x8 over 57s) replicaset-controller (combined from similar events): Error creating: pods "helloworld-deployment-7986cbf64d-pjms2" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
Enter fullscreen mode Exit fullscreen mode

failed to provision 3rd pod as it exceeded quota

  1. How much quota exhausted: 
kubectl describe quota user-compute-quota -n mynamespace Name: user-compute-quota Namespace: mynamespace Resource Used Hard -------- ---- ---- limits.cpu 800m 2 limits.memory 2Gi 2Gi requests.cpu 400m 1 requests.memory 1Gi 1Gi
Enter fullscreen mode Exit fullscreen mode

And they consumed resources as specified in resources specs of the helloworld deployment.

  1. Create a limit ranger object: 
apiVersion: v1 kind: LimitRange metadata: name: limits-quota namespace: mynamespace spec: limits: - default: cpu: 200m memory: 512Mi defaultRequest: cpu: 100m memory: 256Mi type: Container
Enter fullscreen mode Exit fullscreen mode 
kubectl create -f limit_range.yaml
Enter fullscreen mode Exit fullscreen mode
  1. Now retry dep_without_quota.yaml 
kubectl create -f dep_without_quota.yaml
Enter fullscreen mode Exit fullscreen mode
  1. Verify again dep_without_quota replicas/pods: 
kubectl get deploy -n mynamespace NAME READY UP-TO-DATE AVAILABLE AGE nginx-deployment 3/3 3 3 13s
Enter fullscreen mode Exit fullscreen mode

All 3 are in ready states

  1. Finally check the consumption of quota 
kubectl describe quota user-compute-quota -n mynamespace Name: user-compute-quota Namespace: mynamespace Resource Used Hard -------- ---- ---- limits.cpu 600m 2 limits.memory 1536Mi 2Gi requests.cpu 300m 1 requests.memory 768Mi 1Gi
Enter fullscreen mode Exit fullscreen mode

All 3 pods got scheduled using default limits specified in limit_range.yaml This is because the deployment ‘nginx-deployment’ does not have resources spec specified hence the pods created out of it use default specs from LimitRange object.

Folks, if you like my content, would you consider following me on linked in at: https://www.linkedin.com/in/hitesh-pattanayak-52290b160/

Top comments (0)