Using NVIDIA GPU Operator with EKS Anywhere

The NVIDIA GPU Operator allows GPUs to be exposed to applications in Kubernetes clusters much like CPUs. Instead of provisioning a special OS image for GPU nodes with the required drivers and dependencies, a standard OS image can be used for both CPU and GPU nodes. The NVIDIA GPU Operator can be used to provision the required software components for GPUs such as the NVIDIA drivers, Kubernetes device plugin for GPUs, and the NVIDIA Container Toolkit. See the licensing section of the NVIDIA GPU Operator documentation for information on the NVIDIA End User License Agreements.

In the example on this page, a single-node EKS Anywhere cluster on bare metal is used with an Ubuntu 20.04 image produced from image-builder without modifications and Kubernetes version 1.31.

1. Configure an EKS Anywhere cluster spec and hardware inventory

See the Configure for Bare Metal page and the Prepare hardware inventory page for details. If you use cluster spec sample below is used, your hardware inventory definition must have type=cp for the labels field in the hardware inventory for your server.

apiVersion: anywhere.eks.amazonaws.com/v1alpha1 kind: Cluster metadata: name: gpu-test spec: clusterNetwork: cniConfig: cilium: {} pods: cidrBlocks: - 192.168.0.0/16 services: cidrBlocks: - 10.96.0.0/12 controlPlaneConfiguration: count: 1 endpoint: host: "<my-cp-ip>" machineGroupRef: kind: TinkerbellMachineConfig name: gpu-test-cp datacenterRef: kind: TinkerbellDatacenterConfig name: gpu-test kubernetesVersion: "1.31" --- apiVersion: anywhere.eks.amazonaws.com/v1alpha1 kind: TinkerbellDatacenterConfig metadata: name: gpu-test spec: tinkerbellIP: "<my-tb-ip>" osImageURL: "https://<url-for-image>/ubuntu.gz" --- apiVersion: anywhere.eks.amazonaws.com/v1alpha1 kind: TinkerbellMachineConfig metadata: name: gpu-test-cp spec: hardwareSelector: {type: "cp"} osFamily: ubuntu templateRef: {} 

2. Create a single-node EKS Anywhere cluster

• Replace hardware.csv with the name of your hardware inventory file
• Replace cluster.yaml with the name of your cluster spec file

eksctl anywhere create cluster --hardware hardware.csv -f cluster.yaml 

Warning: The recommended number of control plane nodes is 3 or 5 Warning: No configurations provided for worker node groups, pods will be scheduled on control-plane nodes Performing setup and validations Private key saved to gpu-test/eks-a-id_rsa. Use 'ssh -i gpu-test/eks-a-id_rsa <username>@<Node-IP-Address>' to login to your cluster node ✅ Tinkerbell Provider setup is valid ✅ Validate OS is compatible with registry mirror configuration ✅ Validate certificate for registry mirror ✅ Validate authentication for git provider Creating new bootstrap cluster Provider specific pre-capi-install-setup on bootstrap cluster Installing cluster-api providers on bootstrap cluster Provider specific post-setup Creating new workload cluster Installing networking on workload cluster Creating EKS-A namespace Installing cluster-api providers on workload cluster Installing EKS-A secrets on workload cluster Installing resources on management cluster Moving cluster management from bootstrap to workload cluster Installing EKS-A custom components (CRD and controller) on workload cluster Installing EKS-D components on workload cluster Creating EKS-A CRDs instances on workload cluster Installing GitOps Toolkit on workload cluster GitOps field not specified, bootstrap flux skipped Writing cluster config file Deleting bootstrap cluster 🎉 Cluster created! 

3. Install Helm

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \  && chmod 700 get_helm.sh \  && ./get_helm.sh 

4. Add NVIDIA Helm Repository

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \  && helm repo update 

5. Configure kubectl to use EKS Anywhere cluster

• Replace <path-to-cluster-folder> with the directory location where your EKS Anywhere cluster folder is located. This is typically in the same directory in which the eksctl anywhere command was run.
• Replace <cluster-name> with the name of your cluster.

KUBECONFIG=<path-to-cluster-folder>/<cluster-name>-eks-a-cluster.kubeconfig 

6. Install NVIDIA GPU Operator

helm install --wait --generate-name \  -n gpu-operator --create-namespace \  nvidia/gpu-operator 

7. Validate the operator was installed successfully

kubectl get pods -n gpu-operator 

NAME READY STATUS RESTARTS AGE gpu-feature-discovery-6djnw 1/1 Running 0 5m25s gpu-operator-1691443998-node-feature-discovery-master-55cfkzbl5 1/1 Running 0 5m55s gpu-operator-1691443998-node-feature-discovery-worker-dw8m7 1/1 Running 0 5m55s gpu-operator-59f96d7646-7zcn4 1/1 Running 0 5m55s nvidia-container-toolkit-daemonset-c2mdf 1/1 Running 0 5m25s nvidia-cuda-validator-6m4kg 0/1 Completed 0 3m41s nvidia-dcgm-exporter-jw5wz 1/1 Running 0 5m25s nvidia-device-plugin-daemonset-8vjrn 1/1 Running 0 5m25s nvidia-driver-daemonset-6hklg 1/1 Running 0 5m36s nvidia-operator-validator-2pvzx 1/1 Running 0 5m25s 

8. Validate GPU specs

kubectl get node -o json | jq '.items[].metadata.labels' 

{ ... "nvidia.com/cuda.driver.major": "535", "nvidia.com/cuda.driver.minor": "86", "nvidia.com/cuda.driver.rev": "10", "nvidia.com/cuda.runtime.major": "12", "nvidia.com/cuda.runtime.minor": "2", "nvidia.com/gfd.timestamp": "1691444179", "nvidia.com/gpu-driver-upgrade-state": "upgrade-done", "nvidia.com/gpu.compute.major": "7", "nvidia.com/gpu.compute.minor": "5", "nvidia.com/gpu.count": "2", "nvidia.com/gpu.deploy.container-toolkit": "true", "nvidia.com/gpu.deploy.dcgm": "true", "nvidia.com/gpu.deploy.dcgm-exporter": "true", "nvidia.com/gpu.deploy.device-plugin": "true", "nvidia.com/gpu.deploy.driver": "true", "nvidia.com/gpu.deploy.gpu-feature-discovery": "true", "nvidia.com/gpu.deploy.node-status-exporter": "true", "nvidia.com/gpu.deploy.nvsm": "", "nvidia.com/gpu.deploy.operator-validator": "true", "nvidia.com/gpu.family": "turing", "nvidia.com/gpu.machine": "PowerEdge-R7525", "nvidia.com/gpu.memory": "15360", "nvidia.com/gpu.present": "true", "nvidia.com/gpu.product": "Tesla-T4", "nvidia.com/gpu.replicas": "1", "nvidia.com/mig.capable": "false", "nvidia.com/mig.strategy": "single" } 

9. Run Sample App

Create a gpu-pod.yaml file with the following and apply it to the cluster

apiVersion: v1  kind: Pod  metadata:  name: gpu-pod  spec:  restartPolicy: Never   containers:  - name: cuda-container   image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2   resources:  limits:  nvidia.com/gpu: 1 # requesting 1 GPU   tolerations:  - key: nvidia.com/gpu operator: Exists   effect: NoSchedule 

kubectl apply -f gpu-pod.yaml 

10. Confirm Sample App Succeeded

kubectl logs gpu-pod 

[Vector addition of 50000 elements] Copy input data from the host memory to the CUDA device CUDA kernel launch with 196 blocks of 256 threads Copy output data from the CUDA device to the host memory Test PASSED Done