DEV Community

Cover image for Kubernetes Storage Playlist - Part 4: Implementing Amazon S3 Storage with EKS using Terraform and and Kubernetes Manifests
AWS Community Builders profile image Chinmay Tonape
Chinmay Tonape for AWS Community Builders

Posted on

Kubernetes Storage Playlist - Part 4: Implementing Amazon S3 Storage with EKS using Terraform and and Kubernetes Manifests

#kubernetes #aws #eks #s3

In this blog, we’ll explore how to integrate Amazon S3 as a storage solution with Amazon EKS using Terraform and Kubernetes YAML manifests. We will run a simple Nginx container that serves website files stored in an S3 bucket.

This approach leverages the Mountpoint for S3 CSI driver, which provides Kubernetes workloads access to Amazon S3 objects using standard POSIX interfaces.

Understanding Amazon S3 for Kubernetes

What is Amazon S3?

Amazon Simple Storage Service (Amazon S3) is an object storage service designed for scalability, durability, and availability. Unlike traditional block storage (like EBS) or file storage (like EFS), S3 stores data as objects inside buckets, which makes it ideal for static content, logs, and backups.

Architecture Diagram

The architecture of attaching Amazon S3 storage to an EKS cluster revolves around the S3 CSI (Container Storage Interface) driver and the Mountpoint for S3 integration.

At the base layer, an S3 bucket acts as the backend object store where application data is kept. Inside Kubernetes, a StorageClass defines how storage is provisioned and consumed. With S3, however, only static provisioning is currently supported—meaning a PersistentVolume (PV) must be manually created and mapped to an existing S3 bucket, and then a PersistentVolumeClaim (PVC) binds to that PV so workloads can access it.

Once a pod is deployed, Kubernetes mounts the PVC to the container’s filesystem through the S3 CSI driver, which internally uses Mountpoint for S3 to provide file system-like access to objects in the bucket.

While dynamic provisioning—where PVCs automatically trigger creation of new storage volumes—is common for drivers like EBS and EFS, it is not yet available for S3. This makes static provisioning the only option, requiring administrators to pre-define the mapping between PVs and S3 buckets. Security is handled through IAM Roles for Service Accounts (IRSA), ensuring pods have only the minimum required permissions to access specific S3 buckets.

Key Benefits of Using S3 with EKS

  • Scalability: Virtually unlimited storage capacity.
  • Durability: 11 9’s durability guarantees.
  • Cost-Effective: Pay for what you use with no upfront provisioning.
  • Integration: Easy integration with AWS services like Athena, Glue, and CloudFront.

Important Considerations for EKS

  • CSI Driver Requirements: The Mountpoint for S3 CSI driver must be installed.
  • Pod Identity Support: Currently, IRSA (IAM Roles for Service Accounts) is required. Pod Identity is not supported yet.
  • Limitations: Only static provisioning is supported as of now. Dynamic provisioning is on the roadmap.
  • Resource Quotas: Watch for limits like open file descriptors and network throughput when scaling workloads.

Step 1: Provisioning EKS Cluster with Terraform in a VPC

The first step in integrating Amazon S3 with EKS is to provision a Kubernetes cluster that runs securely inside a dedicated VPC. We will use the widely adopted AWS Terraform community modules for both the VPC and EKS setup. Please refer to main module of GitHub repo.

Step 2: Creating the S3 bucket

  • S3 Bucket: Encrypted S3 bucket with versioning and permissions 
#################################################################################### # S3 Bucket for Kubernetes Storage #################################################################################### resource "aws_s3_bucket" "main" { bucket = var.bucket_name tags = { Name = var.bucket_name Environment = var.environment Terraform = "true" } } #################################################################################### # S3 Bucket Versioning #################################################################################### resource "aws_s3_bucket_versioning" "main" { bucket = aws_s3_bucket.main.id versioning_configuration { status = var.versioning_enabled ? "Enabled" : "Disabled" } } #################################################################################### # S3 Bucket Encryption #################################################################################### resource "aws_s3_bucket_server_side_encryption_configuration" "main" { bucket = aws_s3_bucket.main.id rule { apply_server_side_encryption_by_default { sse_algorithm = "AES256" } bucket_key_enabled = true } } #################################################################################### # S3 Bucket Public Access Block #################################################################################### resource "aws_s3_bucket_public_access_block" "main" { bucket = aws_s3_bucket.main.id block_public_acls = true block_public_policy = true ignore_public_acls = true restrict_public_buckets = true }
Enter fullscreen mode Exit fullscreen mode
  • IAM Role: For Mountpoint for S3 CSI Driver 
#################################################################################### # IAM Role for S3 CSI Driver (IRSA) #################################################################################### resource "aws_iam_role" "s3_csi_driver_role" { name = "${var.cluster_name}-s3-csi-driver-role" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Principal = { Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}" } Action = "sts:AssumeRoleWithWebIdentity" Condition = { StringEquals = { "${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}:sub" = "system:serviceaccount:kube-system:s3-csi-driver-sa" "${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}:aud" = "sts.amazonaws.com" } } } ] }) tags = { Name = "${var.cluster_name}-s3-csi-driver-role" Environment = var.environment Terraform = "true" } } #################################################################################### # S3 CSI Driver Policy (Based on AWS Documentation) #################################################################################### resource "aws_iam_policy" "s3_csi_driver_policy" { name = "${var.cluster_name}-s3-csi-driver-policy" description = "IAM policy for S3 CSI Driver based on AWS documentation" policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Action = [ "s3:ListBucket" ] Resource = aws_s3_bucket.main.arn }, { Effect = "Allow" Action = [ "s3:GetObject", "s3:PutObject", "s3:AbortMultipartUpload", "s3:DeleteObject" ] Resource = "${aws_s3_bucket.main.arn}/*" } ] }) tags = { Name = "${var.cluster_name}-s3-csi-driver-policy" Environment = var.environment Terraform = "true" } } #################################################################################### # Attach S3 CSI Driver Policy #################################################################################### resource "aws_iam_role_policy_attachment" "s3_csi_driver_policy" { role = aws_iam_role.s3_csi_driver_role.name policy_arn = aws_iam_policy.s3_csi_driver_policy.arn }
Enter fullscreen mode Exit fullscreen mode
  • Mountpoint for S3 Add-on: AWS Mountpoint for S3 CSI Driver for Kubernetes integration 
#################################################################################### ### S3 Mountpoint CSI Driver Addon (deployed after S3 module) #################################################################################### resource "aws_eks_addon" "s3_mountpoint_csi_driver" { cluster_name = module.eks.cluster_name addon_name = "aws-mountpoint-s3-csi-driver" service_account_role_arn = module.s3.s3_csi_driver_role_arn # Using IRSA for CSI driver, Pod Identity for application pods # Ensure this addon is created after the S3 module creates the IAM role and pod identity association depends_on = [module.s3] tags = { Name = "${var.cluster_name}-s3-mountpoint-csi-driver" Environment = var.environment Terraform = "true" } }
Enter fullscreen mode Exit fullscreen mode

Step 3: S3 Storage Implementation Patterns

Since only static provisioning is supported for S3 today, we’ll define PersistentVolumes (PV) that reference an S3 bucket.

  • storage-class.yaml - StorageClass for S3 CSI driver (requires ${S3_BUCKET_NAME} placeholder) 
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: s3-csi-sc provisioner: s3.csi.aws.com parameters: bucketName: ${S3_BUCKET_NAME} prefix: "k8s-storage/" volumeBindingMode: Immediate allowVolumeExpansion: false
Enter fullscreen mode Exit fullscreen mode
  • persistent-volume.yaml - PersistentVolume for S3 bucket (static provisioning only) 
apiVersion: v1 kind: PersistentVolume metadata: name: s3-pv spec: capacity: storage: 1Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain storageClassName: s3-csi-sc csi: driver: s3.csi.aws.com volumeHandle: s3-csi-driver-volume volumeAttributes: bucketName: ${S3_BUCKET_NAME} prefix: "k8s-storage/"
Enter fullscreen mode Exit fullscreen mode
  • persistent-volume-claim.yaml - PVC using S3 CSI driver 
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: s3-pvc spec: accessModes: - ReadWriteMany storageClassName: s3-csi-sc resources: requests: storage: 1Gi
Enter fullscreen mode Exit fullscreen mode
  • nginx-pod.yaml - Nginx pod with S3 storage mounted and content creation 
apiVersion: v1 kind: Pod metadata: name: nginx-s3-pod namespace: default labels: app: nginx-s3 spec: securityContext: runAsUser: 0 runAsGroup: 0 containers: - name: nginx image: nginx:latest ports: - containerPort: 80 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name command: ["/bin/sh", "-c"] args: - | echo '<h1>Hello from S3 Mountpoint!</h1><p><b>Pod:</b> '$POD_NAME'</p>' > /usr/share/nginx/html/index.html || true sed -i 's/user nginx;/user root;/' /etc/nginx/nginx.conf nginx -g 'daemon off;' volumeMounts: - name: s3-storage mountPath: /usr/share/nginx/html volumes: - name: s3-storage persistentVolumeClaim: claimName: s3-pvc
Enter fullscreen mode Exit fullscreen mode
  • nginx-service.yaml - ClusterIP service to expose nginx on port 80 
apiVersion: v1 kind: Service metadata: name: nginx-s3-service spec: type: ClusterIP ports: - port: 80 targetPort: 80 selector: app: nginx-s3
Enter fullscreen mode Exit fullscreen mode

Deployment Steps For Static Provisioning:

  1. Get S3 bucket name from Terraform: 
cd infrastructure S3_BUCKET_NAME=$(terraform output -raw s3_bucket_name 2>/dev/null || echo "")
Enter fullscreen mode Exit fullscreen mode
  1. Update manifests with S3 values: 
sed "s/\${S3_BUCKET_NAME}/$S3_BUCKET_NAME/g" storage-class.yaml > storage-class-final.yaml sed "s/\${S3_BUCKET_NAME}/$S3_BUCKET_NAME/g" persistent-volume.yaml > persistent-volume-final.yaml
Enter fullscreen mode Exit fullscreen mode
  1. Apply manifests: 
kubectl apply -f storage-class-final.yaml kubectl apply -f persistent-volume-final.yaml kubectl apply -f persistent-volume-claim.yaml kubectl apply -f nginx-pod.yaml kubectl apply -f nginx-service.yaml
Enter fullscreen mode Exit fullscreen mode

Refer to deploy.sh for automated deployment script

$ kubectl get sc,pv,pvc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE storageclass.storage.k8s.io/s3-csi-sc s3.csi.aws.com Delete Immediate false 10m NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE persistentvolume/s3-pv 1Gi RWX Retain Bound default/s3-pvc s3-csi-sc <unset> 10m NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE persistentvolumeclaim/s3-pvc Bound s3-pv 1Gi RWX s3-csi-sc <unset> 10m
Enter fullscreen mode Exit fullscreen mode

S3 Bucket:

S3 Bucket Contents:

Nginx pod accessing S3 for index.html

Verification 

# Check S3 CSI driver status: kubectl get pods -n kube-system -l app=aws-mountpoint-s3-csi-driver # Check deployment status: kubectl get pod nginx-s3-pod kubectl get service nginx-s3-service kubectl get pvc s3-pvc kubectl get pv # Test nginx web server: kubectl port-forward service/nginx-s3-service 8084:80 # Then visit http://localhost:8084 # Check S3 File Persistence kubectl exec nginx-s3-pod -- ls -la /usr/share/nginx/html/ kubectl exec nginx-s3-pod -- cat /usr/share/nginx/html/index.html # Check S3 bucket content: aws s3 ls s3://$S3_BUCKET_NAME/k8s-storage/
Enter fullscreen mode Exit fullscreen mode

Cleanup 

kubectl delete -f nginx-service.yaml kubectl delete -f nginx-pod.yaml kubectl delete -f persistent-volume-claim.yaml kubectl delete -f persistent-volume.yaml kubectl delete -f storage-class.yaml
Enter fullscreen mode Exit fullscreen mode

And then terraform destroy the EKS infrastructure if you are not using it to save costs.

Conclusion

Amazon S3 provides a scalable and cost-effective storage solution for workloads running on EKS. With the Mountpoint for S3 CSI driver, Kubernetes pods can directly mount S3 buckets and serve static files seamlessly. While currently limited to static provisioning, this integration is a powerful way to manage object storage in containerized environments.

In this blog, we provisioned an EKS cluster using Terraform, set up IAM roles with IRSA, deployed the S3 CSI driver, and ran an Nginx container backed by S3 storage.

References

Top comments (0)