Setting up a robust Kubernetes cluster in a production environment is no small feat. In this article, I’ll walk you through my journey of deploying an RKE2 (Rancher Kubernetes Engine 2) cluster in a vSphere environment using a custom Terraform module. This guide will include configurations, explanations, and a peek into the setup process with screenshots.
Why RKE2 on vSphere?
RKE2 provides a lightweight yet powerful Kubernetes distribution ideal for secure production workloads. When combined with vSphere’s virtualization and Terraform’s Infrastructure-as-Code capabilities, you can achieve a flexible, scalable, and automated deployment process.
Tools and Technologies
Terraform: Automates the provisioning of infrastructure.
RKE2: Kubernetes distribution optimized for production environments.
vSphere: Virtualization platform for deploying VMs.
Ansible: Used for post-deployment configuration.
Prerequisites:
Python 3.x
Ansible, Ansible-Core
sshpass and whois (for password management)
Terraform Module Overview
I built a reusable Terraform module to standardize and automate the Kubernetes cluster provisioning. Here's an overview of the main.tf configuration file that calls the module:
Core Module Features
Multi-AZ Clusters: Enables highly available clusters with master and worker nodes spread across multiple availability zones.
Customizable Resources: Easily configure CPU, memory, and storage for master, worker, and storage nodes.
Built-in RKE2 Installation: Installs RKE2 with a choice of CNI plugins (canal, flannel, etc.).
Networking Configuration: Define Kubernetes service and cluster CIDRs.
Storage Options: Support for local storage and optional NFS integration.
Secure Communication: TLS certificates for domain and API access.
main.tf Breakdown
Here are the main aspects of the main.tf file:
Module Invocation
module "rke2_prod_cluster" { source = "./modules/rke2-provisioner" env = "prod" # Environment name domain = var.domain # Domain name multi_az = true # If you want to create multi-az cluster install_rke2 = true # Install RKE2 lh_storage = true # Local storage for worker nodes hashed_pass = var.hashed_pass # Hashed password for user creation cluster_cidr = var.cluster_cidr # Kubernetes cluster CIDR service_cidr = var.service_cidr # Kubernetes service CIDR nfs_enabled = false # Change to true if you want to enable nfs server update_apt = false # Update apt packages by changing to true rke2_token = var.rke2_token rke2_version = "v1.30.5+rke2r1" rke2_cni = "canal" # Alternatives: flannel, calico, cilium kubevip_range_global = join("-", [cidrhost(var.vm_cidr_az1, 50)], [cidrhost(var.vm_cidr_az1, 60)]) # Global IP range for LoadBalancer IPs kubevip_alb_cidr = "${cidrhost(var.vm_cidr_az1, 20)}/32" # IP for Nginx Ingress Controller Service rke2_api_endpoint = cidrhost(var.vm_cidr_az1, 10) # API Server IP ansible_password = var.ansible_password # Ansible user password domain_crt = var.domain_crt # Domain certificate domain_key = var.domain_key # Domain key domain_root_crt = var.domain_root_crt # Root certificate master_node_count = var.master_node_count_prod worker_node_count = var.worker_node_count_prod storage_node_count = var.storage_node_count_prod # Resources worker_node_cpus = 8 worker_node_memory = 8192 worker_node_disk_size = 100 master_node_cpus = 8 master_node_memory = 8192 master_node_disk_size = 50 storage_node_disk_size = 100 nfs_node_disk_size = 50 # AZ1 master_ip_range_az1 = [for i in range(61, 69) : cidrhost(local.vm_cidr_az1, i)] # Master node IP range worker_ip_range_az1 = [for i in range(71, 79) : cidrhost(local.vm_cidr_az1, i)] # Worker node IP range vsphere_datacenter_az1 = var.vsphere_datacenter_az1 # vSphere datacenter name vsphere_host_az1 = var.vsphere_host_az1 # vSphere host name vsphere_resource_pool_az1 = var.vsphere_resource_pool_az1 # vSphere resource pool name vsphere_datastore_az1 = var.vsphere_datastore_az1 # vSphere datastore name vsphere_network_name_az1 = var.vsphere_network_name_az1 # vSphere network name vm_gw_ip_az1 = local.vm_gw_ip_az1 # Gateway IP nfs_ip_az1 = cidrhost(local.vm_cidr_az1, 70) # NFS server IP # AZ3 master_ip_range_az3 = [for i in range(81, 89) : cidrhost(local.vm_cidr_az3, i)] worker_ip_range_az3 = [for i in range(91, 99) : cidrhost(local.vm_cidr_az3, i)] vsphere_datacenter_az3 = var.vsphere_datacenter_az3 vsphere_host_az3 = var.vsphere_host_az3 vsphere_resource_pool_az3 = var.vsphere_resource_pool_az3 vsphere_datastore_az3 = var.vsphere_datastore_az3 vsphere_network_name_az3 = var.vsphere_network_name_az3 vm_gw_ip_az3 = local.vm_gw_ip_az3 } Deployment Walkthrough
Step 1: Initialize Terraform
Run the following commands to initialize Terraform and apply the configuration:
terraform init terraform plan terraform apply Step 2: Verify Resources in vSphere
Confirm the VMs are provisioned in vSphere.
Ensure the network configurations (IP, gateway) match the Terraform parameters.
Step 3: Validate Kubernetes Cluster
After deployment:
SSH into one of the master nodes.
Run kubectl get nodes to ensure all nodes are registered and ready.
Screenshots of the Process
Terraform Apply Output
vSphere Dashboard
Kubernetes Terminal
Lessons Learned
Key Challenges
Configuring multi-AZ setups required precise IP allocation and resource planning.
Ensuring compatibility between Terraform, vSphere, and RKE2 versions.
Tips for Success
Automate Certificate Management: Pre-generate and verify certificates for secure communication.
Test Locally: Run initial setups in a test environment to validate module behavior.
Optimize Resource Allocation: Tailor resource parameters to your workload needs.
Conclusion
Using Terraform and vSphere to deploy an RKE2 Kubernetes cluster offers a highly customizable and scalable solution for production environments. By modularizing the Terraform configuration, this setup can be reused and extended for other environments with minimal changes.
If you've followed along or have feedback, share your experience in the comments below. Checkout the code repository from my GitHub profile. Let's discuss Kubernetes automation at scale!
Top comments (0)