DEV Community

suin
suin

Posted on

Deploying Helm Charts to Multiple Kubernetes Clusters with Cluster API's HelmChartProxy

#kubernetes #helm #capi #caaph

In this article, I'll introduce you to HelmChartProxy, a powerful feature of Cluster API that enables bulk deployment of Helm charts across multiple Kubernetes clusters.

Introduction

When managing multiple Kubernetes clusters, you often need to deploy the same Helm charts across different environments. Traditionally, this meant running Helm commands individually for each cluster. However, with Cluster API's HelmChartProxy, you can now streamline this process with bulk deployments.

We'll explore:

  • Bulk deployment to all clusters
  • Targeted deployment using labels
  • Deployment from private registries

Setting Up the Test Environment

Prerequisites

  • Docker (16GB+ memory recommended)
  • kubectl
  • kind
  • clusterctl

1. Creating the Management Cluster

First, let's create a management cluster using kind. Create the following configuration file:

# kind-cluster-with-extramounts.yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: ipFamily: dual nodes: - role: control-plane extraMounts: - hostPath: /var/run/docker.sock containerPath: /var/run/docker.sock
Enter fullscreen mode Exit fullscreen mode

Create the cluster using this configuration:

kind create cluster --config kind-cluster-with-extramounts.yaml
Enter fullscreen mode Exit fullscreen mode

2. Setting Up Cluster API

Install Cluster API on the management cluster:

export CLUSTER_TOPOLOGY=true clusterctl init --infrastructure docker --addon helm
Enter fullscreen mode Exit fullscreen mode

Wait for all components to be ready:

kubectl wait --for=condition=Available --timeout=300s -n capi-system deployments --all && \ kubectl wait --for=condition=Available --timeout=300s -n capi-kubeadm-bootstrap-system deployments --all && \ kubectl wait --for=condition=Available --timeout=300s -n capi-kubeadm-control-plane-system deployments --all && \ kubectl wait --for=condition=Available --timeout=300s -n capd-system deployments --all && \ kubectl wait --for=condition=Available --timeout=300s -n caaph-system deployments --all
Enter fullscreen mode Exit fullscreen mode

3. Creating Workload Clusters

We'll create two clusters named "muscat" and "delaware":

# Generate manifest for muscat cluster clusterctl generate cluster muscat \ --flavor development \ --kubernetes-version v1.28.0 \ --control-plane-machine-count=3 \ --worker-machine-count=3 \ > muscat.yaml # Generate manifest for delaware cluster clusterctl generate cluster delaware \ --flavor development \ --kubernetes-version v1.28.0 \ --control-plane-machine-count=3 \ --worker-machine-count=3 \ > delaware.yaml # Create the clusters kubectl apply -f muscat.yaml kubectl apply -f delaware.yaml
Enter fullscreen mode Exit fullscreen mode

Get the kubeconfig files:

clusterctl get kubeconfig muscat > muscat-kubeconfig.yaml clusterctl get kubeconfig delaware > delaware-kubeconfig.yaml
Enter fullscreen mode Exit fullscreen mode

4. Setting Up CNI

Install Calico on both clusters:

# Install Calico on muscat kubectl --kubeconfig=muscat-kubeconfig.yaml apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml # Install Calico on delaware kubectl --kubeconfig=delaware-kubeconfig.yaml apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml
Enter fullscreen mode Exit fullscreen mode

Deploying with HelmChartProxy

Deploying to All Clusters

Let's look at an example of deploying nginx to all clusters:

# helm-chart-proxy-nginx.yaml apiVersion: addons.cluster.x-k8s.io/v1alpha1 kind: HelmChartProxy metadata: name: nginx spec: clusterSelector: {} # Empty selector means "select all clusters" repoURL: oci://registry-1.docker.io/bitnamicharts chartName: nginx version: "18.2.5" releaseName: nginx namespace: nginx options: waitForJobs: true atomic: true wait: true timeout: 5m install: createNamespace: true valuesTemplate: |- service: type: NodePort
Enter fullscreen mode Exit fullscreen mode

Deploy and verify:

# Deploy nginx kubectl apply -f helm-chart-proxy-nginx.yaml # Wait for HelmReleaseProxy to be ready kubectl wait --for=condition=Ready --timeout=300s helmreleaseproxy --all # Check pods on muscat kubectl --kubeconfig=muscat-kubeconfig.yaml get pods -n nginx # Check pods on delaware kubectl --kubeconfig=delaware-kubeconfig.yaml get pods -n nginx
Enter fullscreen mode Exit fullscreen mode

Targeted Deployment Using Labels

You can deploy to specific clusters using labels:

# helm-chart-proxy-nginx.yaml apiVersion: addons.cluster.x-k8s.io/v1alpha1 kind: HelmChartProxy metadata: name: nginx spec: clusterSelector: matchLabels: use-nginx: "true" # Only deploy to clusters with this label # ... (other settings remain the same)
Enter fullscreen mode Exit fullscreen mode

Usage:

# Deploy nginx kubectl apply -f helm-chart-proxy-nginx.yaml # Label the muscat cluster kubectl label cluster muscat use-nginx=true # Check deployment status kubectl get helmreleaseproxy -A # Verify nginx is running on muscat kubectl --kubeconfig=muscat-kubeconfig.yaml get pods -n nginx # Verify nginx is not running on delaware kubectl --kubeconfig=delaware-kubeconfig.yaml get pods -n nginx
Enter fullscreen mode Exit fullscreen mode

Deploying from Private Registries

Let's use GitHub Container Registry as an example:

  1. Create a GitHub Personal Access Token (needs read:packages permission)

  2. Set up registry credentials:

# Create config.json with base64 encoded credentials echo -n "YOUR_GITHUB_USERNAME:YOUR_GITHUB_TOKEN" | base64 > auth.txt cat > config.json << EOF { "auths": { "ghcr.io": { "auth": "$(cat auth.txt)" } } } EOF # Create the secret kubectl create secret generic github-creds \ --from-file=config.json \ -n caaph-system
Enter fullscreen mode Exit fullscreen mode
  1. Create HelmChartProxy for private charts: 
apiVersion: addons.cluster.x-k8s.io/v1alpha1 kind: HelmChartProxy metadata: name: private-chart spec: clusterSelector: {} repoURL: oci://ghcr.io/YOUR_GITHUB_USERNAME chartName: YOUR_CHART_NAME version: "0.1.0" credentials: secret: name: github-creds namespace: caaph-system key: config.json # ... (other settings)
Enter fullscreen mode Exit fullscreen mode

Conclusion

HelmChartProxy significantly simplifies the process of deploying Helm charts across multiple clusters. Key benefits include:

  • Reduced operational overhead through bulk deployments
  • Flexible deployment control using labels
  • Support for private registries

If you're managing multiple Kubernetes clusters, consider incorporating HelmChartProxy into your workflow. It can greatly streamline your Helm chart deployment process and make your multi-cluster management more efficient.

Top comments (0)