DEV Community

AWS Community Builders profile image Sayan Moitra
Sayan Moitra for AWS Community Builders

Posted on • Originally published at aws.plainenglish.io on

Securing Kubernetes with Kyverno: A Practical Guide to Deployment with Helm and ArgoCD in EKS

#kubernetesargocd #kyverno #aws #containers

Introduction

Kubernetes has revolutionized container orchestration, but as clusters grow in complexity, security becomes increasingly challenging. Enter Kyverno , a policy engine designed specifically for Kubernetes. Unlike traditional policy engines, Kyverno doesn’t require learning a new language — it uses familiar Kubernetes-style resources to define policies.

In this guide, we’ll explore how to deploy Kyverno in Amazon EKS using Helm charts managed with ArgoCD, providing a GitOps approach to policy management. By the end, you’ll have a fully functional policy enforcement system that can validate, mutate, and generate resources across your EKS clusters.

Kyverno installation

https://cdn-images-1.medium.com/max/1024/1*xPnEhp7osbvnwoU7RzKaKg.png
Kyverno Official Doc — Installation guide

Architecture

https://cdn-images-1.medium.com/max/1024/1*P1J6aQ4zhgZunMZB0D-fdw.png

What is Kyverno?

Kyverno (derived from the Greek word “govern”) is a policy engine built specifically for Kubernetes. It allows cluster administrators to:

  • Validate resources against policies before they’re admitted to the cluster
  • Mutate resources to ensure they conform to organizational standards
  • Generate related resources automatically when certain resources are created
  • Clean up resources when their parents are deleted

All this is achieved using Kubernetes-native custom resources, making Kyverno intuitive for teams already familiar with Kubernetes.

Prerequisites

Before we begin, ensure you have:

  • An EKS cluster up and running
  • kubectl configured to communicate with your cluster
  • Helm v3 installed
  • ArgoCD installed in your cluster
  • A Git repository for storing your Kyverno configurations

Deployment Architecture

We’ll follow a GitOps approach with three main components:

  1. Helm  — To package and template the Kyverno installation
  2. ArgoCD  — To sync configurations from Git and manage the deployment
  3. EKS  — Our Kubernetes environment on AWS

Step 1: Setting Up the Git Repository Structure

First, let’s set up our Git repository with the necessary configuration files:

kyverno-gitops/ ├── applications/ │ └── kyverno.yaml └── helm-values/ └── kyverno-values.yaml
Enter fullscreen mode Exit fullscreen mode

Step 2: Creating the ArgoCD Application

Create the applications/kyverno.yaml file:

apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: kyverno namespace: argocd spec: project: default source: repoURL: https://kyverno.github.io/kyverno/ targetRevision: 2.7.2 chart: kyverno helm: valueFiles: - ../../helm-values/kyverno-values.yaml destination: server: https://kubernetes.default.svc namespace: kyverno syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true
Enter fullscreen mode Exit fullscreen mode

Step 3: Configuring Kyverno with Helm Values

Now, let’s create the helm-values/kyverno-values.yaml file with our desired configuration:

replicaCount: 3 resources: limits: cpu: 1000m memory: 1Gi requests: cpu: 100m memory: 128Mi serviceMonitor: enabled: true namespace: kyverno extraArgs: - "--clientRateLimitQPS=25" - "--clientRateLimitBurst=50" podSecurityContext: runAsNonRoot: true runAsUser: 1000 fsGroup: 1000 metricsService: create: true type: ClusterIP topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app.kubernetes.io/name: kyverno
Enter fullscreen mode Exit fullscreen mode

Step 4: Deploying with ArgoCD

With our configuration files ready, let’s deploy using ArgoCD:

# Apply the ArgoCD application kubectl apply -f applications/kyverno.yaml # Check the status argocd app get kyverno
Enter fullscreen mode Exit fullscreen mode

You can also use the ArgoCD UI to monitor the deployment progress.

https://cdn-images-1.medium.com/max/1024/1*b4uexXnn01k4DYkSRX0KmA.png
Argocd deployment

https://cdn-images-1.medium.com/max/1024/1*iyysU88dJyIKavWW-6xnrw.png
Kyverno policies

Step 5: Verifying the Installation

Once ArgoCD reports the application as “Healthy” and “Synced,” verify the installation:

# Check if Kyverno pods are running kubectl get pods -n kyverno # Verify the CRDs are installed kubectl get crds | grep kyverno
Enter fullscreen mode Exit fullscreen mode

You should see the Kyverno pods running and several Kyverno-related CRDs installed.

Step 6: Creating Your First Policy

Let’s create a simple policy that requires all pods to have resource limits:

apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-resource-limits spec: validationFailureAction: enforce rules: - name: validate-resource-limits match: resources: kinds: - Pod validate: message: "Resource limits are required for all containers." pattern: spec: containers: - resources: limits: memory: "?*" cpu: "?*"
Enter fullscreen mode Exit fullscreen mode

Save this as policies/require-resource-limits.yaml in your Git repository.

Step 7: Managing Policies with ArgoCD

Create another ArgoCD application to manage your policies:

apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: kyverno-policies namespace: argocd spec: project: default source: repoURL: https://github.com/yourusername/kyverno-gitops.git targetRevision: main path: policies destination: server: https://kubernetes.default.svc namespace: default syncPolicy: automated: prune: true selfHeal: true
Enter fullscreen mode Exit fullscreen mode

Apply this application:

kubectl apply -f applications/kyverno-policies.yaml
Enter fullscreen mode Exit fullscreen mode

Step 8: Testing the Policy

Let’s test our policy by trying to create a pod without resource limits:

cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: nginx image: nginx EOF
Enter fullscreen mode Exit fullscreen mode

You should receive an error message indicating that the pod was rejected because it doesn’t specify resource limits.

Now, let’s create a pod that complies with our policy:

cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: nginx image: nginx resources: limits: memory: "256Mi" cpu: "500m" EOF
Enter fullscreen mode Exit fullscreen mode

This pod should be created successfully.

Advanced Configuration: Multi-Cluster Policy Management

For organizations with multiple EKS clusters, you can use ArgoCD’s App of Apps pattern to deploy Kyverno consistently across all clusters:

apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: kyverno-management namespace: argocd spec: project: default source: repoURL: https://github.com/yourusername/kyverno-gitops.git targetRevision: main path: clusters destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: automated: prune: true selfHeal: true
Enter fullscreen mode Exit fullscreen mode

Within your clusters directory, include a separate application for each cluster:

# clusters/prod-cluster.yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: kyverno-prod namespace: argocd spec: project: default source: repoURL: https://github.com/yourusername/kyverno-gitops.git targetRevision: main path: overlays/prod destination: server: https://prod-cluster-api.example.com namespace: kyverno syncPolicy: automated: prune: true selfHeal: true
Enter fullscreen mode Exit fullscreen mode

Performance Tuning for EKS

When running Kyverno in a production EKS environment, consider these performance optimizations:

  1. Set appropriate resource requests and limits to ensure Kyverno pods have sufficient resources without over-provisioning.
  2. Implement pod topology spread constraints to distribute Kyverno pods across availability zones: 
topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app.kubernetes.io/name: kyverno
Enter fullscreen mode Exit fullscreen mode
  1. Configure Kyverno’s rate limits to prevent it from overwhelming the Kubernetes API server: 
extraArgs: - "--clientRateLimitQPS=25" - "--clientRateLimitBurst=50"
Enter fullscreen mode Exit fullscreen mode
  1. Use webhooks failurePolicy wisely : 
webhooks: failurePolicy: Ignore # Use "Fail" in production for critical policies
Enter fullscreen mode Exit fullscreen mode

Monitoring and Alerting

To monitor Kyverno’s health and performance, enable Prometheus metrics:

serviceMonitor: enabled: true namespace: monitoring
Enter fullscreen mode Exit fullscreen mode

Create a simple Prometheus rule to alert on policy violations:

apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: kyverno-alerts namespace: monitoring spec: groups: - name: kyverno.rules rules: - alert: KyvernoPolicyViolations expr: sum(increase(kyverno_policy_results_total{result="fail"}[15m])) > 10 for: 5m labels: severity: warning annotations: summary: "High number of Kyverno policy violations" description: "There have been more than 10 policy violations in the last 15 minutes."
Enter fullscreen mode Exit fullscreen mode

Conclusion

In this guide, we’ve covered how to deploy Kyverno in an EKS environment using Helm and ArgoCD. By leveraging GitOps principles, we’ve created a scalable, version-controlled system for managing Kubernetes policies.

Kyverno provides a powerful yet easy-to-understand approach to policy enforcement in Kubernetes. Its native integration with Kubernetes resources makes it an excellent choice for teams looking to implement policy-as-code without learning complex domain-specific languages.

As you continue your Kyverno journey, explore more advanced policies for security, compliance, and operational best practices. Remember that effective policy management is an iterative process — start small, test thoroughly, and gradually expand your policy coverage as your team becomes more comfortable with the tool.

Additional Resources

Top comments (0)