DEV Community

Apurv Kiri
Apurv Kiri

Posted on • Originally published at apurv.me

Kubernetes GitOps with FluxCD - Part 2 - Secret Management using SOPS

#kubernetes #gitops #fluxcd #sops

In Kubernetes-based GitOps workflows, securely managing sensitive information presents a unique challenge. While GitOps principles encourage storing all cluster configurations in Git repositories, committing plaintext secrets creates significant security risks. SOPS (Secrets OPerationS) offers an elegant solution to this problem when integrated with FluxCD.

In our previous post, we explored how to perform the initial setup of FluxCD in a Kubernetes cluster. Building on that foundation, we'll now address one of the most critical aspects of GitOps implementation: Secret Management.

SOPS, developed by Mozilla, enables encrypting specific values within YAML, JSON, and other configuration formats while keeping the file structure intact. When combined with FluxCD's native support for decryption, this allows teams to safely store encrypted secrets directly in their Git repositories alongside other infrastructure definitions. The secrets are only decrypted at runtime within the Kubernetes cluster, maintaining security while preserving the GitOps workflow.

This article explores how to implement a robust secret management strategy using SOPS and FluxCD. We'll cover the setup process, encryption workflows, integration with different key management systems, and best practices for maintaining secure GitOps operations in production environments.

1. Install SOPS and GPG

Your installation method may differ depending on your operating system. Below is the command for openSUSE Tumbleweed:

sudo zypper in sops gpg2
Enter fullscreen mode Exit fullscreen mode

For other distributions:

  • Ubuntu/Debian: sudo apt install sops gnupg2
  • macOS: brew install sops gnupg
  • Arch Linux: sudo pacman -S sops gnupg

2. Generate Keypair

Next, we'll generate a GPG keypair that will be used for encryption and decryption:

export KEY_NAME="cluster.local" export KEY_COMMENT="For Flux Secrets" gpg --batch --full-generate-key <<EOF %no-protection Key-Type: 1 Key-Length: 4096 Subkey-Type: 1 Subkey-Length: 4096 Expire-Date: 0 Name-Comment: ${KEY_COMMENT} Name-Real: ${KEY_NAME} EOF
Enter fullscreen mode Exit fullscreen mode

This creates a 4096-bit RSA key with no expiration date. The output will look similar to:

gpg: directory '/home/apurv/.gnupg/openpgp-revocs.d' created gpg: revocation certificate stored as '/home/apurv/.gnupg/openpgp-revocs.d/A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4.rev'
Enter fullscreen mode Exit fullscreen mode

Now retrieve the key fingerprint, which we'll need for subsequent steps:

gpg --list-secret-keys "${KEY_NAME}" gpg: checking the trustdb gpg: marginals needed: 3 completes needed: 1 trust model: pgp gpg: depth: 0 valid: 2 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 2u sec rsa4096 2025-02-25 [SCEAR] A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4 uid [ultimate] cluster.local (For Flux Secrets) ssb rsa4096 2025-02-25 [SEA] 8B514FAAC98DF16F61EE487A45E294E6CACB03D4
Enter fullscreen mode Exit fullscreen mode

Take note of the fingerprint (in this example: A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4).

3. Store Private Key as Kubernetes Secret

Now we'll export the private key and store it as a Kubernetes secret that FluxCD can access:

gpg --export-secret-keys --armor "A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4" | kubectl create secret generic sops-gpg \ --namespace=flux-system \ --from-file=sops.asc=/dev/stdin secret/sops-gpg created
Enter fullscreen mode Exit fullscreen mode

Let's verify the secret was created properly:

apurv@oxygen:~> kubectl -n flux-system get secrets NAME TYPE DATA AGE flux-system Opaque 3 25h sops-gpg Opaque 1 38s
Enter fullscreen mode Exit fullscreen mode

Since the local key is unprotected (we used %no-protection for demonstration purposes), we should delete it from our local machine:

gpg --delete-secret-keys "A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4"
Enter fullscreen mode Exit fullscreen mode

4. Configure In-Cluster Secrets Decryption

For production environments, the recommended approach is to store secrets in a separate repository with restricted access. However, for this tutorial, we'll patch the existing cluster manifest to add support for the decryption provider.

Edit cluster/default/flux-system/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources:  - gotk-components.yaml - gotk-sync.yaml +patches: + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: flux-system + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-gpg
Enter fullscreen mode Exit fullscreen mode

This patch configures FluxCD to use SOPS for decryption, referencing our previously created GPG key secret.
Let's commit and push our changes, then verify the deployment:

git commit -m "Configured SOPS decryption for main cluster repo" && git push origin flux get kustomizations flux-system --watch NAME REVISION SUSPENDED READY MESSAGE flux-system main@sha1:c35e4e45 False True Applied revision: main@sha1:c35e4e45
Enter fullscreen mode Exit fullscreen mode

5. Export Public Key and Configure SOPS

We'll export the public key to the cluster directory:

gpg --export --armor "A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4" > ./cluster/default/.sops.pub.asc
Enter fullscreen mode Exit fullscreen mode

Next, create a SOPS configuration file to specify which parts of the YAML files should be encrypted:

cat <<EOF > ./cluster/default/.sops.yaml creation_rules: - path_regex: .*.yaml encrypted_regex: ^(data|stringData)$ pgp: A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4 EOF
Enter fullscreen mode Exit fullscreen mode

This configuration tells SOPS to only encrypt the data and stringData sections of YAML files, leaving metadata intact for Kubernetes to process.
Commit these configuration files:

git add . && git commit -m "Added SOPS public key and config"
Enter fullscreen mode Exit fullscreen mode

6. Validate the Setup

Let's verify our setup by creating a sample secret and ensuring it works properly:

Create ./cluster/default/samplesecret.yaml:

apiVersion: v1 kind: Secret metadata: name: samplesecret namespace: default type: Opaque stringData: message: This is a secret message
Enter fullscreen mode Exit fullscreen mode

Now encrypt this file with SOPS:

sops --encrypt --in-place samplesecret.yaml
Enter fullscreen mode Exit fullscreen mode

After encryption the yaml file looks like below.

apiVersion: v1 kind: Secret metadata: name: samplesecret namespace: default type: Opaque stringData: message: ENC[AES256_GCM,data:H0iMCWDGpqRSQZtjSXTbMlXaRvXQRxqs,iv:/r5ylcoqWd1wnOp1p9ksDUEs+kkPkdQz31I66LXrpxo=,tag:r2m9+dyo9tnBz+ty42cfrA==,type:str] sops: kms: [] gcp_kms: [] azure_kv: [] hc_vault: [] age: [] lastmodified: "2025-02-25T12:00:11Z" mac: ENC[AES256_GCM,data:AuJ9hHq2+Yars67Rw8QhkpjCu8iwqGVsWDJ2/1oEx430yaGSCNXXcZnntek9rsI3xuEzyVwTJcXIE/1yW1bCvPPdWUOel+dY+mLkLSUxio238FbDoGI8mv+7FLpDe4Tn9GZdBbaMyGF6FwXXfQJ8021tLSqK6IJQN8nW2XbT0L0=,iv:KkmRa+HzZ0lI0vsEdqX+ZY6HIpudA/0ABny6OruwuQ8=,tag:6gaSJssqOEQk53DvXCZT0Q==,type:str] pgp: - created_at: "2025-02-25T12:00:11Z" enc: |- -----BEGIN PGP MESSAGE----- hQIMA0XilObKywPUARAArblXLVn7VaJ+Sw2jNqptBPoPaDq7CA4V2LqUoGQUdIiX 6pHZoZmfWCpvXAUTqLQ1xEnZ32XHl+lwEVxLzV6OckkYcZKxMGz1lHZGmK7QghtV Gm32lFvXVOnMfI9uZ96WH9WW+8Dng3VgyBAK/putNG+N3NLkeXa3vrWaaNaHY/Dj aBPK1FqzFAYLbFlGjadA1+1xpFfA4JnE5dLX5HTDIo9DKldJWxxqY6cGo6jTg1lO j+vmLHQicdKrApbMqdq1KyRLWTU21B0cRlfIIK37rX9xESXjUlwPmF27sew2KPsx iuLyofDu/fVD9kFUC/Zkrmog0IgBWaKuUniOBT1KkcPJoJ5AX9m3qyTGjTGNwvVU DWATppNCqnO2NhN2D13sOxL5/BSkJT4HZgdj8oIiQQjv0QwLW6nwlEHcmveiX6aK CRUOR51gp+ja+fAlZnevYUpJMJZFz4TX1LTVXqUd8dJeXePaAsCPzO38Ywrpe83D Z9n9ABYawCohwSb+Nd1/eoU0RBZwRfcD0PumTbWj8mbSMn6cPHJuAY1j1WZKH1/y PM583Vv1zA4NlUJwhAiqI/X0kl22+Qh4tq+tdrhPVLw7P+m7diHG2pyd6jKDZ/K9 D3tY+eeQWcM3EV0hRBN7yA87Rs07lCy0m72PqAtD07qoUbO6jLXipIuE7TgPuZ3U aAEJAhBexKkWWprEJjk+jpt4h8aXyG7wUAeafCWr2kIWz0/kOSnG0STCjuL1kDbu +Ysah6EqMijU1sQBJv9Jn5oQ9eTAiHwN2Brh8F1nCPT+E6Ih6lbiJSLAD8duEa7V Nsgo8cYB0ebx =R3G0 -----END PGP MESSAGE----- fp: A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4 encrypted_regex: ^(data|stringData)$ version: 3.9.4
Enter fullscreen mode Exit fullscreen mode

Lets push these changes.

git add samplesecret.yaml && git commit -m "Added sample secret" git push origin
Enter fullscreen mode Exit fullscreen mode

Let's verify that the secret is created in the cluster:

kubectl get secrets NAME TYPE DATA AGE samplesecret Opaque 1 52s
Enter fullscreen mode Exit fullscreen mode

Verify the content of the secret:

kubectl get secret samplesecret -o jsonpath='{.data.message}' | base64 --decode This is a secret message
Enter fullscreen mode Exit fullscreen mode

What next ?

Future posts will explore advanced GitOps patterns with FluxCD, including:

  • Helm chart automation
  • Image update automation
  • Notification and alerting configuration

Stay tuned for each of these topics.

References

Top comments (0)