DEV Community

Cover image for Vault With Kubernetes 🔐
Omar Ahmed
Omar Ahmed

Posted on • Edited on

Vault With Kubernetes 🔐

#devops #devsecops #vault #kubernetes

What is HashiCorp Vault?

HashiCorp Vault is a secrets & encryption platform for securely storing, generating, and controlling access to sensitive data (API keys, DB creds, TLS certs, tokens, etc.). Teams use it to centralize secrets, issue short-lived credentials on demand, and enforce least-privilege access across apps, CI/CD, and infrastructure

Install Vault Without a DataStorage (Dev Mode) 

helm repo add hashicorp https://helm.releases.hashicorp.com helm repo update helm repo list helm install vault hashicorp/vault \ --namespace vault --create-namespace \ --set 'server.dev.enabled=true' \ --set 'server.dataStorage.enabled=false' \ --set 'ui.enabled=true' kubectl config set-context --current --namespace=vault
Enter fullscreen mode Exit fullscreen mode

Install Vault With a DataStorage (Raft storage) 

cat > values.yaml <<'YAML' server: dev: enabled: false standalone: enabled: true config: | ui = true listener "tcp" { address = "[::]:8200" cluster_address = "[::]:8201" tls_disable = 1 } storage "raft" { path = "/vault/data" } dataStorage: enabled: true size: 1Gi ui: enabled: true serviceType: ClusterIP # use port-forward; change to NodePort/Ingress if you prefer YAML
Enter fullscreen mode Exit fullscreen mode 
helm repo add hashicorp https://helm.releases.hashicorp.com helm repo update helm -n vault install vault hashicorp/vault -f values.yaml --create-namespace --version 0.16.1
Enter fullscreen mode Exit fullscreen mode

Seal/Unseal

When you start a Vault server, it starts in a sealed state. In this state, Vault can access the physical storage, but it cannot decrypt any of the data on it.
Vault encrypts the data using an encryption key (in the keyring) and stores them in its storage backend. To protect this encryption key, Vault encrypts it using another encryption key known as the root key and stores it with the data.
To decrypt the data, Vault needs the root key so that it can decrypt the encryption key. Unsealing is the process of getting access to this root key. Vault encrypts the root key using the unseal key, and stores it alongside all other Vault data.
To summarize, Vault encrypts most data using the encryption key in the keyring. To get the keyring, Vault uses the root key to decrypt it. The root key itself requires the unseal key to decrypt it.

Shamir seals 

Key Shares / Unseal Keys Root Key Encryption Key (protected by root key) ┌─────────────────────┐ │ Jon 🔑 │ ├─────────────────────┤ │ Jane 🔑 │-------------->>>🔑 ---------------------->>🔑 ├─────────────────────┤ Root Key Encryption Key │ James 🔑 │ ├─────────────────────┤ │ Alison 🔑 │ ├─────────────────────┤ │ Pam 🔑 │ └─────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Instead of distributing the unseal key to an operator as a single key, the default Vault configuration uses an algorithm known as Shamir's Secret Sharing to split the key into shares.
Vault requires a certain threshold of shares to reconstruct the unseal key. Vault operators add shares one at a time in any order until Vault has enough shares to reconstruct the key. Then, Vault uses the unseal key to decrypt the root key. This is the Vault unseal process.
To Summarize:

  • Vault never writes the master key to disk.
  • Only encrypted data is stored (plus an encrypted data-encryption key).
  • On startup Vault can’t decrypt anything by itself, so it starts sealed.
  • Unseal = provide a quorum of unseal key shares (Shamir) to reconstruct the master key in memory → that master key decrypts the data-encryption key → Vault becomes unsealed and can serve requests. 
❯❯❯ k get po ⎈ (kind-my-cluster/vault) NAME READY STATUS RESTARTS AGE vault-0 0/1 Running 0 55s vault-agent-injector-6d97459765-4mc6b 1/1 Running 0 55s
Enter fullscreen mode Exit fullscreen mode

vault-0 0/1 Ready (not 1/1) >> because vault is sealed

kubectl -n vault exec -it vault-0 -- vault status # sealed=true as shown below Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 0/3 Unseal Nonce n/a Version 1.8.3 Storage Type raft HA Enabled true
Enter fullscreen mode Exit fullscreen mode 
❯❯❯ k exec -it vault-0 -- /bin/sh ⎈ (kind-my-cluster/vault) / $ vault operator init Unseal Key 1: Q3Awl3Qw+ItIWFyLFVGFiE3OOJ1qHrHC+kQnFD2kwLbE Unseal Key 2: OON236vYc+bf3K45J75lVHy3FmwIRdUT+mGPU3Iq4SX9 Unseal Key 3: N53iBqGPiL9RRkNNVg5DmrlD5dn6R3Vt703y6NhVUB+0 Unseal Key 4: mWPyNPitk8JTlxZByNkyYJefDo+MTpfIEcDn/lkaKuP3 Unseal Key 5: kRHm81+cYIbjO12eFJp0iJP8y3u7CR4NkOAb/4nHS5Kl Initial Root Token: s.bQN8VBeKBga76dAUVmaGyTjJ Vault initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. When the Vault is re-sealed, restarted, or stopped, you must supply at least 3 of these keys to unseal it before it can start servicing requests. Vault does not store the generated master key. Without at least 3 keys to reconstruct the master key, Vault will remain permanently sealed! It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information. / $
Enter fullscreen mode Exit fullscreen mode 
/ $ vault operator unseal Q3Awl3Qw+ItIWFyLFVGFiE3OOJ1qHrHC+kQnFD2kwLbE Key Value --- ----- Seal Type shamir Initialized true Sealed true ###### it is still sealed Total Shares 5 Threshold 3 Unseal Progress 1/3 Unseal Nonce 8932e24e-a03d-59e7-a694-9c9d9d6005f3 Version 1.8.3 Storage Type raft HA Enabled true / $ vault operator unseal OON236vYc+bf3K45J75lVHy3FmwIRdUT+mGPU3Iq4SX9 Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 2/3 Unseal Nonce 8932e24e-a03d-59e7-a694-9c9d9d6005f3 Version 1.8.3 Storage Type raft HA Enabled true / $ vault operator unseal N53iBqGPiL9RRkNNVg5DmrlD5dn6R3Vt703y6NhVUB+0 Key Value --- ----- Seal Type shamir Initialized true Sealed false ######## it is unsealed Total Shares 5 Threshold 3 Version 1.8.3 Storage Type raft Cluster Name vault-cluster-62a5fa0a Cluster ID 97d206e3-d4a2-2c1f-f898-6a95c034b87e HA Enabled true HA Cluster n/a HA Mode standby Active Node Address <none> Raft Committed Index 24 Raft Applied Index 24 / $ vault status Key Value --- ----- Seal Type shamir Initialized true Sealed false #<<<<<<<<<<<######### Total Shares 5 Threshold 3 Version 1.8.3 Storage Type raft Cluster Name vault-cluster-62a5fa0a Cluster ID 97d206e3-d4a2-2c1f-f898-6a95c034b87e HA Enabled true HA Cluster https://vault-0.vault-internal:8201 HA Mode active Active Since 2025-09-03T20:26:58.922335892Z Raft Committed Index 29 Raft Applied Index 29
Enter fullscreen mode Exit fullscreen mode

Vault-0 becomes 1/1 Ready state (now vault is unsealed):

❯❯❯ k get po ⎈ (kind-my-cluster/vault) NAME READY STATUS RESTARTS AGE vault-0 1/1 Running 0 27m vault-agent-injector-6d97459765-4mc6b 1/1 Running 0 27m
Enter fullscreen mode Exit fullscreen mode 
/ $ vault operator init Unseal Key 1: Q3Awl3Qw+ItIWFyLFVGFiE3OOJ1qHrHC+kQnFD2kwLbE Unseal Key 2: OON236vYc+bf3K45J75lVHy3FmwIRdUT+mGPU3Iq4SX9 Unseal Key 3: N53iBqGPiL9RRkNNVg5DmrlD5dn6R3Vt703y6NhVUB+0 Unseal Key 4: mWPyNPitk8JTlxZByNkyYJefDo+MTpfIEcDn/lkaKuP3 Unseal Key 5: kRHm81+cYIbjO12eFJp0iJP8y3u7CR4NkOAb/4nHS5Kl Initial Root Token: s.bQN8VBeKBga76dAUVmaGyTjJ / $ vault login s.bQN8VBeKBga76dAUVmaGyTjJ Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token. Key Value --- ----- token s.bQN8VBeKBga76dAUVmaGyTjJ token_accessor mkdn3cQnsFcO4hmklMrV8Chq token_duration ∞ token_renewable false token_policies ["root"] identity_policies [] policies ["root"] / $
Enter fullscreen mode Exit fullscreen mode 
# Enable Kubernetes Authentication kubectl exec -n vault -it vault-0 -- vault auth enable kubernetes # Create Service Account for App Pods kubectl create namespace webapps kubectl create serviceaccount vault-auth -n webapps # Configure Kubernetes Auth in Vault # Extract required info: SERVICE_ACCOUNT_NAME=vault-auth NAMESPACE=webapps # JWT Token TOKEN_REVIEW_JWT=$(kubectl get secret $(kubectl get serviceaccount $SERVICE_ACCOUNT_NAME -n $NAMESPACE -o jsonpath="{.secrets[0].name}") -n $NAMESPACE -o jsonpath="{.data.token}" | base64 --decode) # Kubernetes API Host KUBE_HOST=$(kubectl config view --raw -o=jsonpath='{.clusters[0].cluster.server}') # Kubernetes CA Cert KUBE_CA_CERT=$(kubectl get secret $(kubectl get serviceaccount $SERVICE_ACCOUNT_NAME -n $NAMESPACE -o jsonpath="{.secrets[0].name}") -n $NAMESPACE -o jsonpath="{.data['ca.crt']}" | base64 --decode) # Configure in Vault: kubectl exec -n vault -it vault-0 -- vault write auth/kubernetes/config \ token_reviewer_jwt="$TOKEN_REVIEW_JWT" \ kubernetes_host="$KUBE_HOST" \ kubernetes_ca_cert="$KUBE_CA_CERT"
Enter fullscreen mode Exit fullscreen mode

Purpose of Kubernetes Auth Configuration:
This command configures Vault to trust and communicate with the Kubernetes API server so that:

  • Vault can validate that service account tokens are legitimate
  • Pods can authenticate to Vault using their Kubernetes service account tokens
  • Vault can verify which namespace and service account a request is coming from Breaking Down Each Parameter: token_reviewer_jwt 
"$(kubectl get secret vault-token-reviewer -o jsonpath='{.data.token}' | base64 -d)"
Enter fullscreen mode Exit fullscreen mode
  • What it is: A service account token that Vault uses to authenticate with the Kubernetes API
  • Why needed: Vault needs to call the Kubernetes TokenReview API to validate incoming service account tokens
  • What it does: Acts as Vault's "identity" when talking to Kubernetes

kubernetes_host

"https://kubernetes.default.svc:443"
Enter fullscreen mode Exit fullscreen mode
  • What it is: The URL of the Kubernetes API server from inside the cluster
  • Why needed: Vault needs to know where to send TokenReview requests
  • kubernetes.default.svc: This is the internal DNS name for the Kubernetes API server

kubernetes_ca_cert

"$(kubectl get secret vault-token-reviewer -o jsonpath='{.data.ca\.crt}' | base64 -d)"
Enter fullscreen mode Exit fullscreen mode
  • What it is: The Certificate Authority certificate for the Kubernetes cluster
  • Why needed: Vault uses this to verify the SSL certificate when connecting to the Kubernetes API
  • Security: Prevents man-in-the-middle attacks

The Authentication Flow:
Here's what happens when a pod tries to authenticate to Vault:

1. Pod → Vault: "Here's my service account token" 2. Vault → Kubernetes API: "Is this token valid?" (using token_reviewer_jwt) 3. Kubernetes API → Vault: "Yes, it belongs to service account X in namespace Y" 4. Vault → Pod: "Authentication successful, here's your Vault token"
Enter fullscreen mode Exit fullscreen mode

Without this configuration:

  • Vault can't verify that service account tokens are real
  • Any pod could potentially forge authentication requests
  • You lose the security boundary that Kubernetes provides

With this configuration:

  • Vault cryptographically verifies each authentication request
  • Only legitimate pods with valid service account tokens can authenticate
  • You get fine-grained access control based on Kubernetes RBAC

This step essentially creates a trust relationship between Vault and your Kubernetes cluster, enabling secure, automated authentication for your workloads.

Create Vault Policy: Create a file myapp-policy.hcl:
A policy in Vault is a set of rules (paths + capabilities) that defines what a token/identity is allowed to do.
Vault is deny-by-default—nothing is allowed unless a policy explicitly grants it.

# Access to read/write secret data path "secret/data/mysql" { capabilities = ["create", "update", "read", "delete", "list"] } path "secret/data/frontend" { capabilities = ["create", "update", "read", "delete", "list"] } # Access to list secrets under the path path "secret/metadata/mysql" { capabilities = ["list"] } path "secret/metadata/frontend" { capabilities = ["list"] }
Enter fullscreen mode Exit fullscreen mode

Apply Vault Policy:

kubectl cp myapp-policy.hcl vault/vault-0:/tmp/myapp-policy.hcl kubectl exec -n vault -it vault-0 -- vault policy write myapp-policy /tmp/myapp-policy.hcl vault policy read <policy name> # show a policy vault policy list # to use the app policy token  export VAULT_TOKEN="$(vault token create -field token -policy=app)"
Enter fullscreen mode Exit fullscreen mode

Create Role in Vault to Map Pod to Policy:

kubectl exec -n vault -it vault-0 -- vault write auth/kubernetes/role/vault-role \ bound_service_account_names=vault-auth \ bound_service_account_namespaces="webapps" \ policies=myapp-policy \ ttl=24h kubectl describe clusterrolebindings vault-server-binding
Enter fullscreen mode Exit fullscreen mode

Store Secrets in Vault:

# Enable KV V2 Engine at the specific path (secret) kubectl exec -n vault -it vault-0 -- vault secrets enable -path=secret -version=2 kv # Store Secrets in Vault kubectl exec -n vault -it vault-0 -- vault kv put secret/mysql MYSQL_DATABASE=bankappdb MYSQL_ROOT_PASSWORD=Test@123 kubectl exec -n vault -it vault-0 -- vault kv put secret/frontend MYSQL_ROOT_PASSWORD=Test@123 kubectl exec -n vault -it vault-0 -- vault kv get secret/frontend
Enter fullscreen mode Exit fullscreen mode

What makes KV v2 different from v1:

  • Versioning: every write creates a new version of the secret.
  • Soft delete & undelete: you can delete specific versions and later restore them.

Example:

/ $ vault secrets enable -path=crds kv-v2 Success! Enabled the kv-v2 secrets engine at: crds/ / $ vault kv put crds/mysql username=root Key Value --- ----- created_time 2025-09-04T12:00:14.567764837Z deletion_time n/a destroyed false version 1 / $ vault kv put crds/mysql username=root password=12345 Key Value --- ----- created_time 2025-09-04T12:00:22.119818763Z deletion_time n/a destroyed false version 2 / $ vault kv get crds/mysql ====== Metadata ====== Key Value --- ----- created_time 2025-09-04T12:00:22.119818763Z deletion_time n/a destroyed false version 2 ====== Data ====== Key Value --- ----- password 12345 username root / $ vault kv get crds/mysql ====== Metadata ====== Key Value --- ----- created_time 2025-09-04T12:00:22.119818763Z deletion_time n/a destroyed false version 2 ====== Data ====== Key Value --- ----- password 12345 username root / $ vault kv metadata get crds/mysql ========== Metadata ========== Key Value --- ----- cas_required false created_time 2025-09-04T12:00:14.567764837Z current_version 2 delete_version_after 0s max_versions 0 oldest_version 0 updated_time 2025-09-04T12:00:22.119818763Z ====== Version 1 ====== Key Value --- ----- created_time 2025-09-04T12:00:14.567764837Z deletion_time n/a destroyed false ====== Version 2 ====== Key Value --- ----- created_time 2025-09-04T12:00:22.119818763Z deletion_time n/a destroyed false / $ vault kv delete crds/mysql Success! Data deleted (if it existed) at: crds/mysql / $ vault kv get crds/mysql ====== Metadata ====== Key Value --- ----- created_time 2025-09-04T12:00:22.119818763Z deletion_time 2025-09-04T12:02:04.56772668Z destroyed false version 2 / $
Enter fullscreen mode Exit fullscreen mode

Create YAML Manifest File (With Below Configurations):
Example annotation block for a pod:

annotations: vault.hashicorp.com/agent-inject: "true" # this should be set to a true or false, default to false vault.hashicorp.com/role: "vault-role" # role name vault.hashicorp.com/agent-inject-secret-MYSQL_ROOT_PASSWORD: "secret/mysql" # agent-inject-secret-<the name of the secret> , this will retrieve the secret from Vault # agent-inject-template-<the name of the secret>: used for rendering a secret vault.hashicorp.com/agent-inject-template-MYSQL_ROOT_PASSWORD: | {{- with secret "secret/mysql" -}} export MYSQL_ROOT_PASSWORD="{{ .Data.data.MYSQL_ROOT_PASSWORD }}" {{- end }}
Enter fullscreen mode Exit fullscreen mode

what to read and how to write.
vault.hashicorp.com/agent-inject-secret-MYSQL_ROOT_PASSWORD: "<vault path>"
Tells the injector which secret to fetch from Vault and that it should write a file named MYSQL_ROOT_PASSWORD under /vault/secrets/ (unless you override the filename/path).
vault.hashicorp.com/agent-inject-template-MYSQL_ROOT_PASSWORD:
Provides a custom template for what to write into that file. Without this, the injector writes a default rendering (key=value lines). With it, you control the exact content.
Template: It renders a shell line into the injected file, like: export MYSQL_ROOT_PASSWORD="supersecret123"

 {{- with secret "secret/mysql" -}} export MYSQL_ROOT_PASSWORD="{{ .Data.data.MYSQL_ROOT_PASSWORD }}" {{- end }}
Enter fullscreen mode Exit fullscreen mode

The sidecar Vault Agent will:
• Auth using the service account token
• Fetch secrets from Vault
• Write them to /vault/secrets/... >>> /vault/secrets/MYSQL_ROOT_PASSWORD inside the pod

UI access

kubectl -n vault get svc vault kubectl -n vault port-forward --address=0.0.0.0 svc/vault 8200:8200
Enter fullscreen mode Exit fullscreen mode

Clean-up steps

helm uninstall vault kubectl -n vault delete pod -l app.kubernetes.io/name=vault --force --grace-period=0 || true kubectl -n vault delete pvc -l app.kubernetes.io/name=vault || true
Enter fullscreen mode Exit fullscreen mode

Top comments (0)