DEV Community

Cover image for Vault Auto-unseal using Transit Secret Engine on Kubernetes
Artur Bartosik
Artur Bartosik

Posted on • Edited on

Vault Auto-unseal using Transit Secret Engine on Kubernetes

#vault #kubernetes #helm

Theoretical introduction

To make the Vault operational once it has been installed, we need to perform two actions:

  • Intialzie Vault
  • Unseal Vault

Unsealing has to happen every time Vault starts. This is because Vault starts with sealed state in which it can't read storage because it doesn't know how to decrypt it.

Initialzing happens once when the server started with new backend. During initialization Vault generates bunch of keys:

  • unseal keys
  • encryption keys
  • root token

As you can easily guess, unseal keys generated during initizlization are used for unsealing the Vault.

We can distinguish three options for unsealing the Vault:

  • Manual unsealing
  • Auto-unseal
  • Transit Unseal - de facto one of Auto-unseal option

Manual unsealing

Manual unsealing is the simplest and doesn't require any additional configuration. Vault generated root key (don’t be confused with root token) and uses an algorithm  Shamir's Secret Sharing to split the key into chunks. During initialization, we can determine how many key shares will be needed to unseal the Vault. This is cloud agnostic and very flexible option but can become painful when you have many Vault clusters, many keys, and many key holders.

Vault Manual unsealing diagram

Auto-unseal

Auto-unseal reduces operational complexity and makes management less painful. In this approach, we delegate the responsibility of securing the unseal key from users to a trusted device or service. Vault with Auto-unseal, takes care of unsealing itself so we no longer need to worry about this as long as the service we have configured is available.

Vault Auto-unseal diagram

Services and devices supported with Auto-unseal you can find in official docs.

Transit Auto-unseal

We said that is one of Auto-unseal option. What makes this Auto-unseal different is that we don’t rely on an external service, but on external Vault itself. In this way, we can place one central Vault that will be responsible for Auto-unsealing other Vault instances.

Vault Transit Auto-unseal diagram

Transit Auto-unseal setup

We are going to isolate our Vaults on the level of namespaces, so let's start with their creation.

# namespace for Vault central kubectl create ns vault # namespace for Vault with Transit Auto-unseal kubectl create ns vault-a
Enter fullscreen mode Exit fullscreen mode

Let’s start with installation of Vault Central. Save below Helm chart values to install Vault in HA mode with default manual unsealing.

server: affinity: "" ha: enabled: true replicas: 2 raft: enabled: true
Enter fullscreen mode Exit fullscreen mode 
# change namespace to Vault central kns vault helm repo add hashicorp https://helm.releases.hashicorp.com helm install vault hashicorp/vault -f vault-central-helm-values.yml
Enter fullscreen mode Exit fullscreen mode

With the below initialization we split the root key into 4 shares (unseal keys). We can also set how many keys are required to reconstruct the root key, which is then used to decrypt the Vault's encryption key.

kubectl exec vault-0 -- vault operator init \ -key-shares=4 \ -key-threshold=2 \ -format=json > vault-central-keys.json
Enter fullscreen mode Exit fullscreen mode

Once is applied Vault generates unseal keys encrypted with base64 and hex and root token responsible for authentication against Vault. We will use it later.

{ "unseal_keys_b64": [ "4Wm5BYsNal+zMbsb3ewNbi6zLtKIOXz3L+NFX7jw0/3T", "miasg31FmPJqx9LrnPaVEuG639fvjAqZF3gp4ZlKw+wK", "EyVw9nQH/T+3zsa4HbPJ2s15l6B5MizMKQlKqs9taFzX", "zc7eU9MEvy9AaV4FPSQe7Jla2LcqSjS8KNPFDlQs0Rcg" ], "unseal_keys_hex": [ "e169b9058b0d6a5fb331bb1bddec0d6e2eb32ed288397cf72fe3455fb8f0d3fdd3", "9a26ac837d4598f26ac7d2eb9cf69512e1badfd7ef8c0a99177829e1994ac3ec0a", "132570f67407fd3fb7cec6b81db3c9dacd7997a079322ccc29094aaacf6d685cd7", "cdcede53d304bf2f40695e053d241eec995ad8b72a4a34bc28d3c50e542cd11720" ], "unseal_shares": 4, "unseal_threshold": 2, "recovery_keys_b64": [], "recovery_keys_hex": [], "recovery_keys_shares": 0, "recovery_keys_threshold": 0, "root_token": "hvs.NbXRWfYNI4PmA860aBlC4onU"
Enter fullscreen mode Exit fullscreen mode

Let’s unseal the first of two Vault instances in the central cluster. Pass two of four unseal_keys_b64 to the Vault to unseal them according to key-threshold.

kubectl exec vault-0 -- vault operator unseal 4Wm5BYsNal+zMbsb3ewNbi6zLtKIOXz3L+NFX7jw0/3T kubectl exec vault-0 -- vault operator unseal miasg31FmPJqx9LrnPaVEuG639fvjAqZF3gp4ZlKw+wK
Enter fullscreen mode Exit fullscreen mode

This same we have to do with the second instance, but before that, we must connect the second Vault to Raft storage cluster.

kubectl exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200 kubectl exec vault-1 -- vault operator unseal 4Wm5BYsNal+zMbsb3ewNbi6zLtKIOXz3L+NFX7jw0/3T kubectl exec vault-1 -- vault operator unseal miasg31FmPJqx9LrnPaVEuG639fvjAqZF3gp4ZlKw+wK
Enter fullscreen mode Exit fullscreen mode

Time to create Transit Secret Engine. This component will generate root key that we will use to Auto-unseal other Vaults.

# separate window kubectl port-forward vault-0 -n vault 8200:8200 # set Vault address to use locally Vault CLI export VAULT_ADDR=http://127.0.0.1:8200 # use 'root_token' generated during Vault initialization vault login # create transit secret  vault secrets enable transit vault write -f transit/keys/autounseal
Enter fullscreen mode Exit fullscreen mode

Save below Vault policy that we will attach to Auto-unseal token.

path "transit/encrypt/autounseal" { capabilities = [ "update" ] } path "transit/decrypt/autounseal" { capabilities = [ "update" ] }
Enter fullscreen mode Exit fullscreen mode 
# create policy with the above definition vault policy write autounseal autounseal-policy.hcl # create token for Auto-unsealing $ vault token create -orphan -policy=autounseal -period=24h Key Value --- ----- token hvs.CAESIP_A7TaC9kt4yUeqg5_bJNiOJElb4UbA01xoV9Rk4ei6Gh4KHGh2cy5zVXpaa3A1MG9uOEZrNXN2a3J0TGl0cHU token_accessor wkTM4nsF0ehkRvIuBD9cedHC token_duration 24h token_renewable true token_policies ["autounseal" "default"] identity_policies [] policies ["autounseal" "default"]
Enter fullscreen mode Exit fullscreen mode

Finally, we have created periodic orphan token which we will use for Auto-unsealing. Orphan means that created token doesn’t have a parent token so can’t be revoked together with ancestor. What is important to note, transit Auto-unseal token is renewed automatically by default.

Now it’s time to prepare Helm chart with the second Vault installation. It will be Vault with transit Auto-unsealing configuration. Check below Helm values file. Provide Vault central address and of course generated in previous step token (root key).

server: standalone: enabled: true config: | disable_mlock = true ui=true storage "file" { path = "/vault/data" } listener "tcp" { address = "127.0.0.1:8200" tls_disable = "true" } seal "transit" { address = "http://vault.vault:8200" token = "hvs.CAESIP_A7TaC9kt4yUeqg5_bJNiOJElb4UbA01xoV9Rk4ei6Gh4KHGh2cy5zVXpaa3A1MG9uOEZrNXN2a3J0TGl0cHU" disable_renewal = "false" key_name = "autounseal" mount_path = "transit/" tls_skip_verify = "true" }
Enter fullscreen mode Exit fullscreen mode 
# change namespace to Vault Auto-unseal kns vault-a helm install vault hashicorp/vault -f vault-auto-unseal-helm-values.yml
Enter fullscreen mode Exit fullscreen mode

The last step is to initialize Vault.

kubectl exec -it vault-0 -- vault operator init Recovery Key 1: FFMLznSZq9wh/0CJwKLJWKkI9BrK/hjF6ySDYl9a19Ie Recovery Key 2: qRfrdpkuEcXsF+dFh1Geru8VHkiL/hWUW+vY25twlwT1 Recovery Key 3: dX8sed7Dv8kI8kfFuYWDeQlagoikEVBpV5lZqH4ORnEh Recovery Key 4: TCCplv+KvZHEOlICQU6eb67hGccufiqcZGkSiGlpQPkx Recovery Key 5: ictL+c9czgMO+ME8qoTcGpgsvymEcORN7MkrpDE28x4a Initial Root Token: hvs.6umGyyta9xrjq0q7Cv09Hr8X Success! Vault is initialized
Enter fullscreen mode Exit fullscreen mode

… and check its status to verify Sealed status.

kubectl exec -it vault-a -- vault status Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 5 Threshold 3 Version 1.12.0 Build Date 2022-10-10T18:14:33Z Storage Type file Cluster Name vault-cluster-7a11a0ae Cluster ID a883d977-e70a-6367-3148-9c7a2c246897 HA Enabled false
Enter fullscreen mode Exit fullscreen mode

Each Vault initialization with configured to Auto-Unseal generates Recovery keys instead of Unseal Keys. Recovery keys can’t be used for unsealing Vault. These keys perform only authorization functions, which allows, for example, generates a new root token.

Final Thoughts

In the DevOps world, we want to automate everything possible and reduce operational complexity anywhere possible. Undoubtedly, Auto-unseal is something that fits into this assumption. However, from a security point of view, it is sometimes good to introduce a manual step with human intervention. In the above sample of Transit Auto-unseal we have exact combination of both approaches - Unsealed manually central cluster and related clusters Auto-unsealed by it.
It is also worth noting, that our solution is cloud-agnostic. We don’t rely on any external service, so we can setup it on-premise. The downside here is the introduction of a very crucial component in the overall deployment - central Vault cluster. Definitely, we have to think, how to ensure high availability and fault tolerance here.

Top comments (0)