In this article, you will learn how to prepare the GitOps process on Kubernetes for the Postgres database and Hashicorp Vault with Argo CD. I guess that you are using Argo CD widely on your Kubernetes clusters for managing standard objects like deployment, services, or secrets. However, our configuration around the apps usually contains several other additional tools like databases, message brokers, or secrets engines. Today, we will consider how to implement the GitOps approach for such tools.

We will do the same thing as described in that article, but fully with the GitOps approach applied by Argo CD. The main goal here is to integrate Postgres with the Vault database secrets engine to generate database credentials dynamically and initialize the DB schema for the sample Spring Boot app. In order to achieve these goals, we are going to install two Kubernetes operators: Atlas and Vault Config. Atlas is a tool for managing the database schema as code. Its Kubernetes Operator allows us to define the schema and apply it to our database using the CRD objects. The Vault Config Operator provided by the Red Hat Community of Practice does a very similar thing but for Hashicorp Vault.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. I will explain the structure of our sample in detail later. So after cloning the Git repository you should just follow my instructions 🙂

How It Works

Before we start, let’s describe our sample scenario. Thanks to the database secrets engine Vault integrates with Postgres and generates its credentials dynamically based on configured roles. On the other hand, our sample Spring Boot app integrates with Vault and uses its database engine to authenticate against Postgres. All the aspects of that scenario are managed in the GitOps style. Argo CD installs Vault, Postgres, and additional operators on Kubernetes via their Helm charts. Then, it applies all the required CRD objects to configure both Vault and Postgres. We keep the whole configuration in a single Git repository in the form of YAML manifests.

Argo CD prepares the configuration on Vault and creates a table on Postgres for the sample Spring Boot app. Our app integrates with Vault through the Spring Cloud Vault project. It also uses Spring Data JPA to interact with the database. Here’s the illustration of our scenario.

Install Argo CD on Kubernetes

Traditionally, we need to start our GitOps exercise by installing Argo CD on the Kubernetes cluster. Of course, we can do it using the Helm chart. In the first step, we need to add the following repository:

$ helm repo add argo https://argoproj.github.io/argo-helm

We will add one parameter to the argo-cm ConfigMap to ignore the MutatingWebhookConfiguration kind. This step is not necessary. It allows us to ignore the specific resource generated by one of the Helm charts used in the further steps. Thanks to that we will have everything in Argo CD in the “green” color 🙂 Here’s the Helm values.yaml file with the required configuration:

configs:  cm:  resource.exclusions: |  - apiGroups:  - admissionregistration.k8s.io  kinds:  - MutatingWebhookConfiguration  clusters:  - "*"

Now, we can install the Argo CD in the argocd namespace using the configuration previously defined in the values.yml file:

$ helm install argo-cd argo/argo-cd \  --version 6.7.8 \  -n argo \  --create-namespace

That’s not all. Since the Atlas operator is available in the OCI-type Helm repository, we need to apply the following Secret in the argocd namespace. By default, Argo CD doesn’t allow the OCI-type repo, so we need to include the enableOCI parameter in the definition.

apiVersion: v1 kind: Secret metadata:  name: ghcr-io-helm-oci  namespace: argocd  labels:  argocd.argoproj.io/secret-type: repository stringData:  name: ariga  url: ghcr.io/ariga  enableOCI: "true"  type: helm

Let’s take a look at the list of repositories in the Argo CD UI dashboard. You should see the “Successful” connection status.

Prepare Configuration Manifests for Argo CD

Config Repository Structure

Let me first explain the structure of our Git config repository. The additional configuration is stored in the apps directory. It includes the CRD objects required to initialize the database schema or Vault engines. In the bootstrap directory, we keep the values.yaml file for each Helm chart managed by Argo CD. It’s all that we need. The bootstrap-via-appset/bootstrap.yaml contains the definition of Argo CD ApplicationSet we need to apply to the Kubernetes cluster. This ApplicationSet will generate all required Argo CD applications responsible for installing the charts and creating CRD objects.

. ├── apps │   ├── postgresql │   │   ├── database.yaml │   │   ├── policies.yaml │   │   ├── roles.yaml │   │   └── schema.yaml │   └── vault │   └── job.yaml ├── bootstrap │   ├── values │   │   ├── atlas │   │   │   └── values.yaml │   │   ├── cert-manager │   │   │   └── values.yaml │   │   ├── postgresql │   │   │   └── values.yaml │   │   ├── vault │   │   │   └── values.yaml │   │   └── vault-config-operator │   │   └── values.yaml └── bootstrap-via-appset  └── bootstrap.yaml

Bootstrap with the Argo CD ApplicationSet

Let’s take a look at the ApplicationSet. It’s pretty interesting (I hope :)). I’m using here some relatively new Argo CD features like multiple sources (Argo CD 2.6) or application sets template patch (Argo CD 2.10). We need to generate an Argo CD Application per each tool we want to install on Kubernetes (1). In the generators section, we define parameters for Vault, PostgreSQL, Atlas Operator, Vault Config Operator, and Cert Manager (which is required by the Vault Config Operator). In the templatePatch section, we prepare a list of source repositories used by each Argo CD Application (2). There is always a Helm chart repo, which refers to our Git repository containing dedicated values.yaml files. For the Vault and PostgreSQL charts, we include another source containing CRDs or additional Kubernetes objects. We will discuss it later.

apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata:  name: bootstrap-config  namespace: argocd spec:  goTemplate: true  generators:  - list:  elements:  - chart: vault  name: vault  repo: https://helm.releases.hashicorp.com  revision: 0.27.0  namespace: vault  postInstall: true  - chart: postgresql  name: postgresql  repo: https://charts.bitnami.com/bitnami  revision: 12.12.10  namespace: default  postInstall: true  - chart: cert-manager  name: cert-manager  repo: https://charts.jetstack.io  revision: v1.14.4  namespace: cert-manager  postInstall: false  - chart: vault-config-operator  name: vault-config-operator  repo: https://redhat-cop.github.io/vault-config-operator  revision: v0.8.25  namespace: vault-config-operator  postInstall: false  - chart: charts/atlas-operator  name: atlas  repo: ghcr.io/ariga  revision: 0.4.2  namespace: atlas  postInstall: false  template:  metadata:  name: '{{.name}}'  annotations:  argocd.argoproj.io/sync-wave: "1"  spec:  syncPolicy:  automated: {}  syncOptions:  - CreateNamespace=true  destination:  namespace: '{{.namespace}}'  server: https://kubernetes.default.svc  project: default  templatePatch: |  spec:  sources:  - repoURL: '{{ .repo }}'  chart: '{{ .chart }}'  targetRevision: '{{ .revision }}'  helm:  valueFiles:  - $values/bootstrap/values/{{ .name }}/values.yaml  - repoURL: https://github.com/piomin/kubernetes-config-argocd.git  targetRevision: HEAD  ref: values  {{- if .postInstall }}  - repoURL: https://github.com/piomin/kubernetes-config-argocd.git  targetRevision: HEAD  path: apps/{{ .name }}  {{- end }}

Once we apply the bootstrap-config ApplicationSet to the argocd namespace, all the magic just happens. You should see five applications in the Argo CD UI dashboard. All of them are automatically synchronized (Argo CD autoSync enabled) to the cluster. It does the whole job. Now, let’s analyze step-by-step what we have to put in that configuration.

The Argo CD ApplicationSet generates five applications for installing all required tools. Here’s the Application generated for installing Vault with Helm charts and applying an additional configuration stored in the apps/vault directory.

apiVersion: argoproj.io/v1alpha1 kind: Application metadata:  name: vault  namespace: argocd spec:  destination:  namespace: vault  server: https://kubernetes.default.svc  project: default  sources:  - chart: vault  helm:  valueFiles:  - $values/bootstrap/values/vault/values.yaml  repoURL: https://helm.releases.hashicorp.com  targetRevision: 0.27.0  - ref: values  repoURL: https://github.com/piomin/kubernetes-config-argocd.git  targetRevision: HEAD  - path: apps/vault  repoURL: https://github.com/piomin/kubernetes-config-argocd.git  targetRevision: HEAD  syncPolicy:  automated: {}  syncOptions:  - CreateNamespace=true 

Configure Vault on Kubernetes

Customize Helm Charts

Let’s take a look at the Vault values.yaml file. We run it in the development mode (single, in-memory node, no unseal needed). We will also enable the UI dashboard.

server:  dev:  enabled: true ui:  enabled: true

With the parameters visible above Argo CD installs Vault in the vault namespace. Here’s a list of running pods:

$ kubectl get po -n vault NAME READY STATUS RESTARTS AGE vault-0 1/1 Running 0 1h vault-agent-injector-7f7f68d457-fvsd2 1/1 Running 0 1h

It also exposes Vault API under the 8200 port in the vault Kubernetes Service.

$ kubectl get svc -n vault NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE vault ClusterIP 10.110.69.159 <none> 8200/TCP,8201/TCP 21h vault-agent-injector-svc ClusterIP 10.111.24.183 <none> 443/TCP 21h vault-internal ClusterIP None <none> 8200/TCP,8201/TCP 21h vault-ui ClusterIP 10.110.160.239 <none> 8200/TCP 21h

For the Vault Config Operator, we need to override the default address of Vault API to vault.vault.svc:8200 (an a). In order to do that, we need to set the VAULT_ADDR env variable in the values.yaml file. We also disable Prometheus monitoring and enable integration with Cert Manager. Thanks to “cert-manager” we don’t need to generate any certificates or keys manually.

enableMonitoring: false enableCertManager: true env:  - name: VAULT_ADDR  value: http://vault.vault:8200

Enable Vault Config Operator

The Vault Config Operator needs to authenticate against Vault API using Kubernetes Authentication. So we need to configure a root Kubernetes Authentication mount point and role. Then we can create more roles or other Vault objects via the operator. Here’s the Kubernetes Job responsible for configuring Kubernetes mount point and role. It uses the Vault image and the vault CLI available inside that image. As you see, it creates the vault-admin role allowed in the default namespace.

apiVersion: batch/v1 kind: Job metadata:  name: vault-admin-initializer  annotations:  argocd.argoproj.io/sync-wave: "3" spec:  template:  spec:  containers:  - name: vault-admin-initializer  image: hashicorp/vault:1.15.2  env:  - name: VAULT_ADDR  value: http://vault.vault.svc:8200  command:  - /bin/sh  - -c  - |  export VAULT_TOKEN=root  sleep 10  vault auth enable kubernetes  vault secrets enable database  vault write auth/kubernetes/config kubernetes_host=https://kubernetes.default.svc:443 kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt  vault write auth/kubernetes/role/vault-admin bound_service_account_names=default bound_service_account_namespaces=default policies=vault-admin ttl=1h  vault policy write vault-admin - <<EOF  path "/*" {  capabilities = ["create", "read", "update", "delete", "list","sudo"]  }   EOF  restartPolicy: Never

Argo CD applies such a Job after installing the Vault chart.

$ kubectl get job -n vault NAME COMPLETIONS DURATION AGE vault-admin-initializer 1/1 15s 1h

Configure Vault via CRDs

Once a root Kubernetes authentication is ready, we can proceed to the CRD object creation. In the first step, we create objects responsible for configuring a connection to the Postgres database. In the DatabaseSecretEngineConfig we set the connection URL, credentials, and the name of a Vault plugin used to interact with the database (postgresql-database-plugin). We also define a list of allowed roles (postgresql-default-role). In the next step, we define the postgresql-default-role DatabaseSecretEngineRole object. Of course, the name of the role should be the same as the name passed in the allowedRoles list in the previous step. The role defines a target database connection name in Vault and the SQL statement for creating new users with privileges.

kind: DatabaseSecretEngineConfig apiVersion: redhatcop.redhat.io/v1alpha1 metadata:  name: postgresql-database-config  annotations:  argocd.argoproj.io/sync-wave: "3"  argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec:  allowedRoles:  - postgresql-default-role  authentication:  path: kubernetes  role: vault-admin  connectionURL: 'postgresql://{{username}}:{{password}}@postgresql.default:5432?sslmode=disable'  path: database  pluginName: postgresql-database-plugin  rootCredentials:  passwordKey: postgres-password  secret:  name: postgresql  username: postgres --- apiVersion: redhatcop.redhat.io/v1alpha1 kind: DatabaseSecretEngineRole metadata:  name: postgresql-default-role  annotations:  argocd.argoproj.io/sync-wave: "3"  argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec:  creationStatements:  - CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}"; GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO "{{name}}";  maxTTL: 10m0s  defaultTTL: 1m0s  authentication:  path: kubernetes  role: vault-admin  dBName: postgresql-database-config  path: database

Once Argo CD applies both DatabaseSecretEngineConfig and DatabaseSecretEngineRole objects, we can verify it works fine by generating database credentials using the vault read command. We need to pass the name of the previously created role (postgresql-default-role). Our sample app will do the same thing but through the Spring Cloud Vault module.

Finally, we can create a policy and role for our sample Spring Boot. The policy requires only the privilege to generate new credentials:

kind: Policy apiVersion: redhatcop.redhat.io/v1alpha1 metadata:  name: database-creds-view  annotations:  argocd.argoproj.io/sync-wave: "3"  argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec:  authentication:  path: kubernetes  role: vault-admin  policy: |  path "database/creds/default" {  capabilities = ["read"]  }

Now, we have everything to proceed to the last step in this section. We need to create a Vault role with the Kubernetes authentication method dedicated to our sample app. In this role, we set the name and location of the Kubernetes ServiceAccount and the name of the Vault policy created in the previous step.

kind: KubernetesAuthEngineRole apiVersion: redhatcop.redhat.io/v1alpha1 metadata:  name: database-engine-creds-role  annotations:  argocd.argoproj.io/sync-wave: "3"  argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec:  authentication:  path: kubernetes  role: vault-admin  path: kubernetes  policies:  - database-creds-view  targetServiceAccounts:  - default  targetNamespaces:  targetNamespaces:  - default

Managing Postgres Schema with Atlas Operator

Finally, we can proceed to the last step in the configuration part. We will use the AtlasSchema CRD object to configure the database schema for our sample app. The object contains two sections: credentials and schema. In the credentials section, we refer to the PostgreSQL Secret to obtain a password. In the schema section, we create the person table with the id primary key.

apiVersion: db.atlasgo.io/v1alpha1 kind: AtlasSchema metadata:  name: sample-spring-cloud-vault  annotations:  argocd.argoproj.io/sync-wave: "4"  argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec:  credentials:  scheme: postgres  host: postgresql.default  user: postgres  passwordFrom:  secretKeyRef:  key: postgres-password  name: postgresql  database: postgres  port: 5432  parameters:  sslmode: disable  schema:  sql: |  create table person (  id serial primary key,  name varchar(255),  gender varchar(255),  age int,  external_id int  );

Here’s the corresponding app @Entity model class in the sample Spring Boot app.

@Entity public class Person {  @Id  @GeneratedValue(strategy = GenerationType.IDENTITY)  private Integer id;  private String name;  private int age;  @Enumerated(EnumType.STRING)  private Gender gender;  private Integer externalId;     // GETTERS AND SETTERS ...   }

Once Argo CD applies the AtlasSchema object, we can verify its status. As you see, it has been successfully executed on the target database.

We can log in to the database using psql CLI and verify that the person table exists in the postgres database:

Run Sample Spring Boot App

Dependencies

For this demo, I created a simple Spring Boot application. It exposes REST API and connects to the PostgreSQL database. It uses Spring Data JPA to interact with the database. Here are the most important dependencies of our app in the Maven pom.xml:

<dependency>  <groupId>org.springframework.cloud</groupId>  <artifactId>spring-cloud-starter-bootstrap</artifactId> </dependency> <dependency>  <groupId>org.springframework.boot</groupId>  <artifactId>spring-boot-starter-data-jpa</artifactId> </dependency> <dependency>  <groupId>org.springframework.cloud</groupId>  <artifactId>spring-cloud-vault-config-databases</artifactId> </dependency> <dependency>  <groupId>org.postgresql</groupId>  <artifactId>postgresql</artifactId>  <scope>runtime</scope> </dependency>

The first of them enables bootstrap.yml processing on the application startup. The third one includes Spring Cloud Vault Database engine support.

Integrate with Vault using Spring Cloud Vault

The only thing we need to do is to provide the right configuration settings. Here’s the minimal set of the required dependencies to make it work without any errors. The following configuration is provided in the bootstrap.yml file:

spring:  application:  name: sample-db-vault  datasource:  url: jdbc:postgresql://postgresql:5432/postgres #(1)  jpa:  hibernate:  ddl-auto: update  cloud:  vault:  config.lifecycle: #(2)  enabled: true  min-renewal: 10s  expiry-threshold: 30s  kv.enabled: false #(3)  uri: http://vault.vault:8200 #(4)  authentication: KUBERNETES #(5)  postgresql: #(6)  enabled: true  role: postgresql-default-role  backend: database  kubernetes: #(7)  role: database-engine-creds-role

Let’s analyze the configuration visible above in the details:

(1) Firstly, we need to set the database connection URL without any credentials. Our application uses standard properties for authentication against the database (spring.datasource.username and spring.datasource.password). Thanks to that, we don’t need to do anything else

(2) As you probably remember, the maximum TTL for the database lease is 10 minutes. We enable lease renewal every 30 seconds. Just for the demo purpose. You will see that Spring Cloud Vault will create new credentials in PostgreSQL every 30 seconds, and the application still works without any errors

(3) Vault KV is not needed here, since I’m using only the database engine

(4) The application is going to be deployed in the default namespace, while Vault is running in the vault namespace. So, the address of Vault should include the namespace name

(5) (7) Our application uses the Kubernetes authentication method to access Vault. We just need to set the role name, which is database-engine-creds-role. All other settings should be left with the default values

(6) We also need to enable postgres database backend support. The name of the backend in Vault is database and the name of the Vault role used for that engine is postgresql-default-role.

Run the App on Kubernetes

Finally, we can run our sample app on Kubernetes by applying the following YAML manifest:

apiVersion: apps/v1 kind: Deployment metadata:  name: sample-app-deployment spec:  selector:  matchLabels:  app: sample-app  template:  metadata:  labels:  app: sample-app  spec:  containers:  - name: sample-app  image: piomin/sample-app:1.0-gitops  ports:  - containerPort: 8080  serviceAccountName: default --- apiVersion: v1 kind: Service metadata:  name: sample-app spec:  type: ClusterIP  selector:  app: sample-app  ports:  - port: 8080

Our app exposes REST API under the /persons path. We can easily test it with curl after enabling port forwarding as shown below:

$ kubectl port-forward svc/sample-app 8080:8080 $ curl http://localhost:8080/persons

Final Thoughts

This article proves that we can effectively configure and manage tools like Postgres database or Hashicorp Vault on Kubernetes with Argo CD. The database schema or Vault configuration can be stored in the Git repository in the form of YAML manifests thanks to Atlas and Vault Config Kubernetes operators. Argo CD applies all required CRDs automatically, which results in the integration between Vault, Postgres, and our sample Spring Boot app.