Deploy with Kustomize, FluxCD and Remote Resources

Introduction

As a Kubernetes user, I chose to use the default tooling available, named Kustomize, to manage my manifests and all modifications I need to do over them for each environment I want to deploy. For that, Kustomize is a powerful solution based on concepts of inheritance (resources) and composition (components).

But one thing I miss the most is the ability to distribute manifests to other people, letting them inherit or compose them as they want. This is one subject where helm is ahead of Kustomize now.

However, Kustomize provides a mechanism to use remote elements, for both resources and components using git under the hood. This solution is perfect, because we can use any git repository as a yaml registry for our reousrces and components 🔥.

But, the downside is FluxCD does not supports so well, for a lot of good reasons (especially when your repositories are private)… In this article, we will see how to bypass this limitation and use remote resources and components, the FluxCD way!

Cluster & FluxCD configuration

In this article, I won't detail so much the cluster & flux configuration… because there is already plenty of documentation and blogposts about that!

For this article, I have set up a cluster using Google Kubernetes Engine (aka GKE) using this documentation and installed FluxCD connected to a GitLab repository with the following command (extracted from the official documentation).

$ flux bootstrap gitlab \ --owner=davinkevin.fr/articles/flux-and-kustomize-remote-base \ --repository=clusters \ --branch=gke \ --path=fluxcd 

NOTE: I don't like to use default branch, so I chose to create a branch for this cluster, so if I have another cluster, I just need another branch 😉. This is some kind of GitLab Flow for environment.

With this, I have a complete Kubernetes Cluster ready for GitOps, using clusters repository as source-of-truth. Every modification made in this repository will be automatically deployed by FluxCD 🤩.

bases repository

I have set up another git repository dedicated to resources and components. It hosts what we usually call base, because it contains the common core of an application and also many components to enable some extra features (at infrastructure or application level).

This repository is available in GitLab and is agnostic of my deployment environment. Again, I only host here the bare definition of my applications, we can compare this to an helm registry. Because it is managed in git, I can use branch & tags for management 🚀.

NOTE: Usually, this kind of yaml repository is only populated by CI from other projects automatically.

PodInfo base publication

I have chosen the well known podinfo application as an example, and I have developed the base and some components associated to it.

. ├── README.md └── podinfo └── base ├── components │   ├── hpa │   │   ├── hpa.yaml │   │   └── kustomization.yaml │   ├── ingress │   │   ├── ingress.yaml │   │   └── kustomization.yaml │   └── redis │   ├── kustomization.yaml │   ├── redis.conf │   └── redis.yaml ├── kustomization.yaml └── podinfo.yaml 

Core PodInfo

This part is the common part of all potential deployment. It contains:

• The podinfo Kubernetes Service
• The podinfo Kubernetes Deployment

And nothing else… if you need to deploy any other part of the app, you have to use components 👇.

Components

In this components, we can find all optional part of our application. As an end-user, I need to declare those I want to use, because by default, they are not applied at all in the common core.

Here, we have:

• hpa with a default preconfigured Horizontal-Pod-Autoscaler if you want the application to scale up and down automatically
• ingress definition to expose the application to the outside world (see official documentation)
• redis component to enable cache on the PodInfo application with default configuration.

It's interesting to note components can be of any sort, hpa and ingress are focused on infrastructure when redis is a feature of the application. Without this redis component, the application is able to work, just without any caching system. This redis component is here to deploy a redis server (of course 😅) but also to enable caching at podinfo level too.

NOTE: If you want a better description of this component feature, I advise you to discover this article from the official documentation

So, we have a complete, environment agnostic resource and components committed to our base repository. Now, It is time to use it!

Publish PodInfo to dev

Back to our cluster, I want to deploy this podinfo application, but of course I want to apply some extra customizations on it, like:

• Define components I want to enable
• Define a domain name for the ingress definition
• Define the required tls configuration to the ingress
• Define an alternative registry for the image

So, to do that, I need to have one (or multiple) kustomization.yaml to define this:

# <clusters-repo>/podinfo-dev/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: podinfo-dev images: - name: podinfo newName: ghcr.io/stefanprodan/podinfo resources: - ../base - ingress components: - ../base/components/redis 

NOTE: The ingress modification is in its own file to keep this one simple enough.

But, what is this ../base folder? There is no base at all in cluster repository. And you are right… we will use FluxCD feature available in flux to include the base repository at the right location during reconciliation. To do that, we need to define multiple flux resources.

podinfo-base flux repository

First, we need a representation of our flux.GitRepository for bases, which is materialized in flux with:

apiVersion: source.toolkit.fluxcd.io/v1beta1 kind: GitRepository metadata: name: podinfo-base-dev namespace: flux-system spec: interval: 5m ref: branch: podinfo url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases secretRef: name: flux-system 

NOTE: Here, I have chosen to define spec.ref.branch: podinfo to follow every commit made to this branch. It is, for development, a continuous deployment system 🚀.

podinfo flux repository

Now, we need to create a flux.GitRepository for manifests in clusters repository and manifests from the bases repository created in the step before. To do that, we are going to use the include feature from FluxCD:

apiVersion: source.toolkit.fluxcd.io/v1beta1 kind: GitRepository metadata: name: podinfo-dev namespace: flux-system spec: interval: 5m include: - repository: name: podinfo-base-dev fromPath: podinfo/base toPath: base ref: branch: gke url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters secretRef: name: flux-system 

The important part is the following:

spec: include: - repository: name: podinfo-base-dev # <1> fromPath: podinfo/base # <2> toPath: base # <3> 

This means the folder podinfo/base (2) previously defined in the podinfo-base-dev flux.GitRepository (1) will be injected in the path base (3) of the current podinfo-dev flux.GitRepository during FluxCD reconciliation… making all resources and components available.

podinfo-dev flux Kustomization

This one is the standard flux.Kustomization you would use in any case. This one is just using the composite flux.GitRepository from the previous step as source (spec.sourceRef):

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1 kind: Kustomization metadata: name: podinfo-dev namespace: flux-system spec: suspend: false interval: 10m0s path: podinfo-dev # file path in the <clusters> repository  prune: true sourceRef: kind: GitRepository namespace: flux-system name: podinfo-dev validation: client 

If we deploy all those files (you can put them in the same one), you will see your application deployed by FluxCD with your provided custom configuration.

❯ kubectl get all -n podinfo-dev NAME READY STATUS RESTARTS AGE pod/redis-869ff7c78b-jjbbk 1/1 Running 0 103m pod/podinfo-6dbf56d6d7-ctxxr 1/1 Running 0 103m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/podinfo ClusterIP 10.43.115.207 <none> 9898/TCP,9999/TCP 103m service/redis ClusterIP 10.43.192.84 <none> 6379/TCP 103m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/redis 1/1 1 1 103m deployment.apps/podinfo 1/1 1 1 103m NAME DESIRED CURRENT READY AGE replicaset.apps/redis-869ff7c78b 1 1 1 103m replicaset.apps/podinfo-6dbf56d6d7 1 1 1 103m 

Publish PodInfo to prod

Now, we want to use again the same base this time for a production environment.

Because we want something relatively stable, we chose to define spec.ref.tag to a specific tag in the bases repository (instead of the podinfo branch).

apiVersion: source.toolkit.fluxcd.io/v1beta1 kind: GitRepository metadata: name: podinfo-base-prod namespace: flux-system spec: interval: 5m ref: tag: podinfo-1.0 url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases secretRef: name: flux-system 

All others flux.Kustomization and flux.GitRepository are the same.

Because we are in production, we also want to enable different modules, here the HorizontalPodAutoscaler. So we have a different kustomization.yaml. Thanks to the full control we have over the kustomization.yaml per environment, we can modify this only in production 😍.

Once everything is published and FluxCD reconciliation is OK, we have:

❯ kubectl get all -n podinfo-prod NAME READY STATUS RESTARTS AGE pod/redis-869ff7c78b-lnll5 1/1 Running 0 113m pod/podinfo-6dbf56d6d7-k8rjl 1/1 Running 0 113m pod/podinfo-6dbf56d6d7-qldr8 1/1 Running 0 113m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/podinfo ClusterIP 10.43.109.213 <none> 9898/TCP,9999/TCP 113m service/redis ClusterIP 10.43.252.54 <none> 6379/TCP 113m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/redis 1/1 1 1 113m deployment.apps/podinfo 2/2 2 2 113m NAME DESIRED CURRENT READY AGE replicaset.apps/redis-869ff7c78b 1 1 1 113m replicaset.apps/podinfo-6dbf56d6d7 2 2 2 113m NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE horizontalpodautoscaler.autoscaling/podinfo Deployment/podinfo 4%/99% 2 4 2 113m 

Conclusion

It was tough, with many yaml but we have achieved our original goal. Being able to publish bare manifests in a central place and allow other team / people to consume them instantaneously with all features available in Kustomize.
We are then able to enjoy all the feature of a GitOps model with a dedicated (yaml) registry, like helm already have 😇.

There is some follow-up improvements to come, like OCI registries (announced in FluxCD 0.32), to replace flux.GitRepository. We still need this issue to be solved to be able to use it with all features provided by Kustomize.

In a more distant future, some other solution like porch could be a good candidate to simplify again our deployment strategy… but it is currently in alpha stage.

I hope you liked this article, you can find all resources used in GitLab, and you can reproduce it in your own cluster! If you have any questions, don't hesitate to comment or ping me on twitter @davinkevin