DEV Community

Meysam
Meysam

Posted on • Originally published at developer-friendly.blog on

cert-manager: All-in-One Kubernetes TLS Certificate Manager

#kubernetes #aws #opentofu #certmanager

Originally published at https://developer-friendly.blog on May 6, 2024.

Kubernetes is a great orchestration tool for managing your applications and all its dependencies. However, it comes with an extensible architecture and with an unopinionated approach to many of the day-to-day operational tasks.

One of these tasks is the management of TLS certificates. This includes issuing as well as renewing certificates from a trusted Certificate Authority. This CA may be a public internet-facing application or an internal service that needs encrypted communication between parties.

In this post, we will introduce the industry de-facto tool of choice for managing certificates in Kubernetes: cert-manager. We will walk you through the installation of the operator, configuring the issuer(s), and receiving a TLS certificate as a Kubernetes Secret for the Ingress or Gateway of your application.

Finally, we will create the Gateway CRD and expose an application securely over HTTPS to the internet.

If that gets you excited, hop on and let's get started!

Introduction

If you have deployed any reverse proxy in the pre-Kubernetes era, you might have, at some point or another, bumped into the issuance and renewal of TLS certificates. The trivial approach, back in the days as well as even today, was to use certbot1. This command-line utility abstracts you away from the complexity of the underlying CA APIs and deals with the certificate issuance and renewal for you.

Certbot is created by the Electronic Frontier Foundation (EFF) and is a great tool for managing certificates on a single server. However, when you're working at scale with many applications and services, you will benefit from the automation and integration that cert-manager2 provides.

cert-manager is a Kubernetes-native tool that extends the Kubernetes API with custom resources for managing certificates. It is built on top of the Operator Pattern3, and is a graduated project of the CNCF4.

With cert-manager, you can fetch and renew your TLS certificates behind automation, passing them along to the Ingress5 or Gateway6 of your platform to host your applications securely over HTTPS without losing the comfort of hosting your applications in a Kubernetes cluster.

With that introduction, let's kick off the installation of cert-manager.

Huge Thanks to You 🤗

If you're reading this, I would like to thank you for the time you spend on this blog 🌹. Whether this is your first time, or you've been here
before and have liked the content and its quality, I truly appreciate thetime you spend here.

As a token of appreciation, and to celebrate with you, I would like to
share the achievements of this blog over the course of ~11 weeks since its launch (the initial commit on Feb 13, 20247).

  • 10 posts published 📚
  • 14k+ words written so far (40k+ including codes) 📝
  • 2.5k+ views since the launch 👀
  • 160+ clicks coming from search engines 🔍

Here are the corresponding screenshots:

performance

total views

visitors

countries
I don't run ads on this blog (yet!? 🤔) and my monetization plan, as of the moment, is nothing! I may switch gear at some point; financial independence and doing this full-time makes me happy honestly ☺️.
But, for now, I'm just enjoying writing in Markdown format and seeing how Material for Mkdocs8 renders rich content from it.

If you are interested in supporting this effort, the GitHub Sponsors program, as well as the PayPal donation link are available at the bottom of all the pages in our website.

Greatly appreciate you being here and hope you keep coming back.

Pre-requisites

Before we start, make sure you have the following set up:

Step 0: Installation

cert-manager comes with a first-class support for Helm chart installation.
This makes the installation rather straightforward.

As mentioned earlier, we will install the Helm chart using FluxCD CRDs.

# cert-manager/namespace.yml apiVersion: v1 kind: Namespace metadata: name: cert-manager
Enter fullscreen mode Exit fullscreen mode 
# cert-manager/repository.yml apiVersion: source.toolkit.fluxcd.io/v1beta2 kind: HelmRepository metadata: name: cert-manager spec: interval: 60m url: https://charts.jetstack.io
Enter fullscreen mode Exit fullscreen mode 
# cert-manager/release.yml apiVersion: helm.toolkit.fluxcd.io/v2beta2 kind: HelmRelease metadata: name: cert-manager spec: chart: spec: chart: cert-manager sourceRef: kind: HelmRepository name: cert-manager version: v1.14.x interval: 30m maxHistory: 10 releaseName: cert-manager targetNamespace: cert-manager timeout: 2m valuesFrom: - kind: ConfigMap name: cert-manager-config
Enter fullscreen mode Exit fullscreen mode

Although not required, it is hugely beneficial to store the Helm values as it is in your VCS. This makes your future upgrades and code reviews easier.

helm repo add jetstack https://charts.jetstack.io helm repo update jetstack helm show values jetstack/cert-manager \ --version v1.14.x > cert-manager/values.yml
Enter fullscreen mode Exit fullscreen mode 
# cert-manager/values.yml # NOTE: truncated for brevity ... # In a production setup, the whole file will be stored in VCS as is! installCRDs: true
Enter fullscreen mode Exit fullscreen mode

Additionally, we will use Kubernetes Kustomize10:

# cert-manager/kustomizeconfig.yml nameReference: - kind: ConfigMap version: v1 fieldSpecs: - path: spec/valuesFrom/name kind: HelmRelease
Enter fullscreen mode Exit fullscreen mode 
# cert-manager/kustomization.yml configurations: - kustomizeconfig.yml configMapGenerator: - files: - values.yaml=./values.yml name: cert-manager-config resources: - namespace.yml - repository.yml - release.yml namespace: cert-manager
Enter fullscreen mode Exit fullscreen mode

Notice the namespace we are instructing Kustomization to place the resources in. The FluCD Kustomization CRD will be created in the flux-system namespace, while the Helm release itself is placed in the cert-manager namespace.

Ultimately, to create this stack, we will create a FluxCD Kustomization resource11:

# cert-manager/kustomize.yml apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: cert-manager namespace: flux-system spec: force: false interval: 10m0s path: ./cert-manager prune: true sourceRef: kind: GitRepository name: flux-system wait: true
Enter fullscreen mode Exit fullscreen mode

You may either advantage from the recursive reconciliation of FluxCD, add it to your root Kustomization or apply the resources manually from your command line.

kubectl apply -f cert-manager/kustomize.yml
Enter fullscreen mode Exit fullscreen mode

Build Kustomization

A good practice is to build your Kustomization locally and optionally apply
them as a dry-run to debug any potential typo or misconfiguration.

 kustomize build ./cert-manager

And the output:

 apiVersion: v1 kind: Namespace metadata: name: cert-manager --- apiVersion: v1 data: values.yaml: | # NOTE: truncated for brevity ... # In a production setup, the whole file will be stored in VCS as is! installCRDs: true kind: ConfigMap metadata: name: cert-manager-config-8b8tf9hfb4 namespace: cert-manager --- apiVersion: helm.toolkit.fluxcd.io/v2beta2 kind: HelmRelease metadata: name: cert-manager namespace: cert-manager spec: chart: spec: chart: cert-manager sourceRef: kind: HelmRepository name: cert-manager version: v1.14.x interval: 30m maxHistory: 10 releaseName: cert-manager targetNamespace: cert-manager timeout: 2m valuesFrom: - kind: ConfigMap name: cert-manager-config-8b8tf9hfb4 --- apiVersion: source.toolkit.fluxcd.io/v1beta2 kind: HelmRepository metadata: name: cert-manager namespace: cert-manager spec: interval: 60m url: https://charts.jetstack.io

Step 1.0: Issuer 101

In general, you can fetch your TLS certificate in two ways: either by verifying your domain using the HTTP01 challenge or the DNS01 challenge. Each have their own pros and cons, but both are just to make sure that you own the domain you're requesting the certificate for. Imagine a world where you could request a certificate for google.com without owning it! 😱

The HTTP01 challenge requires you to expose a specific path on your web server and asking the CA to send a GET request to that endpoint, expecting a specific file to be present in the response.

This is not always possible, especially if you're running a private service.

On a personal note, the HTTP01 feels like a complete hack to me. 😓

As such, in this guide, we'll use the DNS01 challenge. This challenge will create a specific DNS record in your nameserver. You don't specifically
have to manually do it yourself, as that is the whole point of automation that cert-manager will bring to the table.

For the DNS01 challenge, there are a couple of nameserver providers natively supported by cert-manager. You can find the list of supported providers on their website12.

For the purpose of this guide, we will provide examples for two different nameserver providers: AWS Route53 and Cloudflare.

AWS services are the indudstry standard for many companies, and Route53 is one of the most popular DNS services (fame where it's due).

Cloudflare, on the other hand, is handling a significant portion of the internet's traffic and is known for its networking capabilities across the
globe.

If you have other needs, you won't find it too difficult to find support for your nameserver provider in the cert-manager documentation.

Step 1.1: AWS Route53 Issuer

The developer-friendly.blog domain is hosted in Cloudflare and to demonstrate the AWS Route53 issuer, we will make it so that a subdomain will be resolved
by a Route53 Hosted Zone. That way, we can instruct the cert-manager controller to talk to the Route53 API for record creation and domain verfication.

diagram

# hosted-zone/variables.tf variable "root_domain" { type = string default = "developer-friendly.blog" } variable "subdomain" { type = string default = "aws" } variable "cloudflare_api_token" { type = string nullable = false sensitive = true }
Enter fullscreen mode Exit fullscreen mode 
# hosted-zone/versions.tf terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 5.47" } cloudflare = { source = "cloudflare/cloudflare" version = "~> 4.30" } } } provider "cloudflare" { api_token = var.cloudflare_api_token }
Enter fullscreen mode Exit fullscreen mode 
# hosted-zone/main.tf data "cloudflare_zone" "this" { name = var.root_domain } resource "aws_route53_zone" "this" { name = format("%s.%s", var.subdomain, var.root_domain) } resource "cloudflare_record" "this" { for_each = toset(aws_route53_zone.this.name_servers) zone_id = data.cloudflare_zone.this.id name = var.subdomain type = "NS" value = each.value ttl = 1 depends_on = [ aws_route53_zone.this ] }
Enter fullscreen mode Exit fullscreen mode 
# hosted-zone/outputs.tf output "hosted_zone_id" { value = aws_route53_zone.this.zone_id } output "name_servers" { value = aws_route53_zone.this.name_servers }
Enter fullscreen mode Exit fullscreen mode

To apply this stack we'll use OpenTofu.

We could've either separated the stacks to create the Route53 zone beforehand, or we will go ahead and target our resources separately from command line as
you see below.

export TF_VAR_cloudflare_api_token="PLACEHOLDER" export AWS_PROFILE="PLACEHOLDER" tofu plan -out tfplan -target=aws_route53_zone.this tofu apply tfplan # And now the rest of the resources tofu plan -out tfplan tofu apply tfplan
Enter fullscreen mode Exit fullscreen mode

Why Applying Two Times?

The values in a TF for_each must be known at the time of planning, AKA, static values13.
And since that is not the case with aws_route53_zone.this.name_servers, we have to make sure to create the Hosted Zone first before passing its output to another resource.
We should have our AWS Route53 Hosted Zone created as you see in the screenshot
below.

route53

Now that we have our Route53 zone created, we can proceed with the cert-manager configuration.

AWS IAM Role

We now need an IAM Role with enough permissions to create the DNS records to satisfy the DNS01 challenge14.

Make sure you have a good understanding of the
OpenID Connect, the technique we're employing in the trust relationship of the AWS IAM Role.

# route53-iam-role/variables.tf variable "role_name" { type = string default = "cert-manager" } variable "hosted_zone_id" { type = string description = "The Hosted Zone ID that the role will have access to. Defaults to `*`." default = "*" } variable "oidc_issuer_url" { type = string description = "The OIDC issuer URL of the cert-manager Kubernetes Service Account token." nullable = false } variable "access_token_audience" { type = string default = "sts.amazonaws.com" } variable "service_account_name" { type = string default = "cert-manager" description = "The name of the service account." } variable "service_account_namespace" { type = string default = "cert-manager" description = "The namespace of the service account." }
Enter fullscreen mode Exit fullscreen mode 
# route53-iam-role/versions.tf terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 5.47" } tls = { source = "hashicorp/tls" version = "~> 4.0" } } }
Enter fullscreen mode Exit fullscreen mode 
# route53-iam-role/main.tf data "aws_iam_policy_document" "iam_policy" { statement { actions = [ "route53:GetChange", ] resources = [ "arn:aws:route53:::change/${var.hosted_zone_id}", ] } statement { actions = [ "route53:ChangeResourceRecordSets", "route53:ListResourceRecordSets", ] resources = [ "arn:aws:route53:::hostedzone/${var.hosted_zone_id}", ] } statement { actions = [ "route53:ListHostedZonesByName", ] resources = [ "*", ] } } data "aws_iam_policy_document" "assume_role_policy" { statement { actions = [ "sts:AssumeRoleWithWebIdentity" ] effect = "Allow" principals { type = "Federated" identifiers = [ aws_iam_openid_connect_provider.this.arn ] } condition { test = "StringEquals" variable = "${aws_iam_openid_connect_provider.this.url}:aud" values = [ var.access_token_audience ] } condition { test = "StringEquals" variable = "${aws_iam_openid_connect_provider.this.url}:sub" values = [ "system:serviceaccount:${var.service_account_namespace}:${var.service_account_name}", ] } } } data "tls_certificate" "this" { url = var.oidc_issuer_url } resource "aws_iam_openid_connect_provider" "this" { url = var.oidc_issuer_url client_id_list = [ var.access_token_audience ] thumbprint_list = [ data.tls_certificate.this.certificates[0].sha1_fingerprint ] } resource "aws_iam_role" "this" { name = var.role_name inline_policy { name = "${var.role_name}-route53" policy = data.aws_iam_policy_document.iam_policy.json } assume_role_policy = data.aws_iam_policy_document.assume_role_policy.json }
Enter fullscreen mode Exit fullscreen mode 
# route53-iam-role/outputs.tf output "iam_role_arn" { value = aws_iam_role.this.arn } output "service_account_name" { value = var.service_account_name } output "service_account_namespace" { value = var.service_account_namespace } output "access_token_audience" { value = var.access_token_audience }
Enter fullscreen mode Exit fullscreen mode 
tofu plan -out tfplan -var=oidc_issuer_url="KUBERNETES_OIDC_ISSUER_URL" tofu apply tfplan
Enter fullscreen mode Exit fullscreen mode

If you don't know what OpenID Connect is and what we're doing here, you might want to check out our ealier guides on the following topics:

The gist of both articles is that we are providing a means for the two services to talk to each other securely and without storing long-lived credentials.

In essence, one service will issue the tokens (Kubernetes cluster), and the other will trust the tokens of the said service (AWS IAM).

Kubernetes Service Account

Now that we have our IAM role set up, we can pass that information to the cert-manager Deployment. This way the cert-manager will assume that role with the Web Identity Token flow15 (there are five flows in
total).

We will also create a ClusterIssuer CRD to be responsible for fetching the TLS certificates from the trusted CA.

# route53-issuer/variables.tf variable "role_arn" { type = string default = null } variable "kubeconfig_path" { type = string default = "~/.kube/config" } variable "kubeconfig_context" { type = string default = "k3d-k3s-default" } variable "field_manager" { type = string default = "flux-client-side-apply" } variable "access_token_audience" { type = string default = "sts.amazonaws.com" } variable "chart_url" { type = string default = "https://charts.jetstack.io" } variable "chart_name" { type = string default = "cert-manager" } variable "release_name" { type = string default = "cert-manager" } variable "release_namespace" { type = string default = "cert-manager" } variable "release_version" { type = string default = "v1.14.x" }
Enter fullscreen mode Exit fullscreen mode 
# route53-issuer/versions.tf terraform { required_providers { kubernetes = { source = "hashicorp/kubernetes" version = "~> 2.29" } helm = { source = "hashicorp/helm" version = "~> 2.13" } } } provider "kubernetes" { config_path = var.kubeconfig_path config_context = var.kubeconfig_context } provider "helm" { kubernetes { config_path = var.kubeconfig_path config_context = var.kubeconfig_context } }
Enter fullscreen mode Exit fullscreen mode 
# route53-issuer/values.yml.tftpl extraEnv: - name: AWS_ROLE_ARN value: ${sa_role_arn} - name: AWS_WEB_IDENTITY_TOKEN_FILE value: /var/run/secrets/aws/token volumeMounts: - name: token mountPath: /var/run/secrets/aws readOnly: true volumes: - name: token projected: sources: - serviceAccountToken: audience: ${sa_audience} expirationSeconds: 3600 path: token securityContext: fsGroup: 1001
Enter fullscreen mode Exit fullscreen mode 
# route53-issuer/main.tf data "terraform_remote_state" "iam_role" { count = var.role_arn != null ? 0 : 1 backend = "local" config = { path = "../route53-iam-role/terraform.tfstate" } } data "terraform_remote_state" "hosted_zone" { backend = "local" config = { path = "../hosted-zone/terraform.tfstate" } } locals { sa_audience = coalesce(var.access_token_audience, data.terraform_remote_state.iam_role[0].outputs.access_token_audience) sa_role_arn = coalesce(var.role_arn, data.terraform_remote_state.iam_role[0].outputs.iam_role_arn) } resource "helm_release" "cert_manager" { name = var.release_name repository = var.chart_url chart = var.chart_name version = var.release_version namespace = var.release_namespace reuse_values = true values = [ templatefile("${path.module}/values.yml.tftpl", { sa_audience = local.sa_audience, sa_role_arn = local.sa_role_arn }) ] } resource "kubernetes_manifest" "cluster_issuer" { manifest = yamldecode(<<-EOF apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: route53-issuer spec: acme: email: admin@developer-friendly.blog enableDurationFeature: true privateKeySecretRef: name: route53-issuer server: https://acme-v02.api.letsencrypt.org/directory solvers: - dns01: route53: hostedZoneID: ${data.terraform_remote_state.hosted_zone.outputs.hosted_zone_id} region: eu-central-1  EOF  ) }
Enter fullscreen mode Exit fullscreen mode 
# route53-issuer/outputs.tf output "cluster_issuer_name" { value = kubernetes_manifest.cluster_issuer.manifest.metadata.name }
Enter fullscreen mode Exit fullscreen mode 
tofu plan -out tfplan -var=kubeconfig_context="KUBECONFIG_CONTEXT" tofu apply tfplan
Enter fullscreen mode Exit fullscreen mode

If you're wondering why we're changing the configuration of the cert-manager Deployment with a new Helm upgrade, you will find an exhaustive discussion and my comment on the relevant GitHub issue16.

The gist of that conversation is that the cert-manager Deployment won't take into account the eks.amazonaws.com/role-arn annotation on its Service Account, as you'd see the External Secrets Operator would. It won't even consider using the ClusterIssuer.spec.acme.solvers[*].dns01.route53.role field for some reason! 🔫

That's why we're manually passing that information down to its AWS Go SDK17 using the official environment variables18.

This stack allows the cert-manager controller to talk to AWS Route53.

Notice that we didn't pass any credentials, nor did we have to create any IAM User for this communication to work. It's all the power of OpenID Connect and allows us to establish a trust relationship and never have to worry about any credentials in the client service. ✅

Is There a Simpler Way?

Sure there is. If you don't fancy OpenID Connect, there is always the option to pass the credentials around in your environment. That leaves you with the burden of having to rotate them every now and then, but if you're cool with that, there's nothing stopping you from going down that path. You also have the possibility of automating such rotation using less than 10 lines of code in any programming language of course.

All that said, I have to say that I consider this to be an implementation bug16; where cert-manager does not provide you with a clean interface to easily pass around IAM Role ARN. The cert-manager controller SHOULD be able to assume the role it is given with the web identity flow!

Regardless of such shortage, in this section, I'll provide you a simpler way around this.

Bear in mind that I do not recommend this approach, and wouldn't use it in my own environments either. 🤷

The idea is to use our previously deployed ESO and pass the AWS IAM User credentials to the cert-manager controller (easy peasy, no drama!).

# iam-user/variables.tf variable "user_name" { type = string default = "cert-manager" } variable "hosted_zone_id" { type = string description = "The Hosted Zone ID that the role will have access to. Defaults to `*`." default = "*" }
Enter fullscreen mode Exit fullscreen mode 
# iam-user/versions.tf terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 5.47" } } }
Enter fullscreen mode Exit fullscreen mode 
# iam-user/main.tf data "aws_iam_policy_document" "iam_policy" { statement { actions = [ "route53:GetChange", ] resources = [ "arn:aws:route53:::change/${var.hosted_zone_id}", ] } statement { actions = [ "route53:ChangeResourceRecordSets", "route53:ListResourceRecordSets", ] resources = [ "arn:aws:route53:::hostedzone/${var.hosted_zone_id}", ] } statement { actions = [ "route53:ListHostedZonesByName", ] resources = [ "*", ] } } resource "aws_iam_user" "this" { name = var.user_name } resource "aws_iam_access_key" "this" { user = aws_iam_user.this.name } resource "aws_ssm_parameter" "access_key" { for_each = { "/cert-manager/access-key" = aws_iam_access_key.this.id "/cert-manager/secret-key" = aws_iam_access_key.this.secret } name = each.key type = "SecureString" value = each.value }
Enter fullscreen mode Exit fullscreen mode 
# iam-user/outputs.tf output "iam_user_arn" { value = aws_iam_user.this.arn } output "iam_access_key_id" { value = aws_iam_access_key.this.id sensitive = true } output "iam_access_key_secret" { value = aws_iam_access_key.this.secret sensitive = true }
Enter fullscreen mode Exit fullscreen mode

And now let's create the corresponding ClusterIssuer, passing the credentials like a normal human being!

# route53-issuer-creds/variables.tf variable "kubeconfig_path" { type = string default = "~/.kube/config" } variable "kubeconfig_context" { type = string default = "k3d-k3s-default" } variable "field_manager" { type = string default = "flux-client-side-apply" }
Enter fullscreen mode Exit fullscreen mode 
# route53-issuer-creds/versions.tf terraform { required_providers { kubernetes = { source = "hashicorp/kubernetes" version = "~> 2.29" } } } provider "kubernetes" { config_path = var.kubeconfig_path config_context = var.kubeconfig_context }
Enter fullscreen mode Exit fullscreen mode 
# route53-issuer-creds/main.tf data "terraform_remote_state" "hosted_zone" { backend = "local" config = { path = "../hosted-zone/terraform.tfstate" } } resource "kubernetes_manifest" "external_secret" { manifest = yamldecode(<<-EOF apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: route53-issuer-aws-creds namespace: cert-manager spec: data: - remoteRef: key: /cert-manager/access-key secretKey: awsAccessKeyID - remoteRef: key: /cert-manager/secret-key secretKey: awsSecretAccessKey refreshInterval: 5m secretStoreRef: kind: ClusterSecretStore name: aws-parameter-store target: immutable: false  EOF  ) } resource "kubernetes_manifest" "cluster_issuer" { manifest = yamldecode(<<-EOF apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: route53-issuer spec: acme: email: admin@developer-friendly.blog enableDurationFeature: true privateKeySecretRef: name: route53-issuer server: https://acme-v02.api.letsencrypt.org/directory solvers: - dns01: route53: hostedZoneID: ${data.terraform_remote_state.hosted_zone.outputs.hosted_zone_id} region: eu-central-1 accessKeyIDSecretRef: key: awsAccessKeyID name: route53-issuer-aws-creds secretAccessKeySecretRef: key: awsSecretAccessKey name: route53-issuer-aws-creds  EOF  ) }
Enter fullscreen mode Exit fullscreen mode 
# route53-issuer-creds/outputs.tf output "external_secret_name" { value = kubernetes_manifest.external_secret.manifest.metadata.name } output "external_secret_namespace" { value = kubernetes_manifest.external_secret.manifest.metadata.namespace } output "cluster_issuer_name" { value = kubernetes_manifest.cluster_issuer.manifest.metadata.name }
Enter fullscreen mode Exit fullscreen mode

We're now done with the AWS issuer. Let's switch gear for a bit to create the Cloudflare issuer before finally creating a TLS certificate for our desired domain(s).

Step 1.2: Cloudflare Issuer

Since Cloudflare does not have native support for OIDC, we will have to pass an API token to the cert-manager controller to be able to manage the DNS records on our behalf.

That's where the External Secrets Operator comes into play, again. I invite you to take a look at our last week's guide if you haven't done so already.

We will use the ExternalSecret CRD to fetch an API token from the AWS SSM Parameter Store and pass it down to our Kubernetes cluster as a Secret resource.

Notice the highlighted lines.

# cloudflare-issuer/externalsecret.yml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: cloudflare-issuer-api-token spec: data: - remoteRef: key: /cloudflare/api-token secretKey: cloudflareApiToken refreshInterval: 5m secretStoreRef: kind: ClusterSecretStore name: aws-parameter-store target: immutable: false
Enter fullscreen mode Exit fullscreen mode 
# cloudflare-issuer/clusterissuer.yml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: cloudflare-issuer spec: acme: email: meysam@licenseware.io enableDurationFeature: true privateKeySecretRef: name: cloudflare-issuer server: https://acme-v02.api.letsencrypt.org/directory solvers: - dns01: cloudflare: apiTokenSecretRef: key: cloudflareApiToken name: cloudflare-issuer-api-token email: admin@developer-friendly.blog
Enter fullscreen mode Exit fullscreen mode 
# cloudflare-issuer/kustomization.yml resources: - externalsecret.yml - clusterissuer.yml namespace: cert-manager
Enter fullscreen mode Exit fullscreen mode 
# cloudflare-issuer/kustomize.yml apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: cloudflare-issuer namespace: flux-system spec: interval: 5m path: ./cloudflare-issuer prune: true sourceRef: kind: GitRepository name: flux-system wait: true
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f cloudflare-issuer/kustomize.yml
Enter fullscreen mode Exit fullscreen mode

That's all the issuers we aimed to create for today. One for AWS Route53 and another for Cloudflare.

We are now equipped with enough access in our Kubernetes cluster to just create the TLS certificate and never have to worry about how to verify their ownership.

With that promise, let's wrap this up with the easiest part! 😎

Step 2: TLS Certificate

You should have noticed by now that the root developer-friendly.blog will be resolved by Cloudflare as our initial nameserver. We also created a subdomain and a Hosted Zone in AWS Route53 to resolve the aws. subdomain using Route53 as its nameserver.

We can now fetch a TLS certificate for each of them using our newly created ClusterIssuer resource. The rest is the responsibility of the cert-manager to verify the ownership within the cluster through the DNS01 challenge and using the access we've provided it.

# tls-certificates/aws-subdomain.yml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: aws-developer-friendly-blog spec: dnsNames: - '*.aws.developer-friendly.blog' issuerRef: kind: ClusterIssuer name: route53-issuer privateKey: rotationPolicy: Always revisionHistoryLimit: 5 secretName: aws-developer-friendly-blog-tls
Enter fullscreen mode Exit fullscreen mode 
# tls-certificates/cloudflare-root.yml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: developer-friendly-blog spec: dnsNames: - '*.developer-friendly.blog' issuerRef: kind: ClusterIssuer name: cloudflare-issuer privateKey: rotationPolicy: Always revisionHistoryLimit: 5 secretName: developer-friendly-blog-tls
Enter fullscreen mode Exit fullscreen mode 
# tls-certificates/kustomization.yml resources: - cloudflare-root.yml - aws-subdomain.yml namespace: cert-manager
Enter fullscreen mode Exit fullscreen mode 
# tls-certificates/kustomize.yml resources: - cloudflare-root.yml - aws-subdomain.yml namespace: cert-manager
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f tls-certificates/kustomize.yml
Enter fullscreen mode Exit fullscreen mode

It'll take less than a minute to have the certificates issued and stored as Kubernetes Secrets in the same namespace as the cert-manager Deployment.

If you would like the certificates in a different namespace, you're better off creating Issuer instead of ClusterIssuer.

The final result will have a Secret with two keys: tls.crt and tls.key. This will look similar to what you see below.

--- - apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: aws-developer-friendly-blog namespace: cert-manager spec: dnsNames: - "*.aws.developer-friendly.blog" issuerRef: kind: ClusterIssuer name: route53-issuer privateKey: rotationPolicy: Always revisionHistoryLimit: 5 secretName: aws-developer-friendly-blog-tls status: conditions: - lastTransitionTime: "2024-05-04T05:44:12Z" message: Certificate is up to date and has not expired observedGeneration: 1 reason: Ready status: "True" type: Ready notAfter: "2024-07-30T04:44:12Z" notBefore: "2024-05-04T04:44:12Z" renewalTime: "2024-06-29T04:44:12Z" --- apiVersion: v1 data: tls.crt: ...truncated... tls.key: ...truncated... kind: Secret metadata: annotations: cert-manager.io/alt-names: "*.aws.developer-friendly.blog" cert-manager.io/certificate-name: aws-developer-friendly-blog cert-manager.io/common-name: "*.aws.developer-friendly.blog" cert-manager.io/ip-sans: "" cert-manager.io/issuer-group: "" cert-manager.io/issuer-kind: ClusterIssuer cert-manager.io/issuer-name: route53-issuer cert-manager.io/uri-sans: "" labels: controller.cert-manager.io/fao: "true" name: aws-developer-friendly-blog-tls namespace: cert-manager type: kubernetes.io/tls
Enter fullscreen mode Exit fullscreen mode

Step 3: Use the TLS Certificates in Gateway

At this point, we have the required ingredients to host an application within cluster and exposing it securely through HTTPS into the world.

That's exactly what we aim for at this step. But, first, let's create a Gateway CRD that will be the entrypoint to our cluster. The Gateway can be thought of as the sibling of Ingress resource, yet more handsome, more successful, more educated and more charming19.

The key point to keep in mind is that the Gateway API doesn't come with the implementation. Infact, it is unopinionated about the implementation and you can use any networking solution that fits your needs and has support for it.

In our case, and based on the personal preference and tendency of the author 😇, we'll use Cilium as the networking solution, both as the CNI, as well as the implementation for our Gateway API.

We have covered the Cilium installation before, but, for the sake of completeness, here's the way to do it20.

# cilium/playbook.yml - name: Bootstrap the Kubernetes cluster hosts: localhost gather_facts: false become: true environment: KUBECONFIG: ~/.kube/config vars: helm_version: v3.14.4 kube_context: k3d-k3s-default tasks: - name: Install Kubernetes library ansible.builtin.pip: name: kubernetes<30 state: present - name: Install helm binary ansible.builtin.shell: cmd: "{{ lookup('ansible.builtin.url', 'https://git.io/get_helm.sh', split_lines=false) }}" creates: /usr/local/bin/helm environment: DESIRED_VERSION: "{{ helm_version }}" - name: Install Kubernetes gateway CRDs kubernetes.core.k8s: src: "{{ item }}" state: present context: "{{ kube_context }}" loop: - https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml - https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml - name: Install cilium block: - name: Add cilium helm repository kubernetes.core.helm_repository: name: cilium repo_url: https://helm.cilium.io - name: Install cilium helm release kubernetes.core.helm: name: cilium chart_ref: cilium/cilium namespace: kube-system state: present chart_version: 1.15.x kube_context: "{{ kube_context }}" values: gatewayAPI: enabled: true kubeProxyReplacement: true encryption: enabled: true type: wireguard operator: replicas: 1
Enter fullscreen mode Exit fullscreen mode

And now, let's create the Gateway CRD.

# gateway/gateway.yml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: developer-friendly-blog spec: gatewayClassName: cilium listeners: - allowedRoutes: namespaces: from: All name: http port: 80 protocol: HTTP - allowedRoutes: namespaces: from: All name: https port: 443 protocol: HTTPS tls: certificateRefs: - group: "" kind: Secret name: developer-friendly-blog-tls namespace: cert-manager - group: "" kind: Secret name: aws-developer-friendly-blog-tls namespace: cert-manager mode: Terminate
Enter fullscreen mode Exit fullscreen mode

Notice that we did not create the gatewayClassName. It comes as battery-included with Cilium. You can find the GatewayClass as soon as Cilium installation completes with the following command:

kubectl get gatewayclass
Enter fullscreen mode Exit fullscreen mode

GatewayClass is to Gateway as IngressClass is to Ingress.

Also note that we are passing the TLS certificates to this Gateway we have created earlier. That way, the gateway will terminate and offload the SSL/TLS encryption and your upstream service will receive plaintext traffic.

However, if you have set up your mTLS the way we did with Wireguard encryption (or any other mTLS solution for that matter), node-to-node and/or pod-to-pod communications will also be encrypted.

# gateway/http-to-https-redirect.yml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: https-redirect spec: parentRefs: - group: gateway.networking.k8s.io kind: Gateway name: developer-friendly-blog namespace: cert-manager sectionName: http rules: - filters: - requestRedirect: scheme: https statusCode: 301 type: RequestRedirect matches: - path: type: PathPrefix value: /
Enter fullscreen mode Exit fullscreen mode

Though not required, the above HTTP to HTTPS redirect allows you to avoid accepting any plaintext HTTP traffic on your domain.

# gateway/kustomization.yml resources: - gateway.yml - http-to-https-redirect.yml namespace: cert-manager
Enter fullscreen mode Exit fullscreen mode 
# gateway/kustomize.yml apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: gateway namespace: flux-system spec: interval: 5m path: ./gateway prune: true sourceRef: kind: GitRepository name: flux-system wait: true
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f gateway/kustomize.yml
Enter fullscreen mode Exit fullscreen mode

Step 4: HTTPS Application

That's all the things we aimed to do today. At this point, we can create our HTTPS-only application and expose it securely to the wild internet!

# app/deployment.yml apiVersion: apps/v1 kind: Deployment metadata: name: echo-server spec: replicas: 1 selector: matchLabels: app: echo-server template: metadata: labels: app: echo-server spec: containers: - envFrom: - configMapRef: name: echo-server image: ealen/echo-server name: echo-server ports: - containerPort: 80 name: http securityContext: capabilities: add: - NET_BIND_SERVICE drop: - ALL readOnlyRootFilesystem: true securityContext: runAsGroup: 1000 runAsUser: 1000 seccompProfile: type: RuntimeDefault
Enter fullscreen mode Exit fullscreen mode 
# app/service.yml apiVersion: v1 kind: Service metadata: name: echo-server spec: ports: - name: http port: 80 targetPort: http type: ClusterIP
Enter fullscreen mode Exit fullscreen mode 
# app/httproute.yml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: echo-server spec: hostnames: - echo.developer-friendly.blog - echo.aws.developer-friendly.blog parentRefs: - group: gateway.networking.k8s.io kind: Gateway name: developer-friendly-blog namespace: cert-manager sectionName: https rules: - backendRefs: - group: "" kind: Service name: echo-server port: 80 weight: 1 filters: - responseHeaderModifier: set: - name: Strict-Transport-Security value: max-age=31536000; includeSubDomains; preload type: ResponseHeaderModifier matches: - path: type: PathPrefix value: /
Enter fullscreen mode Exit fullscreen mode 
# app/configs.env PORT=80 LOGS__IGNORE__PING=false ENABLE__HOST=true ENABLE__HTTP=true ENABLE__REQUEST=true ENABLE__COOKIES=true ENABLE__HEADER=true ENABLE__ENVIRONMENT=false ENABLE__FILE=false
Enter fullscreen mode Exit fullscreen mode 
# app/kustomization.yml resources: - deployment.yml - service.yml - httproute.yml images: - name: ealen/echo-server newTag: 0.9.2 configMapGenerator: - name: echo-server envs: - configs.env replacements: - source: kind: Deployment name: echo-server fieldPath: spec.template.metadata.labels targets: - select: kind: Service name: echo-server fieldPaths: - spec.selector options: create: true - source: kind: ConfigMap name: echo-server fieldPath: data.PORT targets: - select: kind: Deployment name: echo-server fieldPaths: - spec.template.spec.containers.[name=echo-server].ports.[name=http].containerPort namespace: default
Enter fullscreen mode Exit fullscreen mode 
# app/kustomize.yml apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: app namespace: flux-system spec: interval: 5m path: ./app prune: true sourceRef: kind: GitRepository name: flux-system wait: true
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f app/kustomize.yml
Enter fullscreen mode Exit fullscreen mode

That's everything we had to say for today. We can now easily access our application as follows:

curl -v https://echo.developer-friendly.blog -sSo /dev/null
Enter fullscreen mode Exit fullscreen mode

or...

curl -v https://aws.echo.developer-friendly.blog -sSo /dev/null
Enter fullscreen mode Exit fullscreen mode 
...truncated... * expire date: Jul 30 04:44:12 2024 GMT ...truncated...
Enter fullscreen mode Exit fullscreen mode

Both will show that the TLS certificate is present. signed by a trusted CA, is valid and matches the domain we're trying to access. 🎉

You shall see the same expiry date on your certificate if accessing as follows:

kubectl get certificate \ -n cert-manager \ -o jsonpath='{.items[*].status.notAfter}'
Enter fullscreen mode Exit fullscreen mode 
2024-07-30T04:44:12Z
Enter fullscreen mode Exit fullscreen mode

As you can see, the information we get from the publicly available certificate as well as the one we get internally from our Kubernetes cluster are the same down to the second. 💪

Conclusion

These days, I am never spinning up a Kubernetes cluster without having cert-manager installed on it as its day 1 operation task. It's such a life-saver tool to have in your toolbox and you can rest assured that the TLS certificates in your cluster are always up-to-date and valid.

If you ever had to worry about the expiry date of your certificates before, those days are behind you and you can benefit a lot by employing the cert-manager operator in your Kubernetes cluster. Use it to its full potential and you shall be served greatly.

Hope you enjoyed reading this material.

Until next time 🫡, ciao 🤠 and happy hacking! 🦀
🐧 🐳

Top comments (0)