DEV Community

Ishan Khare
Ishan Khare

Posted on

Kubernetes internals

#kubernetes #jobs #rbac #inclusteraccess

In this blog post we are going to setup a kubernetes cluster with automated certificate generation using certbot. Will cover up some interesting concepts of kubernetes along the way, like:

So let's get started.

Why do we need this?

When using a traditional VM's/instances, one has access to ssh into a fixed instance with an assigned and fixed public IP that the DNS can resolve to. Once the DNS is set to resolve your hostname to your instance, you can install certbot on it and generate the certs on that instance.

With kubernetes, things get a bit tricky. You will still have a set of instances in your cluster, but they aren't directly accessible from outside the cluster. Plus you cannot preempt on which node your nginx or other ingress pod will be scheduled to run. Hence the most straight forward way to setup is doing everything through kubernetes and docker. This will also provide us with a few advantages:

  • The cert generation process will be documented as part of the kubernetes manifest files and Dockerfiles
  • Our solution would work for any number of pods and services. Basically we won't be bogged down due to issues arising because of scalability like when adding more pods or even adding more nodes to our cluster. Things will work seamlessly until we don't create a new cluster.
  • Our solution will be platform independent, just like docker and kubernetes. It will work on any cloud provider GCP, AWS or even Azure(hopefully).

I'll be using GCP for some parts of this blog post but replicating those parts on other cloud platforms is pretty straight forward.

We'll start by reserving a static IP for our cluster and then forward a DNS record to that IP so that certbot can easily resolve to our cluster.

Make sure to keep the region of the static IP and your cluster same.

Now that we have the static IP, we can move on to the kubernetes part - the fun part.

The LoadBalancer service

The first thing that we setup is the load balancer service, we will then use this service to resolve to the pods running our certbot client and later on our own application pods.
Below is the YAML for our LoadBalancer service. It utilizes the static IP we created in the previous step.

svc.yml

apiVersion: v1 kind: Service metadata: name: certbot-lb labels: app: certbot-lb spec: type: LoadBalancer loadBalancerIP: X.X.X.X ports: - port: 80 name: "http" protocol: TCP - port: 443 name: "tls" protocol: TCP selector: app: certbot-generator
Enter fullscreen mode Exit fullscreen mode

Most of it is self explanatory, few fields of interest are:

  • spec.type : LoadBalancer - This spins up a Google Cloud LoadBalancer on GCP and AWS Elastic LoadBalancer on AWS
  • spec.loadBalancerIP - This assign the previously generated static IP to our loadBalancer, now all traffic coming to our IP address is funneled into this load balancer.
  • ports.port - We opened 2 TCP ports, port 80 and port 443 for accepting HTTP and HTTPS traffic respectively.
  • spec.selector - These are the set of labels that allow us to govern which pods can our loadBalancer resolve to. We'll later use the same set of labels in our Job and Pod templates

Let's deploy this service to our cluster.

$ kubectl apply -f svc.yml service "certbot-lb" created
Enter fullscreen mode Exit fullscreen mode

If we see the status of our service now, we should see this

$ kubectl get svc certbot-lb NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE certbot-lb LoadBalancer X.X.X.X X.X.X.X 80:30271/TCP,443:32224/TCP 1m
Enter fullscreen mode Exit fullscreen mode

If you're trying this on minikube, LoadBalancer service is not available. Besides, it makes no sense trying to generate a SSL cert on your local environment.

Next we have to think about where to run our certbot container.

Job Controllers

The Job controllers of kubernetes allows us to schedule pods which run to completion. That is, these are jobs that if finished without error, need not be run again in future. This is exactly what we want when generating SSL certs. Once the certbot process is done and has given us the certificates, we no longer need that container to be running. Also we don't want kubernetes to restart this pod when it exits. However we can ask kubernetes to automatically reschedule the pod in case is case the pod exists with a failure/error.

So lets write a Job spec file that will generate the SSL cert for us using certbot. Certbot provides an official docker container that we can just reuse in our case.

jobs.yml

apiVersion: batch/v1 kind: Job metadata: name: certbot spec: template: metadata: labels: app: certbot-generator spec: containers: - name: certbot image: certbot/certbot command: ["certbot"] args: ["certonly", "--noninteractive", "--agree-tos", "--staging", "--standalone", "-d", "staging.ishankhare.com", "-m", "me@ishankhare.com"] ports: - containerPort: 80 - containerPort: 443 restartPolicy: "Never"
Enter fullscreen mode Exit fullscreen mode

Few important fields here are:

  • spec.template.metadata.labels - This matches with the spec.selector specified in our LoadBalancer service. This brings our pod under our loadBalancer. Now everything coming on port 80 and 443 will be funneled to our pod.
  • spec.template.spec.containers[0].image - The certbot/certbot docker image. Kubernetes will do a docker pull certbot/certbot on the server for us when scheduling this pod.
  • spec.template.spec.containers[0].command - The command to run in the pod.
  • spec.template.spec.containers[0].args - The arguments to the above command. We're using the standalone mode of certbot to generate the certs here as it makes things straightforward. You can read more about this command in certbot docs
  • spec.template.spec.containers[0].ports - We've opened port 80 and 443 for our container.

Before deploying this to our cluster, make sure that the domain you specify is actually pointing to the static IP we setup in the previous steps.

Let's deploy this to our cluster:

$ kubectl apply -f jobs.yml job.batch "certbot" created
Enter fullscreen mode Exit fullscreen mode

This Job will spin-up a single pod for us, we can get that:

$ kubectl get pods NAME READY STATUS RESTARTS AGE certbot-sgd4w 1/1 Running 0 5s
Enter fullscreen mode Exit fullscreen mode

We can now see the STDOUT logs on this pod

$ kubectl logs -f certbot-sgd4w Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator standalone, Installer None Obtaining a new certificate Performing the following challenges: http-01 challenge for staging.ishankhare.com Waiting for verification... Cleaning up challenges IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/staging.ishankhare.com/privkey.pem Your cert will expire on 2019-03-04. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal.
Enter fullscreen mode Exit fullscreen mode

After this, the certbot process exits without error and so does our pod. If we now list our pods

$ kubectl get pods NAME READY STATUS RESTARTS AGE certbot-sgd4w 0/1 Completed 0 45m
Enter fullscreen mode Exit fullscreen mode

We see only a single pod whose status is completed.

We can add .spec.ttlSecondsAfterFinish to our jobs.yml like so:

spec: ttlSecondsAfterFinish: 10

This will automatically garbage collect our certbot pod after its finished.
Only supported in kubernetes v1.12 alpha. Hence not recommended right now.

Where to save the generated SSL certificate?

We currently have 2 options when it comes to saving the certs:

  1. Save it in a mounted volume. (Not recommended)
    • Would require a PersistentVolume and a PersistentVolumeClaim
    • The certificate might be just a few KB in size, but the minimum volume size that GKE allots is 1Gi. Hence this is highly inefficient.
  2. Use kubernetes Secrets. (Recommended)
    • This would also require us to use kubernetes client's in cluster configuration, which is straightforward to use and is usually the recommended way in such cases.
    • Would require us to setup ClusterRole and ClusterRoleBinding.
    • Can be configured to allow specific access to specific people using the concept of rbac (RoleBasedAccessControl)

Why in-cluster access?

To understand this, we need to go a little deeper into our current architecture.
Our current setup looks roughly like this:

Cluster Architecture

As we can see here, the certs generated by our certbot Job Controller Pod are already inside the cluster. We want these cert credentials to be stored on the secrets.

When we fetch/create/modify secrets in normal flow, its usually done through the kubectl client like:

$ kubectl get secrets NAME TYPE DATA AGE default-token-hks8k kubernetes.io/service-account-token 3 1m $ kubectl create secret Create a secret using specified subcommand. Available Commands: docker-registry Create a secret for use with a Docker registry generic Create a secret from a local file, directory or literal value tls Create a TLS secret Usage: kubectl create secret [flags] [options]
Enter fullscreen mode Exit fullscreen mode

But, in our case, we want to store these credentials as secrets from inside a Pod which is itself inside the cluster. This is exactly what I meant by in-cluster access above.

What all do we need for In-Cluster access?

We'll need the first 2 mentioned above in the same Pod that is trying to access the secrets.
Hence its best to extend the certbot/certbot docker image with the above dependencies added in the container image itself.

This will change our cluster architecture a bit. The image below shows the rough cluster arch with our modified setup.

Cluster Architecture with kube-proxy and kubectl

Let's first create our Dockerfile for our extended container:

FROM certbot/certbot COPY ./script.sh /script.sh RUN wget https://storage.googleapis.com/kubernetes-release/release/v1.6.3/bin/linux/amd64/kubectl RUN chmod +x kubectl RUN mv kubectl /usr/local/bin ENTRYPOINT /script.sh
Enter fullscreen mode Exit fullscreen mode

We have also used an extra script.sh in our container. We'll use this script as our entrypoint instead of the default entrypoint of certbot/certbot image.
This allows us to start our auxiliary kube-proxy and use kubectl as we desire. We present the script.sh below:

#!/bin/sh # start kube proxy kubectl proxy & certbot certonly --noninteractive --force-renewal --agree-tos --staging --standalone -d staging.ishankhare.com -m me@ishankhare.com kubectl create secret tls cert --cert=/etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem \ --key=/etc/letsencrypt/live/staging.ishankhare.com/privkey.pem # kill the kubectl process running in background kill %1
Enter fullscreen mode Exit fullscreen mode
  • First we start our kube-proxy as a background process.
  • Next we run our certbot command for generating the cert.
  • The certbot command generates certs for us at /etc/letsencrypt/live/staging.ishankhare.com/, we now use these paths to create the tls type Secret using kubectl create secret tls. This command has the following syntax: 
$ kubectl create secret tls -h Create a TLS secret from the given public/private key pair. The public/private key pair must exist before hand. The public key certificate must be .PEM encoded and match the given private key. Examples: # Create a new TLS secret named tls-secret with the given key pair: kubectl create secret tls tls-secret --cert=path/to/tls.cert --key=path/to/tls.key
Enter fullscreen mode Exit fullscreen mode

Hence our command above will create a secret named cert

  • Finally we kill our kube-proxy background process.

Update our jobs.yml

Since we are now using an updated Dockerfile with own own script as its entrypoint, we can modify the Job manifest file like below:

apiVersion: batch/v1 kind: Job metadata: #labels: # app: certbot-generator name: certbot spec: template: metadata: labels: app: certbot-generator spec: containers: - name: certbot image: ishankhare07/certbot:0.0.6 - containerPort: 80 - containerPort: 443 restartPolicy: "Never"
Enter fullscreen mode Exit fullscreen mode

With this we are ready to run our job now.

First verify that our LoadBalancer service is up:

$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE certbot-lb LoadBalancer 10.11.240.100 X.X.X.X 80:31855/TCP,443:32457/TCP 32m # delete our older job $ kubectl delete job certbot job.batch "certbot" deleted
Enter fullscreen mode Exit fullscreen mode

Now we deploy our new Job file

$ kubectl apply -f jobs.yml job.batch "certbot" created # list the pod created for the job $ kubectl get pods NAME READY STATUS RESTARTS AGE certbot-5nn5h 1/1 Running 0 3s # get STDOUT logs for this pod $ kubectl logs -f certbot-5nn5h Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator standalone, Installer None Obtaining a new certificate Performing the following challenges: http-01 challenge for staging.ishankhare.com Waiting for verification... Cleaning up challenges Starting to serve on 127.0.0.1:8001IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/staging.ishankhare.com/privkey.pem Your cert will expire on 2019-03-13. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal. Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:default" cannot create secrets in the namespace "default"
Enter fullscreen mode Exit fullscreen mode

The certificate generation was successful. But we cannot yet write to Secrets. The last line in the above output shows us that.

Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:default" cannot create secrets in the namespace "default"

Why is that? Because RBAC....

RBAC or Role Based Access Control in kubernetes consists of 2 parts:

  • Role/ClusterRole
  • RoleBinding/ClusterRoleBinding

We will be using ClusterRole and ClusterRoleBinding approach to provide cluster wide access to our Pod. More fine-grained, role-based access can be provided with the Role and RoleBinding approach which can be referred from the docs - https://kubernetes.io/docs/reference/access-authn-authz/rbac/

Let's create 2 files called rbac-cr.yml and rbac-crb.yml with the following contents:
rbac-cr.yml

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: namespace: default name: secret-reader rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "create"]
Enter fullscreen mode Exit fullscreen mode

This allows access to Secrets, in-particular to get, list and create Secrets. The last verb create is what concerns us.

rbac-crb.yml

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: secret-reader subjects: - kind: User name: <your account here> namespace: default - kind: ServiceAccount name: default namespace: default roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io
Enter fullscreen mode Exit fullscreen mode

Fields of interest here are .subjects. It is an array and we have defined two kinds in this.

  1. kind: User : This refers to the current user, who is executing these commands using kubectl. This is required so as to grant the current user enough permissions to grant the .rules.resources and .rules.verbs related access that we have defined in the rbac-cr.yml i.e. our ClusterRole definition.
  2. kind: ServiceAccount : This refers to the account use inside the cluster when our pod will be creating the secrets using the kube-proxy.

We push this to our kubernetes cluster in this particular order only:

$ kubectl apply -f rbac-crb.yml clusterrolebinding.rbac.authorization.k8s.io "secret-reader" created $ kubectl apply -f rbac-cr.yml clusterrole.rbac.authorization.k8s.io "secret-reader" created
Enter fullscreen mode Exit fullscreen mode

The order is important, because we first need to create ClusterRoleBinding (defined in rbac-crb.yml) and then using that binding on our account, we are going to apply a ClusterRole (defined in rbac-cr.yml).

Finally re-run jobs.yml again

I say re-run because since previous job still exists:

$ kubectl get jobs NAME DESIRED SUCCESSFUL AGE certbot 1 1 1h
Enter fullscreen mode Exit fullscreen mode

and because of this job, a stagnant Pod exists as well:

$ kubectl get pods NAME READY STATUS RESTARTS AGE certbot-5nn5h 0/1 Completed 0 1h
Enter fullscreen mode Exit fullscreen mode

If we just try to apply our jobs.yml file now, kubernetes sees that nothing as changed in our yml manifest and chooses to take no action:

$ kubectl apply -f jobs.yml job.batch "certbot" unchanged
Enter fullscreen mode Exit fullscreen mode

Hence we delete and apply our job from scratch:

$ kubectl delete job certbot job.batch "certbot" deleted $ kubectl apply -f jobs.yml job.batch "certbot" created # get the pod created by the above job $ kubectl get pods NAME READY STATUS RESTARTS AGE certbot-c9h6m 1/1 Running 0 2s
Enter fullscreen mode Exit fullscreen mode

We try to see the STDOUT logs of this container

$ kubectl logs -f certbot-c9h6m Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator standalone, Installer None Obtaining a new certificate Performing the following challenges: http-01 challenge for staging.ishankhare.com Waiting for verification... Cleaning up challenges Starting to serve on 127.0.0.1:8001IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/staging.ishankhare.com/privkey.pem Your cert will expire on 2019-03-13. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal. secret "cert" created
Enter fullscreen mode Exit fullscreen mode

The last line says secret "cert" created

Mission accomplished!

We can now see this recently created secret:

$ kubectl get secret cert NAME TYPE DATA AGE cert kubernetes.io/tls 2 9m
Enter fullscreen mode Exit fullscreen mode

If you actually want to get the contents of the cert you can:

$ kubectl get secret cert -o yaml > cert.yml # or $ kubectl get secret cert -o json > cert.json
Enter fullscreen mode Exit fullscreen mode

It is always desirable to redirect the above streams to a file rather than printing them directly to the console.

With this goal achieved, I'll wrap up this post here now. I'll be back with more posts soon detailing on:

  1. Options available for using these certificated once we have created them.
  2. Proper use of Init Containers along with the Job Controllers used in this post and our good old Pods and Deployments to see how we can make these interdependent services wait for the depending services to complete before spawning themselves.

Do let me know what you think of this post and if you have any questions in the comments section below.

This post was originally published on my blog ishankhare.com

Top comments (0)