DEV Community

Cover image for Setting Up a Kubernetes Cluster with a Local Registry and Ingress in Docker using KIND
Piyush Jajoo
Piyush Jajoo

Posted on

Setting Up a Kubernetes Cluster with a Local Registry and Ingress in Docker using KIND

#kind #development #kubernetes #ingress

Kubernetes in Docker (kind) is a tool that allows you to run Kubernetes clusters locally using Docker container "nodes". It's a great tool for developers who want to test their applications in a Kubernetes environment without the overhead of a full-scale cluster.

In this blog post, we'll walk through the steps to set up a kind cluster with a Docker registry where you can push the images and pull from in the Kubernetes cluster. This will mimic what we would do in Cloud Provider managed Kubernetes clusters i.e. pull the images from OCI registries. We will also look at how to install nginx Ingress and create a LoadBalancer service.

Prerequisites

Ensure you have the following installed on your machine:

  • Docker
  • kind, there are many ways to install kind, please refer the link to install as you see fit.
  • kubectl, we will use this to interact with the cluster.

All the scripts discussed in this blog are uploaded to this github repository.

What are we going to create?

  • Multi node kubernetes cluster with control-plane and worker nodes. Examples of worker nodes with labels and taints.
  • Create a local registry to push and pull locally built docker images
  • Deploy Ingress so that we can access the endpoints exposed by the services
  • Validate the setup by installing pods with dummy services

This setup is pretty much what you need to validate your applications locally, this setup will be extremely helpful and cost efficient while developing services for kubernetes environment.

Create k8s cluster with multiple nodes and configure cluster with containerd registry config dir

The following is the Cluster cluster-config.yaml with control-plane and worker nodes. The api-server and other control plane components will be on the node with role control-plane. And nodes with worker role will have your pods. You will observe the controller-manager, api-server, scheduler, etcd, coredns, kindnet, kube-proxy and local-path-provisioner pods will be deployed on the control-plane node by default.

The worker nodes will have kindnet and kube-proxy by default. You will also observe that we are configuring the cluster with containerd registry config.

We are also exposing ports 80/443/5678 so that localhost can hit those port from machine.

kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: "platformwale" # configure cluster with containerd registry config dir enabled containerdConfigPatches: - |- [plugins."io.containerd.grpc.v1.cri".registry] config_path = "/etc/containerd/certs.d" nodes: # control plane node # this comes with taint so that control-plane node will not accept any other pods by default - role: control-plane image: "kindest/node:v1.27.3" # worker nodes # worker with no node labels, so pods with no nodeSelectors will schedule here - role: worker image: "kindest/node:v1.27.3" # worker with node label role=app # pods with nodeSelectors role=app will schedule here - role: worker image: "kindest/node:v1.27.3" labels: role: app # worker with node label role=ingress # pods with nodeSelectors role=ingress and toleration to taint role=ingress:NoSchedule will schedule here - role: worker image: "kindest/node:v1.27.3" labels: role: ingress # extraPortMappings allow the localhost to make requests to the Ingress controller over ports 80/443/5678 extraPortMappings: - containerPort: 80 hostPort: 80 protocol: TCP - containerPort: 443 hostPort: 443 protocol: TCP - containerPort: 5678 hostPort: 5678 protocol: TCP # add taint to the node such that only pods tolerating the taint will be scheduled on this node kubeadmConfigPatches: - | kind: JoinConfiguration nodeRegistration: kubeletExtraArgs: register-with-taints: "role=ingress:NoSchedule"
Enter fullscreen mode Exit fullscreen mode

Create the kind cluster by submitting the cluster-config.yaml file as follows -

kind create cluster --config cluster-config.yaml
Enter fullscreen mode Exit fullscreen mode

You will see something like below on successful creation of kind cluster -

$ kind create cluster --config cluster-config.yaml Creating cluster "platformwale" ... ✓ Ensuring node image (kindest/node:v1.27.3) 🖼 ✓ Preparing nodes 📦 📦 📦 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 ✓ Joining worker nodes 🚜 Set kubectl context to "kind-platformwale" You can now use your cluster with: kubectl cluster-info --context kind-platformwale Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂 $ kubectl cluster-info --context kind-platformwale Kubernetes control plane is running at https://127.0.0.1:58931 CoreDNS is running at https://127.0.0.1:58931/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Enter fullscreen mode Exit fullscreen mode

Validate that you can see the kind cluster you created above -

$ kind get clusters platformwale
Enter fullscreen mode Exit fullscreen mode

Validate that the nodes are created successfully -

$ kubectl get nodes NAME STATUS ROLES AGE VERSION platformwale-control-plane Ready control-plane 9m20s v1.27.3 platformwale-worker Ready <none> 8m55s v1.27.3 platformwale-worker2 Ready <none> 9m1s v1.27.3 platformwale-worker3 Ready <none> 8m56s v1.27.3
Enter fullscreen mode Exit fullscreen mode

Please read this configurationdocumentation to learn more options to configure the KIND cluster.

Create the registry container and configure cluster nodes for the registry

Create the registry container and configure the cluster nodes for registry access as below.

This command will pull the registry container locally and will start the container. This container will be used as the local docker registry.

# start the registry container reg_name='kind-registry' reg_port='5001' if [ "$(docker inspect -f '{{.State.Running}}' "${reg_name}" 2>/dev/null || true)" != 'true' ]; then docker run \ -d --restart=always -p "127.0.0.1:${reg_port}:5000" --name "${reg_name}" \ registry:2 fi
Enter fullscreen mode Exit fullscreen mode

Now add the registry config to the nodes as below. This is necessary because localhost resolves to loopback addresses that are network-namespace local. In other words, localhost in the container is not localhost on the host. We want a consistent name that works from both ends, so we tell containerd to alias localhost:${reg_port} to the registry container when pulling images.

kind_cluster_name="platformwale" reg_port='5001' REGISTRY_DIR="/etc/containerd/certs.d/localhost:${reg_port}" for node in $(kind get nodes --name ${kind_cluster_name}); do docker exec "${node}" mkdir -p "${REGISTRY_DIR}" cat <<EOF | docker exec -i "${node}" cp /dev/stdin "${REGISTRY_DIR}/hosts.toml" [host."http://${reg_name}:5000"] EOF done
Enter fullscreen mode Exit fullscreen mode

Now connect the registry to the cluster network, this allows kind to bootstrap the network and ensure they are on the same network.

reg_name='kind-registry' if [ "$(docker inspect -f='{{json .NetworkSettings.Networks.kind}}' "${reg_name}")" = 'null' ]; then docker network connect "kind" "${reg_name}" fi
Enter fullscreen mode Exit fullscreen mode

Now document the local registry. The standard of defining the local registry is defined in detail in this doc. This is a standard way for cluster configuration tools to record how developer tools should interact with the local registry as well as a standard way for developer tools to read that information when pushing images to the cluster.

reg_port='5001' cat <<EOF | kubectl apply -f - apiVersion: v1 kind: ConfigMap metadata: name: local-registry-hosting namespace: kube-public data: localRegistryHosting.v1: | host: "localhost:${reg_port}" help: "https://kind.sigs.k8s.io/docs/user/local-registry/" EOF
Enter fullscreen mode Exit fullscreen mode

At this point you have finished creating the cluster as well enabled a local docker registry.

Connect to the Private Registry

We will pull a sample app from the remote docker registry, tag it and push it to the local registry we created above. We will then start the pod using the image pulled from the local registry.

Here's an example:

# pull a sample hello-app from remote registry docker pull gcr.io/google-samples/hello-app:1.0 # tag the pulled docker image for local registry docker tag gcr.io/google-samples/hello-app:1.0 localhost:5001/hello-app:1.0 # push the docker image to the local registry docker push localhost:5001/hello-app:1.0
Enter fullscreen mode Exit fullscreen mode

Submit following yaml to create hello-server deployment to use the hello-app docker image from the local registry.

kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: labels: app: hello-server name: hello-server namespace: default spec: replicas: 1 selector: matchLabels: app: hello-server template: metadata: labels: app: hello-server spec: nodeSelector: role: app containers: - image: localhost:5001/hello-app:1.0 imagePullPolicy: IfNotPresent name: hello-app EOF
Enter fullscreen mode Exit fullscreen mode

Validate that the pod is running successfully as below and it runs on platformwale-worker2 node as we have used nodeSelector: role=app -

$ kubectl get po -n default -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES hello-server-bfc485c98-jzxbb 1/1 Running 0 14s 10.244.2.3 platformwale-worker2 <none> <none>
Enter fullscreen mode Exit fullscreen mode

This proves that we have a working kubernetes cluster with multiple nodes as well as we are able to push and pull images from local docker registry.

The local registry can also be used for bootstrapping local development environments faster. Consider you have a big local vagrant or docker-compose setup which pulls huge images, you can setup this local registry on one of the hosted machines in your office network, and then all the engineers in your team can use this registry to pull the images to setup the local environment instead of pulling from public network. This will speed up the environment setup as well as save the network bandwidth.

Deploy Ingress and validate LoadBalancer k8s service

In this section we will deploy nginx Ingress service on the ingress nodepool we created earlier. We will also deploy LoadBalancer k8s service along with sample apps to validate the nginx deployment.

We have modified the public nginx deploy.yaml to tolerate taint -> role:ingress:NoSchedule and use nodeSelector -> role=ingress to deploy the nginx pods on platformwale-worker3 node which we have configured with taints and label such that we only deploy the nginx deployments. Use the modified nginx.yaml as below -

kubectl apply -f https://raw.githubusercontent.com/piyushjajoo/kind-with-local-registry-and-ingress/master/nginx.yaml
Enter fullscreen mode Exit fullscreen mode

Validate all the nginx pods are running -

$ kubectl get pods -n ingress-nginx -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ingress-nginx-admission-create-4kxmk 0/1 Completed 0 34m 10.244.1.2 platformwale-worker3 <none> <none> ingress-nginx-admission-patch-kx4zv 0/1 Completed 1 34m 10.244.1.3 platformwale-worker3 <none> <none> ingress-nginx-controller-57d7c6cb58-g2gdf 1/1 Running 0 34m 10.244.1.4 platformwale-worker3 <none> <none>
Enter fullscreen mode Exit fullscreen mode

Pull following docker image to the local registry, we will use it for the Ingress setup validation as below -

# pull docker image docker pull hashicorp/http-echo:0.2.3 # tag the image for local registry docker tag hashicorp/http-echo:0.2.3 localhost:5001/http-echo:0.2.3 # push the docker image docker push localhost:5001/http-echo:0.2.3
Enter fullscreen mode Exit fullscreen mode

This will also install MetalLB loadbalancer. NOTE: on On macOS and Windows, docker does not expose the docker network to the host. Because of this limitation, containers (including kind nodes) are only reachable from the host via port-forwards, however other containers/pods can reach other things running in docker including loadbalancers. If you are on Mac or Windows, you can skip installing MetalLB.

# install metallb echo "install metallb" kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml # wait until MetalLB pods (controller and speaker) are ready echo "wait until MetalLB pods (controller and speaker) are ready" kubectl wait --namespace metallb-system \ --for=condition=ready pod \ --selector=app=metallb \ --timeout=90s # find the cidr range for kind network echo "find the cidr range for the kind network" output=$(docker network inspect -f '{{.IPAM.Config}}' kind) ipv4_cidr=$(echo "$output" | grep -oE '([0-9]+\.[0-9]+)\.[0-9]+\.[0-9]+/[0-9]+' | head -n 1) ipv4_parts=$(echo "$ipv4_cidr" | cut -d '.' -f 1,2) echo "IPv4 CIDR Range (First 2 Parts): $ipv4_parts" # configure ip address pool echo "configuring ip address pool" kubectl apply -f - <<EOF apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: example namespace: metallb-system spec: addresses: - $ipv4_parts.255.200-$ipv4_parts.255.250 --- apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: empty namespace: metallb-system EOF
Enter fullscreen mode Exit fullscreen mode

Validate the Ingress object using ClusterIP services as well as validate LoadBalancer type service using the script below. You can skip creating LoadBalancer type service below if you are on Mac or Windows.

The script below will deploy pods and setup Ingress object to divert the traffic to the pods based on path configured.

kubectl apply -f - <<EOF kind: Pod apiVersion: v1 metadata: name: foo-app labels: name: foo-app app: http-echo spec: containers: - name: foo-app image: localhost:5001/http-echo:0.2.3 args: - "-text=foo" --- kind: Pod apiVersion: v1 metadata: name: bar-app labels: name: bar-app app: http-echo spec: containers: - name: bar-app image: localhost:5001/http-echo:0.2.3 args: - "-text=bar" --- kind: Service apiVersion: v1 metadata: name: foo-service spec: selector: name: foo-app ports: # Default port used by the image - port: 5678 --- kind: Service apiVersion: v1 metadata: name: bar-service spec: selector: name: bar-app ports: # Default port used by the image - port: 5678 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: /$2 spec: rules: - http: paths: - pathType: Prefix path: /foo(/|$)(.*) backend: service: name: foo-service port: number: 5678 - pathType: Prefix path: /bar(/|$)(.*) backend: service: name: bar-service port: number: 5678 --- kind: Service apiVersion: v1 metadata: name: foo-service-lb spec: type: LoadBalancer selector: name: foo-app app: http-echo ports: # Default port used by the image - port: 5678 --- kind: Service apiVersion: v1 metadata: name: bar-service-lb spec: type: LoadBalancer selector: name: bar-app app: http-echo ports: # Default port used by the image - port: 5678 EOF
Enter fullscreen mode Exit fullscreen mode

Validate the Ingress as below -

# validate cluster ips via Ingress object # should output "foo-app" echo "validating foo-app via Ingress object" curl localhost/foo/hostname # should output "bar-app" echo "validating bar-app via Ingress object" curl localhost/bar/hostname
Enter fullscreen mode Exit fullscreen mode

Validate the LoadBalancer service as below, if you are on Mac or Windows, as mentioned earlier docker doesn't expose the docker network to the host, hence you won't be able to use the LoadBalancer IP directly instead you will need to port forward the service to access.

 ## on linux you can validate LoadBalancer as below # validate load balancer service echo "validating loadbalancer, note on macOS and Windows, docker does not expose the docker network to the host. Because of this limitation, containers (including kind nodes) are only reachable from the host via port-forwards, however other containers/pods can reach other things running in docker including loadbalancers" FOO_LB_IP=$(kubectl get svc/foo-service-lb -n default -o=jsonpath='{.status.loadBalancer.ingress[0].ip}') BAR_LB_IP=$(kubectl get svc/bar-service-lb -n default -o=jsonpath='{.status.loadBalancer.ingress[0].ip}') # should output foo and bar on separate lines  for _ in {1..10}; do curl ${FOO_LB_IP}:5678 curl ${BAR_LB_IP}:5678 done ## on Mac or Windows port-forward and hit the service as below # validate foo-service-lb kubectl port-forward -n default svc/foo-service-lb 5678:5678 # in browser hit localhost:5678 or do curl as below, you will see foo as output $ curl localhost:5678 foo # validate bar-service-lb kubectl port-forward -n default svc/bar-service-lb 5678:5678 # in browser hit localhost:5678 or do curl as below, you will see bar as output $ curl localhost:5678 bar
Enter fullscreen mode Exit fullscreen mode

To setup LoadBalancer on Mac using MetalLB refer this documentation

Cleanup

Destroy the kind cluster as well as registry as below -

# delete kind cluster echo "deleting kind cluster" kind delete cluster --name "platformwale" # delete registry echo "deleting registry" docker rm -f $(docker ps -a | grep registry | awk -F ' ' '{print $1}')
Enter fullscreen mode Exit fullscreen mode

Conclusion

And that's it! You now have a local Kubernetes development environment with a local Docker registry. This setup allows you to build, push, and deploy your Docker images without needing to push them to a public registry. This is useful in setting up local development environments faster and save network bandwidth. You also installed Ingress service to create LoadBalancer k8s service, this is useful to mimic the LoadBalancer behavior as you might need to do in an actual cloud provider.

Resources

Author Notes

Feel free to reach out with any concerns or questions you have. I will make every effort to address your inquiries and provide resolutions. Stay tuned for the upcoming blog in this series dedicated to Platformwale (Engineers who work on Infrastructure Platform teams).

Originally published at https://platformwale.blog on Aug 15, 2023.

Top comments (0)