DEV Community

Simon Aubury
Simon Aubury

Posted on

Confluent Kafka Kubernetes Operator — Setup and Playing!

Originally published Sat Aug 03 2019 at https://medium.com/@simon.aubury/confluent-kafka-kubernetes-operator-setup-and-playing-acd2c3c9fe5e

Confluent Kafka Kubernetes Operator — Setup and Playing!

Confluent released their Kafka Kubernetes Operator this week (as part of their CP 5.3 release). The TL;DR summary is this gives you a straight forward way to deploy your own clustered & secured Apache Kafka on Kubernetes (ZooKeeper, Apache Kafka) along with the cool bits (Kafka Connect, KSQL, Schema Registry, REST Proxy, Control Center).

This is pretty cool as it opens the way for Kafka deployments to on premises (eg., OpenShift or Rancher) and eases the way for clouds deploys (EKS, AKS, GKE). It also automates security configuration setup (setting up kerberos authentication in Kafka is a job for the patient!).

So, let’s see how to see this up.

Machine Setup

I recently got a new Mac — so this was a good opportunity to document the steps necessary to start from a clean machine. This demonstration uses the Google (GCP) Kubernetes Engine

Kubernetes Helm CLI

First — let’s install the kubernetes command line tools (assuming *brew* is already installed)

brew install kubernetes-helmbrew install kubernetes-cli
Enter fullscreen mode Exit fullscreen mode

Google Cloud SDK

I’ll be using the Google (GCP) Kubernetes Engine. You’ll need to install the google-cloud-sdk . Google has great quick-start instructions . In short, download, extract and add to your PATH

Confluent Kubernetes Operator Helm bundle

Next, we actually need to download the Confluent Helm bundle. This Confluent bundle comprises Helm charts, templates, and scripts used to deploy Confluent Operator and Confluent Platform components for your Kubernetes cluster. Remember this is an Enterprise feature (ie., paid for if used in production).

Download and extract the bundle from Confluent Platform Operations from https://docs.confluent.io/current/installation/operator/co-deployment.html

tar xvfz confluent-operator-20190726-v0.65.0.tar.gz cd confluent-operator-20190726-v0.65.0/helm
Enter fullscreen mode Exit fullscreen mode

GCP Kubernetes Engine Provision

Adding a Kubernetes clusters

Navigating to GCP Kubernetes Engine ; click “create cluster”

Adding a Kubernetes clustersAdding a Kubernetes clusters

Add a node pool

For the new cluster; add a lot of capacity. I found 16 nodes of 2 vCPU’s was enough to get going

Add a node poolAdd a node pool

Adding resource quota

Initially I hit a problem where I could not provision enough resource due to a quota on CPUs.

Be sure to increase the CPU quotaBe sure to increase the CPU quota

If this happens to you navigate to IAM Admin Quotas (https://console.cloud.google.com/iam-admin/quotas) and increase to something like 32 for the Compute Engine API CPUs maximum

Update quotaUpdate quota

Setup GCP Client

Now you’ll need to initialises and authenticate your setup

gcloud init gcloud auth list gcloud config set compute/region us-central1 gcloud config set compute/zone us-central1-a gcloud container clusters get-credentials my-kafka-cluster
Enter fullscreen mode Exit fullscreen mode

Next, setup a Service Account

kubectl create serviceaccount tiller -n kube-system kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount kube-system:tiller helm init --service-account tiller
Enter fullscreen mode Exit fullscreen mode

Install Confluent Operator

Good — now my laptop can communicate with a remote GCP Kubernetes Engine. Now to install the Confluent Operator. These steps are pretty much verbatim from https://docs.confluent.io/current/installation/operator/co-deployment.html

helm install -f ./providers/gcp.yaml --name operator --namespace operator --set operator.enabled=true ./confluent-operator kubectl -n operator patch serviceaccount default -p '{"imagePullSecrets": [{"name": "confluent-docker-registry" }]}' kubectl get pods -n operator | grep "manager"
Enter fullscreen mode Exit fullscreen mode

Install ZooKeeper

First — let’s install 3 zookeeper nodes

helm install -f ./providers/gcp.yaml --name zookeeper --namespace operator --set zookeeper.enabled=true ./confluent-operator kubectl get zookeeper zookeeper -ojsonpath='{.status.phase}' -n operator
Enter fullscreen mode Exit fullscreen mode

Setup DNS (Optional)

Not strictly necessary — but if you want to have direct (external) access to your Kafka cluster, you’ll need to provide an accessible DNS that can be pushed into your LISTENER config. The settings are in helm/providers/gcp.yaml

Changes to helm/providers/gcp.yamlChanges to helm/providers/gcp.yaml

Install Kafka brokers

A very similar approach to setting up 3 kafka brokers

helm install -f ./providers/gcp.yaml --name kafka --namespace operator --set kafka.enabled=true ./confluent-operator kubectl get pods -n operator | grep kafka
Enter fullscreen mode Exit fullscreen mode

Install Schema Registry

Now to install a 2 node schema registry

helm install -f ./providers/gcp.yaml --name schemaregistry --namespace operator --set schemaregistry.enabled=true ./confluent-operator kubectl get pods -n operator | grep schemaregistry
Enter fullscreen mode Exit fullscreen mode

Install Kafka Connect

Now for Kafka connect

helm install -f ./providers/gcp.yaml --name connect --namespace operator --set connect.enabled=true ./confluent-operator kubectl get pods -n operator | grep connectors
Enter fullscreen mode Exit fullscreen mode

Install Confluent Control Center 

helm install -f ./providers/gcp.yaml --name controlcenter --namespace operator --set controlcenter.enabled=true ./confluent-operator kubectl get pods -n operator | grep controlcenter
Enter fullscreen mode Exit fullscreen mode

Install Confluent KSQL 

helm install -f ./providers/gcp.yaml --name ksql --namespace operator --set ksql.enabled=true ./confluent-operator kubectl get pods -n operator | grep ksql
Enter fullscreen mode Exit fullscreen mode

Running Workload

After the deployments have completed, the running workloads should look like this

Kubernetes Workloads on GCPKubernetes Workloads on GCP

Does it work — let’s test it …

Test Kafka from within the cluster

Launch a bash session on a kafka broker

kubectl -n operator exec -it kafka-0 bash

Create a local kafka.properties file

cat << EOF > kafka.properties bootstrap.servers=kafka:9071 sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="test" password="test123"; sasl.mechanism=PLAIN security.protocol=SASL_PLAINTEXT EOF
Enter fullscreen mode Exit fullscreen mode

And then run a test

kafka-topics --command-config kafka.properties --bootstrap-server kafka:9092 --list exit
Enter fullscreen mode Exit fullscreen mode

Test local KSQL client against remote KSQL server

Establish a port mapping tunnel

kubectl port-forward svc/ksql 8088:8088 -n operator
Enter fullscreen mode Exit fullscreen mode

And start a ksql session (locally) connecting to a remote server

ksql list topics; exit
Enter fullscreen mode Exit fullscreen mode

Test Confluent Control Center

Now let’s test access to Confluent Control Center. Establish a port mapping tunnel for port 9021

kubectl port-forward svc/controlcenter 9021:9021 -n operator
Enter fullscreen mode Exit fullscreen mode

And navigate to http://localhost:9021

The initial login is admin and the default password is Developer1.

Behold — Confluent Control CenterBehold — Confluent Control Center

Teardown and Cleanup

Be sure to remove the deployment once you’ve finished testing. For GCP Kubernetes Engine it’s pretty much down to deleting the cluster

Things not to do to productionThings not to do to production

Final Thoughts

A managed service anything is preferable to rolling things yourself. However, the Kafka Kubernetes Operator made a complex deployment pretty seem-less. This is pretty battle tested too as Confluent run their own public cloud offering (Confluent Cloud) on Kubernetes.

I had a fully clustered, secured and highly available streaming platform of my own with a few yaml files and some cli magic.

This is an amazing outcome — built in minutes (not days or weeks). This was so much easier than the last time I did this!

Top comments (0)