DEV Community

Cover image for Kafka UI in Action: Monitoring and Managing Kafka Like a Pro
Kneonix
Kneonix

Posted on

Kafka UI in Action: Monitoring and Managing Kafka Like a Pro

#kafka #microservices #monitoring #programming

Why You Need to Monitor Your Kafka Broker

Recently, I was working with Apache Kafka as the message broker for a microservice architecture. I quickly realized that it can be difficult to verify if Kafka is running correctly, especially when the rest of your application hasn’t finished connecting. In scenarios like this, a monitoring tool can be incredibly helpful and that’s where Kafka UI shines.

Image description

What Can You Do With Kafka UI?

Kafka UI is a powerful, lightweight monitoring tool for Apache Kafka. Here's what it offers out of the box:

  • Multi-Cluster Management : Monitor and manage multiple Kafka clusters in one place.
  • Performance Monitoring with Metrics Dashboard : Track essential Kafka metrics through an intuitive dashboard.
  • View Kafka Brokers : Inspect topic and partition assignments, controller status, and broker metadata.
  • View Kafka Topics : See partition count, replication status, and custom topic configurations.
  • View Consumer Groups : Monitor per-partition parked offsets and consumer lag (combined and per partition).
  • Browse Messages : View Kafka messages in JSON, plain text, or Avro formats.
  • Dynamic Topic Configuration : Create and configure new topics dynamically.
  • Configurable Authentication : Secure the UI using GitHub, GitLab, or Google OAuth 2.0.
  • Custom Serialization/Deserialization Plugins : Use built-in serializers like AWS Glue or Smile, or implement your own.
  • Role-Based Access Control (RBAC) : Manage fine-grained access permissions to the UI.
  • Data Masking : Obfuscate sensitive data in topic messages.

How to Set Up a Kafka Broker in Your System

You can run Kafka and Zookeeper in two main ways: Docker and Kubernetes.

Option 1: Docker

Here’s a simple docker-compose.yml setup:

version: "3.8" services: zookeeper: image: confluentinc/cp-zookeeper:latest container_name: zookeeper ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 kafka: image: confluentinc/cp-kafka:latest container_name: kafka ports: - "9092:9092" depends_on: - zookeeper environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
Enter fullscreen mode Exit fullscreen mode

Run:

docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

Option 2: Kubernetes

Create two YAML files: zookeeper.depl.yaml and kafka.depl.yaml.

zookeeper.depl.yaml 

apiVersion: apps/v1 kind: Deployment metadata: name: zookeeper-depl spec: replicas: 1 selector: matchLabels: app: zookeeper template: metadata: labels: app: zookeeper spec: containers: - name: zookeeper image: confluentinc/cp-zookeeper:latest ports: - containerPort: 2181 env: - name: ZOOKEEPER_CLIENT_PORT value: "2181" - name: ZOOKEEPER_TICK_TIME value: "2000" --- apiVersion: v1 kind: Service metadata: name: zookeeper-srv spec: selector: app: zookeeper ports: - name: zookeeper protocol: TCP port: 2181 targetPort: 2181
Enter fullscreen mode Exit fullscreen mode

kafka.depl.yaml 

apiVersion: apps/v1 kind: Deployment metadata: name: kafka-broker-depl spec: replicas: 1 selector: matchLabels: app: kafka-broker template: metadata: labels: app: kafka-broker spec: containers: - name: kafka-broker image: confluentinc/cp-kafka:latest ports: - containerPort: 9092 env: - name: KAFKA_BROKER_ID value: "1" - name: KAFKA_ZOOKEEPER_CONNECT value: "zookeeper-srv:2181" - name: KAFKA_ADVERTISED_LISTENERS value: PLAINTEXT://kafka-broker-srv:9092 - name: KAFKA_LISTENERS value: PLAINTEXT://0.0.0.0:9092 - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR value: "1" - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE value: "false" --- apiVersion: v1 kind: Service metadata: name: kafka-broker-srv spec: selector: app: kafka-broker ports: - name: kafka-broker protocol: TCP port: 9092 targetPort: 9092
Enter fullscreen mode Exit fullscreen mode

Apply them by these command:

kubectl apply -f zookeeper.depl.yaml kubectl apply -f kafka.depl.yaml
Enter fullscreen mode Exit fullscreen mode

How to Set Up Kafka UI

You can deploy Kafka UI via Docker or Kubernetes as well.

Option 1: Docker

Add this to your docker-compose.yml file:

kafka-ui: image: provectuslabs/kafka-ui:latest container_name: kafka-ui ports: - "8080:8080" depends_on: - kafka environment: KAFKA_CLUSTERS_0_NAME: "local" KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:9092"
Enter fullscreen mode Exit fullscreen mode

Then run:

docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

Option 2: Kubernetes

Create a kafkaUI.depl.yaml file:

apiVersion: apps/v1 kind: Deployment metadata: name: kafka-ui-depl spec: replicas: 1 selector: matchLabels: app: kafka-ui template: metadata: labels: app: kafka-ui spec: containers: - name: kafka-ui image: provectuslabs/kafka-ui:latest ports: - containerPort: 8080 env: - name: KAFKA_CLUSTERS_0_NAME value: "local" - name: KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS value: "kafka-broker-srv:9092" --- apiVersion: v1 kind: Service metadata: name: kafka-ui-srv spec: selector: app: kafka-ui ports: - name: kafka-ui protocol: TCP port: 8080 targetPort: 8080
Enter fullscreen mode Exit fullscreen mode

Apply it:

kubectl apply -f kafkaUI.depl.yaml
Enter fullscreen mode Exit fullscreen mode

Viewing the UI

If you’re running this on Kubernetes, forward the port:

kubectl port-forward service/kafka-ui-srv 8080:8080
Enter fullscreen mode Exit fullscreen mode

Then open your browser and go to http://localhost:8080 to monitor and manage your Kafka broker, topics, and consumer groups.

Learn More

Top comments (1)

Collapse
 
_mohamadmohamadi profile image
Mohamad •

so helpful
thanks for sharing that❤️