DEV Community

Cover image for Kafka Made Simple: A Hands-On Quickstart with Docker and Spring Boot
Arata
Arata

Posted on

Kafka Made Simple: A Hands-On Quickstart with Docker and Spring Boot

#kafka #docker #springboot

Apache Kafka is a distributed, durable, real-time event streaming platform. It goes beyond a message queue by providing scalability, persistence, and stream processing capabilities.

In this guide, we’ll quickly spin up Kafka with Docker, explore it with CLI tools, and integrate it into a Spring Boot application.

1. What is Kafka?

Apache Kafka is a distributed, durable, real-time event streaming platform.

It was originally developed at LinkedIn and is now part of the Apache Software Foundation.

Kafka is designed for high-throughput, low-latency data pipelines, streaming analytics, and event-driven applications.

What is an Event?

An event is simply a record of something that happened in the system.

Each event usually includes:

  • Key → identifier (e.g., user ID, order ID).
  • Value → the payload (e.g., “order created with total = $50”).
  • Timestamp → when the event occurred.

Example event:

{ "key": "order-123", "value": { "customer": "Alice", "total": 50 }, "timestamp": "2025-09-19T10:15:00Z" }
Enter fullscreen mode Exit fullscreen mode

What is an Event Streaming Platform?

An event streaming platform is a system designed to handle continuous flows of data — or events — in real time.

Instead of working in batches (processing data after the fact), it allows applications to react as events happen.

2. What Kafka Can Do

Kafka is more than a message queue—it's a real-time event backbone for modern systems.

Messaging Like a Message Queue

Kafka decouples producers and consumers, enabling asynchronous communication between services.
Example:
A banking system publishes transaction events to Kafka. Fraud detection, ledger updates, and notification services consume these events independently.

Event Streaming

Kafka streams data in real time, allowing systems to react instantly.
Example:
An insurance platform streams claim events to trigger automated validation, underwriting checks, and customer updates in real time.

Data Integration

Kafka Connect bridges Kafka with databases, cloud storage, and analytics platforms.
Example:
A semiconductor company streams sensor data from manufacturing equipment into a data lake for predictive maintenance and yield optimization.

Log Aggregation

Kafka centralizes logs from multiple services for monitoring and analysis.
Example:
An industrial automation system sends logs from PLCs and controllers to Kafka, where they’re consumed by a monitoring dashboard for anomaly detection.

Replayable History

Kafka retains events for reprocessing or backfilling.
Example:
An insurance company replays past policy events to train a model that predicts claim risk or customer churn. This avoids relying on static snapshots and gives the model a dynamic, time-aware view of behavior.

Scalable Microservices Communication

Kafka handles high-throughput messaging across distributed services.
Example:
A financial institution uses Kafka to coordinate customer onboarding, KYC checks, and account provisioning across multiple microservices.

3. Architecture

Apache Kafka’s architecture is built for high throughput, fault tolerance, and horizontal scalability. At its core, Kafka relies on a log-based storage model and a distributed broker cluster.

Core Components

  • Producer → Publishes records (events/messages) to topics. Can be idempotent or transactional.

  • Topic → Logical category/feed for messages. Divided into partitions for parallelism.

  • Partition → Ordered, immutable commit log. Records have sequential offsets.

  • Broker → A Kafka server that stores partitions. Clusters have multiple brokers.

  • Consumer → Subscribes to topics and processes messages. Part of a consumer group for scaling.

  • Controller → Special broker role that manages metadata, leader election, and partition assignment.

  • Replication → Each partition has one leader and multiple followers in the ISR (in-sync replicas).

Data Flow

  • Producers send records to brokers.
  • Records are appended to the leader partition log.
  • Followers replicate the leader’s log for durability.
  • Consumers fetch records from leaders, tracking their offsets.

 Architecture Diagram 

 +-----------------+ | Producers | +-----------------+ | | | v v v +------------------------+ | Kafka Cluster | | +---------+ | | | Broker 1| <--------------- Partition 0 Leader | +---------+ | | | Broker 2| <--------------- Partition 0 Follower | +---------+ | | | Broker 3| <--------------- Partition 1 Leader | +---------+ | +------------------------+ | | | v v v +-------------------+ | Consumer Group | |-------------------| | Consumer A → P0 | | Consumer B → P1 | +-------------------+
Enter fullscreen mode Exit fullscreen mode

4. QuickStart with Docker

This configuration sets up a single-node Kafka broker using the KRaft. It’s ideal for development, testing scenarios

name: kafka services: kafka: image: apache/kafka:4.1.0 container_name: kafka environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_LISTENERS: BROKER://:9092,CONTROLLER://:9093 KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: BROKER KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT KAFKA_ADVERTISED_LISTENERS: BROKER://localhost:9092 KAFKA_CLUSTER_ID: "kafka-1" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_LOG_DIRS: /var/lib/kafka/data volumes: - kafka_data:/var/lib/kafka/data ports: - "9092:9092" volumes: kafka_data:
Enter fullscreen mode Exit fullscreen mode

How to Run

Start the Kafka container using:

docker compose up
Enter fullscreen mode Exit fullscreen mode

Kafka will be available at localhost:9092 for producers and consumers, and internally at localhost:9093 for controller communication.

5. Kafka CLI

Before running Kafka commands, log into the Kafka container:

docker container exec -it localhost bash
Enter fullscreen mode Exit fullscreen mode

Create Topic

Create a topic named quickstart with one partition and a replication factor of 1:

/opt/kafka/bin/kafka-topics.sh --create \ --bootstrap-server localhost:9092 \ --replication-factor 1 \ --partitions 1 \ --topic quickstart
Enter fullscreen mode Exit fullscreen mode

List Topic

Check all existing topics:

/opt/kafka/bin/kafka-topics.sh --list \ --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Consume Message

Read messages from the order topic starting from the beginning:

/opt/kafka/bin/kafka-console-consumer.sh \ --bootstrap-server localhost:9092 \ --topic quickstart \ --from-beginning
Enter fullscreen mode Exit fullscreen mode

Send Message

You can send messages to the quickstart topic using either direct input or a file.

Option A: Send a single message 

echo 'This is Event 1' | \ /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:9092 \ --topic quickstart
Enter fullscreen mode Exit fullscreen mode

Option B: Send multiple messages from a file 

echo 'This is Event 2' > messages.txt echo 'This is Event 3' >> messages.txt cat messages.txt | \ /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:9092 \ --topic quickstart
Enter fullscreen mode Exit fullscreen mode

5. Spring Boot Integration

This configuration enables seamless integration between a Spring Boot application and an Apache Kafka broker. It defines both producer and consumer settings for message serialization, deserialization, and connection behavior.

pom.xml 

<!-- spring-web --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <version>3.4.9</version> </dependency> <!-- kafka --> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> <version>3.3.9</version> </dependency> <!-- Lombok(optional) --> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.30</version> <optional>true</optional> </dependency>
Enter fullscreen mode Exit fullscreen mode

applicaiton.yml 

spring: kafka: bootstrap-servers: localhost:9092 template: default-topic: orders consumer: group-id: quickstart-group auto-offset-reset: latest key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer properties: spring.json.trusted.packages: "dev.aratax.messaging.kafka.model" producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
Enter fullscreen mode Exit fullscreen mode

Topic Setup 

@Bean public NewTopic defaultTopic() { return new NewTopic("orders", 1, (short) 1); }
Enter fullscreen mode Exit fullscreen mode

Event Model 

public class OrderEvent { private String id; private Status status; private BigDecimal totalAmount; private Instant createdAt = Instant.now(); private String createdBy; public enum Status { IN_PROGRESS, COMPLETED, CANCELLED } }
Enter fullscreen mode Exit fullscreen mode

Producer Example 

@RestController @RequestMapping("/api") @RequiredArgsConstructor public class OrderEventController { private final KafkaTemplate<String, OrderEvent> kafkaTemplate; @PostMapping("/orders") public String create(@RequestBody OrderEvent event) { event.setId(UUID.randomUUID().toString()); event.setCreatedAt(Instant.now()); kafkaTemplate.sendDefault(event.getId(), event); return "Order sent to Kafka"; } }
Enter fullscreen mode Exit fullscreen mode

Consumer Example 

@Component public class OrderEventsListener { @KafkaListener(topics = "orders") public void handle(OrderEvent event) { System.out.println("Received order: " + event); } }
Enter fullscreen mode Exit fullscreen mode

6. Demo Project

I built a demo project using Spring Boot and Kafka to demonstrate basic producer/consumer functionality.
Check it out on GitHub: springboot-kafka-quickstart

7. Key Takeaways

  • Kafka is more than a message queue—it's a scalable, durable event streaming platform.
  • Events are central to Kafka’s architecture, enabling real-time data flow across systems.
  • Docker makes setup easy, allowing you to spin up Kafka locally for development and testing.
  • Kafka CLI tools help you explore topics, produce messages, and consume events interactively.
  • Spring Boot integration simplifies Kafka usage with built-in support for producers and consumers.
  • Real-world use cases span industries like banking, insurance, semiconductor, and automation.

8. Conclusion

Apache Kafka empowers developers to build reactive, event-driven systems with ease. Whether you're streaming financial transactions, processing insurance claims, or monitoring factory equipment, Kafka provides the backbone for scalable, real-time communication.

With Docker and Spring Boot, you can get started in minutes—no complex setup required. This quickstart gives you everything you need to explore Kafka hands-on and begin building production-grade event pipelines.

Ready to go deeper? Try explore its design/implementation, stream processing, or Kafka Connect integrations next.

Top comments (0)