In this tutorial, you will learn how to build two Spring Boot microservices that communicate asynchronously using Spring Cloud Stream with Apache Kafka. This guide is designed for beginners and includes a working example with detailed explanations for each step.

What You’ll Learn:

• How to build microservices using Spring Boot.
• How to set up Spring Cloud Stream to send and receive messages using Kafka.
• How to run microservices with Kafka as a messaging broker.

Introduction to Spring Cloud Stream and Kafka

Spring Cloud Stream is a framework for building event-driven microservices connected to messaging systems like Kafka or RabbitMQ. It abstracts the messaging infrastructure, allowing developers to focus on writing business logic.

Why Use Spring Cloud Stream with Kafka?

• Asynchronous Communication: Microservices can communicate without direct calls, improving system resilience.
• Decoupling: Services are loosely coupled, as they only communicate through messages.
• Scalability: Kafka handles large volumes of data efficiently.

Prerequisites

Before starting, ensure that you have the following tools installed:

• JDK 17 or later
• Maven (to build the project)
• Kafka and Zookeeper installed (or use Docker to run Kafka)
• IDE (IntelliJ IDEA, Eclipse, etc.)

Step 1: Create the Projects

We will create two microservices:

1. employee-service: Sends employee data to Kafka.
2. department-service: Listens to Kafka and receives employee data.

Step 2: Set Up employee-service

2.1 Create the Project

Go to Spring Initializr and generate a Spring Boot project with the following dependencies:

• Spring Web
• Spring Cloud Stream
• Spring for Apache Kafka

2.2 Configure application.yml

Create a configuration file src/main/resources/application.yml for the employee-service to define Kafka bindings.

spring: cloud: stream: bindings: output: destination: employee-topic content-type: application/json kafka: binder: brokers: localhost:9092 

Explanation:

• output.destination=employee-topic: Specifies the Kafka topic for sending messages.
• brokers=localhost:9092: Defines the Kafka broker address.

2.3 Create the Employee Model

Define an Employee class to represent employee data.

package com.example.employeeservice; public class Employee { private String id; private String name; private String department; // Constructors, getters, and setters public Employee(String id, String name, String department) { this.id = id; this.name = name; this.department = department; } public String getId() { return id; } public String getName() { return name; } public String getDepartment() { return department; } } 

2.4 Create a Message Producer

Create a service that will send employee data to Kafka.

package com.example.employeeservice; import org.springframework.cloud.stream.function.StreamBridge; import org.springframework.stereotype.Service; @Service public class EmployeeProducer { private final StreamBridge streamBridge; public EmployeeProducer(StreamBridge streamBridge) { this.streamBridge = streamBridge; } public void sendEmployee(Employee employee) { streamBridge.send("output", employee); } } 

Explanation:

• StreamBridge: Allows sending messages to a Kafka topic dynamically.
• sendEmployee(): Sends employee data to the employee-topic.

2.5 Create a REST Controller

Create a REST controller to trigger message sending.

package com.example.employeeservice; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RestController; @RestController public class EmployeeController { private final EmployeeProducer employeeProducer; public EmployeeController(EmployeeProducer employeeProducer) { this.employeeProducer = employeeProducer; } @PostMapping("/employees") public String createEmployee(@RequestBody Employee employee) { employeeProducer.sendEmployee(employee); return "Employee sent: " + employee.getName(); } } 

2.6 Create a Dockerfile

Create a Dockerfile for employee-service:

FROM openjdk:17-jdk-alpine WORKDIR /app COPY target/employee-service-0.0.1-SNAPSHOT.jar employee-service.jar EXPOSE 8081 ENTRYPOINT ["java", "-jar", "employee-service.jar"] 

Step 3: Set Up department-service

3.1 Create the Project

Go to Spring Initializr and generate another Spring Boot project with the following dependencies:

• Spring Web
• Spring Cloud Stream
• Spring for Apache Kafka

3.2 Configure application.yml

Create a configuration file src/main/resources/application.yml for department-service:

spring: cloud: stream: bindings: input: destination: employee-topic content-type: application/json kafka: binder: brokers: localhost:9092 

Explanation:

• input.destination=employee-topic: Listens to the same Kafka topic (employee-topic) to receive messages.

3.3 Create the Employee Model

Create the same Employee model as in employee-service to deserialize the received message:

package com.example.departmentservice; public class Employee { private String id; private String name; private String department; // Constructors, getters, and setters public Employee() {} public Employee(String id, String name, String department) { this.id = id; this.name = name; this.department = department; } public String getId() { return id; } public String getName() { return name; } public String getDepartment() { return department; } } 

3.4 Create a Message Consumer

Create a service to consume employee data from Kafka.

package com.example.departmentservice; import org.springframework.context.annotation.Bean; import org.springframework.stereotype.Service; import java.util.function.Consumer; @Service public class EmployeeConsumer { @Bean public Consumer<Employee> input() { return employee -> { System.out.println("Received employee: " + employee.getName() + " from department " + employee.getDepartment()); }; } } 

Explanation:

• @Bean Consumer<Employee> input(): Registers a function that consumes messages from Kafka.

3.5 Create a Dockerfile

Create a Dockerfile for department-service:

FROM openjdk:17-jdk-alpine WORKDIR /app COPY target/department-service-0.0.1-SNAPSHOT.jar department-service.jar EXPOSE 8082 ENTRYPOINT ["java", "-jar", "department-service.jar"] 

Step 4: Set Up Kafka with Docker Compose

Create a docker-compose.yml file to run Kafka and Zookeeper:

version: '3.8' services: zookeeper: image: confluentinc/cp-zookeeper:latest environment: ZOOKEEPER_CLIENT_PORT: 2181 ports: - "2181:2181" kafka: image: confluentinc/cp-kafka:latest environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 ports: - "9092:9092" depends_on: - zookeeper 

Run Kafka and Zookeeper:

docker-compose up -d 

Step 5: Build Docker Images

Navigate to the root directories of each service and run:

For employee-service:

mvn clean package docker build -t employee-service . 

For department-service:

mvn clean package docker build -t department-service . 

Step 6: Create a Docker Compose File for Services

Create a docker-compose.yml file to run both microservices with Kafka:

version: '3.8' services: employee-service: image: employee-service build: context: ./employee-service ports: - "8081:8081" networks: - microservices-net department-service: image: department-service build: context: ./department-service ports: - "8082:8082" networks: - microservices-net kafka: image: confluentinc/cp-kafka:latest ports: - "9092:9092" environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 networks: - microservices-net zookeeper: image: confluentinc/cp-zookeeper:latest environment: ZOOK EEPER_CLIENT_PORT: 2181 ports: - "2181:2181" networks: - microservices-net networks: microservices-net: driver: bridge 

Step 7: Run Docker Compose

Navigate to the directory containing the docker-compose.yml file and run:

docker-compose up --build 

Docker Compose will build and start the containers.

Step 8: Test the Microservices Communication

Use Postman or curl to send employee data to the employee-service:

curl -X POST http://localhost:8081/employees \ -H "Content-Type: application/json" \ -d '{"id":"1", "name":"John Doe", "department":"Engineering"}' 

The department-service should log the received employee data in the console.

Conclusion

You have successfully built two Spring Boot microservices that communicate asynchronously using Spring Cloud Stream and Kafka. This setup demonstrates how to build scalable, event-driven microservices architecture.

Next Steps:

• Add more microservices to the system.
• Implement error handling and retries for message delivery.