This repository contains sample Spring Boot setup for handling multiple event types in single kafka topic.

• Java Functions using Spring Cloud Stream Kafka Streams binder (avro project).
• Avro's unions with schema references.
• Avro Schema registration with gradle.
• Protobuf schema references.
• Unit testing via TopologyTestDriver.

• Java 11
• Docker
• Apache Kafka, Schema Registry

• Start kafka platform.

  docker-compose -f docker/docker-compose.yml up -d

• List containers and check if all are Up.

  docker-compose -f docker/docker-compose.yml ps # Name Command State Ports  # ---------------------------------------------------------------------------------------------------- # broker /etc/confluent/docker/run Up 0.0.0.0:9092->9092/tcp, 0.0.0.0:9101->9101/tcp # control-center /etc/confluent/docker/run Up 0.0.0.0:9021->9021/tcp  # rest-proxy /etc/confluent/docker/run Up 0.0.0.0:8082->8082/tcp  # schema-registry /etc/confluent/docker/run Up 0.0.0.0:8081->8081/tcp  # zookeeper /etc/confluent/docker/run Up 0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp 

• Start one of the sample application:

  • Avro:

    • Register Schemas via gradle task:

      cd avro ./gradlew registerSchemasTask

    • Start application:

      ./gradlew bootRun

    • Check application logs to see that messages are correctly serialized and deserialized. Below sample output:

      # c.r.k.a.a.t.d.DummyTransactionGenerator : Successfully sent TransactionEvent ca0e0357-e76e-4d8c-8015-e545111ea853 of type class com.rbiedrawa.app.avro.events.transactions.TransactionStarted # c.r.k.a.a.t.d.DummyTransactionGenerator : Successfully sent TransactionEvent ca0e0357-e76e-4d8c-8015-e545111ea853 of type class com.rbiedrawa.app.avro.events.transactions.TransactionCompleted # c.r.k.a.a.t.TransactionKStream : Consumed key ca0e0357-e76e-4d8c-8015-e545111ea853 value class com.rbiedrawa.app.avro.events.transactions.TransactionStarted # c.r.k.a.a.t.TransactionKStream : Consumed key ca0e0357-e76e-4d8c-8015-e545111ea853 value class com.rbiedrawa.app.avro.events.transactions.TransactionCompleted

    • Use interactive query to get transaction from kafka state store:

      curl -X GET --location "http://localhost:8080/api/transactions/ca0e0357-e76e-4d8c-8015-e545111ea853"

  • Protobuf:

    • Start application:

      cd proto ./gradlew bootRun

    • Check application logs to see that messages are correctly serialized and deserialized. Below sample output:

      # c.r.k.a.t.d.DummyTransactionGenerator : Successfully sent TransactionEvent d1e869d5-fb9b-4b13-ac43-678693d5910d of type TRANSACTION_STARTED # c.r.k.a.transactions.TransactionKStream : Consumed transaction event of type TRANSACTION_STARTED. TransactionId d1e869d5-fb9b-4b13-ac43-678693d5910d # c.r.k.a.t.d.DummyTransactionGenerator : Successfully sent TransactionEvent d1e869d5-fb9b-4b13-ac43-678693d5910d of type TRANSACTION_COMPLETED # c.r.k.a.transactions.TransactionKStream : Consumed transaction event of type TRANSACTION_COMPLETED. TransactionId d1e869d5-fb9b-4b13-ac43-678693d5910d

    • Use interactive query to get transaction from kafka state store:

      curl -X GET --location "http://localhost:8080/api/transactions/d1e869d5-fb9b-4b13-ac43-678693d5910d"

• Stop docker-compose demo.

  docker-compose -f docker/docker-compose.yml down -v

• Should You Put Several Event Types in the Same Kafka Topic?

• Putting Several Event Types in the Same Topic – Revisited

Distributed under the MIT License. See LICENSE for more information.