Uploaded byconfluent
Building Microservices with Apache Kafka
The document outlines the benefits of using Apache Kafka for building scalable microservices, highlighting its distributed streaming capabilities and various APIs such as Kafka Connect and Kafka Streams. It emphasizes the importance of loose coupling in microservices, improved scalability, and the ability to process data in real-time. Additionally, it discusses the latest features and performance improvements in Kafka, demonstrating its effectiveness as a foundational tool for microservices architecture.
Building Microservices with Apache Kafka
- 1. 1 By Colin McCabe BuildingMicroservices with Apache Kafka™
- 2.
- 3. 3 Roadmap ● Example networkservice • Why microservices? • Why Kafka? ● Apache Kafka background ● How Kafka helps scale microservices ● Kafka APIs • Kafka Connect API • Kafka Streams API ● Wrap up ● New Kafka features and improvements
- 4.
- 5. 5 Single Process First Try:Monolithic Service
- 6. 6 Emailer Second Try: Microserviceswith REST HDFS Connector Metrics Connector Frontend
- 7. 7 Third Try: Microserviceswith Kafka Frontend
- 8. 8 Themes ● Improving Decoupling •Everything in one big app: no decoupling • Microservices with REST: multiple services • Microservices with Kafka: decoupled services sharing data ● Improving Scalability • Everything in one big app: single node • Microservices with REST: one node per service • Microservices with Kafka: scalable microservices
- 9. 9 Apache Kafka ● Adistributed streaming platform ● https://kafka.apache.org/intro ● Kafka was built at LinkedIn around 2010 ● Multi-platform: clients in Java, Scala, C, C++, Python, Go, C#, …
- 10.
- 11. 11 Kafka Concepts: the10,000 foot view ● 4 APIs • Producer • Consumer • Connector • Stream Processor
- 12. 12 Producers and Consumers Producer Consumer ProducerProducer Consumer Consumer Consumer write messages read messages message ● key ● value
- 13. 13 Topics Frontend { ‘story’: ‘my newsstory’, ‘user’: ‘foo’, ‘timestamp’: <time> } ‘views’ topic Backend
- 14. 14 Kafka is Durable Frontend ●Data is replicated to multiple servers and persisted to disk. ● Configurable log retention. ● Consumers can read from any part of the log. ‘views’ topic
- 15. 15 Scaling with Kafka ●Can have multiple producers writing to a topic ● Can have multiple consumers reading from a topic ● Can add new microservices to consume data easily • Example: add more microservices processing views • Organize microservices around data, rather than APIs ● Can add more Kafka brokers to handle more messages and topics • Horizontal scalability
- 16. 16 Scaling a Topicwith Multiple Partitions Frontend events topic Backend Backend Backend
- 17. 17 Load Balancing withMultiple Consumers Frontend emailer consumer group story_emails topic
- 18.
- 19. 19 Connecting to ExternalServices Frontend Kafka Connect API
- 20. 20 Kafka Connect API docs.confluent.io/current/connect/ ConnectorInstance ● Responsible for copying data between Kafka and an external system Connector Task Connector Plugin
- 21. 21 Kafka Streams API kafka.apache.org/ documentation/streams ●Process streams of data. ● Fault-tolerant and scalable.
- 22. 22 Calculating News ReaderMetrics Alice 13 Bob 4 Chao 25 Bob 19 Dave 55 ... Alice europe Bob us Chao asia Bob us Dave europe ... europe 68 us 23 asia 25 ... + = clicks locations clicks per location
- 23. 23 Kafka Streams API ●Inputs and outputs are Kafka streams ● Fault-tolerance, rebalancing, scalability provided by Kafka ● KStream ● KTable
- 24. 24 Joining the Clicksand Location Streams in KStreams KStream<String, Long> userClicksStream = builder.stream(..., "user-clicks-topic"); KTable<String, String> userRegionsTable = builder.table(..., "user-regions-topic") KTable<String, Long> clicksPerRegion = userClicksStream .leftJoin(userRegionsTable, (c, r) -> new RegionWithClicks(r == null ? "UNKNOWN" : r, c)) .map((user, regionWithClicks) -> new KeyValue<>(regionWithClicks.getRegion(), regionWithClicks.getClicks())). reduceByKey((c1, c2) -> c1 + c2, ...); clicksPerRegion.to("clicks-per-region-topic", ...);
- 25.
- 26. 26 New Kafka Featuresand Improvements ● Exactly once semantics in Kafka 0.11 • https://www.confluent.io/blog/exactly-once-semantics- are-possible-heres-how-apache-kafka-does-it/ ● Consumer and producer performance improvements • Up to +20% producer throughput • Up to +50% consumer throughput ● Better CLASSPATH isolation for Kafka Connect connectors
- 27. 27 Conclusion ● The loosecoupling, deployability, and testability of microservices makes them a great way to scale. ● Apache Kafka is an incredibly useful building block for many different microservices. ● Kafka is reliable and does the heavy lifting ● Kafka Connect is a great API for connecting with external databases, Hadoop clusters, and other external systems. ● Kafka Streams can process data in realtime. ● https://www.confluent.io/solutions/microservices/
- 28.