Downloaded 78 times
Uploaded byDatabricks
Introducing Exactly Once Semantics in Apache Kafka with Matthias J. Sax
The document outlines improvements in Apache Kafka's delivery semantics, introducing exactly-once processing capabilities through idempotent and transactional producers. These new features enhance stream processing efficiency, making Kafka more robust for various streaming applications. Users can easily implement these capabilities using the Streams API and configurations for improved performance.
In this document
Powered by AI
Introduction to Kafka's exactly-once semantics. The presentation outlines existing delivery semantics, new features, usage, and a summary.
Kafka offers at-least-once, in-order delivery per partition. Producer retries can lead to duplicates.
Illustrates how duplicate writes can occur with producer-broker interactions in Kafka, highlighting key steps in the process.
The importance of improved stream processing capabilities in Kafka as it underpins a growing part of the data landscape.
Introduction to new features: idempotent producer for exactly-once writes, and transactional producer for atomic writes.
Demonstrates the idempotent producer, ensuring exactly-once writes with sequence numbers and acknowledgments.
Introduction to transactions for multi-partition writes, ensuring atomicity and commit across topics.
Discusses the transactional API for producers with code example demonstrating transaction management.
How to utilize exactly-once semantics via Streams API, idempotent, and transactional producers with configuration parameters.
Highlights the transactional read-process-write-commit pattern used in Kafka Streams and Apache Spark.
Features were made available in Kafka 0.11 (June 2017) with encouragement to try them out.
Summarizes the presentation's insights into Kafka's delivery semantics and improvements. Concludes with a thank you and hiring announcement.
Introducing Exactly Once Semantics in Apache Kafka with Matthias J. Sax
- 1. 1Confidential Introducing ExactlyOnce Semantics inApache® KafkaTM Jason Gustafson, Apurva Mehta, Guozhang Wang, and Sriram Subramaniam Matthias J. Sax | Software Engineer matthias@confluent.io @MatthiasJSax
- 2. 2Confidential Outline • Kafka’sexisting deliverysemantics. • What’s new? • How do you use it? • Summary.
- 3. 3Confidential What Kafka offerstoday • At-least-once,in-order delivery per partition. • Producer retries can introduce duplicates.
- 4. 4Confidential Stream Processing withApache Spark Read-process-write pattern:
- 5. 5Confidential Example: duplicate write ProducerBroker Topic Partition
- 6. 6Confidential Example: duplicate write ProducerBroker Topic Partition send (k,v)
- 7. 7Confidential Example: duplicate write ProducerBroker (k,v) Topic Partition append (k,v)
- 8. 8Confidential Example: duplicate write ProducerBroker (k,v) Topic Partition ack
- 9. 9Confidential Example: duplicate write ProducerBroker (k,v) Topic Partition send (k,v)
- 10. 10Confidential Example: duplicate write ProducerBroker (k,v) (k,v) Topic Partition append (k,v)
- 11. 11Confidential Example: duplicate write ProducerBroker (k,v) (k,v) Topic Partition ack
- 12. 12Confidential Why improve? • Streamprocessing is becoming a bigger part of the data landscape. • Apache Kafka is the foundation for such stream processing. • Strengthening Kafka’ssemantics expands the universe of streaming applications.
- 13.
- 14. 14Confidential What’s new • Idempotentproducer:exactly-once writes. • Transactionalproducer:Atomic writes across multiple partitions. • Exactly-once stream processing: read-process-write.
- 15.
- 16. 16Confidential Example: Idempotent Producer ProducerBroker Topic Partition
- 17. 17Confidential Example: Idempotent Producer ProducerBroker send (k,v) seq = 0 pid = 73 Topic Partition
- 18. 18Confidential Example: Idempotent Producer ProducerBroker append (k,v) seq = 0 pid = 73 Topic Partition (k,v) seq = 0 pid = 73
- 19. 19Confidential Example: Idempotent Producer ProducerBroker ack Topic Partition (k,v) seq = 0 pid = 73
- 20. 20Confidential Example: Idempotent Producer ProducerBroker send (k,v) seq = 0 pid = 73 Topic Partition (k,v) seq = 0 pid = 73
- 21. 21Confidential Example: Idempotent Producer ProducerBroker ack (dup) Topic Partition (k,v) seq = 0 pid = 73
- 22. 22Confidential What’s new Atomic Multi-PartitionWrites (aka “transactions”)
- 23. 23Confidential Atomic Multi-Partition Writes Producer TopicA, Partition 0 Topic B, Partition 0 Topic B, Partition 1
- 24. 24Confidential Atomic Multi-Partition Writes Producer TopicA, Partition 0 m1 m5 m3 m4 m6Topic B, Partition 0 m2Topic B, Partition 1
- 25. 25Confidential Atomic Multi-Partition Writes Producer TopicA, Partition 0 m1 m5 C m3 m4 m6Topic B, Partition 0 m2Topic B, Partition 1 C C atomic commit
- 26. 26Confidential Atomic Multi-Partition Writes Consumer TopicA, Partition 0 m1 m5 C m3 m4 m6Topic B, Partition 0 m2Topic B, Partition 1 C C read committed
- 27. 27Confidential TransactionalAPI producer.initTransactions(); try { producer.beginTransaction(); producer.send(record0); producer.send(record1); producer.sendOffsetsToTxn(…); producer.commitTransaction(); } catch(ProducerFencedException e) { producer.close(); } catch (KafkaException e) { producer.abortTransaction(); }
- 28. 28Confidential How to useexactly-once capabilities: • Streams API (the easiest way to use exactly-once semantics) • Config parameter processing.mode = “exactly_once” • Idempotent Producer • Config parameter enable.idempotence = true • Transactional Producer • Config parameter transactional.id = “my-unique-tid” • And Transactional API (hard to use!) • Transactional Consumer • Config parameter isolation.level = “read_committed” (default: “read_uncommitted”)
- 29. 29Confidential Stream Processing withKafka’s Streams API Transactional read-process-write-commit pattern:
- 30. 30Confidential Stream Processing withApache Spark Transactional read-process-write-commit pattern:
- 31. 31Confidential When to usethis? Available in Kafka 0.11, June 2017. Try it out!
- 32. 32Confidential Putting it together •We understood Kafka’sexisting delivery semantics. • Learned how these have been strengthened. • Learned how the new semantics work. • Saw, it’s easy to use with higher levelAPIs like Kafka Streams or Apache Spark.
- 33.