![Producers - push Kafka Broker org.apache.kafka.clients.producer.KafkaProducer Response => [TopicName [Partition ErrorCode Offset]] Request => RequiredAcks Timeout [TopicName [Partition MessageSetSize MessageSet]]](https://image.slidesharecdn.com/thebestofapachekafkaarchitecture-160530020415/75/The-best-of-Apache-Kafka-Architecture-12-2048.jpg)
The best of Apache Kafka Architecture
- 1. The Best ofApache Kafka Architecture Ranganathan Balashanmugam @ran_than Apache: Big Data 2015
- 2.
- 3. About Me ❏ Graduatedas Civil Engineer. ❏ <dev> 10+ years </dev> ❏ <Thoughtworker from=”India”/> ❏ Organizer of Hyderabad Scalability Meetup with 2000+ members.
- 4. “Form follows function.” -Louis Sullivan
- 5. Gravity Dam Indirasagar Dam,India img src: http://www.montanhydraulik.in
- 6. Forces on agravity dam Dam weight Head Water Tail Water Uplift
- 7. ❏ publish-subscribe messagingservice ❏ distributed commit/write-ahead log “producers produce, consumers consume, in large distributed reliable way -- real time”
- 8. ❏ DBs ❏ Logs ❏Brokers ❏ HDFS “For highly distributed messages, Kafka stands out.” Why Kafka?
- 9. Kafka Vs ________ src:https://softwaremill.com/mqperf/
- 10. Timeline 2011 2012 20132014 2015 Open sourced by LinkedIn, as version 0.6 Graduated from Apache Latest stable - 0.8.2.1 Several Engineers who built Kakfa create Confluent
- 11. A Kafka Message CRCattributes key length key message message length message content kafka.message.Message magic Change requested:KAFKA-2511
- 12. Producers - push Kafka Broker org.apache.kafka.clients.producer.KafkaProducer Response=> [TopicName [Partition ErrorCode Offset]] Request => RequiredAcks Timeout [TopicName [Partition MessageSetSize MessageSet]]
- 13. Topic number of messages time size Removemessages based on kafka.common.Topic
- 14. Partitions kafka.cluster.Partition Serves: Horizontal scaling,Parallel consumer reads
- 15.
- 16. Consumer offsets committing andfetching consumer offsets img src: http://www.reynanprinting.com/photos/undefined/impresion-offset1.jpg
- 17. kafka:// - protocol ●Metadata ● Send ● Fetch ● Offsets ● Offset commit ● Offset fetch “Binary protocol over TCP”
- 18. Mechanical Sympathy "The most amazingachievement of the computer software industry is its continuing cancellation of the steady and staggering gains made by the computer hardware industry." - Henry Peteroski Image source: http://www.theguide2surrey.com
- 19. Persistence “Everything is fastertill the disk IO.”
- 20. Disk faster thanRAM src: http://queue.acm.org/detail.cfm?id=1563874
- 21. Linear Read &Writes On high level there are only two operations: Append to end of log fetch messages from a partition beginning from a particular message id sequential file I/O
- 22. “Let us playpictionary”
- 23. Linux Page Cache “Kafkaate my RAM”
- 24.
- 25. Batching small latency toimprove throughput img src: https://prashanthpanduranga.files.wordpress.com/2015/05/tirupati.jpg
- 26. Compression bandwidth is moreexpensive per-byte to scale than disk I/O, CPU, or network bandwidth capacity within a facility kafka.message.CompressionCodec
- 27. Log compaction img src:http://kafka.apache.org/083/documentation.html kafka.log.LogCleaner, LogCleanerManager
- 28. Message Delivery Atleast onceAtmost once Exactly once
- 29. Replication un-replicated = replicationfactor of one
- 30. Quorum based ● Betterlatency ● To tolerate “f” failures, need “2f+1” replicas
- 31. Primary-backup replication Broker 1 Broker2 Broker 3 Broker 4 Topic 1 Topic 1 Topic 1 Topic 2 Topic 2 Topic 2 Topic 3 Topic 3Topic 3
- 32.
- 33. THANK YOU For questionsor suggestions: Ran.ga.na.than B ranganab@thoughtworks.com @ran_than