Downloaded 174 times
![Example: Espresso • A distributed document store • Primary online data serving platform at LI • Member profile, homepage, InMail, etc [SIGMOD 2013]](https://image.slidesharecdn.com/buildingareplicatedloggingsystemwithapachekafka-150907075312-lva1-app6892/75/Building-a-Replicated-Logging-System-with-Apache-Kafka-58-2048.jpg)
![• Data flow streaming on Kafka and YARN • Stateful processing • Re-processing • Failure Recovery Example: Samza [CIDR 2015]](https://image.slidesharecdn.com/buildingareplicatedloggingsystemwithapachekafka-150907075312-lva1-app6892/75/Building-a-Replicated-Logging-System-with-Apache-Kafka-71-2048.jpg)
Uploaded byGuozhang Wang
Building a Replicated Logging System with Apache Kafka
The document discusses building a replicated logging system using Apache Kafka, emphasizing its role as a distributed messaging system with log-centric data flow for centralized data pipelines. It covers key concepts such as log replication, acknowledgment settings, and failure recovery mechanisms, particularly highlighting Kafka's usage at LinkedIn. The document concludes by asserting that a log-centric data flow facilitates system scalability and real-time data processing.
Building a Replicated Logging System with Apache Kafka
- 1. Building a ReplicatedLogging System with Apache Kafka Guozhang Wang, Joel Koshy, Sriram Subramanian, Kartik Paramasivam Mammad Zadeh, Neha Narkhede, Jun Rao, Jay Kreps, Joe Stein
- 2. We All LoveLogs!
- 6. Apache Kafka • Adistributed messaging system ..that store messages as a log!
- 7. Example: LinkedIn backin 2010 Point-to-Point Pipelines What We Want: A Centralized Data Pipeline
- 8. Log-centric Data Flow •Logical Ordering • Persistent Buffering • “Source-of-Truth”
- 9. Store Messages asa Log 4 5 5 7 8 9 10 11 12... Producer Write Consumer1 Reads (offset 7) Consumer2 Reads (offset 10) Messages 3
- 10. Partition the Logacross Machines Topic 1 Topic 2 Partitions Producers Producers Consumers Consumers Brokers
- 11. Apache Kafka Example: Kafkaat LinkedIn
- 12. “Source-of-Truth” should not belost even when..
- 13.
- 14. Consensus for LogReplication Logs Broker-1 Logs Logs Broker-2 Broker-3 Write Consensus Protocol Consensus Protocol
- 15. Key Idea Separate membershipconfiguration from data replication
- 16. Primary-backup Replication Logs Broker-1 * LogsLogs Broker-2 Broker-3 Write
- 17. Conventional Quorum Commits Logs Broker-1* Logs Logs Broker-2 Broker-3 Write
- 18. Conventional Quorum Commits Logs Broker-1* Logs Logs Broker-2 Broker-3 Write
- 19. Conventional Quorum Commits Logs Broker-1* Logs Logs Broker-2 Broker-3
- 20. Conventional Quorum Commits Logs Broker-1* Logs Logs Broker-2 Broker-3
- 21. • Leader maintainsin-sync-replicas (ISR) • Failed / slow follower => drop from ISR • Caught-up follower => re-join ISR • Producer specifies required ACK based on ISR Configurable ISR Commits
- 22. Example: ACK withall ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 23. Example: ACK withall ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 24. Example: ACK withall ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 25. Example: ACK withall ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 26. Example: ACK withall ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 27. Example: ACK withall ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 28. Example: ACK withall ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 29. Example: ACK withLeader-only Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“leader”) ISR {1, 2, 3}
- 30. Example: ACK withLeader-only Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“leader”) ISR {1, 2, 3}
- 31. Example: ACK withLeader-only Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“leader”) ISR {1, 2, 3}
- 32. Example: ACK withLeader-only Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“leader”) ISR {1, 2, 3}
- 33. Example: ACK withLeader-only Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“leader”) ISR {1, 2, 3}
- 34. Example: ACK withLeader-only Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“leader”) ISR {1, 2, 3}
- 35. Example: Slow Follower Logs Broker-1* Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 36. Example: Slow Follower Logs Broker-1* Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 37. Example: Slow Follower Logs Broker-1* Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 38. Example: Slow Follower Logs Broker-1* Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 39. Example: Slow Follower Logs Broker-1* Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 40. Example: Slow Follower Logs Broker-1* Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2}
- 41. Example: Slow Follower Logs Broker-1* Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2}
- 42. Configurable ISR Commits ACKmode Latency On Failures “no" no network delay some data loss “leader" 1 network roundtrip a few data loss “all" ~2 network roundtrips no data loss
- 43. • Use anembedded controller • Detect broker failure via ZooKeeper • Leader failure => elect new leader from ISR • Leader and ISR persisted in Zookeeper • For Controller fail-over Membership Management
- 44. Example: Broker Failure Logs Broker-1* Logs Logs Broker-2 Broker-3 ISR {1, 2}
- 45. Example: Broker Failure Logs Broker-1 LogsLogs Broker-2 Broker-3
- 46. Example: Broker Failure Logs Broker-1 LogsLogs Broker-2 Broker-3
- 47. Example: Broker Failure Logs Broker-1 LogsLogs Broker-2 Broker-3
- 48. Example: Broker Failure Logs Broker-1 LogsLogs Broker-2 Broker-3 ISR {2}
- 49. Example: Broker Failure Logs Broker-1 LogsLogs Broker-2 Broker-3 ISR {2}
- 50. Example: Broker Failure Logs Broker-1 LogsLogs Broker-2 Broker-3 ISR {2}
- 51. Example: Broker Failure Logs Broker-1 LogsLogs Broker-2 * Broker-3 ISR {2}
- 52. Example: Broker Failure Logs Broker-1 LogsLogs Broker-2 * Broker-3 ISR {2}
- 53. Example: Broker Failure Logs Broker-1 LogsLogs Broker-2 * Broker-3 ISR {2}
- 54. Example: Broker Failure Logs Broker-1 LogsLogs Broker-2 * Broker-3 ISR {2, 3}
- 55. • Overview: Logsand Kafka • Log Replication in Kafka • Kafka Usage at LinkedIn • Conclusion Agenda
- 56.
- 57. Apache Kafka Example: Kafkaat LinkedIn
- 58. Example: Espresso • Adistributed document store • Primary online data serving platform at LI • Member profile, homepage, InMail, etc [SIGMOD 2013]
- 59. Old Espresso Replication DataCenter-1 Storage Node Storage NodeMySQL Replication MySQL MySQL Search Index Hadoop … …Databus Cross-DC Replicator Data Center-1 Storage Node Storage NodeMySQL Replication MySQL MySQL Search Index Hadoop … DatabusCross-DC Replicator
- 60. Problems with MySQL Replication MasterStorage Node P1 Slave Storage Node P2 P3 P4 P5 P6 P1 P2 P3 P4 P5 P6 Binary Log Shipping
- 61. Replicate Logs withKafka Storage Node Kafka Logs P1 Storage Node P2 P3 P4 P5 P6 P1 P2 P3 P4 P5 P6 Kafka Producer Kafka Consumer Kafka Consumer Kafka Producer
- 62. Key-based Log Compaction ... PartitionMessages Segment-3 Segment-4 Segment-6 *
- 63. Key-based Log Compaction d:3 f: 8 b: 0 c: null... Partition Messages c: 3 a: 5 a: 6 a: 5 f: 9 ... Segment-3 Segment-4 b: 2 d: 4a: 1
- 64. Key-based Log Compaction ...d: 3 f: 8 b: 0 c: null a: 5 f: 9 ... Segment-3 Segment-4 c: 3 a: 5 a: 6b: 2 d: 4a: 1 c: 3 a: 5 a: 6b: 2 d: 4a: 1 d: 3 f: 8 b: 0 a: 5 f: 9 New Segment Partition Messages
- 65. Key-based Log Compaction ...d: 3 f: 8 b: 0 c: null a: 5 f: 9 ... Segment-3 Segment-4 c: 3 a: 5 a: 6b: 2 d: 4a: 1 c: 3 a: 6 d: 3 f: 8 b: 0 c: null a: 5 f: 9 New Segment Partition Messages
- 66. Key-based Log Compaction ...d: 3 f: 8 b: 0 c: null a: 5 f: 9 ... Segment-3 Segment-4 c: 3 a: 5 a: 6b: 2 d: 4a: 1 d: 3 b: 0 a: 5 f: 9 New Segment Partition Messages
- 67. Key-based Log Compaction ...d: 3 f: 8 b: 0 c: null a: 5 f: 9 ... Segment-3 Segment-4 c: 3 a: 5 a: 6b: 2 d: 4a: 1 d: 3 b: 0 a: 5 f: 9 New Segment Partition Messages
- 68. New Espresso Replication DataCenter-1 Storage Node Storage Node Storage Node Kafka Logs MySQL MySQL MySQL Data Center-n Storage Node Storage Node Storage Node Kafka Logs MySQL MySQL MySQL Kafka MirrorMaker Search Index Hadoop … … Search Index Hadoop … * In Progress
- 69.
- 70. Apache Kafka Example: Kafkaat LinkedIn
- 71. • Data flowstreaming on Kafka and YARN • Stateful processing • Re-processing • Failure Recovery Example: Samza [CIDR 2015]
- 72.
- 73.
- 74.
- 75. Kafka Kafka Samza StateProces s Protoc ol StateProces s Protoc ol StateProces s Protoc ol Samza ProcessingKafka Changlog StateProces s Protoc ol
- 76. Take-aways • Log-centric dataflow helps scaling your systems • Kafka: replicated log streams for real-time platforms
- 77.
- 78. Take-aways • Log-centric dataflow helps scaling your systems • Kafka: replicated log streams for real-time platforms THANKS!
Editor's Notes
- #2 Thank you. And good morning, today I am going to talk about Kafka, and how it can be built as a general replicated log streams for a wide use of scalable systems. This is a joint work from the Apache Kafka community.
- #3 First of all, being in this room, I think it is safe to say “we all love logs”. Logs have been around almost as long as this research community.
- #4 No-overwrite in POSTGRES ARIES: Write-Ahead-Logging in the 80’s Today, reading the 50 page Aries pager has been the must-to-do for every single database graduate student including myself.
- #5 Similarly, Log-Structured storage architecture.
- #6 Replicated State Machine And in all these examples, the log is used as the source of truth data change log to scale the systems while providing durability and consistency.
- #7 So that is all good stuff about logs, but where is Kafka is this big picture. Well, Kafka is an Apache open sourced distributed messaging system that stores messages as a commit log.
- #8 Data-serving websites, LinkedIn has a lot of data We have this variety of data and and we need to build all these products around such data. Messaging: ActiveMQ User Activity: In house log aggregation Logging: Splunk Metrics: JMX => Zenoss Database data: Databus, custom ETL
- #9 This idea of using logs for data flow has been floating around LinkedIn, log-centric fashion. Take all the organization's data and put it into a central log for real-time subscription. Data integration, replication, real-time stream processing.
- #10 Disks are fast when used sequentially File system caching
- #11 Topic = message stream Topic has partitions, partitions are distributed to brokers
- #13 higher availability and durability
- #14 evenly distributed
- #15 replicated log => replicated state machine
- #17 One of the replicas is leader, leader evenly spread All writes go to leader Leader propagates writes to followers in order Leader decides when to commit message
- #22 The size of the ISR is decoupled from the size of the replica set, hence the number of replicas and acknowledgements are independent.
- #23 ack=3
- #29 committed messages to consumer messages are committed is independent of the ack chosen by the producer.
- #30 ack=1
- #31 ack=1
- #36 ack=3, follower slow
- #42 under replicated partitions
- #45 ack=3, broker failure
- #61 load balancing cluster expansion
- #62 load balancing cluster expansion
- #69 This is a major initiative and will put Kafka on the critical path for site latency sensitive data paths which also require much higher message delivery guarantees.
- #71 Data standardization, site monitoring
- #72 Data flow graph. Flow rate may overwhelm query processor: batch processing, sampling, synopsis, etc In-memory storage constraints: single-pass algorithms, no stream backtracking
- #74 WAL
- #76 Streaming on Message Pipes