The need to handle large streams of data reliably and at scale has become a necessity in this age of big data and real-time applications. At the core of this revolution is Apache Kafka, a distributed, durable, highly scalable event streaming system used for building streaming applications and real-time pipelines. In this article, we will explore Kafka’s core architectural concepts, show how modern data engineering teams use it, examine practical production practices and configurations, and highlight concrete use cases from Netflix, LinkedIn, and Uber.
1. What is Kafka?
Apache Kafka is a distributed event streaming platform that exposes a durable, partitioned, append-only log. Producers write events to named topics, which are split into partitions for scale; consumers read from partitions independently and maintain offsets to track progress. Kafka was designed for high throughput, horizontal scalability, and fault tolerance, and it’s widely used for log aggregation, stream processing, event sourcing, and building real-time applications. (Apache Kafka)
2. Core concepts
Topics & partitions
A topic is a named stream of records. Each topic is split into partitions, which are the units of parallelism. Partitions are ordered, and each record in a partition has an offset (a monotonically increasing sequence number).
Brokers, clusters, and leaders
A broker is a Kafka server. Brokers form a cluster; each partition has one leader (handles reads/writes) and zero or more followers (replicas) that copy the leader’s data.
Replication & fault tolerance
Replication factor (RF) controls how many copies of each partition exist. If you set RF=3 and one broker fails, followers can be promoted to leader to maintain availability.
Producers & consumers
Producers publish messages to topics. Consumers join consumer groups; Kafka ensures each partition is consumed by at most one consumer in the group (parallelism + load balancing). Offsets let consumers resume from a known position.
Exactly-once and delivery guarantees
Kafka supports at-least-once delivery by default. Using idempotent producers, transactions, and Streams’ exactly-once semantics you can get end-to-end exactly-once guarantees for many topologies. The Kafka documentation explains these guarantees and client configs in depth. (Apache Kafka)
3. Storage model & delivery semantics
Kafka’s storage model is an append-only log persisted to local disk. Each partition is stored as a sequence of segment files. Kafka leverages OS page cache and sequential disk writes to achieve very high throughput. Retention policies (time or size) and log-compaction (keep last value per key) let you tune storage semantics: use time-based retention for metrics/history and compaction for changelog or stateful topics. Core docs provide details on retention, compaction, and log segments. (Apache Kafka)
4. Kafka ecosystem
- Kafka Connect includes ready-made connectors (JDBC, S3, HDFS, etc.) for ingestion and export.
- Kafka Streams is a client library for writing stream processing apps without a separate cluster.
- ksqlDB is a SQL interface for stream processing. These components let teams build an end-to-end streaming platform without stitching many disparate tools. Confluent and the Apache Kafka project provide extensive guides for platform design and enterprise patterns. (Confluent)
5. Data engineering patterns & applications
Ingestion (high throughput)
Common pattern: front-end services → Kafka producers → ingest topic. Use batching, compression (snappy/lz4), and asynchronous sends to maximize producer throughput.
Stream processing & enrichment
Stream processing frameworks (Kafka Streams, Flink, Spark Structured Streaming) subscribe to topics, enrich events (join with lookups), and write results to downstream topics or data stores.
Change Data Capture (CDC)
CDC tools (Debezium, Maxwell) publish database changes to Kafka topics. This enables low-latency replication, audit logs, and event sourcing.
Event sourcing & materialized views
Use Kafka as the canonical event store; build materialized views using stream processors. For example: user actions → events → aggregated metrics stored in a database for queries.
6. Production practices
Cluster sizing and partitioning
- Partitioning determines parallelism; plan partitions per topic so consumers can scale.
- Replication factor: production clusters commonly use RF=3 for durability. Confluent’s enterprise guidance helps decide cluster strategies and operational patterns. (Confluent)
Retention & tiered storage
For very large clusters, use tiered storage to offload older segments to cheaper object stores (S3, GCS). Uber and others have implemented tiered storage to manage petabytes affordably. (Uber)
Monitoring & observability
Track broker CPU/disk, network, under-replicated partitions, consumer lag, JVM GC, and request latencies. Expose metrics via JMX and ship to Prometheus/Grafana or your metrics platform.
Security: encryption & ACLs
Enable TLS for in-transit encryption, SASL/Kerberos or OAuth for authentication, and Kafka ACLs for authorization. Uber published work on securing Kafka at scale which covers authz/authn practices. (Uber)
Upgrades & rolling restarts
Use rolling upgrades (broker one-by-one), ensure no single point of failure for Zookeeper (if used) or use KRaft-mode Kafka (no Zookeeper) in recent versions. Produce and consume during rolling restarts to minimize downtime.
7. Use Cases: Netflix, LinkedIn, Uber
LinkedIn originally developed Kafka to power activity streams and log ingestion. Over time, LinkedIn scaled Kafka to handle hundreds of billions to trillions of messages per day, building a broad ecosystem and contributing many improvements back to the project. Their engineering posts describe Kafka’s history and operational lessons. (LinkedIn Engineering)
Netflix
Netflix uses Kafka as the central event bus (Keystone pipeline), powering telemetry, real-time analytics, and change propagation between microservices. Kafka enabled unified event collection and multiple consumers (analytics, monitoring, personalization) reading the same events. Confluent’s case summaries and Netflix tech blog detail how Kafka supports both batch and stream needs. (Netflix Tech Blog)
Uber
Uber runs Kafka at enormous scale to support more than 300 microservices, dynamic pricing, and real-time analytics. They’ve engineered tiered storage and consumer proxies to make Kafka both scalable and operable across the organization. Uber engineering posts describe security hardening and tiered storage adoption for cost and capacity management. (Uber)
8. Example code & configuration
Minimal Python producer (kafka-python)
from kafka import KafkaProducer producer = KafkaProducer( bootstrap_servers=['broker1:9092','broker2:9092'], value_serializer=lambda v: v.encode('utf-8'), acks='all', # wait for all replicas compression_type='lz4', retries=5 ) producer.send('payments', key=b'user-123', value='{"amount":49.99,"currency":"USD"}') producer.flush() Minimal Python consumer (kafka-python)
from kafka import KafkaConsumer consumer = KafkaConsumer( 'payments', bootstrap_servers=['broker1:9092'], group_id='payments-processor', auto_offset_reset='earliest', enable_auto_commit=False ) for msg in consumer: process(msg.key, msg.value) consumer.commit() # or use transactional processing server.properties snippets (broker)
# basic broker settings broker.id=1 listeners=PLAINTEXT://0.0.0.0:9092 log.dirs=/var/lib/kafka/logs num.partitions=6 default.replication.factor=3 log.retention.hours=168 9. Simple architecture diagrams
Topic partitioning & replication (ASCII)
Topic: payments Partitions: p0, p1, p2 Brokers: 1,2,3 p0: leader@1 replicas: [1 (L),2,3] p1: leader@2 replicas: [2 (L),3,1] p2: leader@3 replicas: [3 (L),1,2] Typical pipeline
Producers --> Kafka (ingest topics) --> Stream Processing (Kafka Streams/Flink) --> Output topics --> Data sinks (DBs, S3, dashboards) Conclusion & further reading
Apache Kafka is a flexible, high-performance foundation for event streaming and real-time data engineering. Applied correctly, with thought to partitioning, replication, retention, observability, and security, it enables organizations to build resilient streaming platforms used by LinkedIn, Netflix, Uber, and many others.
Resources
- Apache Kafka Documentation — architecture, clients, and config. (Apache Kafka)
- Confluent blog & platform guides — practical design and enterprise patterns. (Confluent)
- LinkedIn engineering — Kafka origin and scale case studies. (LinkedIn Engineering)
- Netflix TechBlog — Kafka in their Keystone pipeline and pipeline evolution. (Netflix Tech Blog)
- Uber engineering posts — tiered storage and securing Kafka at scale. (Uber)
Top comments (0)