Big Data Fundamentals: kafka example

Building a Robust Streaming Data Lake with Apache Hudi on Kafka

1. Introduction

Modern data platforms are increasingly driven by real-time insights. A common engineering challenge is building a data lake capable of handling high-velocity, high-volume data streams while simultaneously supporting low-latency analytical queries. Traditional batch-oriented data lakes struggle with this paradigm. We recently faced this issue while ingesting clickstream data for a large e-commerce platform – peaking at 500K events/second, requiring sub-second query latency for real-time personalization, and demanding strict data consistency. Simply appending data to Parquet files in S3 wasn’t scalable or query-friendly. This led us to adopt Apache Hudi as a core component of our streaming data lake, built on top of Kafka. This post details our architecture, performance tuning, and operational considerations for this system. We’re dealing with approximately 5TB of daily ingestion, schema evolution happening weekly, and a requirement for near-real-time dashboards and machine learning feature stores.

2. What is Apache Hudi in Big Data Systems?

Apache Hudi is a data lake storage framework that brings database-like features to data lakes. It’s not a database itself, but rather a layer on top of existing storage (S3, GCS, Azure Blob Storage) that enables incremental processing, ACID transactions, and efficient data updates/deletes. From an architectural perspective, Hudi sits between our streaming ingestion layer (Kafka) and our analytical compute engines (Spark, Presto/Trino). It provides a copy-on-write (CoW) or merge-on-read (MoR) storage format, allowing for efficient querying of the latest data while maintaining historical versions. We primarily use the Parquet file format within Hudi, leveraging its columnar storage and compression capabilities. Hudi’s protocol-level behavior involves writing data to a timeline of log files that track changes, enabling efficient snapshot isolation and time travel queries.

3. Real-World Use Cases

• Clickstream Analytics: Ingesting user clickstream data from Kafka for real-time personalization, A/B testing, and funnel analysis.
• Change Data Capture (CDC): Capturing database changes via Debezium and landing them in Hudi for near real-time data synchronization and reporting.
• Streaming ETL: Performing transformations on streaming data (e.g., enriching with geo-location data) before landing it in the data lake.
• Machine Learning Feature Pipelines: Building and serving real-time features for fraud detection and recommendation systems. Hudi’s time travel capabilities are crucial for training models on historical data.
• Log Analytics: Aggregating and analyzing application logs for monitoring, troubleshooting, and security auditing.

4. System Design & Architecture

graph LR A[Kafka] --> B(Hudi Ingestion - Spark Structured Streaming); B --> C{Hudi Table (S3)}; C --> D[Spark Batch Processing]; C --> E[Trino/Presto]; C --> F[ML Feature Store]; subgraph Data Lake C end subgraph Compute Engines D E F end style A fill:#f9f,stroke:#333,stroke-width:2px style C fill:#ccf,stroke:#333,stroke-width:2px 

This diagram illustrates a typical Hudi-based data lake architecture. Kafka serves as the central message bus. Spark Structured Streaming consumes data from Kafka and writes it to Hudi tables stored in S3. Spark batch jobs perform further transformations and aggregations. Trino/Presto provides low-latency SQL querying capabilities. An ML Feature Store consumes data from Hudi for real-time feature serving. We deploy this on AWS EMR, leveraging its managed Spark and Hadoop ecosystem. Partitioning is key: we partition by event time (day granularity) and a hash of the user ID to distribute data evenly across executors.

5. Performance Tuning & Resource Management

Hudi performance is heavily influenced by Spark configuration. Here are some key settings:

• spark.sql.shuffle.partitions: Set to 200-400 based on cluster size. Too few partitions lead to large tasks; too many create overhead.
• fs.s3a.connection.maximum: Set to 1000 to maximize S3 throughput.
• hudi.write.operation.retry.max: Set to 3-5 to handle transient S3 errors.
• hudi.write.lock.concurrency: Increase to 10-20 for higher write concurrency.
• spark.driver.memory: 16g
• spark.executor.memory: 8g
• spark.executor.cores: 5

We also leverage Hudi’s compaction feature to optimize file sizes. Compaction merges small Parquet files into larger ones, improving query performance. We schedule compaction jobs nightly using Airflow. Monitoring S3 request latency and Spark task duration is crucial for identifying bottlenecks. We use Datadog to track these metrics and alert on anomalies. Choosing the right Hudi table type (CoW vs. MoR) is also critical. We initially used CoW, but switched to MoR for better write performance, accepting a slight increase in read latency.

6. Failure Modes & Debugging

• Data Skew: Uneven distribution of data across partitions can lead to task imbalance and OOM errors. Solution: Salt the key used for partitioning to distribute data more evenly.
• Out-of-Memory Errors: Insufficient executor memory. Solution: Increase executor memory or reduce the amount of data processed per task.
• Job Retries: Transient S3 errors or network issues. Solution: Increase retry counts and implement exponential backoff.
• DAG Crashes: Often caused by bugs in Spark code or Hudi configuration. Solution: Examine Spark UI for error messages and stack traces. Enable Hudi’s detailed logging for more insights.

We use the Spark UI extensively for debugging. The Hudi timeline view within the Spark UI provides valuable information about the state of the Hudi table and the progress of write operations. Datadog alerts notify us of job failures and performance degradation.

7. Data Governance & Schema Management

We integrate Hudi with the AWS Glue Data Catalog for metadata management. Schema evolution is handled using Avro schemas, registered in a Schema Registry (Confluent Schema Registry). We enforce schema compatibility using the Schema Registry’s compatibility rules. Hudi’s support for schema evolution allows us to add new columns without breaking existing queries. Data quality checks are performed using Great Expectations, integrated into our CI/CD pipeline.

8. Security and Access Control

We use AWS IAM roles to control access to S3 buckets and other AWS resources. We leverage AWS Lake Formation to manage fine-grained access control to Hudi tables. Data encryption is enabled at rest using S3 server-side encryption (SSE-S3). Audit logging is enabled on S3 buckets to track data access and modifications.

9. Testing & CI/CD Integration

We use Great Expectations to validate data quality at each stage of the pipeline. DBT tests are used to validate data transformations. We have a dedicated staging environment where we deploy changes before promoting them to production. Automated regression tests are run after each deployment to ensure that the pipeline is functioning correctly. Pipeline linting is performed using a custom script to enforce coding standards and best practices.

10. Common Pitfalls & Operational Misconceptions

• Incorrect Partitioning: Leads to data skew and poor query performance. Symptom: Long task durations, OOM errors. Mitigation: Carefully choose partitioning keys based on data distribution.
• Insufficient Compaction: Results in a large number of small files, degrading query performance. Symptom: Slow query response times. Mitigation: Schedule regular compaction jobs.
• Ignoring Hudi Timeline: Failing to understand the Hudi timeline makes debugging difficult. Symptom: Difficulty identifying the root cause of write failures. Mitigation: Learn to use the Hudi timeline view in the Spark UI.
• Over-provisioning Resources: Leads to unnecessary costs. Symptom: Low resource utilization. Mitigation: Right-size resources based on workload requirements.
• Lack of Schema Enforcement: Results in data quality issues. Symptom: Invalid data, query failures. Mitigation: Enforce schema validation using a Schema Registry.

11. Enterprise Patterns & Best Practices

• Data Lakehouse vs. Warehouse: Hudi enables a data lakehouse architecture, combining the flexibility of a data lake with the reliability and performance of a data warehouse.
• Batch vs. Micro-batch vs. Streaming: Choose the appropriate processing mode based on latency requirements. We use micro-batching for near real-time ingestion.
• File Format Decisions: Parquet is generally the best choice for analytical workloads.
• Storage Tiering: Use S3 Glacier for archiving historical data.
• Workflow Orchestration: Airflow is essential for scheduling and managing complex data pipelines.

12. Conclusion

Apache Hudi has been instrumental in building a robust and scalable streaming data lake. It provides the necessary features for handling high-velocity data streams, supporting low-latency queries, and ensuring data consistency. Next steps include benchmarking different compaction strategies, introducing schema enforcement at the Kafka level, and migrating to a more efficient file format like Apache Iceberg for even greater performance and scalability. Continuous monitoring and optimization are crucial for maintaining a healthy and reliable data platform.