Uploaded byCloudera, Inc.
Moving Beyond Lambda Architectures with Apache Kudu
The document discusses the Lambda architecture, its advantages and disadvantages, and how Kudu can serve as an alternative. The Lambda architecture marries batch and real-time processing by using separate batch, speed, and serving layers. While it provides scalability, maintaining two code bases is complex. Kudu can fill the gap by enabling both fast analytics on frequently updated data through its ability to support updates, scans and lookups simultaneously. Examples of how Kudu has been used by Xiaomi to simplify their analytics pipeline and reduce latency are provided. The document cautions against premature optimization and advocates optimizing only as needed.
Moving Beyond Lambda Architectures with Apache Kudu
- 1. 1© Cloudera, Inc.All rights reserved. Michael Crutcher Director, Product Management - Storage Lambda Architecture
- 2. 2© Cloudera, Inc.All rights reserved. Agenda • Big Data Challenges • What is Lambda? • Lambda Advantages and Disadvantages • Kudu as a Lambda alternative
- 3. 3© Cloudera, Inc.All rights reserved. Big Data Challenges
- 4. 4© Cloudera, Inc.All rights reserved. “Something interesting is happening” The world’s largest taxi company owns ZERO vehicles. The world’s largest accommodation provider owns ZERO real estate. The world’s most popular media owner creates ZERO content. The world’s leading music platform owns no music.
- 5. 5© Cloudera, Inc.All rights reserved. Data is now a strategic asset Instrumentation Consumerization Experimentation Today, everything that can be measured will be measured. Today, data IS the application. Today, becoming data-driven is a business imperative.
- 6. 6© Cloudera, Inc.All rights reserved. “It will soon be technically feasible & affordable to record & store everything…” — New York Times “Digital technologies will, in the near future, accomplish many tasks once considered uniquely human.” . — Second Machine Age Data is abundant, diverse & shared freely As is how we store, process and analyze it Streaming Machine Learning BI ETL Modeling
- 7. 7© Cloudera, Inc.All rights reserved. The new analytics paradigm Understand why it happened Change what happens next Determine what happened Make it happen consistently
- 8. 8© Cloudera, Inc.All rights reserved. So Why Big Data? What does the reporting look like at your business today? What if it could happen in half the time, or half that time? What data are you looking at? What data do you want to know about your customers? How can you best use external data? Too often data is archived, combined, or simplified to save space and strain on systems. Once data is combined we loose the ability to dig deeper. Better Business Forecasting Better Views of CustomersFull Fidelity Data Access
- 9. 9© Cloudera, Inc.All rights reserved. What is Lambda architecture?
- 10. 10© Cloudera, Inc.All rights reserved. What is Lambda Architecture? Batch Layer Serving Layer Speed Layer New Data Data Lake (HDFS) Precompute Views Stream or Micro Batch Increment Views Data Application “Real-time” Increment Batch Recompute Merge Hadoop Storm/Spark HBase Impala
- 11. 11© Cloudera, Inc.All rights reserved. Batch Layer • Manages the master data set, an immutable, append-only set of raw data • Pre-computes views of the data • “Traditionally” this has been in HDFS and processed with Map/Reduce • There has already been some shift to cloud based object storage and processing in other frameworks like Spark
- 12. 12© Cloudera, Inc.All rights reserved. Speed Layer • This layer ingests streaming data or micro-batches • Spark and Storm are traditionally used • In some cases micro-batches are directly ingested into NoSQL data stores like HBase • This data is periodically expunged • In many “Lambda-like” architectures I’ve seen, this layer is used to provide an “active partition” that provides a limited window of mutability
- 13. 13© Cloudera, Inc.All rights reserved. Serving Layer • As you might guess from the name, this is the layer that serves data • It would be unusual for raw data to be served directly • This could be an application written directly against a data store like HBase • It could be a SQL engine on top of a file system, Impala + Parquet is an example
- 14. 14© Cloudera, Inc.All rights reserved. What is a Kappa Architecture? Batch Layer Serving Layer Speed Layer New Data Data Lake (HDFS) Precompute Views Stream or Micro Batch Increment Views Data Application “Real-time” Increment Batch Recompute Merge Hadoop Storm/Spark HBase Impala
- 15. 15© Cloudera, Inc.All rights reserved. Everything Has a New Name Batch Layer Serving Layer Speed Layer New Data Data Lake (HDFS) Precompute Views Stream or Micro Batch Increment Views Data Application “Real-time” Increment Batch Recompute Merge System of Record (OLTP) Operational Data Store Derived Tables (EDW) In-Memory Database Star/Snowflake, Cubes, or In-Memory Tables
- 16. 16© Cloudera, Inc.All rights reserved. The Log as Storage • Lambda and Kappa architectures are both predicated on immutable source data • Data can be modeled as a series of events recorded at specific points in time about entities • Updates are modeled as new events and the current or historic value associated with an entity can be reconstructed through the collected events • Kappa calls this ordered set of events “a log”, it’s safe to say they didn’t invent this term A B C D B E F A G B 1 10 Ordered over Time
- 17. 17© Cloudera, Inc.All rights reserved. Is Raw Data the Right Logical Model? • It’s possible to derive many higher level logical abstractions from raw data • As an example, I could construct a customer account balance from raw account activity data • This doesn’t mean it’s a good idea A B C B A C B A A C t0 t12Account Activity +$10 +$20 +$15 -$10 +$35 -$5 +$25 +$15 -$20 +$10 Easy: What was the last account event for Customer C? Harder: What is the account balance for Customer A at t12?
- 18. 18© Cloudera, Inc.All rights reserved. There are Only Two Hard CS Problems 1) Cache invalidation 2) Naming things -- Phil Karlton
- 19. 19© Cloudera, Inc.All rights reserved. Data Engineering has one hard problem • When should I denormalize to maximize performance? • When should I normalize to minimize maintenance problems? Denormalize Everything! Normalize Everything! I wish things were faster! I wish things were easier to maintain!
- 20. 20© Cloudera, Inc.All rights reserved. Lambda Advantages and Disadvantages
- 21. 21© Cloudera, Inc.All rights reserved. Lambda Advantages • Marries diverse strengths of existing open source software into a unified architecture • Provides scalability via the batch layer • Provides real time performance via the speed layer
- 22. 22© Cloudera, Inc.All rights reserved. Lambda Disadvantages • Complexity • Many moving parts • Restatement is difficult • Two code bases must be kept in sync • Proper failure handling is complex
- 23. 23© Cloudera, Inc.All rights reserved. Lambda Complexity Batch Layer Serving Layer Speed Layer New Data Data Lake (HDFS) Precompute Views Stream or Micro Batch Increment Views Data Application “Real-time” Increment Batch Recompute Merge Hadoop Storm/Spark HBase Impala Code must be kept in sync Restatement is difficult
- 24. 24© Cloudera, Inc.All rights reserved. Lambda Complexity Batch Layer Serving Layer Speed Layer New Data Data Lake (HDFS) Precompute Views Stream or Micro Batch Increment Views Data Application “Real-time” Increment Batch Recompute Merge Hadoop Storm/Spark HBase Impala Hmm… this data looks fishy Problem Here? Here? Here? Here? Here? Here? Here?
- 25. 25© Cloudera, Inc.All rights reserved. The Log as Storage • The idea of representing data as immutable log information is not new and is not without tradeoffs: • Space amplification: how many bytes of data are stored, relative to how many logical bytes the database contains • Write amplification: how many bytes of data are written by the database compared to the number of bytes changed by the user • Read amplification: how many bytes the database has to physically read to return values to the user compared to the bytes returned • Complexity: am I solving a CS problem or a customer problem? • These are not simple issues and there’s no straightforward “right” answer
- 26. 26© Cloudera, Inc.All rights reserved. Premature Optimization Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. --Donald Knuth
- 27. 27© Cloudera, Inc.All rights reserved. Gap Filling vs. Optimization • Some Lambda implementations are deployed on big data systems that don’t require significant optimization to deliver desired SLAs • Often, Lambda architectures are used to fill the very stark difference in workload processing capabilities of technologies that are used typically used for the batch (long scan) and fast layers (quick point lookups) • Anecdotally, Lambda architectures seem to be deployed much more often with current generation open source technology than they were with legacy commercial offerings • Part of this is because of data volume, variety, and velocity caused by our increasingly data driven world, but I think part of this is also because legacy technologies haven’t had as stark of a difference in what workloads they’re optimal for • Are you deploying a Lambda architecture because you need to squeeze out all of the performance possible, or because you have a mixed workload that can’t be deployed on one single storage technology?
- 28. 28© Cloudera, Inc.All rights reserved. Gap Filling v2: Lack of Mutability • Some Lambda implementations aim to fill the gap of the lack of mutability in HDFS • Raw, master data should be immutable, but in the real world raw data could potentially need to be adjusted • Sensors could have been miscalibrated, data may have been incorrectly entered, raw data might be an approximation before finalization, etc. • Derived aggregations might more efficiently modified in place, vs. recalculated from raw data, recalculating all of history is often not practically possible Incoming Data (Messaging System) New Partition Most Recent Partition Historic Data HBase Parquet File • Wait for running operations to complete • Define new Impala partition referencing the newly written Parquet file Reporting Request Impala on HDFS
- 29. 29© Cloudera, Inc.All rights reserved. Kudu as a Lambda alternative
- 30. 30© Cloudera, Inc.All rights reserved. HDFS Fast Scans, Analytics and Processing of Stored Data Fast On-Line Updates & Data Serving Arbitrary Storage (Active Archive) Fast Analytics (on fast-changing or frequently-updated data) Kudu: Fast Analytics on Fast-Changing Data New storage engine enables new Hadoop use cases Unchanging Fast Changing Frequent Updates HBase Append-Only Real-Time Kudu Kudu fills the Gap Modern analytic applications often require complex data flow & difficult integration work to move data between HBase & HDFS Analytic Gap Pace of Analysis PaceofData
- 31. 31© Cloudera, Inc.All rights reserved. Kudu Increases the Value of Time Series Data Time Series Inserts, updates, scans, lookups Workload Examples Stream market data, fraud detection & prevention, risk monitoring Time series data is most valuable if you can analyze it to change outcomes in real time. Kudu simulateneously enables: • Time series data inserted/updated as it arrives • Analytic scans to find trends on fresh time series data • Lookups to quickly visit the point in time where an event occurred for further investigation
- 32. 32© Cloudera, Inc.All rights reserved. Kudu can help spot problems before they happen. Real-time data inserts with the ability to analyze trends identifies potential problems. Kudu identifies trouble through: • Extreme scale, allowing better historic trend analysis • Fast inserts to enable an up-to-date view of your business • Fast scans identify/flag undesired states for remedy Kudu Keeps Your Business Operational Machine Data Analytics Inserts, scans, lookups Workload Examples Network threat detection, IoT, predictive maintenance and failure detection
- 33. 33© Cloudera, Inc.All rights reserved. More Versatility in Online Reporting Online Reporting Inserts, updates, scans, lookups Workload Examples “Active” Reporting Online reporting has traditionally been limited by data volume and analytic capabilitiy, keeping only recent data designed for granular queries. Kudu adds online reporting versatility through: • Fast inserts and updates to keep data fresh • Fast lookups and analytic scans in one data store
- 34. 34© Cloudera, Inc.All rights reserved. Xiaomi use case • World’s 4th largest smart-phone maker (most popular in China) • Gather important RPC tracing events from mobile app and backend service. • Service monitoring & troubleshooting tool. High write throughput • >5 Billion records/day and growing Query latest data and quick response • Identify and resolve issues quickly Can search for individual records • Easy for troubleshooting
- 35. 35© Cloudera, Inc.All rights reserved. Xiaomi big data analytics pipeline Large ETL pipeline delays ● High data visibility latency (from 1 hour up to 1 day) ● Data format conversion woes Ordering issues ● Log arrival (storage) not exactly in correct order ● Must read 2 – 3 days of data to get all of the data points for a single day
- 36. 36© Cloudera, Inc.All rights reserved. Xiaomi big data analytics pipeline Simplified with Kudu Low latency ETL pipeline ● ~10s data latency ● For apps that need to avoid direct backpressure or need ETL for record enrichment Direct zero-latency path ● For apps that can tolerate backpressure and can use the NoSQL APIs ● Apps that don’t need ETL enrichment for storage / retrieval OLAP scan Side table lookup Result store
- 37. 37© Cloudera, Inc.All rights reserved. Conclusions • Lambda has a real place in big data architectures • Optimize as needed, but beware of the cost of premature optimization • Kudu is designed to be a simple solution for when you need a data store that’s updatable and provides “good enough” performance for analytic and real time workloads simultaneously