Downloaded 101 times
Uploaded byYugabyteDB
Distributed Databases Deconstructed: CockroachDB, TiDB and YugaByte DB
The document discusses distributed SQL databases, particularly focusing on Amazon Aurora and Google Spanner, comparing their architectures and features. It highlights the advantages of distributed SQL in terms of scalability, fault tolerance, and globally consistent writes while addressing common issues associated with traditional SQL databases. Additionally, it introduces YugabyteDB, an open-source alternative inspired by Spanner, and its design principles aimed at achieving high performance and consistency in distributed environments.
Distributed Databases Deconstructed: CockroachDB, TiDB and YugaByte DB
- 1. 1© 2019 Allrights reserved. Distributed SQL Databases Deconstructed Understanding Amazon Aurora, Google Spanner & the Spanner Derivatives
- 2. 2© 2019 Allrights reserved. Introduction  Amey Banarse Principal Data Architect, YugabyteDB ♦ Pivotal ♦ FINRA University of Pennsylvania @ameybanarse http://about.me/amey
- 3. 3© 2019 Allrights reserved. Types of Data Stores Today’s Focus OLAP OLTP Write once, Read many Few concurrent sessions Long running, ad-hoc queries Large table scans Petabyte-scale data storage Mixed reads & writes Many concurrent sessions Single-digit ms query latency Point reads & short-range scans Terabyte-scale data storage
- 4. 4© 2019 Allrights reserved. Today’s Focus Types of Data Stores Open Source Proprietary OLAP OLTP NoSQL SQL SQLNoSQL Google BigTable Amazon Aurora Google Spanner Google BigQuery
- 5. 5© 2019 Allrights reserved. Why Devs 😍 SQL? 1. Query Flexibility 💪 – Model data once, change queries as business changes – Balance modeling richness with performance needs 2. Rich Ecosystem 🔌 – Data modeling & query examples – Developer IDEs & data visualization tools – Easy to reuse & build integrations 3. Universal Standard for Data Access 😇 – Learn once, use forever
- 6. 6© 2019 Allrights reserved. Why Devs 😡 SQL? 1. Large Dataset? 📈 – No horizontal write scalability – Use manually sharded SQL or non-transactional NoSQL 2. Infrastructure Failures? 🚨 – No native failover & repair, SPOF w/ Single Node DB – Use complex replication schemes 3. Multi-Region/Geo-Distributed App? 🌏 – Multi-master deployment is the only option – Data inconsistency w/ Last Writer Wins (LWW) conflict resolution
- 7. 7© 2019 Allrights reserved. Distributed SQL = Keep 😍 & Remove 😡 1. SQL Features – ACID, JOINs, foreign keys, serializable isolation 2. Horizontal Write Scalability – Scale write throughput by adding/removing nodes 3. Fault Tolerance With High Availability – Native failover & repair 4. Globally Consistent Writes – Lower end user latency and tolerate region failures 5. Low Read Latency – Strongly consistent (aka correct) reads
- 8. 8© 2019 Allrights reserved. What’s the fastest growing service in AWS?
- 9. 9© 2019 Allrights reserved. Amazon Aurora https://www.allthingsdistributed.com/2019/03/Amazon-Aurora-design-cloud-native-relational-database.html
- 10. 10© 2019 Allrights reserved. Excerpts from Vogels blog post https://www.allthingsdistributed.com/2019/03/Amazon-Aurora-design-cloud-native-relational-database.html
- 11. 11© 2019 Allrights reserved. What database powers Google AdWords and Google Play?
- 12. 12© 2019 Allrights reserved. “At Google, Spanner supports tens of millions of queries per second and runs some of our most critical services, including AdWords and Google Play.” https://ai.google/research/pubs/pub39966 Google Cloud Spanner
- 13. 13© 2019 Allrights reserved. Distributed SQL Architectures - Aurora vs Spanner Amazon Aurora Google Cloud Spanner “A highly available MySQL and PostgreSQL-compatible relational database service” Available on AWS since 2015 “The first horizontally scalable, strongly consistent, relational database service” Available on Google Cloud since 2017 Shared Storage Shared Nothing
- 14. 14© 2019 Allrights reserved. #1 SQL Features and Completeness
- 15. 15© 2019 Allrights reserved. Depth of SQL Support ✓ MySQL and PostgreSQL-compatible Subset of MySQL/PostgreSQL features Amazon Aurora Google Cloud Spanner https://cloud.google.com/spanner/docs/migrating-p ostgres-spanner
- 16. 16© 2019 Allrights reserved. Aurora vs Spanner Feature Amazon Aurora Google Spanner SQL Features ✓ Horizontal Write Scalability ✓ Fault Tolerance with HA ✓ Globally Consistent Writes ✓ Low Read Latency
- 17. 17© 2019 Allrights reserved. #2 Horizontal Write Scalability
- 18. 18© 2019 Allrights reserved. Amazon Aurora Single Node SQL on Multi-Zone Distributed Storage SQL APP INSERT ROW ❌ Add Primary Instances for Write Scaling ✓ Add Read Replicas for Read Scaling
- 19. 19© 2019 Allrights reserved. Google Spanner Multi-Node SQL on Multi-Region Distributed Storage SQL APP INSERT ROW3 ✓ Add Primary Instances for Write Scaling ✓ Add Read Replicas for Read Scaling INSERT ROW1
- 20. 20© 2019 Allrights reserved. Aurora vs Spanner Feature Amazon Aurora Google Spanner SQL Features ✓ Horizontal Write Scalability ❌ ✓ Fault Tolerance with HA Globally Consistent Writes Low Read Latency
- 21. 21© 2019 Allrights reserved. #3 Fault Tolerance with HA
- 22. 22© 2019 Allrights reserved. Amazon Aurora Native Failover & Repair Through Primary Auto Election SQL APP ✓ HA When Primary Instance Fails ✓ HA When Read Replica Fails INSERT ROW
- 23. 23© 2019 Allrights reserved. Google Spanner Native Failover & Repair Through Shard Leader Auto Election SQL APP INSERT ROW1 ✓ HA When Any Primary Node Fails ✓ HA When Read Replica Fails INSERT ROW3
- 24. 24© 2019 Allrights reserved. Aurora vs Spanner Feature Amazon Aurora Google Spanner SQL Features ✓ Horizontal Write Scalability ❌ ✓ Fault Tolerance with HA ✓ ✓ Globally Consistent Writes Low Read Latency
- 25. 25© 2019 Allrights reserved. #4 Global Write Consistency
- 26. 26© 2019 Allrights reserved. Amazon Aurora Multi-Master Last Writer Wins Conflict Resolution Leads to Inconsistencies SQL APP SET BALANCE = BALANCE - 10 SQL APP SET BALANCE = BALANCE - 100 Asynchronous Replication Region 1 Region 2
- 27. 27© 2019 Allrights reserved. Google Spanner Purpose-Built for Globally Consistent Writes SQL APP SET BALANCE = BALANCE - 10 SQL APP SET BALANCE = BALANCE - 100
- 28. 28© 2019 Allrights reserved. Aurora vs Spanner Feature Amazon Aurora Google Spanner SQL Features ✓ Horizontal Write Scalability ❌ ✓ Fault Tolerance with HA ✓ ✓ Globally Consistent Writes ❌ ✓ Low Read Latency
- 29. 29© 2019 Allrights reserved. #5 Low Read Latency
- 30. 30© 2019 Allrights reserved. Amazon Aurora Strongly Consistent Reads Served By Primary Instance SQL APP READ ROW
- 31. 31© 2019 Allrights reserved. Google Spanner Strongly Consistent Reads Served By Shard Leaders w/o Read Quorum SQL APP READ ROW1
- 32. 32© 2019 Allrights reserved. Aurora vs Spanner Feature Amazon Aurora Google Spanner SQL Features ✓ Horizontal Write Scalability ❌ ✓ Fault Tolerance with HA ✓ ✓ Global Write Consistency ❌ ✓ Low Read Latency ✓ ✓
- 33. 33© 2019 Allrights reserved. Battle of Architectures - Spanner Beats Aurora No Performance & Availability Bottlenecks Scale to Large Clusters while Remaining Highly Available Built for Geo-Distributed Apps Future Proofs Data Tier at Global Businesses Complex to Engineer Needs Clock Skew Tracking Across Instances
- 34. 34© 2019 Allrights reserved. Analyzing Open Source Spanner Inspired Derivatives
- 35. 35© 2019 Allrights reserved. Spanner Brought to Life in Open Source
- 36. 36© 2019 Allrights reserved. YugabyteDB Design Principles • CP in CAP Theorem • Consistent • Partition Tolerant • HA on failures (new leader elected in seconds) • ACID Transactions • Single-row linearizability • Multi-row ACID • Serializable & Snapshot • No bottlenecks even for geo-distributed rows • High Performance • All layers in C++ to ensure high perf • Run on large memory machines • Optimized for SSDs • Deploy Anywhere • No IaaS specific dependencies • No atomic clocks • Bare metal, VM and Kubernetes
- 37. 37© 2019 Allrights reserved. Design Follows a Layered Approach tablet 1’ tablet 1’ tablet 1’ Self-Healing, Fault-Tolerant Auto Sharding & Rebalancing ACID Transactions Global Data Distribution High Throughput, Low Latency YCQL SQL-Based Flexible Schema API YSQL Globally Distributed Postgres API
- 38. 38© 2019 Allrights reserved. YugabyteDB Architecture
- 39. Design Goals ✓ Highlyscalable & resilient ✓ Transactional - strong consistency ✓ All layers in C++ for high performance ✓ No dependencies on external systems ✓ Cloud-native – online re-configuration
- 40. API Goals ✓ Multi-model ✓Start with well known APIs ✓ Extend to fill functionality gaps ✓ APIs supported ● PostgreSQL-compatible YSQL ● Cassandra Query Language-compatible YCQL
- 41. 41© 2019 Allrights reserved. YugabyteDB Deployment Topologies
- 42. 42© 2019 Allrights reserved. Distributed SQL = Keep 😍 & Remove 😡 1. SQL Completeness and Features 2. Replication Protocol 3. Clock Skew Tracking 4. Geo-Distributed Transactions
- 43. 43© 2019 Allrights reserved. Spanner vs. its Open Source Derivatives Feature Google Spanner YugabyteDB CockroachDB TiDB Cost Expensive Free Free Free SQL API Compatibility Replication Protocol Clock Skew Tracking Geo-Distributed Txns Tunable Read Latency
- 44. 44© 2019 Allrights reserved. SQL API Compatibility
- 45. 45© 2019 Allrights reserved. PostgreSQL Transformed into Distributed SQL
- 46. 46© 2019 Allrights reserved. Depth of SQL Support - YugabyteDB • SQL Features • Data Types • Referential Integrity (Foreign Keys) • Built-in Functions • Expressions • JSON Column Type • Secondary Indexes • JOINs • Transactions • Views • Advanced SQL Features • Partial Indexes • Stored Procedures • Triggers • PG Extensions
- 47. 47© 2019 Allrights reserved. Spanner vs. its Open Source Derivatives Feature Google Spanner YugabyteDB CockroachDB TiDB Cost Expensive Free Free + Proprietary Free SQL API Compatibility Proprietary PostgreSQL PostgreSQL No Stored Procedures MySQL No Foreign Keys Replication Protocol Clock Skew Tracking Transaction Manager Tunable Read Latency
- 48. 48© 2019 Allrights reserved. Replication Protocol
- 49. 49© 2019 Allrights reserved. Every Table is Automatically Sharded tablet 1’ … … … … … … … … … … … … … … … SHARDING = AUTOMATIC PARTITIONING OF TABLES
- 50. 50© 2019 Allrights reserved. Replication Done at Shard Level tablet 1’ Tablet Peer 1 on Node X Tablet #1 Tablet Peer 2 on Node Y Tablet Peer 3 on Node Z
- 51. 51© 2019 Allrights reserved. Replication uses a Consensus algorithm tablet 1’ Raft Leader Uses Raft Algorithm First elect Tablet Leader
- 52. 52© 2019 Allrights reserved. Writes in Raft Consensus tablet 1’ Raft Leader Writes processed by leader: Send writes to all peers Wait for majority to ack Write
- 53. 53© 2019 Allrights reserved. Reads in Raft Consensus tablet 1’ Raft Leader Reads handled by leader Uses Leader Leases* for performance optimization Read *Leader Leases: https://blog.yugabyte.com/low-latency-reads-in-geo-distributed-sql-with-raft-leader-leases/
- 54. 54© 2019 Allrights reserved. Spanner vs. its Open Source Derivatives Feature Google Spanner YugabyteDB CockroachDB TiDB Cost Expensive Free Free + Proprietary Free SQL API Compatibility Proprietary PostgreSQL PostgreSQL No Stored Procedures MySQL No Foreign Keys Replication Protocol Paxos Raft Raft Raft Clock Skew Tracking Geo-Distributed Txns Tunable Read Latency
- 55. 55© 2019 Allrights reserved. Transactions and Clock Skew Tracking
- 56. 56© 2019 Allrights reserved. Multi-Shard Transactions tablet 1’ k1 and k2 may belong to different shards BEGIN TXN UPDATE k1 UPDATE k2 COMMIT Belong to different Raft groups on completely different nodes
- 57. 57© 2019 Allrights reserved. What do Distributed Transactions need? tablet 1’ Updates should get written at the same physical time Raft Leader Raft Leader BEGIN TXN UPDATE k1 UPDATE k2 COMMIT But how will nodes agree on time?
- 58. 58© 2019 Allrights reserved. Use a Physical Clock tablet 1’ You would need an Atomic Clock or two lying around Atomic Clocks are highly available, globally synchronized clocks with tight error bounds Most of my physical clocks are never synchronized Jeez! I’m fresh out of those.
- 59. 59© 2019 Allrights reserved. Hybrid Logical Clock or HLC tablet 1’ Combine coarsely-synchronized physical clocks with Lamport Clocks to track causal relationships (physical component, logical component) synchronized using NTP a monotonic counter Nodes update HLC on each Raft exchange for things like heartbeats, leader election and data replication
- 60. 60© 2019 Allrights reserved. Spanner vs. its Open Source Derivatives Feature Google Spanner YugabyteDB CockroachDB TiDB Cost Expensive Free Free + Proprietary Free SQL API Compatibility Proprietary PostgreSQL PostgreSQL No Stored Procedures MySQL No Foreign Keys Replication Protocol Paxos Raft Raft Raft Clock Skew Tracking TrueTime Atomic Clock Hybrid Logical Clock + Max Clock Skew Hybrid Logical Clock + Max Clock Skew Single Timestamp Gen ⇒ No Tracking Needed Geo-Distributed Txns ✓ ✓ ✓ Not Recommended Given Single (Region) Timestamp Generator Tunable Read Latency
- 61. 61© 2019 Allrights reserved. Miscellaneous
- 62. 62© 2019 Allrights reserved. Spanner vs. its Open Source Derivatives Feature Google Spanner YugabyteDB CockroachDB TiDB Cost Expensive Free Free + Proprietary Free SQL API Compatibility Proprietary PostgreSQL PostgreSQL No Stored Procedures MySQL No Foreign Keys Replication Protocol Paxos Raft Raft Raft Clock Skew Tracking TrueTime Atomic Clock Hybrid Logical Clock + Max Clock Skew Hybrid Logical Clock + Max Clock Skew Single Timestamp Gen ⇒ No Tracking Geo-Distributed Txns ✓ ✓ ✓ Not Recommended Given Single (Region) Timestamp Generator Tunable Read Latency ✓ ✓ ❌ ❌
- 63. 63© 2019 Allrights reserved. Read more at blog.yugabyte.com blog.yugabyte.com/distributed-postgresql-on-a-google-spanner-architecture-storage-layer Storage Layer blog.yugabyte.com/distributed-postgresql-on-a-google-spanner-architecture-query-layer Query Layer
- 64. 64© 2019 Allrights reserved. tablet 1’ www.distributedsql.org
- 65. 65© 2019 Allrights reserved. Questions? Join Slack Discussions yugabyte.com/slack Check out on GitHub https://github.com/YugaByte/yugabyte-db docs.yugabyte.com/quick-start
- 66. 66© 2019 Allrights reserved. Relevant Research Publications ● Bigtable: http://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf ● Spanner: ○ http://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi20 12.pdf ● Megastore: ○ http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36971.pdf ● Raft algorithm ○ http://ramcloud.stanford.edu/raft.pdf ○ https://raft.gixthub.io/ ○ http://openlife.cc/system/files/3-modifications-for-Raft-consensus.pdf
- 67. 67© 2019 Allrights reserved. Relevant Research Publications ● David Alves, Todd Lipcon, Vijay Garg. Technical Report: HybridTime - Accessible Global Consistency with High Clock Uncertainty. http://pdsl.ece.utexas.edu/david/hybrid-time-tech-report-01.pdf ● Sandeep Kulkarni, Murat Demirbas, Deepak Madeppa, Bharadwaj Avva, and Marcelo Leone. Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases. http://www.cse.buffalo.edu/tech-reports/2014-04.pdf ● Michael J. Cahill, Uwe Röhm, Alan D. Fekete. Serializable Isolation for Snapshot Databases (2008). https://courses.cs.washington.edu/courses/cse444/08au/544M/READING-LIST/fekete-sigmod2008.p df ● Murat Demirbas, Sandeep Kulkarni. Beyond TrueTime: Using AugmentedTime for Improving Spanner. http://www.cse.buffalo.edu/~demirbas/publications/augmentedTime.pdf ● Dahlia Malkhi Jean-Philippe Martin. Spanner’s Concurrency Control. (2) Ittay Eyal. Fault Tolerant Transaction Architectures https://www.cs.cornell.edu/~ie53/publications/DC-col51-Sep13.pdf
- 68. 68© 2019 Allrights reserved. Relevant Research Publications ● Coordination Avoidance in Database Systems Peter Bailis, Alan Fekete, Michael J. Franklin, Ali Ghodsi, Joseph M. Hellerstein, Ion Stoica http://www.bailis.org/papers/ca-vldb2015.pdf ● RocksDB - RocksDB: A High Performance Embedded Key-Value Store for Flash Storage - Data@Scale , https://www.youtube.com/watch?v=plqVp_OnSzg ● Schema-Agnostic Indexing with Azure DocumentDB (VLDB paper): paper describes the Microsoft Azure’s DocumentDB capabilities, including document representation, query language, document indexing approach, core index support, and early production experiences ● MergeOperator on RocksDB - https://github.com/facebook/rocksdb/wiki/Merge-Operator-Implementation ● Cluster scheduling blog post from cambridge: http://www.cl.cam.ac.uk/research/srg/netos/camsas/blog/2016-03-09-scheduler-architectures.html
- 69. 69© 2019 Allrights reserved. Questions? Join Slack Discussions yugabyte.com/slack Check out on GitHub https://github.com/YugaByte/yugabyte-db docs.yugabyte.com/quick-start
- 70. 70© 2019 Allrights reserved. Jepsen.io Testing Jepsen is an effort to improve the safety of distributed databases, queues, consensus systems, etc. led by Kyle Kingsbury “YugabyteDB now passes tests for snapshot isolation, linearizable counters, sets, registers, and systems of registers, as long as clocks are well-synchronized” Jepsen YugaByteDB Analysis: https://jepsen.io/analyses/yugabyte-db-1.1.9
- 71. 71© 2019 Allrights reserved. Functional Architecture DOCDB Spanner-Inspired Distributed Document Store CLOUD NEUTRAL No Specialized Hardware Needed YSQL PostgreSQL-Compatible Distributed SQL API tablet 1’ tablet 1’