Download as KEY, PPTX
Uploaded byBradford Stephens
Building a Business on Hadoop, HBase, and Open Source Distributed Computing
The document discusses the building of a business focused on open-source distributed computing, emphasizing the importance of scalability and the transition from traditional RDBMS to more flexible and efficient data handling solutions. It outlines the challenges and sacrifices associated with various data processing methods, including Hadoop and NoSQL technologies like HBase. The goal is to create a platform capable of analyzing any data with linear costs while addressing issues of latency and structure.
Building a Business on Hadoop, HBase, and Open Source Distributed Computing
- 1. Building a Businesson Open Source Distributed Computing company: www.visibletechnologies.com blog: www.roadtofailure.com twitter: @lusciouspear
- 2. Social Media andScaling
- 3. Social Media andScaling •Scalability Matters Now.
- 4. Social Media andScaling •Scalability Matters Now. •SM produces large, complex data
- 5. Social Media andScaling •Scalability Matters Now. •SM produces large, complex data •Anyone can collect the web
- 6. Social Media andScaling •Scalability Matters Now. •SM produces large, complex data •Anyone can collect the web •Make a Twitter in a few days
- 7. Social Media andScaling •Scalability Matters Now. •SM produces large, complex data •Anyone can collect the web •Make a Twitter in a few days •Easy to get TBs of data
- 8. Social Media andScaling •Scalability Matters Now. •SM produces large, complex data •Anyone can collect the web •Make a Twitter in a few days •Easy to get TBs of data •Big Data enabling new fields for companies
- 9.
- 10. What Visible Does •BIand Brand Management on Social Media
- 11. What Visible Does •BIand Brand Management on Social Media •Listen, Monitor, Engage
- 15.
- 16. Old Product: RDBMS •Afew MSSQL servers on boxes
- 17. Old Product: RDBMS •Afew MSSQL servers on boxes •Lots of ETL
- 18. Old Product: RDBMS •Afew MSSQL servers on boxes •Lots of ETL •Several TB, inserts slow, deletes impossible, random fail
- 19.
- 20. Why RDBMS Bad •Nonlinearscale cost
- 21. Why RDBMS Bad •Nonlinearscale cost •Used as a storage abstraction
- 22. Why RDBMS Bad •Nonlinearscale cost •Used as a storage abstraction •Mainly Select, Join, Group, Count
- 23. Why RDBMS Bad •Nonlinearscale cost •Used as a storage abstraction •Mainly Select, Join, Group, Count •Specialized Scale-Out ones ‘meh’
- 24. Why RDBMS Bad •Nonlinearscale cost •Used as a storage abstraction •Mainly Select, Join, Group, Count •Specialized Scale-Out ones ‘meh’ •Impedance Mismatch - Try to be High- Throughput, Low-Latency
- 25. Why RDBMS Bad •Nonlinearscale cost •Used as a storage abstraction •Mainly Select, Join, Group, Count •Specialized Scale-Out ones ‘meh’ •Impedance Mismatch - Try to be High- Throughput, Low-Latency •Swiss-army knife, unstable, transactions, advanced SQL, tuning
- 26.
- 27.
- 28. Why OSS? •Previously allMS •It exists!
- 29. Why OSS? •Previously allMS •It exists! •Scaling + Licensing = No
- 30. Why OSS? •Previously allMS •It exists! •Scaling + Licensing = No •Can’t build a platform without source
- 31. Why OSS? •Previously allMS •It exists! •Scaling + Licensing = No •Can’t build a platform without source •It’s Enterprise Now!
- 32. Goals for NewPlatform
- 33. Goals for NewPlatform •“Golden Timeline”
- 34. Goals for NewPlatform •“Golden Timeline” •Search/Analyze *any* data
- 35. Goals for NewPlatform •“Golden Timeline” •Search/Analyze *any* data •Linear Cost
- 36. Goals for NewPlatform •“Golden Timeline” •Search/Analyze *any* data •Linear Cost •Not Hacked Together
- 37. Goals for NewPlatform •“Golden Timeline” •Search/Analyze *any* data •Linear Cost •Not Hacked Together •“Collect the Social Internet”
- 38.
- 39. HOW TO SCALE •Whatmakes you special?
- 40. HOW TO SCALE •Whatmakes you special? •What are you willing to sacrifice?
- 41. HOW TO SCALE •Whatmakes you special? •What are you willing to sacrifice? •How will you structure the data?
- 42.
- 43. Avoiding Impedance Mismatch •Most problems can be divided into High or Low latency
- 44. Avoiding Impedance Mismatch •Most problems can be divided into High or Low latency •Get a lot of data eventually, or a little now
- 45. Avoiding Impedance Mismatch •Most problems can be divided into High or Low latency •Get a lot of data eventually, or a little now •MapReduce vs. Sharding/Indexing
- 46. Ecosystem Compiled Pig Cascading Hive Processing Katta / Applications Raw Zookeeper MapReduce Processing Structured HBase Storage Unstructured Hadoop DFS Storage
- 47. Simple Workflow Semantic Unstructured Hadoop Collect Analysis Analysis Structured Analysis Hadoop + Store in HBase HBase Store in Indexing Hadoop Lucene+ Load/ Pull Solr+ Replicate Indexes Katta Shards Search
- 48. Unstructured Processing Cluster Semantic Unstructured Structured Internet Collect Store Analysis Analysis HBase HTML XML Records
- 49.
- 50. Hadoop + MR •Special:Crunch web-scale data fast
- 51. Hadoop + MR •Special:Crunch web-scale data fast •Sacrifice: Low-Latency, Transactions, Random Access, Updates
- 52. Hadoop + MR •Special:Crunch web-scale data fast •Sacrifice: Low-Latency, Transactions, Random Access, Updates •Structure: Chunked flat files
- 53. Structured Processing Cluster Enriched Data Structured Analysis Unstructured Store in Cluster HBase Store in Search Indexing Hadoop Cluster HBase Records Sharded Lucene Index Lucene Index
- 54. Document Structure ContentID: 00BAC189 Title:Iron Maiden Rules Body: I think Janick Gers is an amazing guitarist blah blah PostDT: 20090718 ParentID: 0FDEADBEEF Permalink: www.roadtofailure.com/post?=20
- 55.
- 56. HBase •Special: Scalable random/sequential access almost as fast as RDBMS
- 57. HBase •Special: Scalable random/sequential access almost as fast as RDBMS •Sacrifice: Joins, Secondary Indexes, Transactions (kind of)
- 58. HBase •Special: Scalable random/sequential access almost as fast as RDBMS •Sacrifice: Joins, Secondary Indexes, Transactions (kind of) •Structure: BigTable - column oriented
- 59. Search Cluster Lucene Load/ Pull Indexes from Replicate Indexes HDFS Shards Search Lucene Lucene Indexes Indexes
- 60.
- 61.
- 62. Katta + Solr •Special:Sharded search
- 63. Katta + Solr •Special:Sharded search •Sacrifice: Consistency, high-throughput
- 64. Katta + Solr •Special:Sharded search •Sacrifice: Consistency, high-throughput •Structure: Reverse index
- 65.
- 66. BI •Group, Sort, Filter,Count, Sum
- 67. BI •Group, Sort, Filter,Count, Sum •Semi-additive (Avg) rare but not hard
- 68. BI •Group, Sort, Filter,Count, Sum •Semi-additive (Avg) rare but not hard •MapReduce Jobs
- 69. BI •Group, Sort, Filter,Count, Sum •Semi-additive (Avg) rare but not hard •MapReduce Jobs •Faceted Search
- 70.
- 73.
- 74.
- 75.
- 76. Challenges •Scaling Search •Understanding Latency •Whatdo we need ‘now’? Can customers wait for big data?
- 77. Challenges •Scaling Search •Understanding Latency •Whatdo we need ‘now’? Can customers wait for big data? •Monitoring
- 78. Recap: Rules forScaling
- 79. Recap: Rules forScaling •RDBMS is not a Swiss-Army Knife
- 80. Recap: Rules forScaling •RDBMS is not a Swiss-Army Knife •Know your sacrifices
- 81. Recap: Rules forScaling •RDBMS is not a Swiss-Army Knife •Know your sacrifices •Know your specialness
- 82. Recap: Rules forScaling •RDBMS is not a Swiss-Army Knife •Know your sacrifices •Know your specialness •Know your data structure
- 83. Recap: Rules forScaling •RDBMS is not a Swiss-Army Knife •Know your sacrifices •Know your specialness •Know your data structure •Ponder Latency
- 84.
- 85.
- 86. What Next? •HBase Analytics? •“Whatwould make a bank trust it”
- 87. What Next? •HBase Analytics? •“Whatwould make a bank trust it” •Teach people to think about data
- 88.
- 89. The End company: www.visibletechnologies.com blog: www.roadtofailure.com twitter: @lusciouspear bradfordstephens@gmail.com