Download as PDF, PPTX
Uploaded byBTI360
Intro to Apache Spark
The document introduces Apache Spark as a powerful solution for handling big data, highlighting its key features such as resilient distributed datasets that enable fault-tolerant operations on large clusters. It contrasts Spark with earlier technologies like MapReduce and Hadoop, emphasizing Spark's advantages in speed and ease of use for various applications, including ETL, machine learning, and analytics. The document also outlines the creation and management of RDDs (Resilient Distributed Datasets) and shared variables in Spark.
Intro to Apache Spark
- 1. All product imagesowned by respective companies/institutions Intro to Apache
- 2. Takeaways To understand: • Why we have big data today • What big data problems Spark solves • How Spark approaches big data differently But most of all… to feel comfortable trying Spark out!
- 3.
- 5. Why does big data exist?
- 6. All product imagesowned by respective companies/institutions
- 7. 7.2 B 6.8B 1.44 B 300 M 236 M 3.5 B / day All product images owned by respective companies/institutions
- 8. When data issmall it’s cute and cuddly, easy to contain…
- 9. When data getsbig, we need tools to help us.
- 10. What tools can help?
- 11. 2002 - MapReduce@ Google 2004 – MapReduce Paper 2006 – Hadoop @ Yahoo 2011 – Hadoop Released
- 12.
- 13. But MapReduce falls short…
- 14. Hadoop’s Limitations Lacksone thing to succeed for: • Iterative Queries • Interactive Queries Fast data sharing
- 15. Image courtesy of:http://workinganalytics.com/
- 16. a better way. We need…
- 17. We need… fault tolerance and speed.
- 18. We need… a better data abstraction.
- 19. The Solution.. • Resilient Distributed Datasets – A distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner.
- 20. 2002 - MapReduce@ Google 2004 – MapReduce Paper 2006 – Hadoop @ Yahoo 2013 – Spark @ Apache 2014 – Spark 1.0 Released 2011 – Hadoop Released 2011 – Hadoop Released2009 – Spark at UC Berkeley
- 21. 2002 - MapReduce@ Google 2004 – MapReduce Paper 2006 – Hadoop @ Yahoo 2013 – Spark @ Apache 2014 – Spark 1.0 Released 2009 – Spark at UC Berkeley 2011 – Hadoop Released
- 22.
- 23.
- 25. Why Spark? Fast ImageCredit: http://pixabay.com/en/tunnel-light-speed-fast-auto-101976/
- 26. Why Spark? Fast GeneralPurpose Image Credit: http://www.freestockphotos.biz/stockphoto/9182
- 27. Why Spark? Fast GeneralPurpose Easy Image Credit: http://upload.wikimedia.org/wikipedia/commons/9/92/Easy_button.JPG
- 28. Why Spark? Fast GeneralPurpose Easy Streaming Image Credit: http://pixabay.com/en/faucet-water-bad-sanitaryblock-686958/
- 29. Why Spark? Fast GeneralPurpose Easy Streaming Adoption All product images owned by respective companies/institutions
- 30. All product imagesowned by respective companies/institutions Use Cases
- 31. Spark Use Cases ETL
- 32. Spark Use Cases ETL Machine Learning
- 33. Spark Use Cases ETL Machine Learning Analytics Table Credit: http://www.wsj.com/articles/SB10001424052970203914304576630742911364206
- 34. Spark Use Cases ETL Machine Learning Analytics Modeling Table Credit: http://www.wsj.com/articles/SB10001424052970203914304576630742911364206
- 35. Spark Use Cases ETL Machine Learning Analytics Modeling Data Mining Table Credit: http://www.wsj.com/articles/SB10001424052970203914304576630742911364206
- 36. Spark Modules ImageCredit: http://www.numaq.com
- 37. Basics All productimages owned by respective companies/institutions
- 38.
- 39. Creating RDDs • From practically any data source – HDFS – Local file system – S3 – NoSQL (Cassandra, Hbase, …) – JDBC • From any collection • Transform an existing RDD
- 40. Text File We start withsome data. Put it in a form Spark understands… File RDD Read File
- 41. Text File File RDD Read File RDDs: • Computationblueprint • Lazy: Hold instructions – not data
- 42. Text File File RDD Word RDD Word Count RDD Read File SplitWords Count Words Transformations chain operations together Nothing actually computed yet…
- 43. Text File File RDD Word RDD Word Count RDD All Word Counts Read File SplitWords Count Words Store Result Actions compute results. Why is laziness good?
- 44. Text File File RDD Word RDD Word Count RDD All Word Counts Read File SplitWords Count Words Store Result Top 10 Words Only compute what we need Allows you to: - Focus more on algorithm - Worry less about performance
- 45. Text File File RDD Word RDD Word Count RDD All Word Counts Read File SplitWords Count Words Store Result Top 10 Words “A” Word RDD Words starting with “A” By default, RDDs recomputed each use
- 46. Word RDD Text File File RDD Word Count RDD All Word Counts Read File SplitWords Count Words Store Result Top 10 Words “A” Word RDD Words starting with “A” For better performance… Persist reused RDDs Word RDD
- 47. Word RDD Text File File RDD Word Count RDD All Word Counts Read File SplitWords Count Words Store Result Top 10 Words “A” Word RDD Words starting with “A” RDDs are fault tolerant.
- 48. Text File File RDD Word RDD Word Count RDD All Word Counts Read File SplitWords Count Words Store Result Top 10 Words “A” Word RDD Words starting with “A” RDDs are fault tolerant. Lineage allows recreation.
- 49. Once more, with code
- 50. Word Count Example val input = sc.textFile(”hdfs://...") // HadoopRDD //Transformation val words = input.flatMap(line => line.split(" ")) //FlatMappedRDD //Transformation val result = words.map(word => (word, 1)).reduceByKey((acc, curr) => acc + curr) //Action val collectedResult = result.collect()
- 51. Image courtesy ofhttp://blog.jetoile.fr
- 52. Cluster Basics Allproduct images owned by respective companies/institutions
- 53. Image courtesy ofhttps://spark.apache.org
- 54. Image courtesy ofhttps://spark.apache.org Creates RDDs Executes code on cluster Connects our program to Spark “Main”
- 55. Image courtesy ofhttps://spark.apache.org Acquires cluster resources YARN, Mesos, Standalone…
- 56. Image courtesy ofhttps://spark.apache.org Spawns executors Performs tasks
- 57. Image courtesy ofhttps://spark.apache.org Managed by SparkManaged by You
- 58. All product imagesowned by respective companies/institutions In Action
- 59.
- 60. More Information on Spark • https://spark.apache.org/docs/latest/index.html • http://www.cs.berkeley.edu/~matei/papers/2013/sosp_spark_streaming.pdf • http://www.cs.berkeley.edu/~matei/papers/2010/hotcloud_spark.pdf • http://www.cs.berkeley.edu/~matei/papers/2012/hotcloud_spark_streaming.pdf • http://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf • https://www.usenix.org/conference/nsdi12/technical-sessions/presentation/zaharia • http://www.meetup.com/Washington-DC-Area-Spark-Interactive/ • https://spark-summit.org/
- 61. Shared Variables ● Broadcast variables o Allows user to keep a read-only variable cached on each machine vs shipping it with tasks. o e.g. lookup table ● Accumulators o workers can “add” to using associative operations o only driver can read o used for § counters § sums