Download as PDF, PPTX
Uploaded byJuan Pedro Moreno
Introduction to Apache Spark
The document introduces Apache Spark and its core concepts, including resilient distributed datasets (RDDs), transformations, and actions. It outlines Spark's architecture, data processing capabilities, and the functionality of Spark SQL and Spark Streaming. Additionally, it provides resources for further learning and practical workshops on the topic.
Introduction to Apache Spark
- 1.
- 2. Who are we? JuanPedro Moreno Scala Software Engineer at 47Degrees @juanpedromoreno Fran Pérez Scala Software Engineer at 47Degrees @FPerezP Workshop repo: https://github.com/47deg/spark-workshop
- 3. Roadmap • Intro BigData and Spark • Spark Architecture • Resilient Distributed Datasets (RDDs) • Transformations and Actions on Data using RDDs • Overview Spark SQL and DataFrames • Overview Spark Streaming • Spark Architecture and Cluster Deployment
- 4. Apache Spark Overview https://github.com/apache/spark •Fast and general engine for large-scale data processing • Speed • Ease of Use • Generality • Runs Everywhere http://spark.apache.org
- 5. Spark Architecture Scala JavaPython R Spark SQL Spark Streaming MLlib GraphX DataFrames API RDD API Spark Core Hadoop HDFS Cassandra JSON MySQL … DATA SOURCES
- 6. Spark Core Concepts DriverProgram Worker Node Worker Node Cluster Manager SparkContext Executor Executor Cache Cache Task Task Task Task Hadoop YARN Standalone Apache Mesos
- 7. Spark Core Concepts •Executor: A process launched for an application on a worker node. Each application has its own executors. • Jobs: A parallel computation consisting of one or multiple stages that gets spawned in response to a Spark action. • Stages: Smaller set of tasks that each job is divided into. • Tasks: A unit of work that will be sent to one executor.
- 8. Resilient Distributed Datasets •Immutable. • Partitioned collection. • Operates in parallel. • Customizable.
- 9. RDDs - Partitions •A Partition is one of the different chunks that a RDD is splitted on and that is sent to a node • The more partitions we have, the more parallelism we get • Each partition is candidate to be spread out to different worker nodes Error, ts, msg1 Warn, ts, msg2 Error, ts, msg1 Info, ts, msg8 Warn, ts, msg2 Info, ts, msg8 Error, ts, msg3 Info, ts, msg5 Info, ts, msg5 Error, ts, msg4 Warn, ts, msg9 Error, ts, msg1 RDD with 4 partitions
- 10. RDDs - Partitions RDDwith 8 partitions P1 P2 P3 P4 P5 P6 P7 P8 Worker Node Executor Worker Node Executor Worker Node Executor Worker Node Executor P1 P5 P2 P6 P3 P7 P4 P8
- 11. RDDs - Operations Transformations •Lazy operations. They don’t return a value, but a pointer to a new RDD. Actions • Non-lazy operations. They apply an operation to a RDD and return a value or write data to an external storage system.
- 12. RDDs - Transformations Aset of some of the most popular Spark transformations: • map • flatMap • filter • groupByKey • reduceByKey
- 13. RDDs - Actions Aset of some of the most popular Spark actions: • reduce • collect • foreach • saveAsTextFile
- 14. Transformations and Actions WithVisual Mnemonics, better. Thanks to Jeffrey Thompson • http://data-frack.blogspot.com.es/2015/01/visual-mnemonics-for- pyspark-api.html • https://github.com/jkthompson/pyspark-pictures • http://nbviewer.ipython.org/github/jkthompson/pyspark-pictures/ blob/master/pyspark-pictures.ipynb
- 15. Practice - Part1 && Part 2
- 16. Overview Spark SQLand DataFrames • Works with structured and semistructured data • DataFrame simplifies working with structured data • Read/Write from structure data like JSON, Hive tables, Parquet, etc. • SQL inside your Spark App • Best Performance and more powerful operations API
- 17.
- 18. Overview Spark Streaming •Streaming Applications • DStreams or Discretized Streams • Continuous Series of RDDs, grouped by batches Kafka Spark StreamingR e c e i v e r s Flume HDFS batches of input data Spark Core HDFS/S3 Database Kinesis Dashboard Twitter
- 19. Resources • Official docs- http://spark.apache.org/docs/latest • Learning Spark - http://shop.oreilly.com/product/0636920028512.do • Databricks Spark Knowledge Base - https://goo.gl/wMy7Se • Community packages for Spark - http://spark-packages.org/ • Apache Spark Youtube channel - https://www.youtube.com/user/ TheApacheSpark • API through pictures - https://goo.gl/JMDeqJ • 47 Degrees Blog - http://www.47deg.com/blog/tags/spark
- 20.