Download as PDF, PPTX
![ASASPARKAPPLICATION By adding spark-core and other Spark modules as project dependencies and using Spark API inside the application code def main(args: Array[String]) { val conf = new SparkConf() .setAppName("Sample Application") .setMaster("local") val sc = new SparkContext(conf) val logData = sc.textFile("/tmp/spark/README.md") val lines = textFile.filter(line => line contains "Spark") lines.collect() sc.stop() }](https://image.slidesharecdn.com/prez-spark-160402193358/75/Introduction-to-Apache-Spark-34-2048.jpg)
![BYSUBMITTING SPARKAPPLICATION TO SPARK-SUBMIT ./bin/sparksubmit class <mainclass> master <masterurl> deploymode <deploymode> conf <key>=<value> ... # other options <applicationjar> [applicationarguments]](https://image.slidesharecdn.com/prez-spark-160402193358/75/Introduction-to-Apache-Spark-35-2048.jpg)
Uploaded bySamy Dindane
Introduction to Apache Spark
Apache Spark is a fast distributed data processing engine that runs in memory. It can be used with Java, Scala, Python and R. Spark uses resilient distributed datasets (RDDs) as its main data structure. RDDs are immutable and partitioned collections of elements that allow transformations like map and filter. Spark is 10-100x faster than Hadoop for iterative algorithms and can be used for tasks like ETL, machine learning, and streaming.
Introduction to Apache Spark
- 1.
- 2. OUTLINE History of "BigData" engines Apache Spark: What is it and what's special about it? Apache Spark: What is used for? Apache Spark: API Tools and software usually used with Apache Spark Demo
- 3.
- 4.
- 5. HISTORYOF"BIG DATA"ENGINES 2011 -Hadoop MapReduce: Batch, in-disk processing 2011 - Apache Storm: Realtime 2014 - Apache Tez 2014 - Apache Spark: Batch and near-realtime, in- memory processing 2015 - Apache Flink: Realtime, in-memory processing
- 6.
- 7. WHY SPARK? Most machinelearning algorithms are iterative; each iteration can improve the results With disk-based approach, each iteration's output is written to disk making the processing slow
- 8. HADOOPMAPREDUCE EXECUTION FLOW SPARKEXECUTION FLOW
- 9. Spark is adistributed data processing engine Started in 2009 Open source & written in Scala Compatible with Hadoop's data
- 10. It runs onmemory and on disk Run 10 to 100 times faster than Hadoop MapReduce Can be written in Java, Scala, Python & R Supports batch and near-realtime workflows (micro- batches)
- 11. Spark has fourmodules:
- 12.
- 13. CAPTURE ANDEXTRACTDATA Data cancome from several sources: Databases Flat files Web and mobile applications' logs Data feeds from social media IoT devices
- 15. TRANSFORMDATA Data in ananalytics pipeline needs transformation Check and correct quality issues Handle missing values Cast fields into specific data types Compute derived fields Split or merge records for more granularity Join with other datasets Restructure data
- 17. STORE DATA Data canthen be stored in several ways As self describing files (Parquet, JSON, XML) SQL databases Search databases (Elasticsearch, Solr) Key-value stores (HBase, Cassandra)
- 19. QUERY,ANALYZE,VISUALIZE With Spark Shell,notebooks, Kibana, etc.
- 21.
- 22.
- 23. RESILENTDISTRIBUTEDDATASETS RDD's are thefundamental data unit in Spark Resilient: If data in memory is lost, it can be recreated Distributed: Stored in memory across the cluster Dataset: The initial data can come from a file or created programmatically
- 24. RDD'S Immutable and partionnedcollection of elements Basic operations: map, filter, reduce, persist Several implementations: PairRDD, DoubleRDD, SequenceFileRDD
- 25. HISTORY 2011 (Spark release)- RDD API 2013 - introduction of the DataFrame API: Add the concept of schema and allow Spark to manage it for more efficient serialization and deserialization 2015 - introduction of the DataSet API
- 26.
- 27. TRANSFORMATIONS Create a newdataset from an RDD, like filter, map, reduce
- 28. ACTIONS: Return a valueto the driver program after running a computation on the dataset
- 29.
- 30.
- 31. HOW TO RUNSPARKPROGRAMS? InsideSpark Shell Using a notebook As a Spark application By submitting Spark application to spark-submit
- 32.
- 33. USING ANOTEBOOK There aremany Spark notebooks, we are going to use http://spark-notebook.io/ sparknotebook open http://localhost:9000/
- 34. ASASPARKAPPLICATION By adding spark-coreand other Spark modules as project dependencies and using Spark API inside the application code def main(args: Array[String]) { val conf = new SparkConf() .setAppName("Sample Application") .setMaster("local") val sc = new SparkContext(conf) val logData = sc.textFile("/tmp/spark/README.md") val lines = textFile.filter(line => line contains "Spark") lines.collect() sc.stop() }
- 35.
- 36. TERMINOLOGY SparkContext: A connectionto a Spark context Worker node: Node that runs the program in a cluster Task: A unit of work Job: Consists of multiple tasks Executor: Process in a worker node, that runs the tasks
- 37.
- 38.
- 39. Simple: Uses manyservers as one big computer Reliable: Detects failures, has redundant storage Fault-tolerant: Auto-retry, self-healing Scalable: Scales (almost) lineary with disks and CPU
- 40.
- 42. APACHE ZOOKEEPER ZOOKEEPERISADISTRIBUTED,OPEN-SOURCE COORDINATION SERVICEFORDISTRIBUTEDAPPLICATIONS
- 43. Coordination: Needed whenmultiple nodes need to work together Examples: Group membership Locking Leaders election Synchronization Publisher/subscriber
- 44. APACHE MESOS Mesos isbuilt using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API's for resource management and scheduling across entire datacenter and cloud environments.
- 45. A cluster managerthat: Runs distributed applications Abstracts CPU, memory, storage, and other resources Handles resource allocation Handles applications' isolation Has a Web UI for viewing the cluster's state
- 46. NOTEBOOKS Spark Notebook: Allowsperforming reproducible analysis with Scala, Apache Spark and more Apache Zeppelin: A web-based notebook that enables interactive data analytics
- 48. THE END APACHE SPARK Isa fast distributed data processing engine Runs on memory Can be used with Java, Scala, Python & R Its main data structure is a Resilient Distributed Dataset
- 49.