Uploaded byTaras Matyashovsky
JEEConf 2015 - Introduction to real-time big data with Apache Spark
The document provides an introduction to Apache Spark, highlighting its capabilities as a fast and general-purpose platform for large-scale data processing. It discusses key components such as the RDD API, data processing techniques, and integration use cases while emphasizing Spark's performance advantages over Hadoop. The document also covers project developments, resources for learning Spark, and lessons learned from practical applications.
JEEConf 2015 - Introduction to real-time big data with Apache Spark
- 1. Introduction to Real-time BigData with Apache Spark
- 2.
- 3. Spark Fast and general-purpose clustercomputing platform for large-scale data processing
- 4. Why Spark? As ofmid 2014, Spark is the most active Big Data project http://www.slideshare.net/databricks/new-direction-for-spark-in-2015-spark-summit-east Contributors per month to Spark
- 5.
- 6. Time to Sort100TB http://www.slideshare.net/databricks/new-direction-for-spark-in-2015-spark-summit-east
- 7. Why Spark isFaster? Spark processes data in-memory while Hadoop persists back to the disk after a map/reduce action
- 10.
- 11.
- 12. Core Concepts automatically distributedata across cluster and parallelize operations performed on them
- 13.
- 14.
- 16. RDD API Transformations: • filter() •map() • flatMap() • distinct() • union() • intersection() • subtract() • etc. Actions: • collect() • reduce() • count() • countByValue() • first() • take() • top() • etc. http://www.slideshare.net/databricks/strata-sj-everyday-im-shuffling-tips-for-writing-better-spark-programs
- 18.
- 19. Requirements Analytics about Morning@Lohikaevents: • unique participants by companies • most loyal participants • participants by position • etc. https://github.com/tmatyashovsky/spark-samples-jeeconf-kyiv
- 20. Data Format Simple CSVfiles all fields are optional First Name Last Name Company Position Email Present Vladimir Tsukur GlobalLogic Tech/Team Lead flushdia@gmail.com 1 Mikalai Alimenkou XP Injection Tech Lead mikalai.alimenkou@ xpinjection.com 1 Taras Matyashovsky Lohika Software Engineer taras.matyashovsky@ gmail.com 0 https://github.com/tmatyashovsky/spark-samples-jeeconf-kyiv
- 21.
- 22.
- 23.
- 24. Distributed collection ofdata organized into named columns Data Frame
- 26. Data Frame API •selecting columns • joining different data sources • aggregation, e.g. sum, count, average • filtering
- 28. Plan Optimization &Execution http://web.eecs.umich.edu/~prabal/teaching/resources/eecs582/armbrust15sparksql.pdf
- 29.
- 30.
- 32.
- 34.
- 35.
- 36. Use Cases • supplementNeo4j database used to store/query big dimensions • supplement RDBMS for querying of high volumes of data
- 37. Use Cases • representexisting computational graph as flow of Spark-based operations • predictive analytics based on Spark MLib component
- 38. Lessons Learned • Sparksimplicity is deceptive • Each use case is unique • Be really aware: • Databricks blog • Mailing lists & Jira • Pull requests Spark is kind of magic
- 39. Spark is ona Rise
- 40.
- 41. Project Tungsten • thelargest change to Spark’s execution engine since the project’s inception • focuses on substantially improving the efficiency of memory and CPU for Spark applications • sun.misc.Unsafe https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html
- 42.
- 43. References https://www.linkedin.com/pulse/decoding-buzzwords-big-data-predictive-analytics- business-gordon http://www.ibmbigdatahub.com/infographic/four-vs-big-data http://www.thoughtworks.com/insights/blog/hadoop-or-not-hadoop http://blog.cloudera.com/blog/2014/05/apache-spark-resource-management-and-yarn-app- models/ Learning Spark, byHolden Karau, Andy Konwinski, Patrick Wendell and Matei Zaharia (early release ebook from O'Reilly Media) https://spark-prs.appspot.com/#all https://www.gitbook.com/book/databricks/databricks-spark-knowledge-base/details http://insidebigdata.com/2015/03/06/8-reasons-apache-spark-hot/ https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark http://databricks.com/blog/2014/10/10/spark-petabyte-sort.html http://databricks.com/blog/2014/11/05/spark-officially-sets-a-new-record-in-large-scale- sorting.html http://web.eecs.umich.edu/~prabal/teaching/resources/eecs582/armbrust15sparksql.pdf http://www.slideshare.net/databricks/strata-sj-everyday-im-shuffling-tips-for-writing- better-spark-programs http://www.slideshare.net/databricks/new-direction-for-spark-in-2015-spark-summit-east http://www.slideshare.net/databricks/spark-sqlsse2015public https://spark.apache.org/docs/latest/running-on-mesos.html http://spark.apache.org/docs/latest/cluster-overview.html http://www.techrepublic.com/article/can-anything-dim-apache-spark/ http://spark-packages.org/
Editor's Notes
- #9 Real-time, streaming Structures which could not be decomposed to key-value pairs Jobs/algorithms which do not yield to the MapReduce programming model
- #22 Functional Programming API Drawback - limited opportunities for automatic optimization
- #32 Cluster Manager: Standalone, Apache Mesos, Hadoop Yarn Cluster Manager should be chosen and configured properly Monitoring via web UI(s) and metrics Web UI: master web UI worker web UI driver web UI - available only during execution history server - spark.eventLog.enabled = true Metrics based on Coda Hale Metrics library. Can be reported via HTTP, JMX, and CSV files.
- #33 Serialization: default and Kryo Tune Executor Memory Fraction: RDD Storage (60%), Shuffle and Aggregation Buffers (20%), User code (20%) Tune storage level: store in memory and/or on disk store as unserialized/serialized objects replicate each partition on 1 or 2 cluster nodes store in Tachyon Level of Parallelism: spark.task.cpus 1 task per partition using 1 core to execute spark.default.parallelism can be controlled: repartition() and coalescence() functions degree of parallelism as a operations parameter storage system matters Data locality: check data locality via UI configure data locality settings if needed spark.locality.wait timeout execute certain jobs on a driver spark.localExecution.enabled
- #34 API can be experimental or used just for development Spark Java API can be not up-to-date as Scala API is main focus