Uploaded byEdureka!
Apache Spark Training | Spark Tutorial For Beginners | Apache Spark Certification | Edureka
The document outlines Edureka's Apache Spark certification training, focusing on real-time analytics use cases across various sectors such as government, healthcare, telecommunications, banking, and stock markets. It elaborates on building a movie recommendation system using Apache Spark, detailing aspects like data input, collaborative filtering, machine learning with Spark MLlib, and result storage. Additionally, it highlights the growing job trends associated with Apache Spark, emphasizing its effectiveness and efficiency for data processing.
Apache Spark Training | Spark Tutorial For Beginners | Apache Spark Certification | Edureka
- 1.
- 2. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING What to expect? Use Cases Of Real Time Analytics Movie Recommendation System Using Spark What Is Spark? Getting Movie Dataset Spark Streaming Collaborative Filtering Spark MLlib Fetching Results Storing Results
- 3. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Use Cases of Real Time Analytics
- 4. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Use Cases of Real Time Analytics Government Government agencies perform Real Time Analysis mostly in the field of national security. Countries need to continuously keep a track of all the military and police agencies for updates regarding threats to security.
- 5. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Use Cases of Real Time Analytics Healthcare Healthcare domain uses Real Time analysis to continuously check the medical status of critical patients. Hospitals on the look out for blood and organ transplants need to stay in a real-time contact with each other during emergencies. Getting medical attention on time is a matter of life and death for patients.
- 6. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Use Cases of Real Time Analytics Telecommunications Companies revolving around services in the form of calls, video chats and streaming use real time analysis to reduce customer churn and stay ahead of competition. They also extract measurements of jitter and delay in mobile networks to improve customer experiences.
- 7. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Use Cases of Real Time Analytics Banking Banking transacts with almost all of the world’s money. It becomes very important to ensure fault tolerant transactions across the whole system. Fraud detection is made possible through real time analytics in banking.
- 8. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Use Cases of Real Time Analytics Stock Market Stock brokers use real time analytics to predict movement of stock portfolios. Companies re-think their business model after using real time analytics to analyze the market demand for their brand.
- 9. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Movie Recommendation System
- 10. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Movie Recommendation System
- 11. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Movie Recommendation System Problem Statement To build a Movie Recommendation System which recommends movies based on a user’s preferences using Apache Spark. Process huge amount of data Easy to use Fast processing Our Requirements: Input from multiple sources Apache Spark is the perfect tool to implement our Movie Recommendation System.
- 12. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Use Case – Flow Diagram Huge amount of Movie Rating data 1 Data from Streaming / HDFS 2 Getting Input using Spark Streaming 3 4 Machine Learning Using MLlib Train the data Evaluate ALS Generate Recommendations Fetching Results using Spark SQL 5 Storing Results in RDBMS System for Websites 6
- 13. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING What is Spark?
- 14. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING What is Spark? Apache Spark is an open-source cluster-computing framework for real time processing developed by the Apache Software Foundation. Spark provides an interface for programming entire clusters with implicit data parallelism and fault-tolerance. It was built on top of Hadoop MapReduce and it extends the MapReduce model to efficiently use more types of computations. Reduction in time Parallel Serial Figure: Data Parallelism In Spark Figure: Real Time Processing In Spark Figure: Support for multiple source formats Figure: Lazy Evaluation
- 15. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Spark Features Deployment Powerful Caching Polyglot Features 100x faster than for large scale data processing Simple programming layer provides powerful caching and disk persistence capabilities Can be deployed through Mesos, Hadoop via Yarn, or Spark’s own cluster manger Can be programmed in Scala, Java, Python and R Speed vs
- 16. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Movie Dataset
- 17. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Movie Dataset User Ratings from BookMyShow
- 18. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Movie Dataset Movie Ratings In Our Dataset
- 19. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Getting Dataset
- 20. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Getting Dataset For our Movie Recommendation System, we can get user ratings from many popular websites like IMDB, Rotten Tomatoes and Times Movie Ratings. This dataset is available in many formats such as CSV files, text files and databases. We can either stream the data live from the websites or download and store them in our local file system or HDFS. Figure: Various File Formats
- 21. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Spark Streaming
- 22. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Spark Streaming
- 23. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Spark Streaming Spark Streaming is used for processing real-time streaming data Spark Streaming enables high-throughput and fault-tolerant stream processing of live data streams
- 24. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Collaborative Filtering
- 25. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Collaborative Filtering We will use Collaborative Filtering (CF) to predict the ratings for users for particular movies based on their ratings for other movies. We then collaborate this with other users’ rating for that particular movie. Movie Alice Bob Carol Dave Shutter Island 4 3 5 1 Fight Club 5 4 4 2 Dark Knight 5 3 4 21 4 3 5 Home Alone 4 4 5 5 Figure: Predicting the rating of Dave for Dark Knight and Carol for 21 using Collaborative Filtering ? ?
- 26. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Spark MLlib
- 27. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Spark MLlib Spark MLlib is used to perform machine learning in Apache Spark. Machine learning in Spark is implemented using Spark’s MLlib. MLlib stands for Machine Learning Library. Train Data using Alternating Least Squares (ALS) Generate Recommendations using Collaborative Filtering Machine Learning Using Spark MLlib Figure: Machine Learning Flow Diagram Machine Learning Tools ML Algorithms Featurization Pipelines Persistence Utilities Figure: Machine Learning Tools
- 28. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Fetching Results
- 29. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Spark SQL for Fetching Results Machine Learning Output Spark SQL Results To get the results from our Machine Learning, we need to use Spark SQL’s DataFrame, Dataset and SQL Service. The results in Machine Learning needs to be stored in a RDBMS so that our web application can display the recommendations to a particular use.
- 30. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Ratings for Movies Ratings of Movies for User 77 Figure: User 77’s ratings for different movies
- 31. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Recommended Movies Total Number of Recommendations for User 77 Top Movie Recommendations for User 77 Figure: Movies recommended for User 77
- 32. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Storing Results
- 33. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Storing Results The results for our Movie Recommendation System can be stored either locally or into external storage systems. We can store the Recommended Movies along with the Ratings in a text file or a CSV file. We should prefer storing the results into an RDBMS system so that we can access it directly from our web application and display recommendations and top movies.
- 34. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Spark Job Trends
- 35. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Spark Job Trends The following is the Job Trend of Apache Spark across the world. Spark has almost thrice the average number of jobs in comparison to its competitors and is the market leader from 2014. Source: www.indeed.com
- 36.
- 37. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Summary Real Time Analytics Spark MLlibSpark Streaming Movie Recommendation System Spark Job Trends What is Spark?
- 38. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Conclusion
- 39. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Conclusion Congrats! We have hence demonstrated the power of Apache Spark in Real-Time Analysis. The hands-on examples will give you the required confidence to work on any future projects you encounter in Apache Spark.
- 40. www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATIONTRAINING Thank You … Questions/Queries/Feedback