Uploaded byEdureka!
MapReduce Tutorial | What is MapReduce | Hadoop MapReduce Tutorial | Edureka
The document discusses Hadoop MapReduce and YARN. It provides an overview of MapReduce concepts like parallel processing and data locality. An example of vote counting is used to illustrate the MapReduce approach. Key components of MapReduce like Map, Reduce, and YARN are explained. The YARN application workflow is described through 8 steps from client submission to application completion. Hands-on MapReduce programming and learning resources are also mentioned.
MapReduce Tutorial | What is MapReduce | Hadoop MapReduce Tutorial | Edureka
- 1.
- 2. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Agenda for today’s Session 1. What is Hadoop MapReduce? 2. MapReduce In Nutshell 3. Advantages of MapReduce 4. Hadoop MapReduce Approach with an Example 5. Hadoop MapReduce/YARN Components 6. YARN With MapReduce 7. Yarn Application Workflow 8. MapReduce Program with Hands On
- 3. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Hadoop Components 2 main Hadoop Components Storage Processing
- 4. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING MapReduce: Data Processing Using Programming Big Data Result Hadoop MapReduce is the processing component of Apache Hadoop It processes data parallelly in distributed environment
- 5. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING MapReduce In Nutshell MapReduce FeaturesLarge Scale Distributed Model Used in Function Design Pattern Parallel Programming A Program Model Classification Analytics Recommendation Index and Search Map Reduce Classification Eg: Top N records Analytics Eg: Join, Selection Recommendation Eg: Sort Summarization Eg: Inverted Index Implemented Google Apache Hadoop HDFS Pig Hive HBase For
- 6. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING 2 Biggest Advantages of MapReduce
- 7. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Data is processed in parallel Processing becomes fast Advantage 1: Parallel Processing Slave A Slave B Slave C Slave D Slave E Master Data
- 8. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Moving Data to processing is very costly In MapReduce, we move processing to Data Advantage 2: Data Locality - Processing to Storage Slave A Slave B Slave C Slave D Slave E Data Master
- 9. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Traditional vs MapReduce Way
- 10. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Election Votes Counting Election Votes Casting Votes is stored at different Booths Result Centre has the details of all the Booths Data Booth A Booth B Booth C Booth D Booth E Result Centre
- 11. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Election Votes Counting – Traditional Way Counting – Traditional Approach Votes are moved to Result Centre for counting Moving all the votes to Centre is costly Result Centre is over-burdened Counting takes time Data Booth A Booth B Booth C Booth D Booth E Result Centre
- 12. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Result Centre Data Hadoop MapReduce To the Rescue! Hadoop MapReduce Doesn’t Follow This Approach Booth A Booth B Booth C Booth D Booth E
- 13. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Election Votes Counting – MapReduce Way Booth A Booth B Booth C Booth D Booth E Result Centre Counting – MapReduce Approach Votes are counted at individual booths Booth-wise results are sent back to the result centre Final Result is declared easily and quickly using this way Votes
- 14. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING MapReduce In Detail
- 15. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING MapReduce Way
- 16. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Anatomy of a MapReduce Program MapReduce Map: Reduce: (K1, V1) List (K2, V2) (K2, list (V2)) List (K3, V3) Key Value
- 17. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Let us take an example to understand MapReduce Way
- 18. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING MapReduce Way – Word Count Process
- 19. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Executing a MapReduce Program
- 20. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING MapReduce Using Yarn
- 21. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING YARN – Moving beyond MapReduce BATCH (MapReduce) INTERACTIVE (Text) ONLINE (HBase) STREAMING (Storm, S4, …) GRAPH (Giraph) IN-MEMORY (Spark) HPC MPI (OpenMPI) OTHER (Search) (Weave..)
- 22. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Hadoop 2.x Daemons
- 23. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Hadoop 2.x MapReduce Yarn Components ApplicationMaster » One per application » Short life » Coordinates and Manages MapReduce Jobs » Negotiates with Resource Manager to schedule tasks » The tasks are started by NodeManager(s) Job History Server » Maintains information about submitted MapReduce jobs after their ApplicationMaster terminates Client » Submits a MapReduce Job Resource Manager » Cluster Level resource manager » Long Life, High Quality Hardware Node Manager » One per Data Node » Monitors resources on Data Node Container » Created by NM when requested » Allocates certain amount of resources (memory, CPU etc.) on a slave node
- 24. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING YARN Application Workflow in MapReduce
- 25. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING YARN Workflow Node Manager Node Manager Node Manager Node Manager Node Manager Node Manager Node Manager Node Manager Node Manager Node Manager Node Manager Node Manager Container 1.2 Container 1.1 Container 2.1 Container 2.2 Container 2.3 App Master 2 App Master 1 Scheduler Applications Manager (AsM) Resource Manager
- 26. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Application Workflow Execution Sequence : 1. Client submits an application Client RM NM AM 1
- 27. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Application Workflow Execution Sequence : 1. Client submits an application 2. RM allocates a container to start AM Client RM NM AM 1 2
- 28. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Application Workflow Execution Sequence : 1. Client submits an application 2. RM allocates a container to start AM 3. AM registers with RM Client RM NM AM 1 2 3
- 29. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Application Workflow Execution Sequence : 1. Client submits an application 2. RM allocates a container to start AM 3. AM registers with RM 4. AM asks containers from RM Client RM NM AM 1 2 3 4
- 30. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Application Workflow Execution Sequence : 1. Client submits an application 2. RM allocates a container to start AM 3. AM registers with RM 4. AM asks containers from RM 5. AM notifies NM to launch containers Client RM NM AM 1 2 3 4 5
- 31. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Application Workflow Execution Sequence : 1. Client submits an application 2. RM allocates a container to start AM 3. AM registers with RM 4. AM asks containers from RM 5. AM notifies NM to launch containers 6. Application code is executed in container Client RM NM AM 1 2 3 4 5 6
- 32. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Application Workflow Execution Sequence : 1. Client submits an application 2. RM allocates a container to start AM 3. AM registers with RM 4. AM asks containers from RM 5. AM notifies NM to launch containers 6. Application code is executed in container 7. Client contacts RM/AM to monitor application’s status Client RM NM AM 1 2 3 4 5 7 6
- 33. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Application Workflow Execution Sequence : 1. Client submits an application 2. RM allocates a container to start AM 3. AM registers with RM 4. AM asks containers from RM 5. AM notifies NM to launch containers 6. Application code is executed in container 7. Client contacts RM/AM to monitor application’s status 8. AM unregisters with RM Client RM NM AM 1 2 3 4 5 7 8 6
- 34. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Learning Resources Hadoop Tutorial: www.edureka.co/blog/hadoop-tutorial MapReduce Tutorial: www.edureka.co/blog/mapreduce-tutorial MapReduce Interview Questions: www.edureka.co/blog/interview-questions/hadoop-interview-questions-mapreduce
- 35. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATIONTRAINING Thank You … Questions/Queries/Feedback