Uploaded byEdureka!
Apache Sqoop Tutorial | Sqoop: Import & Export Data From MySQL To HDFS | Hadoop Training | Edureka
The document explains Sqoop, an import-export tool used for transferring bulk data between RDBMS and HDFS. It highlights the problems with RDBMS, the need for Sqoop, its features such as full and incremental load, security integration, and its command-line interface. Additionally, it provides commands for importing and exporting data from MySQL to HDFS and vice versa.
In this document
Powered by AI
Presentation covers: Problems with RDBMS, Need for Sqoop, Introduction, Features, Architecture, Commands & Demo.
Highlights issues with Relational Databases: tedious data importing, difficulty with large datasets, unstructured data storage, and time-consuming tasks.
Explains the need for Sqoop due to data analysis requirements, command line interface benefits, and high performance.
Defines Sqoop as a tool for transferring bulk data between HDFS and RDBMS, supporting both import and export operations.
Covers Sqoop features including full and incremental loads, parallel import/export, compression, Kerberos security, and direct HIVE loading.
Presents the architectural framework of Sqoop, illustrating its components and interactions.
Explains how the Sqoop import and export operations work, focusing on data movement between systems.
Introduces the commands available in Sqoop for data import and export, as well as managing databases.
Provides a command syntax for importing data from MySQL to Hadoop, including parameters for connection and table specifications.
Details the command syntax for exporting data from HDFS back to MySQL, outlining the necessary connection parameters.
Describes the `list databases` command in Sqoop to view databases in RDBMS, demonstrating its usage with a command example.
Explains how to list tables available in a specific RDBMS using a Sqoop command, including an example output.
Details the `codegen` command in Sqoop that automatically generates Data Access Object (DAO) classes and source code.
Apache Sqoop Tutorial | Sqoop: Import & Export Data From MySQL To HDFS | Hadoop Training | Edureka
- 2. Copyright © 2017,edureka and/or its affiliates. All rights reserved. Topics to be covered… Problems with RDBMS Need for Sqoop Introduction to Sqoop Features of Sqoop Sqoop Architecture Sqoop Commands & Demo using MySQL 1 2 3 4 5 6
- 3. Copyright © 2017,edureka and/or its affiliates. All rights reserved. Problems with Relational Database Data Importing was Tedious task Difficult to handle large datasets Cant store Unstructured Data Time Consuming Task RDBMS
- 4. Copyright © 2017,edureka and/or its affiliates. All rights reserved. Need for Sqoop Data Analysis Command Line Interface High Performance
- 5. Copyright © 2017,edureka and/or its affiliates. All rights reserved. What is Sqoop? RDBMS HDFS Import Tool used to transfer bulk data between HDFS & Relational Database Servers
- 6. Copyright © 2017,edureka and/or its affiliates. All rights reserved. What is Sqoop? RDBMS HDFS Export Tool used to transfer bulk data between HDFS & Relational Database Servers
- 7. Copyright © 2017,edureka and/or its affiliates. All rights reserved. Features of Sqoop Full Load Incremental Load Parallel Import/Export Compression Kerberos Security Integration Data loading directly to HIVE
- 8. Copyright © 2017,edureka and/or its affiliates. All rights reserved. Sqoop Architecture
- 9. Copyright © 2017,edureka and/or its affiliates. All rights reserved. How Sqoop Import & Export Works?
- 10.
- 11. IMPORT EXPORT LIST DATABASE LIST TABLES CODEGEN SqoopImport Command Import Import each table of the RDBMS in Hadoop $ sqoop import --connect jdbc:mysql://<ip address>/<database name> --table <mysql_table name> --username <username_for_mysql_user> --password <Password> --target-dir <target directory where data needs to be imported>
- 12. IMPORT EXPORT LIST DATABASE LIST TABLES CODEGEN SqoopExport Command Exports data from HDFS to RDBMS Export $ sqoop export --connect jdbc:mysql://<ip address>/<database name> --table <mysql_table name> --username <username_for_mysql_user> --password <Password> --export-dir <directory name where data needs to be exported>
- 13. IMPORT EXPORT LIST DATABASE LIST TABLES CODEGEN SqoopList Database It lists the databases present in the relational database Sqoop $ sqoop list databases --connect jdbc:mysql://<ip address>/<database name> --table <mysql_table name> --username <username_for_mysql_user> --password <Password>
- 14. IMPORT EXPORT LIST DATABASE LIST TABLES CODEGEN SqoopList Tables ID Name Age Address 1 Jino 25 Kottayam 2 Neha 22 Dharwad 3 Chaitra 23 Bangalore It lists the available tables in the database $ sqoop list tables --connect jdbc:mysql://<ip address>/<database name> --table <mysql_table name> --username <username_for_mysql_user> --password <Password>
- 15. IMPORT EXPORT LIST DATABASE LIST TABLES CODEGEN SqoopCodegen Generates DAO Class automatically Generates Java Class file Source code can be recreated $ sqoop codegen --connect jdbc:mysql://<ip address>/<database name> --table <mysql_table name> --username <username_for_mysql_user> --password <Password>