Using Sqoop2 to import mysql data to HDFS
On Sqoop2 server:MySQL JDBC Driver:
cp mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar /var/lib/sqoop2/mysql-connector-java.jar
Oracle JDBC Driver:
cp ojdbc6.jar /var/lib/sqoop2/oracle-connector-java.jar
Microsoft SQL Server JDBC Driver:
cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop2/
cd /var/lib/sqoop2
chown sqoop2:sqoop2 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
chmod 755 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
restart sqoop2 service
On Sqoop2 client:
MySQL JDBC Driver:
cp mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar /var/lib/sqoop2/mysql-connector-java.jar
Oracle JDBC Driver:
cp ojdbc6.jar /var/lib/sqoop2/oracle-connector-java.jar
Microsoft SQL Server JDBC Driver:
cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop2/
cd /var/lib/sqoop2
chown sqoop2:sqoop2 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
chmod 755 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
(
For using Sqoop 1 client, you should add sqoop 1 clientservice to client node as gateway role, then put JDBC Driver to /var/lib/sqoop
MySQL JDBC Driver:
cp mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar /var/lib/sqoop/mysql-connector-java.jar
Oracle JDBC Driver:
cp ojdbc6.jar /var/lib/sqoop/oracle-connector-java.jar
Microsoft SQL Server JDBC Driver:
cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop/
cd /var/lib/sqoop
chown sqoop:sqoop mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
chmod 755 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
sudo -u hdfs hadoop fs -mkdir /user/sqoop
sudo -u hdfs hadoop fs -chown sqoop:sqoop /user/sqoop
sqoop list-databases --connect jdbc:mysql://mysql-server-ip --username username --password
sqoop list-tables --connect jdbc:mysql://mysql-server-ip/databasename --username username --password
)
sqoop2 (you can also use HUE web UI > sqoop transfer)
sqoop:000> set server --host sqoop2-server-ip
sqoop:000> show connector --all
sqoop:000> create connection --cid 1
Creating connection for connector with> Please fill following values to create new connection object
Name: mysql-conection
Connection configuration
JDBC Driver> JDBC Connection String: jdbc:mysql://remote-mysql-server/dbname
Username: username
Password: *******
JDBC Connection Properties:
There are currently 0 values in the map:
entry#
Security> Max connections: 10
New connection was successfully created with validation status FINE and persistent> sqoop:000> create job --xid 1 --type import
Creating job for connection with> Please fill following values to create new job object
Name: export-job1
Database configuration
Schema name: dbname
Table name: tablename
Table SQL statement:
Table column names:
Partition column name:
Nulls in partition column:
Boundary query:
Output configuration
Storage type:
0 : HDFS
Choose: 0
Output format:
0 : TEXT_FILE
1 : SEQUENCE_FILE
Choose: 0
Compression format:
0 : NONE
1 : DEFAULT
2 : DEFLATE
3 : GZIP
4 : BZIP2
5 : LZO
6 : LZ4
7 : SNAPPY
Choose: 0
Output directory: /user/sqoop2/output
Throttling resources
Extractors:
Loaders:
New job was successfully created with validation status FINEand persistent> sqoop:000> start job --jid 1
sqoop:000> status job --jid 1
JDBC Driver> JDBC Connection String: jdbc:mysql://remote-mysql-server/dbname
JDBC Driver> JDBC Connection String: jdbc:sqlserver://remote-mysql-server/dbname
JDBC Driver> JDBC Connection String:
页:
[1]