云中漫步 发表于 2018-10-1 11:52:11

Using Sqoop2 to import mysql data to HDFS

  On Sqoop2 server:
  MySQL JDBC Driver:
  cp mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar /var/lib/sqoop2/mysql-connector-java.jar
  Oracle JDBC Driver:
  cp ojdbc6.jar /var/lib/sqoop2/oracle-connector-java.jar
  Microsoft SQL Server JDBC Driver:
  cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop2/
  cd /var/lib/sqoop2
  chown sqoop2:sqoop2 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
  chmod 755 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
  restart sqoop2 service
  On Sqoop2 client:
  MySQL JDBC Driver:
  cp mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar /var/lib/sqoop2/mysql-connector-java.jar
  Oracle JDBC Driver:
  cp ojdbc6.jar /var/lib/sqoop2/oracle-connector-java.jar
  Microsoft SQL Server JDBC Driver:
  cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop2/
  cd /var/lib/sqoop2
  chown sqoop2:sqoop2 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
  chmod 755 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
  (
  For using Sqoop 1 client, you should add sqoop 1 clientservice to client node as gateway role, then put JDBC Driver to /var/lib/sqoop
  MySQL JDBC Driver:
  cp mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar /var/lib/sqoop/mysql-connector-java.jar
  Oracle JDBC Driver:
  cp ojdbc6.jar /var/lib/sqoop/oracle-connector-java.jar
  Microsoft SQL Server JDBC Driver:
  cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop/
  cd /var/lib/sqoop
  chown sqoop:sqoop mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
  chmod 755 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
  sudo -u hdfs hadoop fs -mkdir /user/sqoop
  sudo -u hdfs hadoop fs -chown sqoop:sqoop /user/sqoop
  sqoop list-databases --connect jdbc:mysql://mysql-server-ip --username username --password
  sqoop list-tables --connect jdbc:mysql://mysql-server-ip/databasename --username username --password
  )
  sqoop2 (you can also use HUE web UI > sqoop transfer)
  sqoop:000> set server --host sqoop2-server-ip
  sqoop:000> show connector --all
  sqoop:000> create connection --cid 1

  Creating connection for connector with>  Please fill following values to create new connection object
  Name: mysql-conection
  Connection configuration

  JDBC Driver>  JDBC Connection String: jdbc:mysql://remote-mysql-server/dbname
  Username: username
  Password: *******
  JDBC Connection Properties:
  There are currently 0 values in the map:
  entry#

  Security>  Max connections: 10

  New connection was successfully created with validation status FINE and persistent>  sqoop:000> create job --xid 1 --type import

  Creating job for connection with>  Please fill following values to create new job object
  Name: export-job1
  Database configuration
  Schema name: dbname
  Table name: tablename
  Table SQL statement:
  Table column names:
  Partition column name:
  Nulls in partition column:
  Boundary query:
  Output configuration
  Storage type:
  0 : HDFS
  Choose: 0
  Output format:
  0 : TEXT_FILE
  1 : SEQUENCE_FILE
  Choose: 0
  Compression format:
  0 : NONE
  1 : DEFAULT
  2 : DEFLATE
  3 : GZIP
  4 : BZIP2
  5 : LZO
  6 : LZ4
  7 : SNAPPY
  Choose: 0
  Output directory: /user/sqoop2/output
  Throttling resources
  Extractors:
  Loaders:

  New job was successfully created with validation status FINEand persistent>  sqoop:000> start job --jid 1
  sqoop:000> status job --jid 1

  JDBC Driver>  JDBC Connection String: jdbc:mysql://remote-mysql-server/dbname

  JDBC Driver>  JDBC Connection String: jdbc:sqlserver://remote-mysql-server/dbname

  JDBC Driver>  JDBC Connection String:

页: [1]
查看完整版本: Using Sqoop2 to import mysql data to HDFS