hadoop hbase hive 集群安装

zzbb

一：卸载redhat操作系统默认jdk
1：查找安装默认安装jdk
   rpm -qa | grep java
2：删除jdk
   rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.21.b17.el6.x86_64
二：安装oracle jdk
1：使用root账号安装
2：创建目录：/usr/java
3：下载jdk存放到/usr/java目录：jdk-6u33-linux-x64.bin
4：给安装文件添加执行权限：
   chmod +x jdk-6u43-linux-x64.bin
5：执行jdk安装包
   ./jdk-6u43-linux-x64.bin
6：在/etc/profile文件中添加环境变量
export JAVA_HOME=/usr/java/jdk1.6.0_43
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib/rt.jar
export PATH=$PATH:$JAVA_HOME/bin
7：配置生效，执行下面命令
source /etc/profile
三：主机分配，在每一个机器的的/etc/hosts文件中添加下面四行内容
192.168.205.23 inm1
192.168.205.24 inm2
192.168.205.25 inm3
192.168.205.26 inm4
192.168.205.27 inm5
192.168.205.28 inm6
192.168.205.29 inm7
192.168.205.30 inm8
192.168.205.31 inm9
192.168.205.32 inm10

四：关闭所有机器防火墙
chkconfig iptables off
service iptables stop
五：在每台机器上创建hadoop用户组合hadoop用户
1：创建用户组：groupadd hadoop
2：创建用户：useradd -g hadoop hadoop
3：修改密码：passwd hadoop
六：在master.hadoop机器上配置SSH
[hadoop@master ~]$ ssh-keygen -t rsa -P ""
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): /home/hadoop/.ssh/id_rsa
[hadoop@master ~]cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
[hadoop@master ~]chmod 700 ~/.ssh/
[hadoop@master ~]chmod 600 ~/.ssh/authorized_key
验证
[hadoop@master ~]ssh localhost
[hadoop@master ~]ssh inm1
复制ssh配置到其它机器
[hadoop@master ~]ssh-copy-id -i $HOME/.ssh/id_rsa.pub hadoop@inm2
[hadoop@master ~]ssh-copy-id -i $HOME/.ssh/id_rsa.pub hadoop@inm3

七：zookeeper三节点集群安装
1：使用三台服务器安装zookeeper，安装在hadoop用户上
   192.168.205.24、192.168.205.25、192.168.205.26
2：使用cloudera版本zookeeper：zookeeper-3.4.5-cdh4.2.0.tar.gz
3：解压并修改目录名称
   tar -zxf zookeeper-3.4.5-cdh4.2.0.tar.gz
   mv zookeeper-3.4.5-cdh4.2.0/ zookeeper
4：配置zookeeper，在conf目录下创建zoo.cfg文件，添加文件内容
   tickTime=2000  
initLimit=5   
syncLimit=2   
dataDir=/homt/hadoop/storage/zookeeper/data
dataLogDir=/homt/hadoop/storage/zookeeper/logs   
clientPort=2181
server.1=inm2:2888:3888   
server.2=inm3:2888:3888   
   server.3=inm4:2888:3888
5：创建zookeeper的数据文件和日志存放目录
   /home/hadoop/storage/zookeeper/data
   /home/hadoop/storage/zookeeper/logs
   在/home/hadoop/storage/zookeeper/data目录中创建文件myid，添加内容为：1
6：复制安装的zookeeper和storage目录到inm3和inm4机器上。
   scp -r zookeeper inm4:/home/hadoop
   scp -r storage inm4:/home/hadoop
   修改inm3机器上myid文件中值为2
   修改inm3机器上myid文件中值为3
7：启动服务器
   ./bin/zkServer.sh start
8：验证安装
   ./bin/zkCli.sh -server inm3:2181  
八：安装hadoop-2.0.0-cdh4.2.0
用户hadoop账号进入系统
1：解压tar -xvzf hadoop-2.0.0-cdh4.2.0.tar.gz ，修改目录名称：mv hadoop-2.0.0-cdh4.2.0 hadoop
2：配置Hadoop环境变量：修改vi ~/.bashrc，在文件最后面加上如下配置：
export HADOOP_HOME=/home/hadoop/hadoop
export HIVE_HOME=/home/hadoop/hive
export HBASE_HOME=/home/hadoop/hbase
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$HIVE_HOME/bin
3：使配置生效
   source .bashrc
4：修改HADOOP_HOME/etc/hadoop目录下mastes和slaves文件
   masters文件内容：
   inm1
   slaves文件内容：
   inm2
   inm3
   inm4
5：修改HADOOP_HOME/etc/hadoop/core-site.xml文件配置
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://inm1:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
<description>Size of read/write buffer used in SequenceFiles.</description>
</property>
<property>
<name>io.native.lib.available</name>
<value>true</value>
</property>
</configuration>
6：修改HADOOP_HOME/etc/hadoop/hdfs-site.xml文件配置
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/storage/hadoop/tmp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/storage/hadoop/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/storage/hadoop/data</value>
</property>
<property>
<name>dfs.block.size</name>
<value>67108864</value>
<description>HDFS blocksize of 64MB for large file-systems.</description>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>inm1:50070</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
7：修改HADOOP_HOME/etc/hadoop/mapred-site.xml文件配置
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>inm1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>inm1:19888</value>
</property>
</configuration>
8：修改HADOOP_HOME/etc/hadoop/yarn-site.xml文件配置
<configuration>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>inm1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>inm1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>inm1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>inm1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>inm1:8088</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
$YARN_HOME/share/hadoop/yarn/*,$YARN_HOME/share/hadoop/yarn/lib/*,
$YARN_HOME/share/hadoop/mapreduce/*,$YARN_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/hadoop/storage/yarn/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/hadoop/storage/yarn/logs</value>
</property>
<property>
<description>Where to aggregate logs</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/home/hadoop/storage/yarn/logs</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/user</value>
</property>
</configuration>
9：同步hadoop工程到inm2,inm3,inm4机器上面
scp -r hadoop inm2:/home/hadoop
scp -r hadoop inm2:/home/hadoop
scp -r hadoop inm2:/home/hadoop
10：格式文件系统
hadoop namenode -format
11：启动hdfs和yarn，启动脚本在HADOOP_HOME/sbin目录中
./start-hdfs.sh
./start-yarn.sh
八：安装hbase-0.94.2-cdh4.2.0
1：解压tar -xvzf hbase-0.94.2-cdh4.2.0.tar.gz ，修改目录名称：mv hbase-0.94.2-cdh4.2.0.tar.gz hbase
2：修改HBASE_HOME/conf/regionservers文件，添加运行HRegionServer进程的机器名称。
   inm2
inm3
inm4
3：修改HBASE_HOME/conf/hbase-site.xml文件
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://inm1/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.tmp.dir</name>
<value>/home/hadoop/storage/hbase</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>inm2,inm3,inm4</value>
</property>
</configuration>
4：同步hbase工程到inm2,inm3,inm4机器上面
scp -r hbase inm2:/home/hadoop
scp -r hbase inm2:/home/hadoop
scp -r hbase inm2:/home/hadoop
5：在inm1上启动hbase集群
HBASE_HOME/bin/start-hbase.sh
6：执行hbase shell进入hbase console。执行list命令验证安装。
九：安装hive-0.10.0-cdh4.2.0
1：解压tar -xvzf hive-0.10.0-cdh4.2.0.tar.gz ，修改目录名称：mv hive-0.10.0-cdh4.2.0 hive
2：修改HIVE_HOME/conf/hive-site.xml文件
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.205.31:3306/hive?useUnicode=true&amp;characterEncoding=UTF-8</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive2013</value>
<description>password to use against metastore database</description>
</property>
<property>
<name>mapred.job.tracker</name>
<value>inm1:8031</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>hive.aux.jars.path</name>
<value>file:///home/hadoop/hive/lib/zookeeper-3.4.5-cdh4.2.0.jar,
file:///home/hadoop/hive/lib/hive-hbase-handler-0.10.0-cdh4.2.0.jar,
file:///home/hadoop/hive/lib/hbase-0.94.2-cdh4.2.0.jar,
file:///home/hadoop/hive/lib/guava-11.0.2.jar</value>
</property>
<property>
<name>hive.querylog.location</name>
<value>/home/hadoop/storage/hive/querylog</value>
<description>
Location of Hive run time structured log file
</description>
</property>
<property>
<name>hive.support.concurrency</name>
<description>Enable Hive's Table Lock Manager Service</description>
<value>true</value>
</property>
<property>
<name>hive.zookeeper.quorum</name>
<description>Zookeeper quorum used by Hive's Table Lock Manager</description>
<value>inm2,inm3,inm4</value>
</property>
<property>
<name>hive.hwi.listen.host</name>
<value>inm1</value>
<description>This is the host address the Hive Web Interface will listen on</description>
</property>
<property>
<name>hive.hwi.listen.port</name>
<value>9999</value>
<description>This is the port the Hive Web Interface will listen on</description>
</property>
<property>
<name>hive.hwi.war.file</name>
<value>lib/hive-hwi-0.10.0-cdh4.2.0.war</value>
<description>This is the WAR file with the jsp content for Hive Web Interface</description>
</property>

</configuration>
3：添加mysql驱动修改HIVE_HOME/lib目录。
4：进入hive console，执行show databases，验证安装是否成功！