hadoop详细安装和配置

jacky4955

hadoop版本：hadoop-2.2.0-cdh5.0.0-beta-1
jdk版本：jdk-7u40-linux-x64
环境准备：
10.95.3.100 master1
10.95.3.101 master2
10.95.3.103 slave1
10.95.3.104 slave2
说明：本安装没有进行hdfs HA配置，NameNode和SecondaryNameNode都在master1上，其他三个节点作为DN节点。
安装步骤（JDK的安装省略）：
1、添加hadoop用户
   root身份登录master1
   添加用户：useradd –d /home/hadoop -m hadoop
   给hadoop用户设置密码：passwd hadoop
2、配置hosts
   以root用户登录各台机器，vi /etc/hosts文件，修改如下：

#127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.95.3.100              master1
10.95.3.101              master2
10.95.3.103              slave1
10.95.3.104              slave2


3、配置master1到各台机器的免密码登录
   以hadoop用户登录master1，进入/home/hadoop目录下，执行:
   ssh-keygen -t rsa
   注意一定不要输入密码，这样，在/home/hadoop/.ssh目录下将会生成id_rsa，  id_rsa.pub两个文件
   然后再分别执行：
   ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@master1
   ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@master2
   ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@slave1
   ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@slave2
这样，就把master1的公钥加入到了master的authorized_keys文件中。可以测试下从master1 ssh到其他各台机器是否不需要输入密码。
   注意：文件夹的权限
   sudo chmod 755 .ssh/
   sudo chmod 644 .ssh/authorized_keys
4、解压hadoop-2.2.0-cdh5.0.0-beta-1.tar.gz，设置hadoop的安装目录为/dp/hadoop
   设置环境变量：vi ~/.bashrc

export HADOOP_HOME=/dp/hadoop
export HADOOP_PID_DIR=/dp/hadoop_pid_dir
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=/dp/hadoop
export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$ANT_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$ZOOKEEPER_HOME/bin:$FLUME_HOME/bin:$SQOOP_HOME/bin:$OOZIE_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:.


5、进入/dp/hadoop/etc/hadoop目录，修改配置文件
core-site.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master1</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>10080</value>
</property>
<property>
<name>fs.trash.checkpoint.interval</name>
<value>10080</value>
</property>
<property>
<name>topology.script.file.name</name>
<value>/dp/hadoop/etc/hadoop/rack.py</value>
</property>
<property>
<name>topology.script.number.args</name>
<value>6</value>
</property>
<property>
<name>hadoop.security.group.mapping</name>
<value>org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback</value>
</property>
<!--
<property>
<name>hadoop.native.lib</name>
<value>false</value>
<description>Should native hadoop libraries, if present, be used.</description>
</property>
-->
<!--
<property>
<name>ha.zookeeper.quorum</name>
<value>master:2181,slave2:2181,slave6:2181</value>
</property>      
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
-->
</configuration>


hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.blocksize</name>
<value>16m</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/dp/data/hadoop</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>master1:50070</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master1:50090</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.max.xcievers</name>
<value>1000000</value>
</property>
<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>104857600</value>
<description>
Specifies the maximum amount of bandwidth that each datanode
can utilize for the balancing purpose in term of
the number of bytes per second.
</description>
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>/dp/hadoop/etc/hadoop/excludes</value>
<description>Names a file that contains a list of hosts that are
not permitted to connect to the namenode.  The full pathname of the
file must be specified.  If the value is empty, no hosts are
excluded.</description>
</property>

上述的配置要手工创建excludes文件。
map-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master1:19888</value>
</property>
<!--        <property>
<name>mapreduce.history.server.delegationtoken.renewer</name>
<value>true</value>
</property>
-->
<property>
<name>mapreduce.output.fileoutputformat.compress</name>
<value>true</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.compress.type</name>
<value>BLOCK</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<property>
<name>mapreduce.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>


yarn-site.xml

<configuration>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master1:8088</value>
</property>
<property>
<name>yarn.nm.liveness-monitor.expiry-interval-ms</name>
<value>10000</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
$YARN_HOME/share/hadoop/yarn/*,$YARN_HOME/share/hadoop/yarn/lib/*,
$YARN_HOME/share/hadoop/mapreduce/*,$YARN_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/dp/data/yarn/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/dp/data/yarn/logs</value>
</property>
<property>
<description>Where to aggregate logs</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/dp/data/yarn/logs</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/user</value>
</property>
<property>
<description>Amount of physical memory, in MB, that can be allocated
for containers.</description>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>


6、配置机架感知,在/dp/hadoop/etc/hadoop下添加rack.data rack.py文件
rack.data

default /rack/default
10.95.3.101 master2 /rack/rack1
10.95.3.103 slave1  /rack/rack1
10.95.3.104 slave2  /rack/rack1


rack.py

#!/bin/env python
import sys,os,time
pwd = os.path.realpath( __file__ )
rack_file = os.path.dirname(pwd) + "/rack.data"
rack_list = [ l.strip().split() for l in open(rack_file).readlines() if len(l.strip().split()) > 1 ]
rack_map = {}
for item in rack_list:
for host in item[:-1]:
rack_map[host] = item[-1]
rack_map['default'] = 'default' in rack_map and rack_map['default'] or '/default/rack'
rack_result = [av in rack_map and rack_map[av] or rack_map['default'] for av in sys.argv[1:]]
#print rack_map, rack_result
print ' '.join( rack_result )
f = open('/tmp/rack.log','a+')
f.writelines( "[%s] %sn" % (time.strftime("%F %T"),str(sys.argv)))
f.close()


7、修改slaves文件：

master2
slave1
slave


8、将master1上配置好的hadoop拷贝到其他节点上去
9、执行start-dfs.sh启动hadoop，注意如果防火墙没关闭，有可能造成DN连接不上NN
10、执行hadoop dfsadmin -report命令，查看hadoop是否正常启动