hadoop完全分布模式安装配置

fswdnr

　　hadoop有三种运行方式：单价模式，伪分布式，完全分布模式
在此介绍下完全分布模式。
安装前应先将VMware和linux系统安装好（我安装的VMware 9.0 和redhat 5），
安装VMware Tools，调通网络，桥接，NAT都行，我用是桥接方式。
########################################################################
步骤：
1、配置hosts文件
2、建立hadoop运行账号
3、配置ssh免密码连入
4、下载并解压hadoop安装包
5、配置namenode，修改site文件
6、配置hadoop-env.sh文件
7、配置masters和slaves文件
8、向各个节点复制hadoop
9、格式化mamenode
10、启动hadoop
11、使用jps检验各个后台进程是否成功启动
########################################################################
1、配置hosts文件，修改所有节点的/etc/hosts,使彼此之间都能把主机名解析为IP
   

[iyunv@red1 ~]# sudo gedit /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost2.localdomain2 localhost2
172.21.42.84 red1
172.21.42.82 red2

　　
2、在所有的节点上都建立运行hadoop的账号
 

  [iyunv@red1 ~]# useradd hadoop
[iyunv@red1 ~]# passwd hadoop
　　3、ssh配置，注意要以hadoop用户的主目录下进行操作，每个节点相同
   

   [hadoop@red1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
9b:82:ea:58:b4:e0:35:d7:ff:19:66:a6:ef:ae:0e:d2 hadoop@red1
[hadoop@red1 /]$ cd /home/hadoop/.ssh/
[hadoop@red1 .ssh]$ ls
id_rsa  id_rsa.pub  known_hosts
[hadoop@red1 ~]$ cp id_rsa.pub authorized_keys

　　同理：在其他节点上进行相应操作。
   
分发ssh公钥，把各个节点的authorized_keys的内容互相拷贝加入到对方的此文件中，然后可以实现免密码彼此ssh连入
  

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAmzXjSVazkfYs6WDxSnopXf41mKDMKj
D4m7ZwH9+peZVQAWIzGAlkVn6Osn/dSZAKhdPqf8dTQfgtw8BkgapqCq3dWCImeEOl
T6Hw452+AH55BiDnzleuOLueh+itq639DCFty8zMSj0H14GVc0KMfsQxnJqsCfF7EY
pwVWzmFI8bbMeoynQA1m5QIJEJC8AsLKy8/MZhy9GaGKTv/BkDwpkbF1LsLew7Mg35
/ddeX7z8Fsx7nS2rko8/rAxRO47JEFVw2VXVeWfdPPXNVbTXppu0StV+Pdaffell6D
jKizHf8aX87MRHFK+8DiTBD5+BsZsmw9ynbjJasxMl05Kxnw== hadoop@red1
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAn86HahOIgUA7FeIwgfvJx8IdvRFuCy
x8MEdFqhFThLBE49RoG0wkmc06j2uSb/Kq9xTrIQrev0dP0i9QPUCBn8DaQ80AylE8
uqOPivXqFLd9Qni/mAqASaOkHM5SWU8fPJAZIljq5nJ+zSUXC381isqwYkX63lPJZu
traCglEy6yJoqfvvR5TKrAm9nLNCl3SYRoH/nT1+Vt+ywVaaFkTzvZUbiAs54krz3E
on8n8tFAJvlMAvZdmoy+s/bBUTTsR4SPwG7teRXTrJBD2zHAJnzVMu8AJe3TipyXBh
PlT5Gfu8RgfhTMx/S2rCIfezqi4pc5vFKY5u0UJdbF7cTC6w== hadoop@red2
　　4、下载hadoop压缩包并解压
  

[hadoop@red1 ~]$ tar -zxvf hadoop-0.20.2.tar.gz /home/hadoop/
　　5、修改配置文件
(1) 修改core-site.xml文件

[hadoop@red1 ~]# cd /home/hadoop/hadoop-0.20.2/conf/
[hadoop@red1 conf]# sudo gedit core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://red1:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop-0.20.2/tmp</value>
</property>
</configuration>

　　
(2) 修改mapred-site.xml
 

[hadoop@red1 conf]# sudo gedit mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>red1:9001</value>
</property>
</configuration>

　　
(3) 修改 hdfs-site.xml 文件

[hadoop@red1 conf]# sudo gedit hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

　　6、配置hadoop-env.sh文件
在hadoop-env.sh文件中配置jdk的环境变量

# The java implementation to use.  Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
export JAVA_HOME=/usr/java/jdk1.6.0_03

　　7、配置masters和slaves文件
     masters文件中添加 red1
 [hadoop@red1 conf]$ sudo gedit masters
     slaves文件中添加  red2 
 [hadoop@red1 conf]$ sudo gedit slaves
　　8、向各个节点复制hadoop

[hadoop@red1 conf]$ scp -r /home/hadoop/hadoop-0.20.2 red2:/home/hadoop

　　9、格式化mamenode
       

[hadoop@red1 hadoop-0.20.2]$ bin/hadoop namenode -format
14/01/21 16:35:09 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = red1/172.21.42.84
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
14/01/21 16:35:09 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop
14/01/21 16:35:09 INFO namenode.FSNamesystem: supergroup=supergroup
14/01/21 16:35:09 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/01/21 16:35:09 INFO common.Storage: Image file of size 96 saved in 0 seconds.
14/01/21 16:35:09 INFO common.Storage: Storage directory .../hadoop-hadoop/dfs/name has been successfully formatted.
14/01/21 16:35:09 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at red1/172.21.42.84
************************************************************/

　　
 没有waring和error，出现has been successfully，格式成功。
10、启动hadoop
 

        [hadoop@red1 ~]$ start-all.sh
starting namenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-namenode-red1.out
red2: starting datanode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-red2.out
red1: starting secondarynamenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-secondarynamenode-red1.out
starting jobtracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-jobtracker-red1.out
red2: starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-red2.out
　　
11、使用jps检验各个后台进程是否成功启动
 namenode节点

[hadoop@red1 ~]# /usr/java/jdk1.6.0_03/bin/jps
27561 Jps
7377 NameNode
7606 JobTracker
7523 SecondaryNameNode

　　datanode节点

[hadoop@red2 ~]$ /usr/java/jdk1.6.0_03/bin/jps
14745 Jps
6658 TaskTracker
6577 DataNode

　　成功。