Hadoop 2.2.0 在Red Hat Enterprise Linux 6.1 上的分布式配置(VMware虚拟机，1个namenode，2个datano
这两天参考了一批文章，终于把Hadoop配置好了！！！参考：
http://blog.iyunv.com/licongcong_0224/article/details/12972889http://www.ituring.com.cn/article/63927http://www.iyunv.com/xia520pi/archive/2012/05/16/2503949.html
1、虚拟机情况
3台，每个1G内存，20G硬盘，均为NAT网络，均有VMware Tools
IP:
192.168.220.131 hadoop0
192.168.220.133 hadoop1
192.168.220.134 hadoop2
账户密码设置：mlx/123456
(使用root账户也是可以的，不过不安全，最好使用普通账户)
注意：关闭防火墙(需root权限)
service iptables stop
chkconfig iptables off
2、改hosts
先使用ifconfig查看每个机器的ip
$ ifconfig
eth0 Link encap:EthernetHWaddr 00:0C:29:FC:94:01
inet addr:192.168.220.131Bcast:192.168.220.255Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fefc:9401/64 Scope:Link
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
RX packets:12741 errors:0 dropped:0 overruns:0 frame:0
TX packets:8027 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2209242 (2.1 MiB)TX bytes:862255 (842.0 KiB)
Interrupt:19 Base address:0x2024
lo Link encap:Local Loopback
inet addr:127.0.0.1Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNINGMTU:16436Metric:1
RX packets:2666 errors:0 dropped:0 overruns:0 frame:0
TX packets:2666 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4520281 (4.3 MiB)TX bytes:4520281 (4.3 MiB)
可以看到hadoop0的ip为192.168.220.131
同理，查hadoop1,hadoop2的ip
然后将
192.168.220.131 hadoop0
192.168.220.133 hadoop1
192.168.220.134 hadoop2
放入每台机器的/etc/hosts 文件的末尾（注意：这个需要用root账户）
3、在每台机器上安装JDK
这里看我的另一篇文章即可：
http://www.iyunv.com/xysmlx/p/3551619.html
4、master与slave之间的SSH互联
4.1、master向slave的SSH互联
先用以下命令生成rsa密钥
ssh-keygen -t rsa -P ''
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
然后修改authorized_keys的权限(非常重要，否则无法进行ssh连接)
chmod 600 ~/.ssh/authorized_keys
可以尝试ssh localhost登陆，如果不需要密码，则目前成功
mlx@hadoop0 sbin]$ ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 43:3d:d0:2c:13:de:b1:c4:da:72:34:ba:c9:a3:a2:64.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Last login: Mon Feb 17 15:01:01 2014 from hadoop1
$
然后将rsa文件拷到另外两台机器上面
scp ~/.ssh/id_rsa.pub mlx@192.168.220.133:~/
scp ~/.ssh/id_rsa.pub mlx@192.168.220.134:~/
然后在另外两台机器上面分别执行以下命令：（注意：权限修改chmod非常重要）
chmod 700 ~/.ssh
cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
然后用命令测试ssh：ssh 用户@IP
$ ssh mlx@hadoop1
Last login: Mon Feb 17 13:52:52 2014 from hadoop0
$
如果不需密码，则成功
4.2、slave向master的ssh连接
同4.1，将4.1做的反过来即可
注意：必须进行权限修改chmod，否则会失败
测试：
$ ssh mlx@hadoop0
Last login: Mon Feb 17 16:13:56 2014 from localhost
$
5、Hadoop的安装与配置
这里是把hadoop在master(hadoop0)上面配置好，然后复制到hadoop1和hadoop2上面
5.1、Hadoop的安装
先从hadoop官网下载hadoop文件hadoop-2.2.0.tar.gz
然后将hadoop-2.2.0.tar.gz复制到/usr中
然后解压hadoop-2.2.0.tar.gz
tar -zxf /usr/hadoop-2.2.0.tar.gz
将解压好的hadoop-2.2.0文件夹改名为hadoop
然后在/etc/profile末尾添加以下内容来改环境变量：(root账户)
# set hadoop path
export HADOOP_HOME=/usr/hadoop
export PATH=$HADOOP_HOME/bin:$PATH
然后重启profile
source /etc/profile
5.2、Hadoop的配置
注意5.2.4-5.2.7都是在之间插入代码
5.2.2-5.2.7文件均在/usr/hadoop/etc/hadoop下
5.2.1、创建文件夹
mkdir /usr/hadoop/tmp
mkdir /usr/hadoop/dfs
mkdir /usr/hadoop/dfs/name
mkdir /usr/hadoop/dfs/data
5.2.2、hadoop-env.sh
修改
export JAVA_HOME=/usr/java/jdk1.7.0_51
5.2.3、yarn-env.sh
修改
if [ "$JAVA_HOME" != "" ]; then
#echo "run java in $JAVA_HOME"
JAVA_HOME=/usr/java/jdk1.7.0_51
fi
5.2.4、core-site.xml
fs.defaultFS
hdfs://hadoop0:9000
io.file.buffer.size
131072
hadoop.tmp.dir
file:/usr/hadoop/tmp
Abase for other temporary directories.
hadoop.proxyuser.mlx.hosts
*
hadoop.proxyuser.mlx.groups
*
5.2.5、hdfs-site.xml
dfs.namenode.secondary.http-address
hadoop0:9001
dfs.namenode.name.dir
file:/usr/hadoop/dfs/name
dfs.datanode.data.dir
file:/usr/hadoop/dfs/data
dfs.replication
1
dfs.webhdfs.enabled
true
5.2.6、mapred-site.xml
mapreduce.framework.name
yarn
mapreduce.jobhistory.address
hadoop0:10020
mapreduce.jobhistory.webapp.address
hadoop0:19888
5.2.7、yarn-site.xml
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.nodemanager.aux-services.mapreduce.shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
yarn.resourcemanager.address
hadoop0:8032
yarn.resourcemanager.scheduler.address
hadoop0:8030
yarn.resourcemanager.resource-tracker.address
hadoop0:8031
yarn.resourcemanager.admin.address
hadoop0:8033
yarn.resourcemanager.webapp.address
hadoop0:8088
5.3、将hadoop0配置的hadoop文件夹复制到hadoop1和hadoop2上
scp -r /usr/hadoop root@服务器IP:/usr/
即：
scp -r /usr/hadoop root@hadoop1:/usr/
scp -r /usr/hadoop root@hadoop2:/usr/
5.4、改hadoop0上的slaves文件
将
localhost
改为
192.168.220.133
192.168.220.134
5.5、改权限
每一个机器都要做
chown -R mlx:hadoop hadoop
以及
chmod g-w /usr/hadoop
6、启动Hadoop
6.1、格式化HDFS
hadoop namenode -format
可以看到最后几行为：
14/02/17 15:54:10 INFO namenode.NameNode: registered UNIX signal handlers for
Formatting using clusterid: CID-630c8102-043a-46ca-b9dd-c2c12a96965d
14/02/17 15:54:11 INFO namenode.HostFileManager: read includes:
HostSet(
)
14/02/17 15:54:11 INFO namenode.HostFileManager: read excludes:
HostSet(
)
14/02/17 15:54:11 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
14/02/17 15:54:11 INFO util.GSet: Computing capacity for map BlocksMap
14/02/17 15:54:11 INFO util.GSet: VM type = 32-bit
14/02/17 15:54:11 INFO util.GSet: 2.0% max memory = 966.7 MB
14/02/17 15:54:11 INFO util.GSet: capacity = 2^22 = 4194304 entries
14/02/17 15:54:11 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
14/02/17 15:54:11 INFO blockmanagement.BlockManager: defaultReplication = 1
14/02/17 15:54:11 INFO blockmanagement.BlockManager: maxReplication = 512
14/02/17 15:54:11 INFO blockmanagement.BlockManager: minReplication = 1
14/02/17 15:54:11 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
14/02/17 15:54:11 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks= false
14/02/17 15:54:11 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
14/02/17 15:54:11 INFO blockmanagement.BlockManager: encryptDataTransfer = false
14/02/17 15:54:11 INFO namenode.FSNamesystem: fsOwner = mlx (auth:SIMPLE)
14/02/17 15:54:11 INFO namenode.FSNamesystem: supergroup = supergroup
14/02/17 15:54:11 INFO namenode.FSNamesystem: isPermissionEnabled = true
14/02/17 15:54:11 INFO namenode.FSNamesystem: HA Enabled: false
14/02/17 15:54:11 INFO namenode.FSNamesystem: Append Enabled: true
14/02/17 15:54:11 INFO util.GSet: Computing capacity for map INodeMap
14/02/17 15:54:11 INFO util.GSet: VM type = 32-bit
14/02/17 15:54:11 INFO util.GSet: 1.0% max memory = 966.7 MB
14/02/17 15:54:11 INFO util.GSet: capacity = 2^21 = 2097152 entries
14/02/17 15:54:11 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/02/17 15:54:11 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
14/02/17 15:54:11 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
14/02/17 15:54:11 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
14/02/17 15:54:11 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
14/02/17 15:54:11 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
14/02/17 15:54:11 INFO util.GSet: Computing capacity for map Namenode Retry Cache
14/02/17 15:54:11 INFO util.GSet: VM type = 32-bit
14/02/17 15:54:11 INFO util.GSet: 0.029999999329447746% max memory = 966.7 MB
14/02/17 15:54:11 INFO util.GSet: capacity = 2^16 = 65536 entries
14/02/17 15:54:11 INFO common.Storage: Storage directory /usr/hadoop/dfs/name has been successfully formatted.
14/02/17 15:54:11 INFO namenode.FSImage: Saving image file /usr/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
14/02/17 15:54:11 INFO namenode.FSImage: Image file /usr/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 195 bytes saved in 0 seconds.
14/02/17 15:54:11 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
14/02/17 15:54:11 INFO util.ExitUtil: Exiting with status 0
14/02/17 15:54:11 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop0/192.168.220.131
************************************************************/
6.2、启动hadoop
到/usr/hadoop/sbin下，输入
./start-all.sh
$ ./start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on
hadoop0: starting namenode, logging to /usr/hadoop/logs/hadoop-mlx-namenode-hadoop0.out
192.168.220.134: starting datanode, logging to /usr/hadoop/logs/hadoop-mlx-datanode-hadoop2.out
192.168.220.133: starting datanode, logging to /usr/hadoop/logs/hadoop-mlx-datanode-hadoop1.out
Starting secondary namenodes
hadoop0: starting secondarynamenode, logging to /usr/hadoop/logs/hadoop-mlx-secondarynamenode-hadoop0.out
starting yarn daemons
starting resourcemanager, logging to /usr/hadoop/logs/yarn-mlx-resourcemanager-hadoop0.out
192.168.220.134: starting nodemanager, logging to /usr/hadoop/logs/yarn-mlx-nodemanager-hadoop2.out
192.168.220.133: starting nodemanager, logging to /usr/hadoop/logs/yarn-mlx-nodemanager-hadoop1.out
6.3、查看是否启动
在hadoop0下输入jps
$ jps
11696 Jps
11140 NameNode
11450 ResourceManager
11315 SecondaryNameNode
在hadoop1下输入jps
$ jps
6917 Jps
6168 NodeManager
6062 DataNode
$
在hadoop2下输入jps
$ jps
6536 DataNode
6742 Jps
6641 NodeManager
$
在hadoop0下输入hadoop dfsadmin -report
$ hadoop dfsadmin -report
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Configured Capacity: 37073182720 (34.53 GB)
Present Capacity: 28097224704 (26.17 GB)
DFS Remaining: 28097175552 (26.17 GB)
DFS Used: 49152 (48 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 2 (2 total, 0 dead)
Live datanodes:
Name: 192.168.220.134:50010 (hadoop2)
Hostname: hadoop2
Decommission Status : Normal
Configured Capacity: 18536591360 (17.26 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 4417183744 (4.11 GB)
DFS Remaining: 14119383040 (13.15 GB)
DFS Used%: 0.00%
DFS Remaining%: 76.17%
Last contact: Mon Feb 17 15:55:36 CST 2014
Name: 192.168.220.133:50010 (hadoop1)
Hostname: hadoop1
Decommission Status : Normal
Configured Capacity: 18536591360 (17.26 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 4558774272 (4.25 GB)
DFS Remaining: 13977792512 (13.02 GB)
DFS Used%: 0.00%
DFS Remaining%: 75.41%
Last contact: Mon Feb 17 15:55:36 CST 2014
7、维护Hadoop
7.1、重新格式化HDFS
先到hadoop0的/usr/hadoop/sbin下执行stop-all.sh
然后在三台机器上输入：
rm -rf /usr/hadoop/tmp
rm -rf /usr/hadoop/dfs
mkdir /usr/hadoop/tmp
mkdir /usr/hadoop/dfs
mkdir /usr/hadoop/dfs/name
mkdir /usr/hadoop/dfs/data
rm -rf /tmp/hadoop*
然后在hadoop0上执行
hadoop namenode -format
7.2、重启计算机后打开hadoop
到/usr/hadoop/sbin下执行start-all.sh
7.3、关闭hadoop
到/usr/hadoop/sbin下执行stop-all.sh
7.4、查看日志（最好的排错方法）
日志在/usr/hadoop/logs下，开.log文件即可看日志信息
7.5、结束任务
hadoop job -kill jobid
7.6、执行任务
hadoop jar matrix.jar MartrixMultiplication /input/M.data /output
页:
[1]