2018-07-08期 Hadoop单节点伪分布式集群配置
一、安装介质下载地址：http://archive.apache.org/dist/hadoop/core/
安装版本：hadoop-2.4.1.tar.gz
二、安装步骤
1、解压hadoop-2.4.1.tar.gz
# tar -xvf hadoop-2.4.1.tar.gz -C /usr/local/apps/
# pwd
/usr/local/apps/hadoop-2.4.1
# ll
total 52
drwxr-xr-x. 2 67974 users4096 Jun 202014 bin
drwxr-xr-x. 3 67974 users4096 Jun 202014 etc
drwxr-xr-x. 2 67974 users4096 Jun 202014 include
drwxr-xr-x. 3 67974 users4096 Jun 202014 lib
drwxr-xr-x. 2 67974 users4096 Jun 202014 libexec
-rw-r--r--. 1 67974 users 15458 Jun 202014 LICENSE.txt
-rw-r--r--. 1 67974 users 101 Jun 202014 NOTICE.txt
-rw-r--r--. 1 67974 users1366 Jun 202014 README.txt
drwxr-xr-x. 2 67974 users4096 Jun 202014 sbin
drwxr-xr-x. 4 67974 users4096 Jun 202014 share
#
2、修改配置文件
# cd /usr/local/apps/hadoop-2.4.1/etc/hadoop/
--修改hadoop-env.sh
# vi hadoop-env.sh
# The only required environment variable is JAVA_HOME.All others are
# optional.When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
# The java implementation to use.
export JAVA_HOME=/usr/local/apps/jdk1.7.0_80/
# The jsvc implementation to use. Jsvc is required to run secure datanodes.
#export JSVC_HOME=${JSVC_HOME}
--修改core-site.xml
# vi core-site.xml
fs.defaultFS
hdfs://hadoop-server01:9000/
hadoop.tmp.dir
/usr/local/apps/hadoop-2.4.1/tmp/
--修改hdfs-site.xml
# vi hdfs-site.xml
dfs.replication
1
--修改mapred-site.xml
# mv mapred-site.xml.template mapred-site.xml
# vi mapred-site.xml
mapreduce.framework.name
yarn
--修改yarn-site.xml
# vi yarn-site.xml
yarn.resourcemanager.hostname
hadoop-server01
yarn.nodemanager.aux-services
mapreduce-shuffle
--修改slaves
# vi slaves
hadoop-server01
hadoop-server02
hadoop-server03
3、启动服务
--格式化
# cd /usr/local/apps/hadoop-2.4.1/bin/
# ./hadoop namenode -format
18/06/15 00:44:09 INFO util.GSet: capacity = 2^15 = 32768 entries
18/06/15 00:44:09 INFO namenode.AclConfigFlag: ACLs enabled? false
18/06/15 00:44:09 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1300855425-192.168.1.201-1529048649163
18/06/15 00:44:09 INFO common.Storage: Storage directory /usr/local/apps/hadoop-2.4.1/tmp/dfs/name has been successfully formatted.
18/06/15 00:44:09 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
18/06/15 00:44:09 INFO util.ExitUtil: Exiting with status 0
18/06/15 00:44:09 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop-server01/192.168.1.201
************************************************************/
3.1 手动启动
（1）启动HDFS
# ./hadoop-daemon.sh start namenode
# ./hadoop-daemon.sh start datanode
# ./hadoop-daemon.sh start secondarynamenode
# jps
28993 Jps
28925 SecondaryNameNode
4295 DataNode
4203 NameNode
--访问地址
http://192.168.1.201:50070/
（2）启动yarn
# ./yarn-daemon.sh start resourcemanager
# ./yarn-daemon.sh start nodemanager
# jps
29965 NodeManager
28925 SecondaryNameNode
29062 ResourceManager
4295 DataNode
4203 NameNode
3.2 脚本自动启动
--前提条件需要配置ssh免密登录
# ssh-keygen
# ssh-copy-id hadoop-server01
# shh hadoop-server01
（1）启动HDFS
# ./start-dfs.sh
# jps
31538 Jps
31423 SecondaryNameNode
31271 DataNode
31152 NameNode
（2）启动yarn
# ./start-yarn.sh
# jps
32009 Jps
31423 SecondaryNameNode
31271 DataNode
31697 NodeManager
31593 ResourceManager
31152 NameNode
备注：本文档所有配置都采用主机名配置，因此需要先配置hosts文件 ,非windows下配置/etc/hosts windows下配置/windows/systemm32/drivers/etc/hosts文件，配置格式 ：IP 主机名
页:
[1]