大数据系列12：Hadoop2 – 全新的Hadoop
wget http://mirrors.hust.edu.cn/apache/hadoop/common/stable2/hadoop-2.2.0.tar.gztar -xzvf hadoop-2.2.0.tar.gz
cd hadoop-2.2.0
vi etc/hadoop/hadoop-env.sh
修改：
export JAVA_HOME=/home/ysc/jdk1.7.0_17
vi etc/hadoop/slaves
修改localhost为host001
vi etc/hadoop/core-site.xml
增加：
<property>
<name>fs.defaultFS</name>
<value>hdfs://host001:9000</value>
</property>
cp etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml
vi etc/hadoop/mapred-site.xml
增加：
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.cluster.local.dir</name>
<value>/home/ysc/mapreduce/local</value>
</property>
vi etc/hadoop/yarn-site.xml
增加：
<property>
<name>yarn.resourcemanager.hostname</name>
<value>host001</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
vi etc/hadoop/hdfs-site.xml
增加：
<property>
<name>dfs.name.dir</name>
<value>/home/ysc/dfs/filesystem/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/ysc/dfs/filesystem/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
sudo vi /etc/profile
增加：
export HADOOP_PREFIX=/home/ysc/hadoop-2.2.0
export HADOOP_COMMON_HOME=/home/ysc/hadoop-2.2.0
export HADOOP_MAPRED_HOME=/home/ysc/hadoop-2.2.0
export HADOOP_CONF_DIR=/home/ysc/hadoop-2.2.0/etc/hadoop
export HADOOP_HDFS_HOME=/home/ysc/hadoop-2.2.0
export HADOOP_YARN_HOME=/home/ysc/hadoop-2.2.0
source /etc/profile
格式化：
bin/hdfs namenode -format
启动dfs：
sbin/start-dfs.sh
启动yarn：
sbin/start-yarn.sh
启动historyserver：
sbin/mr-jobhistory-daemon.sh start historyserver
运行wordcount：
echo "APDPlat is a java open source project, Application Product Development Platform." > text1.txt
echo "Yang Shangchuan is the founder of APDPlat which won the "2013 Outstanding Open Source Project" award." > text2.txt
bin/hadoop fs -mkdir input
bin/hadoop fs -put text1.txt input
bin/hadoop fs -put text2.txt input
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount input output
访问管理页面：
ResourceManager: http://host001:8088
NodeManager: http://host001:8042
NameNode:http://host001:50070
NameNode:http://host001:50070
停止dfs：
sbin/stop-dfs.sh
停止yarn：
sbin/stop-yarn.sh
停止historyserver：
sbin/mr-jobhistory-daemon.sh stop historyserver
注意：要把这里的主机host001和用户ysc替换为你自己的，在/etc/hosts文件中要把127.0.0.1 localhost改为实际IP地址 host001
APDPlat旗下十大开源项目
页:
[1]