ubuntu 14.04.1 hadoop 2.6-jonybay
sudo addgroup hadoopsudo adduser --ingroup hadoop hadoop
sudo vim /etc/sodoers
hadoop ALL=(ALL)ALL
sudo apt-get install openjdk-7-jre oenjdk-7-jdk
wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
sudo tar xvf hadoop-2.6.0.tar.gz -C /usr/local/
sudo chown -R hadoop:hadoop hadoop-2.6.0
sudo vim etc/hadoop/hadoop-evn.sh
export JAVA_HOME="/usr/lib/jvm/java-7-openjdk-amd64"
使用hadoop帐号登入
ssh-keygen -t rsa -P ""
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
vim /etc/hosts
127.0.0.1 localhost
192.168.0.140 h1
192.168.0.141 h2
192.168.0.142 h3
192.168.0.143 h4
设置： slaves、core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml
1, 文件 slave
cd /usr/local/hadoop-2.6.0/etc/hadoop
vim slaves
将原来 localhost 删除，把所有Slave的主机名写上，每行一个。
2, 文件 core-site.xml ，将原本的如下内容：
fs.defaultFS
hdfs://h1:9000
hadoop.tmp.dir
file:/usr/local/hadoop-2.6.0/tmp
Abase for other temporary directories.
3, 文件hdfs-site.xml，因为只有一个Slave，所以dfs.replication的值设为1。
dfs.namenode.secondary.http-address
h1:50090
dfs.namenode.name.dir
file:/usr/local/hadoop-2.6.0/tmp/dfs/name
dfs.datanode.data.dir
file:/usr/local/hadoop-2.6.0/tmp/dfs/data
dfs.replication
3
4, 文件mapred-site.xml，这个文件不存在，首先需要从模板中复制一份：
cp mapred-site.xml.template mapred-site.xml 然后配置修改如下：
mapreduce.framework.name
yarn
5, 文件yarn-site.xml：
yarn.resourcemanager.hostname
h1
yarn.nodemanager.aux-services
mapreduce_shuffle
KVM虚拟机克隆
sudo virt-clone -o h1 -n h2 -f /kvm/os/h2.img
sudo virt-clone -o h1 -n h3 -f /kvm/os/h2.img
sudo virt-clone -o h1 -n h4 -f /kvm/os/h2.img
然后在h上就可以启动hadoop了。
cd /usr/local/hadoop/
bin/hdfs namenode -format # 首次运行需要执行初始化，后面不再需要
sbin/start-dfs.sh
sbin/start-yarn.sh
jps查看h1启动了NameNode、SecondrryNameNode、ResourceManager进程
h2 h3 h4 启动了DataNode和NodeManager进程
关闭Hadoop集群也是在h1节点上执行：
sbin/stop-dfs.sh
sbin/stop-yarn.sh
可以在h1节点上通过命令bin/hdfs dfsadmin -report查看DataNode是否正常启动
hadoop@h1:/usr/local/hadoop-2.6.0$ bin/hdfs dfsadmin -report
Configured Capacity: 154960711680 (144.32 GB)
Present Capacity: 140631048192 (130.97 GB)
DFS Remaining: 140630974464 (130.97 GB)
DFS Used: 73728 (72 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Live datanodes (3):
Name: 192.168.0.141:50010 (h2)
Hostname: h2
Decommission Status : Normal
Configured Capacity: 51653570560 (48.11 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 4776554496 (4.45 GB)
DFS Remaining: 46876991488 (43.66 GB)
DFS Used%: 0.00%
DFS Remaining%: 90.75%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 18 22:22:40 CST 2015
Name: 192.168.0.142:50010 (h3)
Hostname: h3
Decommission Status : Normal
Configured Capacity: 51653570560 (48.11 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 4776558592 (4.45 GB)
DFS Remaining: 46876987392 (43.66 GB)
DFS Used%: 0.00%
DFS Remaining%: 90.75%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 18 22:22:40 CST 2015
Name: 192.168.0.143:50010 (h4)
Hostname: h4
Decommission Status : Normal
Configured Capacity: 51653570560 (48.11 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 4776550400 (4.45 GB)
DFS Remaining: 46876995584 (43.66 GB)
DFS Used%: 0.00%
DFS Remaining%: 90.75%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 18 22:22:40 CST 2015
页:
[1]