hadoop hdfs balance

yinian

转载：http://hi.baidu.com/hovlj_1130/blog/item/20200da530603af99052eed9.html

hadoop hdfs balance

最近集群的某些dump节点容量报警，hadoop dfsadmin -report可以发现，报警的节点，均为日志上传至hdfs的节点，在hadoop fs -put时，大部分hdfs文件都直接保存在了put的节点.
因此需要使用hadoop Balancer Administrator Guide.

下面是E文描述：

The balancer is a tool that  balances disk space usage on an HDFS cluster when some
datanodes become full or when new empty nodes join the cluster. The tool is deployed as
an application program that can be run by the cluster administrator on a live HDFS
cluster while application adding and deleting files.

SYNOPSIS

To start:
bin/start-balancer.sh [-threshold <threshold>]
Example: bin/ start-balancer.sh
start the balancer with a default threshold of 10%
bin/ start-balancer.sh -threshold 5
start the balancer with a threshold of 5%

To stop:
bin/ stop-balancer.sh

DESCRIPTION

The threshold parameter is a fraction in the range of (0%, 100%) with a default value of
10%. The threshold sets a target for whether the cluster is balanced. A cluster is balanced
if for each datanode, the utilization of the node (ratio of used space at the node to total
capacity of the node) differs from the utilization of the cluster (ratio of used space in the
cluster to total capacity of the cluster) by no more than the threshold value. The smaller
the threshold, the more balanced a cluster will become. It takes more time to run the
balancer for small threshold values. Also for a very small threshold the cluster may not be
able to reach the balanced state when applications write and delete files concurrently.

The  tool moves  blocks from  highly utilized datanodes  to  poorly utilized datanodes
iteratively. In each iteration a datanode moves or receives no more than the lesser of 10G
bytes or the threshold fraction of its capacity. Each iteration runs no more than 20
minutes. At the end of each iteration, the balancer obtains updated datanodes information
from the namenode.

A system property that limits the balancer’s use of bandwidth is defined in the default
configuration file:

<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>1048576</value>
<description>  Specifies the maximum bandwidth that each datanode can utilize for the
balancing purpose in term of the number of bytes per second. </description>
</property>

This property determines the maximum speed at which a block will be moved from one
datanode to another. The default value is 1MB/s. The higher the bandwidth, the faster a
cluster  can reach the  balanced  state,  but  with  greater competition   with application
processes. If an administrator changes the value of this property in the configuration file,
the change is observed when HDFS is next restarted.

在使用start-balancer.sh时，如果在hdfs-site.xml里面没有配置dfs.balance.bandwidthPerSec，那么集群hdfs内部默认使用1M/S的速度移动数据(so slowly...)，我们可以通过在hdfs-site.xml里面配置dfs.balance.bandwidthPerSec来加快balance的速度。

最开始我们配置的是20m/S
<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>20971520</value>
<description>  Specifies the maximum bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second. </description>
</property>

然后结果是导致job运行变得不稳定，出现一些意外的长map单元，某些reduce时间处理变长(整个集群负载满满的情况下，外加20m/s的balance)，在前天的中国hadoop年会上听淘宝调整的为10m/s，需要调整后实验，看看情况如何。

另外再修改dfs.balance.bandwidthPerSec参数后，需要在namenode上运行stop-dfs.sh start-dfs.sh重启hdfs生效。