Hadoop常见问题及解决方案

黄智勇

　　问题一：
2010-10-18 01:18:45,050 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /usr/local/hadoop/tmp/dfs/data: namenode namespaceID = 1501733340; datanode namespaceID = 1262603975
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:233)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:148)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)

解决方案：
1、停止集群
$./stop-all.sh
2、删除在hdfs中配置的data目录下面的所有数据;
$ rm -rf /usr/local/hadoop/filesystem/data/
3、重新格式化namenode
$ ./hadoop namenode -format
4、重新启动hadoop集群
$./start-all.sh

问题二：
WARNING : There are about 1 missing blocks. Please check the log or run fsck.
解决方案：
$ bin/hadoop fsck /
/home/zhaozheng/hdfs/README.txt: CORRUPT block blk_4085337189286784361

/home/zhaozheng/hdfs/README.txt: MISSING 1 blocks of total size 1366 B.Status: CORRUPT
Total size: 1366 B
Total dirs: 0
Total files: 1
Total blocks (validated): 1 (avg. block size 1366 B)
********************************
CORRUPT FILES: 1
MISSING BLOCKS: 1
MISSING SIZE:  1366 B
CORRUPT BLOCKS:  1
********************************
Minimally replicated blocks: 0 (0.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks:  0 (0.0 %)
Default replication factor: 2
Average block replication: 0.0
Corrupt blocks:  1
Missing replicas:  0
Number of data-nodes:  2
Number of racks:  1

$bin/hadoop dfs -rm /home/zhaozheng/hdfs/README.txt
$bin/hadoop fsck /
.Status: HEALTHY
Total size: 4 B
Total dirs: 12
Total files: 1
Total blocks (validated): 1 (avg. block size 4 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks:  0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks:  0
Missing replicas:  0 (0.0 %)
Number of data-nodes:  2
Number of racks:  1

问题三:
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /usr/local/hadoop/tmp/mapred/system. Name node is in safe mode.
解决方案：
$bin/hadoop dfsadmin -safemode leave #关闭safe mode
Safe mode is OFF