hadoop on windows with Eclipse

15468

　　http://ebiquity.umbc.edu/Tutorials/Hadoop/00%20-%20Intro.html

hadoop无法正常启动(1) 执行  $ bin/hadoop start-all.sh之后,无法启动. 异常一    Exception in thread "main" java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority. localhost:      at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:214) localhost:      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:135) localhost:      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:119) localhost:      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:481) 解决方法： 此时是没有配置conf/mapred-site.xml的缘故.  在0.21.0版本上是配置mapred-site.xml,在之前的版本是配置core-site.xml,0.20.2版本中配置mapred-site.xml无效,只能配置core-site.xml文件 <configuration>   <property>     <name>fs.default.name</name>     <value>hdfs://localhost:9000</value>   </property>   <property>     <name>mapred.job.tracker</name>     <value>hdfs://localhost:9001</value>   </property>   <property>     <name>dfs.replication</name>     <value>1</value>   </property> </configuration>   hadoop无法正常启动(2) 异常二、   starting namenode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-namenode-aist.out localhost: starting datanode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-datanode-aist.out localhost: starting secondarynamenode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-secondarynamenode-aist.out localhost: Exception in thread "main" java.lang.NullPointerException localhost:      at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134) localhost:      at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156) localhost:      at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160) localhost:      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:131) localhost:      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:115) localhost:      at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:469) starting jobtracker, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-jobtracker-aist.out localhost: starting tasktracker, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-tasktracker-aist.out     解决方法： 同上         hadoop无法正常启动(3) 异常三、   starting namenode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-namenode-aist.out localhost: starting datanode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-datanode-aist.out localhost: Error: JAVA_HOME is not set. localhost: starting secondarynamenode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-secondarynamenode-aist.out localhost: Error: JAVA_HOME is not set. starting jobtracker, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-jobtracker-aist.out localhost: starting tasktracker, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-tasktracker-aist.out localhost: Error: JAVA_HOME is not set.   解决方法: 请在$hadoop/conf/hadoop-env.sh文件中配置JDK的环境变量   export  JAVA_HOME=/home/xixitie/jdk export  CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar   hadoop无法正常启动(4)   异常四:mapred-site.xml配置中使用hdfs://localhost:9001，而不使用localhost:9001的配置 异常信息如下： 11/04/20 23:33:25 INFO security.Groups: Group mapping impl=org.apache.hadoop.sec                                                                             urity.ShellBasedUnixGroupsMapping; cacheTimeout=300000 11/04/20 23:33:25 WARN fs.FileSystem: "localhost:9000" is a deprecated filesyste                                                                             m name. Use "hdfs://localhost:9000/" instead. 11/04/20 23:33:25 WARN conf.Configuration: mapred.task.id is deprecated. Instead                                                                             , use mapreduce.task.attempt.id 11/04/20 23:33:25 WARN fs.FileSystem: "localhost:9000" is a deprecated filesyste                                                                             m name. Use "hdfs://localhost:9000/" instead. 11/04/20 23:33:25 WARN fs.FileSystem: "localhost:9000" is a deprecated filesyste                                                                             m name. Use "hdfs://localhost:9000/" instead.   解决方法： mapred-site.xml配置中使用hdfs://localhost:9000，而不使用localhost:9000的配置     <property>     <name>fs.default.name</name>     <value>hdfs://localhost:9000</value>   </property>   <property>     <name>mapred.job.tracker</name>     <value>hdfs://localhost:9001</value>   </property>     hadoop无法正常启动(5) 异常五、no namenode to stop 问题的解决: 异常信息如下： 11/04/20 21:48:50 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 0 time(s). 11/04/20 21:48:51 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 1 time(s). 11/04/20 21:48:52 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 2 time(s). 11/04/20 21:48:53 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 3 time(s). 11/04/20 21:48:54 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 4 time(s). 11/04/20 21:48:55 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 5 time(s). 11/04/20 21:48:56 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 6 time(s). 11/04/20 21:48:57 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 7 time(s). 11/04/20 21:48:58 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                             .1:9000. Already tried 8 time(s).   解决方法: 这个问题是由namenode没有启动起来引起的,为什么no namenode to stop,可能之前的一些数据对namenode有影响, $ bin/hadoop namenode -format $bin/hadoop start-all.sh   hadoop无法正常启动(6) 异常五、no datanode to stop 问题的解决: 有时数据结构出现问题会产生无法启动datanode的问题。 然后用 hadoop namenode -format  重新格式化后仍然无效，/tmp中的文件并没有清楚。 其实还需要清除/tmp/hadoop*里的文件。         hadoop  fs -rmr /tmp            stop-all.sh           rm -rf /tmp/hadoop*           hadoop namenode -format            start-all.sh   搞了好几天终于把window下hadoop配置给搞好了，真不是搞技术的料啊！！！！