Hadoop Pipes程序运行Server failed to authenticate错误解决
运行hadoop自带的pipes examples没有问题,自己写个却在jobtracker界面中报了Server failed to authenticate. Exiting错误.去日志中看下,完整异常如下:1.job日志
java.io.IOException
at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188)
at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194)
at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149)
at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:390)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
2.task日志
2012-10-31 15:37:10,622 ERROR org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:250)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:299)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:320)
at org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:121)
2012-10-31 15:37:10,626 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2012-10-31 15:37:10,667 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException
at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188)
at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194)
at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149)
at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:390)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
2012-10-31 15:37:12,224 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
看来是权限的问题,那为什么我运行hadoop自带的pipes example就不会有问题呢？如此应该是环境方面的问题了.于是跑到$HADOOP_HOME/src/examples/org/pipes中的Makefile.in和configure看到的hadoop环境是默认的/usr/local/include和/usr/local/lib(前提是pipes，libhdfs,utils等已经安装).而我自己用c++写的mapreduce所配置的hadoop编译环境是$HADOOP_HOME/c++/include和$HADOOP_HOME/c++/lib，这个是hadoop默认的。于是改下自己的hadoop编译环境,发现便可以正常运行。
这里也有一篇解决方法,但针对的是你用hadoop自带的pipes example也报这个错误。则需要采用这个方法了。http://www.linuxquestions.org/questions/linux-software-2/hadoop-1-0-3-pipes-server-failed-to-authenticate-4175429779/。英文的,同时为了防止被删除或者被墙,我这里也记录下。
1.在/etc/profile添加一个变量,命令如下：
export LIB=-lcrypto
2.修改$HADOOP_HOME/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java:
patch如下：
Index: src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java
===================================================================
--- src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java(revision 1340233)
+++ src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java(working copy)
@@ -613,10 +613,10 @@
}
}
-private <T> String getEnumValues(Enum<? extends T>[] e) {
+private String getEnumValues(Enum<?>[] e) {
StringBuilder sb = new StringBuilder();
String sep = "";
- for (Enum<? extends T> v : e) {
+ for (Enum<?> v : e) {
sb.append(sep);
sb.append(v.name());
sep = "|";
3.安装c++/utils,到$(HADOOP_HOME)/src/c++/utils运行如下命令:
./configure
make install
4.安装c++/pipes,到$(HADOOP_HOME)/src/c++/pipes运行如下命令:
./configure
make install
5.再你的Makefile中使用如下设置:
-I$(HADOOP_HOME)/src/c++/install/include
-L$(HADOOP_HOME)/src/c++/install/lib -lhadooputils -lhadooppipes -lcrypto -lssl -lpthread
或者你不用修改你的Makefile,只要将$(HADOOP_HOME)/src/c++/install下的东西替换到$HADOOP_HOME/c++即可.
更多文章、感悟、分享、勾搭，请用微信扫描:
页:
[1]