hadoop2.2.0遇到64位操作系统平台报错
遇到的问题$ hadoop dfs -put ./in
DEPRECATED: Use of this script to executehdfs command is deprecated.
Instead use the hdfs command for it.
Java HotSpot(TM) 64-BitServer VM warning: You have loaded library/app/hadoop/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might havedisabled stack guard. The VM will try to fix the stack guard now.
It's highly recommendedthat you fix the library with 'execstack -c <libfile>', or link it with'-z noexecstack'.
13/10/24 04:08:55 WARNutil.NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-java classes where applicable
put: `in': No such file or directory
查看本地文件
$ file /app/hadoop/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0
/app/hadoop/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0:ELF 32-bit LSB shared object, Intel 80386,version 1 (SYSV), dynamically linked, not stripped
貌似是32位和64位的原因
http://mail-archives.apache.org/mod_mbox/hadoop-user/201208.mbox/%3C19AD42E3F64F0F468A305399D0DF39D92EA4521578@winops07.win.compete.com%3E
http://www.mail-archive.com/common-issues@hadoop.apache.org/msg52576.html
操作系统64位，软件是32位。悲剧了。。。装好的集群没法用。
解决方法：重新编译hadoop
解决方法，就是重新编译hadoop软件：
载程序代码
机器得连网，如果没联网找可以联网的机器下载，但是编译时还是要下载一些东西，所以，实在不行。最好找相同平台（可以是虚拟机）能上网的机器做下面工作，弄好了再拷回来。
# svn checkout'http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0'
都下载到这了：
$ ls
BUILDING.txt hadoop-common-project hadoop-maven-plugins hadoop-tools
dev-support hadoop-dist hadoop-minicluster hadoop-yarn-project
hadoop-assemblies hadoop-hdfs-project hadoop-project pom.xml
hadoop-client hadoop-mapreduce-project hadoop-project-dist
安装开发环境
1.必要的包
# yum install svn
# yum install autoconfautomake libtool cmake
root@hadoop01 ~]# yum install ncurses-devel
root@hadoop01 ~]# yum install openssl-devel
root@hadoop01 ~]# yum install gcc*
2.安装maven
下载，并解压
http://maven.apache.org/download.cgi
# mvapache-maven-3.1.1 /usr/local/
将/usr/local/apache-maven-3.1.1/bin加到环境变量中
3.安装protobuf
没装 protobuf,后面编译做不完，结果如下：
---hadoop-maven-plugins:2.2.0:protoc (compile-protoc) @ hadoop-common ---
failed:java.io.IOException: Cannot run program "protoc": error=2, No suchfile or directory
stdout: []
……………………
Apache Hadoop Main................................ SUCCESS
Apache Hadoop Project POM......................... SUCCESS
Apache Hadoop Annotations......................... SUCCESS
Apache Hadoop Assemblies.......................... SUCCESS
Apache Hadoop Project Dist POM.................... SUCCESS
Apache Hadoop Maven Plugins....................... SUCCESS
Apache Hadoop Auth................................ SUCCESS
Apache Hadoop Auth Examples....................... SUCCESS
Apache HadoopCommon .............................. FAILURE
Apache Hadoop NFS................................. SKIPPED
Apache Hadoop Common Project...................... SKIPPED
Apache Hadoop HDFS................................ SKIPPED
Apache Hadoop HttpFS.............................. SKIPPED
Apache Hadoop HDFS BookKeeperJournal ............. SKIPPED
Apache Hadoop HDFS-NFS............................ SKIPPED
Apache Hadoop HDFS Project........................ SKIPPED
安装protobuf过程
下载：https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
https://code.google.com/p/protobuf/downloads/list
# pwd
/soft/protobuf-2.5.0
依次执行下面的命令即可
./configure
make
make check
make install
# protoc--version
libprotoc 2.5.0
4.cmake安装
CMAKE报错：
main:
Created dir:/soft/hadoop/hadoop-tools/hadoop-pipes/target/native
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
CMake Error at /usr/share/cmake/Modules/FindOpenSSL.cmake:66(MESSAGE):
Could NOT find OpenSSL
Call Stack (most recent call first):
CMakeLists.txt:20(find_package)
-- Configuring incomplete, errors occurred!
Apache Hadoop Gridmix............................. SUCCESS
Apache Hadoop Data Join........................... SUCCESS
Apache Hadoop Extras.............................. SUCCESS
Apache Hadoop Pipes ...............................FAILURE
Apache Hadoop Tools Dist.......................... SKIPPED
Apache Hadoop Tools............................... SKIPPED
Apache Hadoop Distribution........................ SKIPPED
Apache Hadoop Client.............................. SKIPPED
Apache Hadoop Mini-Cluster........................ SKIPPED
需要安装
root@hadoop01 ~]# yum install ncurses-devel
root@hadoop01 ~]# yum install openssl-devel
编译hadoop
$ pwd
/soft/hadoop
$ ls
BUILDING.txt hadoop-client hadoop-hdfs-project hadoop-minicluster hadoop-tools
dev-support hadoop-common-project hadoop-mapreduce-project hadoop-project hadoop-yarn-project
hadoop-assemblies hadoop-dist hadoop-maven-plugins hadoop-project-dist pom.xml
$ mvn package -Pdist,native -DskipTests -Dtar
目前的2.2.0 的Source Code 压缩包解压出来的code有个bug 需要patch后才能编译。否则编译hadoop-auth 会提示上面错误。
解决办法如下：
修改下面的pom文件。该文件在hadoop源码包下寻找：
hadoop-common-project/hadoop-auth/pom.xml
打开上面的的pom文件，在54行加入如下的依赖：（如果没有的话）
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty</artifactId>
<scope>test</scope>
</dependency>
编译是个很耗时的工作呀。。。。
下面是做完成功的结果
Reactor Summary:
Apache Hadoop Main................................ SUCCESS
Apache Hadoop Project POM......................... SUCCESS
Apache Hadoop Annotations......................... SUCCESS
Apache Hadoop Assemblies.......................... SUCCESS
Apache Hadoop Project Dist POM.................... SUCCESS
Apache Hadoop Maven Plugins....................... SUCCESS
Apache Hadoop Auth................................ SUCCESS
Apache Hadoop Auth Examples....................... SUCCESS
Apache Hadoop Common.............................. SUCCESS
Apache Hadoop NFS................................. SUCCESS
Apache Hadoop Common Project...................... SUCCESS
Apache Hadoop HDFS................................ SUCCESS
Apache Hadoop HttpFS.............................. SUCCESS
Apache Hadoop HDFS BookKeeperJournal ............. SUCCESS
Apache Hadoop HDFS-NFS............................ SUCCESS
Apache Hadoop HDFS Project........................ SUCCESS
hadoop-yarn....................................... SUCCESS
hadoop-yarn-api................................... SUCCESS
hadoop-yarn-common................................ SUCCESS
hadoop-yarn-server................................ SUCCESS
hadoop-yarn-server-common......................... SUCCESS
hadoop-yarn-server-nodemanager.................... SUCCESS
hadoop-yarn-server-web-proxy...................... SUCCESS
hadoop-yarn-server-resourcemanager................ SUCCESS
hadoop-yarn-server-tests.......................... SUCCESS
hadoop-yarn-client................................ SUCCESS
hadoop-yarn-applications.......................... SUCCESS
hadoop-yarn-applications-distributedshell ......... SUCCESS
hadoop-mapreduce-client........................... SUCCESS
hadoop-mapreduce-client-core...................... SUCCESS
hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS
hadoop-yarn-site.................................. SUCCESS
hadoop-yarn-project............................... SUCCESS
hadoop-mapreduce-client-common.................... SUCCESS
hadoop-mapreduce-client-shuffle................... SUCCESS
hadoop-mapreduce-client-app....................... SUCCESS
hadoop-mapreduce-client-hs........................ SUCCESS
hadoop-mapreduce-client-jobclient................. SUCCESS
hadoop-mapreduce-client-hs-plugins................ SUCCESS
Apache Hadoop MapReduce Examples.................. SUCCESS
hadoop-mapreduce.................................. SUCCESS
Apache Hadoop MapReduce Streaming................. SUCCESS
Apache Hadoop Distributed Copy.................... SUCCESS
Apache Hadoop Archives............................ SUCCESS
Apache Hadoop Rumen............................... SUCCESS
Apache Hadoop Gridmix............................. SUCCESS
Apache Hadoop Data Join........................... SUCCESS
Apache Hadoop Extras.............................. SUCCESS
Apache Hadoop Pipes............................... SUCCESS
Apache Hadoop Tools Dist.......................... SUCCESS
Apache Hadoop Tools............................... SUCCESS
Apache Hadoop Distribution........................ SUCCESS
Apache Hadoop Client.............................. SUCCESS
Apache Hadoop Mini-Cluster........................ SUCCESS
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time: 29:07.811s
Finished at: Thu Oct 24 09:43:18 CST2013
Final Memory: 78M/239M
------------------------------------------------------------------------
使用用编译好的软件再执行一次
$ hadoop dfs -put ./in
DEPRECATED: Use of this script to executehdfs command is deprecated.
Instead use the hdfs command for it.
13/10/24 15:12:53 WARNutil.NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-java classes where applicable
put: `in': No such file or directory
加入环境变量
export HADOOP_HOME=/usr/hadoop/hadoop-2.2.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native
hadoop为2.2.0，操作系统为oracle linux 6.3 64位。
$ hadoop dfs -put ./in
DEPRECATED: Use of this script to executehdfs command is deprecated.
Instead use the hdfs command for it.
13/10/24 15:12:53 WARNutil.NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-java classes where applicable
put: `in': No such file or directory
最后一行“put:`in': No such file or directory”先不管，肯定是语法命令有问题。
先解决“WARN util.NativeCodeLoader: Unable to loadnative-hadoop library for your platform... using builtin-java classes whereapplicable”
备注：我的hadoop环境是自己编译的，因为64位操作系统，hadoop2.2.0貌似只有32位的软件。
解决过程
1.开启debug
$ export HADOOP_ROOT_LOGGER=DEBUG,console
$ hadoop dfs -put./in
DEPRECATED: Use of this script to executehdfs command is deprecated.
Instead use the hdfs command for it.
13/10/24 16:11:31 DEBUG util.Shell: setsidexited with exit code 0
13/10/24 16:11:31 DEBUGlib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRateorg.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess withannotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time,value=, about=,type=DEFAULT, always=false, sampleName=Ops)
13/10/24 16:11:31 DEBUGlib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRateorg.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure withannotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time,value=, about=,type=DEFAULT, always=false, sampleName=Ops)
13/10/24 16:11:31 DEBUGimpl.MetricsSystemImpl: UgiMetrics, User and group related metrics
13/10/24 16:11:32 DEBUGsecurity.Groups: Creating new Groupsobject
13/10/24 16:11:32 DEBUGutil.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
13/10/24 16:11:32 DEBUGutil.NativeCodeLoader: Failed to load native-hadoopwith error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
13/10/24 16:11:32 DEBUGutil.NativeCodeLoader: java.library.path=/usr/java/jdk1.7.0_45/lib:/app/hadoop/hadoop-2.2.0/lib/native:/app/hadoop/hadoop-2.2.0/lib/native
13/10/24 16:11:32 WARNutil.NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-java classes where applicable
13/10/24 16:11:32 DEBUG security.JniBasedUnixGroupsMappingWithFallback:Falling back to shell based
13/10/24 16:11:32 DEBUGsecurity.JniBasedUnixGroupsMappingWithFallback: Group mappingimpl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
13/10/24 16:11:32 DEBUG security.Groups:Group mappingimpl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback;cacheTimeout=300000
13/10/24 16:11:32 DEBUGsecurity.UserGroupInformation: hadoop login
13/10/24 16:11:32 DEBUGsecurity.UserGroupInformation: hadoop login commit
13/10/24 16:11:32 DEBUGsecurity.UserGroupInformation: using local user:UnixPrincipal: hadoop
13/10/24 16:11:32 DEBUGsecurity.UserGroupInformation: UGI loginUser:hadoop (auth:SIMPLE)
13/10/24 16:11:33 DEBUGhdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
13/10/24 16:11:33 DEBUGhdfs.BlockReaderLocal: dfs.client.read.shortcircuit = false
13/10/24 16:11:33 DEBUGhdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
13/10/24 16:11:33 DEBUGhdfs.BlockReaderLocal: dfs.domain.socket.path =
13/10/24 16:11:33 DEBUGimpl.MetricsSystemImpl: StartupProgress, NameNode startup progress
13/10/24 16:11:33 DEBUG retry.RetryUtils:multipleLinearRandomRetry = null
13/10/24 16:11:33 DEBUG ipc.Server:rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=classorg.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWrapper,rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@2e41d9a2
13/10/24 16:11:34 DEBUGhdfs.BlockReaderLocal: Both short-circuit local reads and UNIX domain socketare disabled.
13/10/24 16:11:34 DEBUG ipc.Client: Theping interval is 60000 ms.
13/10/24 16:11:34 DEBUG ipc.Client:Connecting to localhost/127.0.0.1:8020
13/10/24 16:11:34 DEBUG ipc.Client: IPCClient (2141757401) connection to localhost/127.0.0.1:8020 from hadoop:starting, having connections 1
13/10/24 16:11:34 DEBUG ipc.Client: IPCClient (2141757401) connection to localhost/127.0.0.1:8020 from hadoop sending#0
13/10/24 16:11:34 DEBUG ipc.Client: IPCClient (2141757401) connection to localhost/127.0.0.1:8020 from hadoop gotvalue #0
13/10/24 16:11:34 DEBUGipc.ProtobufRpcEngine: Call: getFileInfo took 82ms
13/10/24 16:11:34 DEBUG ipc.Client: IPCClient (2141757401) connection to localhost/127.0.0.1:8020 from hadoop sending#1
13/10/24 16:11:34 DEBUG ipc.Client: IPCClient (2141757401) connection to localhost/127.0.0.1:8020 from hadoop gotvalue #1
13/10/24 16:11:34 DEBUGipc.ProtobufRpcEngine: Call: getFileInfo took 4ms
put: `.': No such file or directory
13/10/24 16:11:34 DEBUG ipc.Client:Stopping client
13/10/24 16:11:34 DEBUG ipc.Client: IPCClient (2141757401) connection to localhost/127.0.0.1:8020 from hadoop: closed
13/10/24 16:11:34 DEBUG ipc.Client: IPCClient (2141757401) connection to localhost/127.0.0.1:8020 from hadoop:stopped, remaining connections 0
上述debug中的错误 :
Failed to load native-hadoop with error:java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
为了解决这个错误，尝试过很多种办法，很多都是对环境变量的修改。都是一筹莫展。。
加入环境变量
export HADOOP_HOME=/usr/hadoop/hadoop-2.2.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native
2.解决方法
最后详细读了官方的NativeLibraries文档。
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/NativeLibraries.html
Either download a hadoop release, whichwill include a pre-built version of the native hadoop library, or build yourown version of the native hadoop library. Whether you download or build, thename for the library is the same: libhadoop.so
发现人家要的名字是这个libhadoop.so，检查我的目录，有libhadoop.so.1.0.0这个。看了官方编译的软件，确实有那个libhadoop.so文件，但只是个link，所以照做
$ ln -slibhadoop.so.1.0.0 libhadoop.so
$ ln -s libhdfs.so.0.0.0libhdfs.so
问题解决了。
$ hadoop dfs -put./in
DEPRECATED: Use of this script to executehdfs command is deprecated.
Instead use the hdfs command for it.
put: `.': No such file or directory
页:
[1]