32位hadoop编译实现与64位操作系统兼容
没有安装过集群的朋友，可能没有发现，hadoop版本没有64位的，我们在安装hadoop之前需要将hadoop源码包进行编译，否则lib下的部分jar包无法使用【有人可能会说hadoop不分操作系统的bit数，这个问题我有怎么会悄悄告诉你呢！！！！哈哈，开玩笑，接下来，给大家分享一下我第一次编译出现的糗事】如果不编译会出现啥问题呢？？你可以看俺
遇到的问题描述：
# hadoop dfs -put ./in
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
14/07/17 17:07:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
put: `./in': No such file or directory
原因查找：
查看本地文件：
# file /usr/local/hadoop/lib/native/libhadoop.so.1.0.0
/usr/local/hadoop/lib/native/libhadoop.so.1.0.0: ELF 32-bit LSB shared object,
Intel 80386, version 1 (SYSV), dynamically linked, not stripped
是32位的hadoop，安装在了64位的linux系统上。lib包编译环境不一样，所以不能使用。
悲剧了，装好的集群没法用
【编译环境准备】
要重新指定yum源
网上下载了version-groups.conf到/etc/yum/，删除系统原有的源文件
1. 安装必要的包：
# yum install autoconfautomake libtool cmake ncurses-devel openssl-devel gcc* --nogpgcheck
2. 安装maven,下载并解压。（版本不能比这新，有压缩包）
http://maven.apache.org/download.cgi //下载对应的压缩包
apache-maven-3.2.1-bin.tar
# tar -xvf apache-maven-3.2.1-bin.tar
# tar -xvf apache-maven-3.2.1-bin.tar
# ln -s /usr/local/apache-maven-3.2.1/ /usr/local/maven
# vim /etc/profile //添加环境变量中
export MAVEN_HOME=/usr/local/maven
export PATH=$MAVEN_HOME/bin:$PATH
3. 安装protobuf（版本不要变，有压缩包）
https://code.google.com/p/protobuf/downloads/detail?name=protobuf-2.5.0.tar.gz
下载：protobuf-2.5.0.tar.gz 并解压
# pwd
/root/protobuf-2.5.0
# ./configure --prefix=/usr/local/protoc/
# make
# make check
# make install
bin目录下执行 protoc --version
libprotoc 2.5.0
安装成功。
添加环境变量：
vi /etc/profile
export MAVEN_HOME=/usr/local/maven
export JAVA_HOME=/usr/java/latest
export HADOOP_HOME=/usr/local/hadoop
export PATH=.:/usr/local/protoc/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
4. 编译hadoop（将hadoop源码包解压就会出现如下的包）
# pwd
/data/release-2.2.0
# ls
BUILDING.txt hadoop-common-project hadoop-maven-plugins hadoop-tools
dev-support hadoop-dist hadoop-minicluster hadoop-yarn-project
hadoop-assemblies hadoop-hdfs-project hadoop-project pom.xml
hadoop-client hadoop-mapreduce-project hadoop-project-dist
# mvn package -Pdist,native -DskipTests -Dtar （往后看不要急着编译，这是maven命令）
..............编译需要较长时间大概1个小时左右。
如果出现如下错误：
Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hadoop-auth: Compilation failure: Compilation failure:
/home/hduser/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java: error: cannot access AbstractLifeCycle
class file for org.mortbay.component.AbstractLifeCycle not found
/home/hduser/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java: error: cannot access LifeCycle
class file for org.mortbay.component.LifeCycle not found
/home/hduser/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java: error: cannot find symbol
symbol: method start()
location: variable server of type Server
/home/hduser/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java: error: cannot find symbol
->
需要修改源码下边的hadoop-common-project/hadoop-auth/pom.xml
vi ~/hadoop-common-project/hadoop-auth/pom.xml
# vim /data/release-2.2.0/hadoop-common-project/hadoop-auth/pom.xml
在第55行下添加：
56 <dependency>
57 <groupId>org.mortbay.jetty</groupId>
58 <artifactId>jetty-util</artifactId>
59 <scope>test</scope>
60 </dependency>
保存退出，重新编译即可。
最后编译成功：
--- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-minicluster ---
Building jar: /data/release-2.2.0/hadoop-minicluster/target/hadoop-minicluster-2.2.0-javadoc.jar
------------------------------------------------------------------------
Reactor Summary:
Apache Hadoop Main ................................ SUCCESS [ 1.386 s]
Apache Hadoop Project POM ......................... SUCCESS [ 1.350 s]
Apache Hadoop Annotations ......................... SUCCESS [ 2.732 s]
Apache Hadoop Assemblies .......................... SUCCESS [ 0.358 s]
Apache Hadoop Project Dist POM .................... SUCCESS [ 2.048 s]
Apache Hadoop Maven Plugins ....................... SUCCESS [ 3.450 s]
Apache Hadoop Auth ................................ SUCCESS [ 16.114 s]
Apache Hadoop Auth Examples ....................... SUCCESS [ 13.317 s]
Apache Hadoop Common .............................. SUCCESS
Apache Hadoop NFS ................................. SUCCESS [ 16.925 s]
Apache Hadoop Common Project ...................... SUCCESS [ 0.044 s]
Apache Hadoop HDFS ................................ SUCCESS
Apache Hadoop HttpFS .............................. SUCCESS [ 28.601 s]
Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [ 27.589 s]
Apache Hadoop HDFS-NFS ............................ SUCCESS [ 3.966 s]
Apache Hadoop HDFS Project ........................ SUCCESS [ 0.044 s]
hadoop-yarn ....................................... SUCCESS [ 52.846 s]
hadoop-yarn-api ................................... SUCCESS [ 41.700 s]
hadoop-yarn-common ................................ SUCCESS [ 25.945 s]
hadoop-yarn-server ................................ SUCCESS [ 0.105 s]
hadoop-yarn-server-common ......................... SUCCESS [ 8.436 s]
hadoop-yarn-server-nodemanager .................... SUCCESS [ 15.659 s]
hadoop-yarn-server-web-proxy ...................... SUCCESS [ 3.647 s]
hadoop-yarn-server-resourcemanager ................ SUCCESS [ 12.495 s]
hadoop-yarn-server-tests .......................... SUCCESS [ 0.684 s]
hadoop-yarn-client ................................ SUCCESS [ 5.266 s]
hadoop-yarn-applications .......................... SUCCESS [ 0.102 s]
hadoop-yarn-applications-distributedshell ......... SUCCESS [ 2.666 s]
hadoop-mapreduce-client ........................... SUCCESS [ 0.093 s]
hadoop-mapreduce-client-core ...................... SUCCESS [ 20.092 s]
hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [ 2.783 s]
hadoop-yarn-site .................................. SUCCESS [ 0.225 s]
hadoop-yarn-project ............................... SUCCESS [ 36.636 s]
hadoop-mapreduce-client-common .................... SUCCESS [ 16.645 s]
hadoop-mapreduce-client-shuffle ................... SUCCESS [ 3.058 s]
hadoop-mapreduce-client-app ....................... SUCCESS [ 9.441 s]
hadoop-mapreduce-client-hs ........................ SUCCESS [ 5.482 s]
hadoop-mapreduce-client-jobclient ................. SUCCESS [ 7.615 s]
hadoop-mapreduce-client-hs-plugins ................ SUCCESS [ 2.473 s]
Apache Hadoop MapReduce Examples .................. SUCCESS [ 6.183 s]
hadoop-mapreduce .................................. SUCCESS [ 6.454 s]
Apache Hadoop MapReduce Streaming ................. SUCCESS [ 4.802 s]
Apache Hadoop Distributed Copy .................... SUCCESS [ 27.635 s]
Apache Hadoop Archives ............................ SUCCESS [ 2.850 s]
Apache Hadoop Rumen ............................... SUCCESS [ 6.092 s]
Apache Hadoop Gridmix ............................. SUCCESS [ 4.742 s]
Apache Hadoop Data Join ........................... SUCCESS [ 3.155 s]
Apache Hadoop Extras .............................. SUCCESS [ 3.317 s]
Apache Hadoop Pipes ............................... SUCCESS [ 9.791 s]
Apache Hadoop Tools Dist .......................... SUCCESS [ 2.680 s]
Apache Hadoop Tools ............................... SUCCESS [ 0.036 s]
Apache Hadoop Distribution ........................ SUCCESS [ 20.765 s]
Apache Hadoop Client .............................. SUCCESS [ 6.476 s]
Apache Hadoop Mini-Cluster ........................ SUCCESS [ 0.215 s]
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time: 16:32 min
Finished at: 2014-07-18T01:18:24+08:00
Final Memory: 117M/314M
------------------------------------------------------------------------
此时编译好的文件位于 ~/hadoop-dist/target/hadoop-2.2.0/ 目录中
拷贝hadoop-2.2.0到安装目录下，/usr/local/ 重新修改其配置文件，重新并格式化，启动，即可。
到此已经不会报错，可以使用。
# hadoop dfs -put ./in
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
put: `.': No such file or directory
# file /usr/local/hadoop/lib/native/libhadoop.so.1.0.0
/usr/local/hadoop/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
测试使用：上传一个文件，下载一个文件，查看上传文件的内容：
# cat wwn.txt
# This is a text txt
# by coco
# 2014-07-18
# hdfs dfs -mkdir /test
# hdfs dfs -put wwn.txt /test
# hdfs dfs -cat /test/wwn.txt
# hdfs dfs -get /test/wwn.txt /tmp
# hdfs dfs -rm /test/wwn.txt
# ll
总用量 6924
-rw-r--r-- 1 root root 70 7月 18 11:50 wwn.txt
# hadoop dfs -ls /test
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Found 2 items
-rw-r--r-- 2 root supergroup 6970105 2014-07-18 11:44 /test/gc_comweight.txt
-rw-r--r-- 2 root supergroup 59 2014-07-18 14:56 /test/hello.txt
到此我们的hdfs文件系统已经能正常使用。
按照在64位CentOS上编译 Hadoop 2.2.0的步骤，进行对hadoop2.2在rhel6.2上进行编译，
cd hadoop-2.2.0-src
mvn package -DskipTests -Pdist,native -Dtar
大概10分钟左右时报错如下：
Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-project: Execution ……………………
使用以下命令从新编译：
mvn package -DskipTests -Pdist,native -Dtar -Dmaven.javadoc.skip=true
大概40分钟后，编译完成
INFO] --- maven-source-plugin:2.1.2:test-jar-no-fork (hadoop-java-sources) @ hadoop-minicluster ---
No sources in project. Archive not created.
--- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ hadoop-minicluster ---
--- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-minicluster ---
Skipping javadoc generation
------------------------------------------------------------------------
Reactor Summary:
Apache Hadoop Main ................................ SUCCESS
Apache Hadoop Project POM ......................... SUCCESS
Apache Hadoop Annotations ......................... SUCCESS
Apache Hadoop Assemblies .......................... SUCCESS
Apache Hadoop Project Dist POM .................... SUCCESS [ 43.968 s]
Apache Hadoop Maven Plugins ....................... SUCCESS [ 46.198 s]
Apache Hadoop Auth ................................ SUCCESS
Apache Hadoop Auth Examples ....................... SUCCESS [ 11.632 s]
Apache Hadoop Common .............................. SUCCESS
Apache Hadoop NFS ................................. SUCCESS [ 54.758 s]
Apache Hadoop Common Project ...................... SUCCESS
Apache Hadoop HDFS ................................ SUCCESS
Apache Hadoop HttpFS .............................. SUCCESS [ 11.402 s]
Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS
Apache Hadoop HDFS-NFS ............................ SUCCESS
Apache Hadoop HDFS Project ........................ SUCCESS
hadoop-yarn ....................................... SUCCESS
hadoop-yarn-api ................................... SUCCESS [ 43.147 s]
hadoop-yarn-common ................................ SUCCESS
hadoop-yarn-server ................................ SUCCESS
hadoop-yarn-server-common ......................... SUCCESS
hadoop-yarn-server-nodemanager .................... SUCCESS [ 42.176 s]
hadoop-yarn-server-web-proxy ...................... SUCCESS
hadoop-yarn-server-resourcemanager ................ SUCCESS
hadoop-yarn-server-tests .......................... SUCCESS
hadoop-yarn-client ................................ SUCCESS
hadoop-yarn-applications .......................... SUCCESS
hadoop-yarn-applications-distributedshell ......... SUCCESS
hadoop-mapreduce-client ........................... SUCCESS
hadoop-mapreduce-client-core ...................... SUCCESS
hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS
hadoop-yarn-site .................................. SUCCESS
hadoop-yarn-project ............................... SUCCESS [ 18.240 s]
hadoop-mapreduce-client-common .................... SUCCESS
hadoop-mapreduce-client-shuffle ................... SUCCESS
hadoop-mapreduce-client-app ....................... SUCCESS
hadoop-mapreduce-client-hs ........................ SUCCESS
hadoop-mapreduce-client-jobclient ................. SUCCESS
hadoop-mapreduce-client-hs-plugins ................ SUCCESS
Apache Hadoop MapReduce Examples .................. SUCCESS
hadoop-mapreduce .................................. SUCCESS
Apache Hadoop MapReduce Streaming ................. SUCCESS
Apache Hadoop Distributed Copy .................... SUCCESS [ 21.522 s]
Apache Hadoop Archives ............................ SUCCESS
Apache Hadoop Rumen ............................... SUCCESS
Apache Hadoop Gridmix ............................. SUCCESS
Apache Hadoop Data Join ........................... SUCCESS
Apache Hadoop Extras .............................. SUCCESS
Apache Hadoop Pipes ............................... SUCCESS
Apache Hadoop Tools Dist .......................... SUCCESS
Apache Hadoop Tools ............................... SUCCESS
Apache Hadoop Distribution ........................ SUCCESS
Apache Hadoop Client .............................. SUCCESS [ 16.353 s]
Apache Hadoop Mini-Cluster ........................ SUCCESS
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time: 35:15 min
Finished at: 2014-11-25T22:43:39+08:00
Final Memory: 95M/368M
------------------------------------------------------------------------
页:
[1]