MAC下hadoop开发环境搭建系列（二）

xiayu

 
Hadoop的安装，hadoop有三种安装方式：
1）Standalone 单机模式，仅用于调试
2）Pseudo-Distributed 伪分布模式，单节点上同时启动NameNode、DataNode、ResourceManager、NodeManager、Secondary Namenode等进程，模拟分布式运行的各个节点。
3）Fully-Distributed 完全分布式模式，真正的hadoop集群，由各司其职的多个节点组成。
 
HADOOP 单机模式安装
 
跟生产一致，hadoop版本选的是cloudera5.0.0版本，下载地址：http://archive.cloudera.com/cdh5/cdh/5/


＃解压安装包
tar zxvf hadoop-2.3.0-cdh5.0.0.tar
＃将hadoop拷贝到安装路径
cp -r hadoop-2.3.0-cdh5.0.0  ~/hadoop
#设置环境变量
cd ~/hadoop
vi  etc/hadoop/hadoop-env.sh
＃增加以下配置
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home
export HADOOP_PREFIX=~/hadoop
#使配置生效
source etc/hadoop/hadoop-env.sh
#这样单机模式就安装好了,验证安装结果是否正确

 
MacBook-Pro:hadoop Administrator$ bin/hadoop version
Hadoop 2.3.0-cdh5.0.0
Subversion git://github.sf.cloudera.com/CDH/cdh.git -r 8e266e052e423af592871e2dfe09d54c03f6a0e8
Compiled by jenkins on 2014-03-28T04:29Z
Compiled with protoc 2.5.0
From source with checksum fae92214f92a3313887764456097e0
This command was run using /Users/Administrator/hadoop/share/hadoop/common/hadoop-common-2.3.0-cdh5.0.0.jar
也可以试着运行案例程序：
MacBook-Pro:hadoop Administrator$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.0.jar pi 10 100
Number of Maps  = 10
Samples per Map = 100
2015-12-17 11:29:44.665 java[1301:55605] Unable to load realm info from SCDynamicStore
2015-12-17 11:30:29,406 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
2015-12-17 11:30:29,768 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - session.id is deprecated. Instead, use dfs.metrics.session-id
.......
2015-12-17 11:30:31,483 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1365)) - Job job_local671517986_0001 running in uber mode : false
2015-12-17 11:30:31,486 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1372)) -  map 100% reduce 100%
2015-12-17 11:30:31,488 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1383)) - Job job_local671517986_0001 completed successfully
2015-12-17 11:30:31,517 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1390)) - Counters: 30
File System Counters
FILE: Number of bytes read=3063418
FILE: Number of bytes written=5427585
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=10
Map output records=20
Map output bytes=180
Map output materialized bytes=280
Input split bytes=1470
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=280
Reduce input records=20
Reduce output records=0
Spilled Records=40
Shuffled Maps =10
Failed Shuffles=0
Merged Map outputs=10
GC time elapsed (ms)=99
Total committed heap usage (bytes)=4572839936
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1300
File Output Format Counters
Bytes Written=109
Job Finished in 1.778 seconds
Estimated value of Pi is 3.14800000000000000000
 
纪录实际操作过程                                 
内容在个人公众号mangrendd同步更新