hadoop 安装和配置

dryu999

　　一、安装 JDK
　　$chmod +x jdk-6u38-linux-i586.bin
　　$./jdk-6u38-linux-i586.bin
　　安装完后设置java 环境变量如下
　　命令 ：/home路径下
　　$vi /etc/profile
　　然后添加如下语句
　　export JAVA_HOME=/home/java/jdk1.6.0_38
　　export PATH=$JAVA_HOME/bin:$PATH
　　export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
　　然后source /etc/profile 使其生效
　　二、ssh 的安装和配置
　　$实现ssh无密码登陆
　　$sudo apt-get install ssh
　　$ssh-keygen 
　　直接回车，完成后会在~/.ssh/生成两个文件：id_dsa 和id_dsa.pub。这两个是成对
　　出现，类似钥匙和锁。再把id_dsa.pub 追加到授权key 里面(当前并没有authorized_keys 
　　$cat~/.ssh/id_rsa.pub>>~/.ssh/authorized_keys完成后可以实现无密码登录本机
　　linux-6fp1:/useradd hadoop
　　linux-6fp1:/home # mkdir hadoop
　　hadoop@linux-6fp1:~> mkdir .ssh
　　hadoop@linux-6fp1:~> ls
　　hadoop@linux-6fp1:~> cd .ssh
　　hadoop@linux-6fp1:~/.ssh> ssh-keygen -t dsa -P '' -f id_dsa
　　Generating public/private dsa key pair.
　　Your identification has been saved in id_dsa.
　　Your public key has been saved in id_dsa.pub.
　　The key fingerprint is:
　　32:32:2a:39:a0:8f:c5:6e:f7:59:9b:b9:1a:ef:71:a6 hadoop@linux-6fp1
　　The key's randomart image is:
　　+--[ DSA 1024]----+
　　|                 |
　　|                 |
　　|                 |
　　|                 |
　　|.   o o S        |
　　|oo . o o         |
　　|= +   . o o      |
　　| B. .  = O       |
　　|..o. .+oE.       |
　　+-----------------+
　　hadoop@linux-6fp1:~/.ssh> ls
　　id_dsa  id_dsa.pub
　　hadoop@linux-6fp1:~/.ssh> cat id_dsa.pub >>authorized_keys
　　hadoop@linux-6fp1:~/.ssh> ll
　　total 12
　　-rw-r--r-- 1 hadoop users 607 2012-12-25 14:39 authorized_keys
　　-rw------- 1 hadoop users 668 2012-12-25 14:38 id_dsa
　　-rw-r--r-- 1 hadoop users 607 2012-12-25 14:38 id_dsa.pub
　　hadoop@linux-6fp1:~/.ssh> ssh -version
　　OpenSSH_5.1p1, OpenSSL 0.9.8h 28 May 2008
　　Bad escape character 'rsion'.
　　hadoop@linux-6fp1:~/.ssh> ssh localhost
　　The authenticity of host 'localhost (::1)' can't be established.
　　RSA key fingerprint is a4:93:5e:af:9f:52:41:ce:c8:44:8f:46:60:5f:68:95.
　　Are you sure you want to continue connecting (yes/no)? yes
　　Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
　　Have a lot of fun...
　　三、Hadoop 安装与配置
　　1.下载Hadoop
　　http://labs.mop.com/apache-mirror/hadoop/common/hadoop-1.0.0/
　　2.解压Hadoop
　　tar xzvf hadoop-0.21.0.tar.gz
　　3. 添加Hadoop Bin到环境变量中
　　修改 hadoop-env.sh
　　配置JDK即可
　　export JAVA_HOME=/home/java/jdk1.6.0_38 
　　vi conf/core-site.xml
　　<configuration>       
　　<property>    
　　<name>fs.default.name</name>    
　　<value>hdfs://master:9000</value>    
　　</property>    
　　<property>  
　　<name>hadoop.tmp.dir</name>  
　　<value>/home/hadoop/tmp</value>  
　　</property>  
　　</configuration>    
　　vi conf/hdfs-site.xml
　　<configuration>       
　　<property>    
　　<name>dfs.replication</name>    
　　<value>1</value>    
　　</property>    
　　</configuration>    
　　vi conf/mapred-site.xml
　　<configuration>       
　　<property>    
　　<name>mapred.job.tracker</name>    
　　<value>localhost:9001</value>    
　　</property>    
　　</configuration>    
　　vi conf/master
　　localhost 改为ip 192.168.1.102
　　vi conf/slaves 
　　localhost 改为  192.168.1.102
　　启动 Hadoop
　　1.格式化文件系统
　　hadoop namenode –format
　　2. 启动hadoop
　　启动关闭所有服务 start-all.sh/stop-all.sh
　　启动关闭HDFS: start-dfs.sh/stop-dfs.sh
　　启动关闭MapReduce:  start-mapred.sh/stop-mapred.sh
　　3. 用jps命令查看进程,确保有   namenode,dataNode,JobTracker,TaskTracker
　　hadoop@linux-w99d:~/hadoop-0.20.2/bin> ./hadoop namenode -format  
　　12/12/30 14:26:41 INFO namenode.NameNode: STARTUP_MSG: 
　　/************************************************************
　　STARTUP_MSG: Starting NameNode
　　STARTUP_MSG:   host = master/192.168.1.102
　　STARTUP_MSG:   args = [-format]
　　STARTUP_MSG:   version = 0.20.2
　　STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
　　************************************************************/
　　12/12/30 14:26:41 INFO namenode.FSNamesystem: fsOwner=hadoop,users,dialout,video
　　12/12/30 14:26:41 INFO namenode.FSNamesystem: supergroup=supergroup
　　12/12/30 14:26:41 INFO namenode.FSNamesystem: isPermissionEnabled=true
　　12/12/30 14:26:41 INFO common.Storage: Image file of size 96 saved in 0 seconds.
　　12/12/30 14:26:42 INFO common.Storage: Storage directory /home/hadoop/tmp/dfs/name has been successfully formatted.
　　12/12/30 14:26:42 INFO namenode.NameNode: SHUTDOWN_MSG: 
　　/************************************************************
　　SHUTDOWN_MSG: Shutting down NameNode at master/192.168.1.102
　　************************************************************/
　　hadoop@linux-w99d:~/hadoop-0.20.2/bin> ./start-all.sh
　　starting namenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-namenode-linux-w99d.out
　　192.168.1.102: datanode running as process 5354. Stop it first.
　　192.168.1.102: secondarynamenode running as process 5463. Stop it first.
　　jobtracker running as process 5530. Stop it first.
　　192.168.1.102: tasktracker running as process 5644. Stop it first.
　　hadoop@linux-w99d:~/hadoop-0.20.2/bin> jps
　　6229 NameNode
　　5644 TaskTracker
　　6536 Jps
　　5463 SecondaryNameNode
　　5530 JobTracker
　　5354 DataNode
　　hadoop@linux-w99d:~/hadoop-0.20.2/bin> ./hadoop dfsadmin -report
　　Configured Capacity: 138087571456 (128.6 GB)
　　Present Capacity: 130494750735 (121.53 GB)
　　DFS Remaining: 130494726144 (121.53 GB)
　　DFS Used: 24591 (24.01 KB)
　　DFS Used%: 0%
　　Under replicated blocks: 0
　　Blocks with corrupt replicas: 0
　　Missing blocks: 0
　　-------------------------------------------------
　　Datanodes available: 1 (1 total, 0 dead)
　　Name: 192.168.1.102:50010
　　Decommission Status : Normal
　　Configured Capacity: 138087571456 (128.6 GB)
　　DFS Used: 24591 (24.01 KB)
　　Non DFS Used: 7592820721 (7.07 GB)
　　DFS Remaining: 130494726144(121.53 GB)
　　DFS Used%: 0%
　　DFS Remaining%: 94.5%
　　Last contact: Sun Dec 30 14:28:08 CST 2012