Hadoop May not run daemons as root. Please specify HADOOP_TASKTRACKER_USER

LOCKLOSE

原文地址：http://www.migrate2cloud.com/blog/hadoop-cluster-with-hadoop-0-20-and-ubuntu-10-04


HADOOP Cluster on AWS EC2 with hadoop-0.20 and ubuntu-10.04
Friday, June 17th, 2011 | Posted in Amazon EC2, Cloud computing
Let’s start with a small introduction- what is hadoop ?. Hadoop is an open-source project administered by the Apache Software Foundation. Apache Hadoop is a Java software framework that supports data-intensive distributed applications under a free license. It enables applications to work with thousands of nodes and petabytes of data. Hadoop was inspired by Google’s MapReduce and Google File System (GFS) papers.

Technically, Hadoop consists of two key services: reliable data storage using the Hadoop Distributed File System (HDFS) and high-performance parallel data processing using a technique called MapReduce.

Dealing with big data requires two things:

Inexpensive, reliable storage; and
New tools for analyzing unstructured and structured data.
Hadoop creates clusters of machines and coordinates work among them. Clusters can be built with inexpensive computers.If one fails, Hadoop continues to operate the cluster without losing data or interrupting work, by shifting work to the remaining machines in the cluster.

HDFS manages storage on the cluster by breaking incoming files into pieces, called “blocks,” and storing each of the blocks redundantly across the pool of servers.

The main services running in a hadoop cluster will be

1)namenode

2)jobtracker

3)secondarynamenode

These three will be running only on a single node(machine) ; that machine is the central machine which controls the cluster.

4)datanode

5)tasktracker

These two services will be running on all other nodes in the cluster.

HDFS has a master/slave architecture. An HDFS cluster consists of a single NameNode, a master server that manages the file system namespace and regulates access to files by clients. In addition, there are a number of DataNodes, usually one per node in the cluster, which manage storage attached to the nodes that they run on.

Above the file systems comes the MapReduce  engine, which consists of one Job Tracker, to which client applications submit MapReduce jobs. The Job Tracker pushes work out to available Task Tracker nodes in the cluster, striving to keep the work as close to the data as possible.

The only purpose of the secondary name-node is to perform periodic checkpoints. The secondary name-node periodically downloads current name-node image and edits log files, joins them into new image and uploads the new image back to the (primary and the only) name-node.

Now Let us have a look at how to build a hadoop cluster using Cloudera hadoop-0.20 on ubuntu-10.04

You should install sun –jdk  first. Then add the following repositories to the apt sources list.

vim /etc/apt/sources.list.d/cloudera.list

1
deb http://archive.cloudera.com/debian lucid-cdh3u0 contrib
2

3
deb-src http://archive.cloudera.com/debian lucid-cdh3u0 contrib
Import key

1
curl -s http://archive.cloudera.com/debian/archive.key | apt-key add -
Then run

1
apt-get update
For Namenode/Jobtracker ( These two services should run only on a single central machine in the cluster)

1
apt-get install hadoop –yes
2

3
apt-get install hadoop-0.20-namenode
4

5
apt-get install hadoop-0.20-jobtracker
6

7
apt-get install hadoop-0.20-secondarynamenode
Configuration

vim /etc/hadoop/conf/hadoop-env.sh

Append these

1
export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.24/   ( your java home comes here )
2

3
export HADOOP_CONF_DIR=/etc/hadoop/conf
4

5
export HADOOP_HOME=/usr/lib/hadoop-0.20
6

7
export HADOOP_NAMENODE_USER=hdfs
8

9
export HADOOP_SECONDARYNAMENODE_USER=hdfs
10

11
export HADOOP_DATANODE_USER=hdfs
12

13
export HADOOP_JOBTRACKER_USER=mapred
14

15
export HADOOP_TASKTRACKER_USER=mapred
16

17
export HADOOP_IDENT_STRING=hadoop
vim /etc/hadoop/conf/core-site.xml

1
<?xml version="1.0"?>
2

3
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
4

5
<!-- Put site-specific property overrides in this file. -->
6

7
<configuration>
8

9
<property>
10

11
<name>fs.default.name</name>
12

13
<value>hdfs://< ip address of this machine >:8020</value>
14

15
</property>
16

17
</configuration>
vim /etc/hadoop/conf/hdfs-site.xml


1
<?xml version="1.0"?>
2

3
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
4

5
<!-- Put site-specific property overrides in this file. -->
6

7
<configuration>
8

9
<property>
10

11
<name>dfs.name.dir</name>
12

13
<value>/var/lib/hadoop-0.20/name</value>
14

15
</property>
16

17
<property>
18

19
<name>dfs.data.dir</name>
20

21
<value>/var/lib/hadoop-0.20/data</value>
22

23
</property>
24

25
<property>
26

27
<name>dfs.replication</name>
28

29
<value>2</value>
30

31
</property>
32

33
</configuration>
vim /etc/hadoop/conf/mapred-site.xml

1
<?xml version="1.0"?>
2

3
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
4

5
<!-- Put site-specific property overrides in this file. -->
6

7
<configuration>
8

9
<property>
10

11
<name>mapred.job.tracker</name>
12

13
<value>< ip address of this machine >:8021</value>
14

15
</property>
16

17
<property>
18

19
<name>mapred.system.dir</name>
20

21
<value>/var/lib/hadoop-0.20/system</value>
22

23
</property>
24

25
<property>
26

27
<name>mapred.local.dir</name>
28

29
<value>/var/lib/hadoop-0.20/mapred</value>
30

31
</property>
32

33
</configuration>
——————————————————————————————————————————————

1
mkdir  / var/lib/hadoop-0.20/name
2

3
mkdir  / var/lib/hadoop-0.20/data
4

5
mkdir  / var/lib/hadoop-0.20/system
6

7
mkdir  / var/lib/hadoop-0.20/mapred
8

9
chown -R hdfs /var/lib/hadoop-0.20/name
10

11
chown -R hdfs /var/lib/hadoop-0.20/data
12

13
chown -R mapred /var/lib/hadoop-0.20/mapred
Now format NameNode

1
yes Y | /usr/bin/hadoop namenode –format
Start namenode

1
/etc/init.d/hadoop-0.20-namenode start
Check the log Files for error:

less /usr/lib/hadoop-0.20/logs/hadoop-hadoop-namenode-<ip>.log

Also you can check whether the Namenode process is up or not using the command

1
# jps
Start the SecondaryNamenode

1
/etc/init.d/hadoop-0.20-secondarynamenode start
Log: less /usr/lib/hadoop-0.20/logs/hadoop-hadoop-secondarynamenode-<ip>.log

1
sudo -u hdfs hadoop fs -mkdir /var/lib/hadoop-0.20/system
2

3
sudo -u hdfs hadoop fs -chown mapred /var/lib/hadoop-0.20/system
Now Start the JobTracker

1
/etc/init.d/hadoop-0.20-jobtracker start
Log : less /usr/lib/hadoop-0.20/logs/hadoop-hadoop-jobtracker-ip-10-108-39-34.log

Now  jps  command will show the three processes up

# jps

19233 JobTracker

18994 SecondaryNameNode

18871 NameNode

For Datanode/Tasktracker ( These two services should be running on all the other machines in the cluster )

1
apt-get install hadoop-0.20-datanode
2

3
apt-get install hadoop-0.20-tasktracker
Configuration

vim /etc/hadoop/conf/core-site.xml


1
<?xml version="1.0"?>
2

3
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
4

5
&nbsp;
6

7
<!-- Put site-specific property overrides in this file. -->
8

9
&nbsp;
10

11
<configuration>
12

13
<property>
14

15
<name>fs.default.name</name>
16

17
<value>hdfs://< ip address of the namenode >:8020</value>
18

19
</property>
20

21
</configuration>
vim /etc/hadoop/conf/hdfs-site.xml

1
<?xml version="1.0"?>
2

3
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
4

5
&nbsp;
6

7
<!-- Put site-specific property overrides in this file. -->
8

9
&nbsp;
10

11
<configuration>
12

13
<property>
14

15
<name>dfs.name.dir</name>
16

17
<value>/var/lib/hadoop-0.20/name</value>
18

19
</property>
20

21
<property>
22

23
<name>dfs.data.dir</name>
24

25
<value>/var/lib/hadoop-0.20/data</value>
26

27
</property>
28

29
<property>
30

31
<name>dfs.replication</name>
32

33
<value>2</value>
34

35
</property>
36

37
</configuration>
vim /etc/hadoop/conf/mapred-site.xml

1
<?xml version="1.0"?>
2

3
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
4

5
&nbsp;
6

7
<!-- Put site-specific property overrides in this file. -->
8

9
&nbsp;
10

11
<configuration>
12

13
<property>
14

15
<name>mapred.job.tracker</name>
16

17
<value>< ip address of jobtracker  >:8021</value>
18

19
</property>
20

21
<property>
22

23
<name>mapred.system.dir</name>
24

25
<value>/var/lib/hadoop-0.20/system</value>
26

27
</property>
28

29
<property>
30

31
<name>mapred.local.dir</name>
32

33
<value>/var/lib/hadoop-0.20/mapred</value>
34

35
</property>
36

37
</configuration>
———————————————————————————————————————————————

1
mkdir  /var/lib/hadoop-0.20/data/
2

3
chown -R hdfs /var/lib/hadoop-0.20/data
4

5
mkdir /var/lib/hadoop-0.20/mapred
6

7
chown -R mapred /var/lib/hadoop-0.20/mapred
Start the DataNode

1
/etc/init.d/hadoop-0.20-datanode start
Log : less /usr/lib/hadoop-0.20/logs/hadoop-hadoop-datanode-<ip>.log

Start the TaskTracker

1
/etc/init.d/hadoop-0.20-tasktracker start
Log: less /usr/lib/hadoop-0.20/logs/hadoop-hadoop-tasktracker-<ip>.log

You can now check the interface

http://< namenode-ip >:50070   – for HDFS overview

and

http://< jobtracker –ip>:50030  – for Mapreduce overview