hadoop-2.0 alpha standalone install

chenkehao

　　看了一堆不太相关的东西...
　　其实只要解压运行即可,形如hadoop-0.20.xx,不过要注意jar的位置.

hadoop jar share/hadoop/hadoop-mapreduce/hadoop-example-xx.jar wordcount input output
 　　接下来将进行cluster布署,加深了解新架构的动作流程.
　　12/06/17 11:59:01 WARN util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty

12/06/17 11:59:01 WARN conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id

12/06/17 11:59:01 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=

12/06/17 11:59:01 INFO util.NativeCodeLoader: Loaded the native-hadoop library

12/06/17 11:59:01 INFO input.FileInputFormat: Total input paths to process : 1

12/06/17 11:59:01 WARN snappy.LoadSnappy: Snappy native library not loaded

12/06/17 11:59:02 INFO mapreduce.JobSubmitter: number of splits:1

12/06/17 11:59:02 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class

12/06/17 11:59:02 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar

12/06/17 11:59:02 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class

12/06/17 11:59:02 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name

12/06/17 11:59:02 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class

12/06/17 11:59:02 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir

12/06/17 11:59:02 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir

12/06/17 11:59:02 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps

12/06/17 11:59:02 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class

12/06/17 11:59:02 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class

12/06/17 11:59:02 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir

12/06/17 11:59:02 WARN conf.Configuration: file:/tmp/hadoop-hadoop/mapred/staging/hadoop2008898472/.staging/job_local_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.

12/06/17 11:59:02 WARN conf.Configuration: file:/tmp/hadoop-hadoop/mapred/staging/hadoop2008898472/.staging/job_local_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.

12/06/17 11:59:02 WARN conf.Configuration: file:/tmp/hadoop-hadoop/mapred/local/localRunner/job_local_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.

12/06/17 11:59:02 WARN conf.Configuration: file:/tmp/hadoop-hadoop/mapred/local/localRunner/job_local_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.

12/06/17 11:59:02 INFO mapreduce.Job: The url to track the job: http://localhost:8080/

12/06/17 11:59:02 INFO mapreduce.Job: Running job: job_local_0001

12/06/17 11:59:02 INFO mapred.LocalJobRunner: OutputCommitter set in config null

12/06/17 11:59:02 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

12/06/17 11:59:02 INFO mapred.LocalJobRunner: Waiting for map tasks

12/06/17 11:59:02 INFO mapred.LocalJobRunner: Starting task: attempt_local_0001_m_000000_0

12/06/17 11:59:02 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@2bc3f5

12/06/17 11:59:02 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

12/06/17 11:59:02 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

12/06/17 11:59:02 INFO mapred.MapTask: soft limit at 83886080

12/06/17 11:59:02 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

12/06/17 11:59:02 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

12/06/17 11:59:02 INFO mapred.LocalJobRunner:

12/06/17 11:59:02 INFO mapred.MapTask: Starting flush of map output

12/06/17 11:59:02 INFO mapred.MapTask: Spilling map output

12/06/17 11:59:02 INFO mapred.MapTask: bufstart = 0; bufend = 2055; bufvoid = 104857600

12/06/17 11:59:02 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26213684(104854736); length = 713/6553600

12/06/17 11:59:02 INFO mapred.MapTask: Finished spill 0

12/06/17 11:59:02 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of committing

12/06/17 11:59:02 INFO mapred.LocalJobRunner: map

12/06/17 11:59:02 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.

12/06/17 11:59:02 INFO mapred.LocalJobRunner: Finishing task: attempt_local_0001_m_000000_0

12/06/17 11:59:02 INFO mapred.LocalJobRunner: Map task executor complete.

12/06/17 11:59:03 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@da3a1e

12/06/17 11:59:03 INFO mapred.Merger: Merging 1 sorted segments

12/06/17 11:59:03 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1823 bytes

12/06/17 11:59:03 INFO mapred.LocalJobRunner:

12/06/17 11:59:03 WARN conf.Configuration: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords

12/06/17 11:59:03 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of committing

12/06/17 11:59:03 INFO mapred.LocalJobRunner:

12/06/17 11:59:03 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now

12/06/17 11:59:03 INFO output.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/cc/hadoop/standalone/hadoop-2.0.0-alpha/out/wc/_temporary/0/task_local_0001_r_000000

12/06/17 11:59:03 INFO mapred.LocalJobRunner: reduce > reduce

12/06/17 11:59:03 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.

12/06/17 11:59:03 INFO mapreduce.Job: Job job_local_0001 running in uber mode : false

12/06/17 11:59:03 INFO mapreduce.Job:  map 100% reduce 100%

12/06/17 11:59:03 INFO mapreduce.Job: Job job_local_0001 completed successfully

12/06/17 11:59:03 INFO mapreduce.Job: Counters: 27

    File System Counters

        FILE: Number of bytes read=544748

        FILE: Number of bytes written=798104

        FILE: Number of read operations=0

        FILE: Number of large read operations=0

        FILE: Number of write operations=0

    Map-Reduce Framework

        Map input records=31

        Map output records=179

        Map output bytes=2055

        Map output materialized bytes=1836

        Input split bytes=121

        Combine input records=179

        Combine output records=131

        Reduce input groups=131

        Reduce shuffle bytes=0

        Reduce input records=131

        Reduce output records=131

        Spilled Records=262

        Shuffled Maps =0

        Failed Shuffles=0

        Merged Map outputs=0

        GC time elapsed (ms)=111

        CPU time spent (ms)=0

        Physical memory (bytes) snapshot=0

        Virtual memory (bytes) snapshot=0

        Total committed heap usage (bytes)=242360320

    File Input Format Counters

        Bytes Read=1366

    File Output Format Counters

        Bytes Written=1326

　　ref:
Hadoop0.23.0初探1

Hadoop0.23.0初探2


　　Hadoop 0.23.x/NameNode federation原理、编译、安装、体验