Hadoop MapReduce编程 API入门系列之二次排序（十六）
2016-12-12 17:04:32,012 INFO - Initializing JVM Metrics with processName=JobTracker, sessionId=2016-12-12 17:04:33,056 WARN - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-12-12 17:04:33,059 WARN - No job jar file set. User>
2016-12-12 17:04:33,083 INFO - Total input paths to process : 1
2016-12-12 17:04:33,161 INFO - number of splits:1
2016-12-12 17:04:33,562 INFO - Submitting tokens for job: job_local1173601391_0001
2016-12-12 17:04:34,242 INFO - The url to track the job: http://localhost:8080/
2016-12-12 17:04:34,244 INFO - Running job: job_local1173601391_0001
2016-12-12 17:04:34,247 INFO - OutputCommitter set in config null
2016-12-12 17:04:34,264 INFO - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2016-12-12 17:04:34,371 INFO - Waiting for map tasks
2016-12-12 17:04:34,373 INFO - Starting task: attempt_local1173601391_0001_m_000000_0
2016-12-12 17:04:34,439 INFO - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-12 17:04:34,667 INFO - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@65bb90dc
2016-12-12 17:04:34,676 INFO - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/data/secondarySort/secondarySort.txt:0+120
2016-12-12 17:04:34,762 INFO - (EQUATOR) 0 kvi 26214396(104857584)
2016-12-12 17:04:34,763 INFO - mapreduce.task.io.sort.mb: 100
2016-12-12 17:04:34,763 INFO - soft limit at 83886080
2016-12-12 17:04:34,763 INFO - bufstart = 0; bufvoid = 104857600
2016-12-12 17:04:34,763 INFO - kvstart = 26214396; length = 6553600
2016-12-12 17:04:34,771 INFO - Map output collector>
2016-12-12 17:04:34,789 INFO -
2016-12-12 17:04:34,789 INFO - Starting flush of map output
2016-12-12 17:04:34,789 INFO - Spilling map output
2016-12-12 17:04:34,789 INFO - bufstart = 0; bufend = 216; bufvoid = 104857600
2016-12-12 17:04:34,790 INFO - kvstart = 26214396(104857584); kvend = 26214328(104857312); length = 69/6553600
2016-12-12 17:04:34,809 INFO - Finished spill 0
2016-12-12 17:04:34,818 INFO - Task:attempt_local1173601391_0001_m_000000_0 is done. And is in the process of committing
2016-12-12 17:04:34,838 INFO - map
2016-12-12 17:04:34,838 INFO - Task 'attempt_local1173601391_0001_m_000000_0' done.
2016-12-12 17:04:34,838 INFO - Finishing task: attempt_local1173601391_0001_m_000000_0
2016-12-12 17:04:34,839 INFO - map task executor complete.
2016-12-12 17:04:34,846 INFO - Waiting for reduce tasks
2016-12-12 17:04:34,846 INFO - Starting task: attempt_local1173601391_0001_r_000000_0
2016-12-12 17:04:34,864 INFO - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-12 17:04:34,950 INFO - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@59b59452
2016-12-12 17:04:34,954 INFO - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@73d5cf65
2016-12-12 17:04:34,974 INFO - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-12-12 17:04:35,011 INFO - attempt_local1173601391_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-12-12 17:04:35,048 INFO - localfetcher#1 about to shuffle output of map attempt_local1173601391_0001_m_000000_0 decomp: 254 len: 258 to MEMORY
2016-12-12 17:04:35,060 INFO - Read 254 bytes from map-output for attempt_local1173601391_0001_m_000000_0
2016-12-12 17:04:35,123 INFO - closeInMemoryFile -> map-output of>
2016-12-12 17:04:35,125 INFO - EventFetcher is interrupted.. Returning
2016-12-12 17:04:35,126 INFO - 1 / 1 copied.
2016-12-12 17:04:35,126 INFO - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-12-12 17:04:35,136 INFO - Merging 1 sorted segments
2016-12-12 17:04:35,137 INFO - Down to the last merge-pass, with 1 segments left of total>
2016-12-12 17:04:35,139 INFO - Merged 1 segments, 254 bytes to disk to satisfy reduce memory limit
2016-12-12 17:04:35,139 INFO - Merging 1 files, 258 bytes from disk
2016-12-12 17:04:35,140 INFO - Merging 0 segments, 0 bytes from memory into reduce
2016-12-12 17:04:35,141 INFO - Merging 1 sorted segments
2016-12-12 17:04:35,142 INFO - Down to the last merge-pass, with 1 segments left of total>
2016-12-12 17:04:35,143 INFO - 1 / 1 copied.
2016-12-12 17:04:35,150 INFO - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2016-12-12 17:04:35,158 INFO - Task:attempt_local1173601391_0001_r_000000_0 is done. And is in the process of committing
2016-12-12 17:04:35,160 INFO - 1 / 1 copied.
2016-12-12 17:04:35,160 INFO - Task attempt_local1173601391_0001_r_000000_0 is allowed to commit now
2016-12-12 17:04:35,166 INFO - Saved output of task 'attempt_local1173601391_0001_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/secondarySort/_temporary/0/task_local1173601391_0001_r_000000
2016-12-12 17:04:35,167 INFO - reduce > reduce
2016-12-12 17:04:35,167 INFO - Task 'attempt_local1173601391_0001_r_000000_0' done.
2016-12-12 17:04:35,167 INFO - Finishing task: attempt_local1173601391_0001_r_000000_0
2016-12-12 17:04:35,168 INFO - reduce task executor complete.
2016-12-12 17:04:35,248 INFO - Job job_local1173601391_0001 running in uber mode : false
2016-12-12 17:04:35,249 INFO - map 100% reduce 100%
2016-12-12 17:04:35,251 INFO - Job job_local1173601391_0001 completed successfully
2016-12-12 17:04:35,271 INFO - Counters: 33
File System Counters
FILE: Number of bytes read=1186
FILE: Number of bytes written=394623
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=18
Map output records=18
Map output bytes=216
Map output materialized bytes=258
Input split bytes=145
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=258
Reduce input records=18
Reduce output records=18
Spilled Records=36
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=534773760
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=120
File Output Format Counters
Bytes Written=115
页:
[1]