Hadoop的JVM重用

lishenghan

Hadoop中有个参数是mapred.job.reuse.jvm.num.tasks，默认是1，表示一个JVM上最多可以顺序执行的task数目（属于同一个Job）是1。也就是说一个task启一个JVM。
　　 
比如我配的是每个slave节点最多同时运行8个map和8个reduce。那么在map阶段，slave节点会启动最多8个JVM用于map。如下：
 

root@slave1:~# jps
28291 Child
28290 Child
28281 Child
28293 Child
28277 Child
1487 DataNode
28298 Child
28273 Child
28272 Child
1636 TaskTracker
28799 Jps
root@slave1:~# ps -e |grep java

1487 ?       00:53:26 java	TaskTracker
1636 ?       00:12:42 java	DataNode
28272?        00:00:35 java	Child
28273?        00:00:35 java	Child
28277?        00:00:36 java	Child
28281?        00:00:36 java	Child
28290?        00:00:36 java	Child
28291?        00:00:37 java	Child
28293?        00:00:36 java	Child
28298?        00:00:36 java	Child

其中前两个是固定的进程。
 
为每个task启动一个新的JVM将耗时1秒左右，对于运行时间较长（比如1分钟以上）的job影响不大，但如果都是时间很短的task，那么频繁启停JVM会有开销。
 
如果我们想使用JVM重用技术来提高性能，那么可以将mapred.job.reuse.jvm.num.tasks设置成大于1的数。这表示属于同一job的顺序执行的task可以共享一个JVM，也就是说第二轮的map可以重用前一轮的JVM，而不是第一轮结束后关闭JVM，第二轮再启动新的JVM。
 
那么最多一个JVM能顺序执行多少个task才关闭呢？这个值就是mapred.job.reuse.jvm.num.tasks。如果设置成-1，那么只要是同一个job的task（无所谓多少个），都可以按顺序在一个JVM上连续执行。
 
如果task属于不同的job，那么JVM重用机制无效，不同job的task需要不同的JVM来运行。
 
注意：
JVM重用技术不是指同一Job的两个或两个以上的task可以同时运行于同一JVM上，而是排队按顺序执行。
一个tasktracker最多可以同时运行的task数目由mapred.tasktracker.map.tasks.maximum和mapred.tasktracker.reduce.tasks.maximum
决定，并且这两个参数在mapred-site.xml中设置。其他方法，如在JobClient端通过命令行
-Dmapred.tasktracker.map.tasks.maximum=number或者conf.set("mapred.tasktracker.map.tasks.maximum","number")设置都是无效的。
 
附《Hadoop权威指南》上的介绍：

Task JVM Reuse

Hadoop runs tasks in their own Java VirtualMachine to isolate them from other running tasks. The overhead of starting anew JVM for each task can take around a second, which for jobs that run for aminute or so is insignificant. However, jobs that have a large number of veryshort-lived tasks (these are usually map tasks), or that have lengthy initialization,can see performance gains when the JVM is reused for subsequent tasks.

 

With task JVM reuse enabled, tasks do not run concurrentlyin a single JVM. The JVM runs tasks sequentially. Tasktrackers can, however, runmore than one task at a time, but this is always done in separate JVMs. Theproperties for controlling the tasktrackers number of map task slots and reducetask slots are discussed in “Memory” on page 269.

The property for controlling task JVM reuseis mapred.job.reuse.jvm.num.tasks: it specifies the maximum number of tasks to run for a given job for eachJVM launched; the default is 1 (see Table 6-4). Tasks from different jobs are always run in separate JVMs. If theproperty is set to –1, there is no limit to the number of tasks from the same jobthat may share a JVM. The method setNumTasksToExecutePerJvm() on JobConfcan also be used to configure this property.

 

Tasks that are CPU-boundmay also benefit from task JVM reuse by taking advantage of runtimeoptimizations applied by the HotSpot JVM. After running for a while, the HotSpotJVM builds up enough information to detect performance-critical sections in thecode and dynamically translates the Java byte codes of these hot spots intonative machine code. This works well for long-running processes, but JVMs thatrun for seconds or a few minutes may not gain the full benefit of HotSpot. Inthese cases, it is worth enabling task JVM reuse.

 

Another place where ashared JVM is useful is for sharing state between the tasks of a job. Bystoring reference data in a static field, tasks get rapid access to the shareddata.