Hadoop下遇到OOM：unable to create new native thread
在hadoop下执行任务出现了这个异常：Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.kill(JvmManager.java:553)
at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.killJvmRunner(JvmManager.java:317)
at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.killJvm(JvmManager.java:297)
at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.taskKilled(JvmManager.java:289)
at org.apache.hadoop.mapred.JvmManager.taskKilled(JvmManager.java:158)
at org.apache.hadoop.mapred.TaskRunner.kill(TaskRunner.java:801)
at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:3274)
at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.jobHasFinished(TaskTracker.java:3246)
at org.apache.hadoop.mapred.TaskTracker.purgeTask(TaskTracker.java:2281)
at org.apache.hadoop.mapred.TaskTracker.fatalError(TaskTracker.java:3666)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at com.sun.proxy.$Proxy1.fatalError(Unknown Source)
at org.apache.hadoop.mapred.Child.main(Child.java:310)
然后，为了了解这个异常，我开始了百度之旅：
也就是说在压力环境下拥有大量的线程，或者本地内存耗尽时，企图创建新的线程时抛出。
而系统能创建的线程数的计算公式如下：
(MaxProcessMemory - JVMMemory - ReservedOsMemory) / (ThreadStackSize) = Number of threads
MaxProcessMemory 指的是一个进程的最大内存
JVMMemory JVM内存
ReservedOsMemory 保留的操作系统内存
ThreadStackSize 线程栈的大小
表象便是当JVM的heap size设置过大时，thread的创建数量便会减少。
因为当JVM申请分配heap size的内存的时候，会马上占有这些内存，因为这些内存是连续的。
所以会导致没有办法创建本地的线程。
JVM的堆和进程所需的栈都会消耗进程地址空间(address space)，而地址空间则取决于操作系统。
而进程栈和一些库也会需要这个寻址空间。
有2个方法可以提高创建线程的数量：
1.减少JVM的heap size；
2.减少单个线程栈的大小，在JVM启动中使用-Xss参数。目前JDK1.4中每个线程栈所大小是256K,1.5是1M.
在Linux下
查了一下man ulimit，有下面解释
-a All current limits are reported
-c The maximum size of core files created
-d The maximum size of a process data segment
-f The maximum size of files created by the shell
-l The maximum size that may be locked into memory
-m The maximum resident set size (has no effect on Linux)
-n The maximum number of open file descriptors (most systems do not allow this value to be set)
-p The pipe size in 512-byte blocks (this may not be set)
-s The maximum stack size
-t The maximum amount of cpu time in seconds
-u The maximum number of processes available to a single user
-v The maximum amount of virtual memory available to the shell
其中：只有“max user processes”会对所有application能创建总的线程数有限制
refer to：
http://www.iyunv.com/topic/654172
http://www.iyunv.com/topic/503349
http://bbs.csdn.net/topics/350248404
页:
[1]