hadoop集群跑mr程序报错beyond physical memory limits
跑etl任务，报错：Container is running beyond physical memory limits. Current usage: 2.7 GB of 2.5 GB physical memory used; 4.4 GB of 7.5 GB virtual memory used. Killing container.
发现是一个小时任务打开了太多文件（大约7000多个，每个进程占用的buffer太多，导致内存不足）
解决办法：
修改程序设置，将每个buffer设置占用更小内存，以下是修改后成功运行mr程序的配置
"orc.strip.size":1024*1024,
"orc.block.size":16*1024*1024,
"orc.row.index.stride":500,
"orc.compress.size":8*1024,
页:
[1]