Automatic task slots assignment in Hadoop MapReduce

• DOI: 10.1145/2377978.2377982
• Publication Date: 2012-09-25T19:48:43Z
• Authors: Kun Wang; Ben Tan; Juwei Shi; Bo Yang
• Keywords: 0202 electrical engineering, electronic engineering, information engineering; 02 engineering and technology

In this paper, we address the problem caused by fixed assignment of task slots in Hadoop MapReduce. It is infeasible to manually configure optimal since characteristics various workloads are different. We design and implement an automatic control mechanism dynamically assign based on resource utilization each Task Tracker node. The takes lag period into account. can improve cluster-wide avoid contention. Experimental results show that our implementation adjust capacity setting runtime. some case such as Word Count, outperforms current with configuration found manual tuning.