Search NASA⌕ Search

DOE OSTI · 1335904

YARNsim: Simulating Hadoop YARN

Abstract

Despite the popularity of the Apache Hadoop system, its success has been limited by issues such as single points of failure, centralized job/task management, and lack of support for programming models other than MapReduce. The next generation of Hadoop, Apache Hadoop YARN, is designed to address these issues. In this paper, we propose YARNsim, a simulation system for Hadoop YARN. YARNsim is based on parallel discrete event simulation and provides protocol-level accuracy in simulating key components of YARN. YARNsim provides a virtual platform on which system architects can evaluate the design and implementation of Hadoop YARN systems. Also, application developers can tune job performance and understand the tradeoffs between different configurations, and Hadoop YARN system vendors can evaluate system efficiency under limited budgets. To demonstrate the validity of YARNsim, we use it to model two real systems and compare the experimental results from YARNsim and the real systems. The experiments include standard Hadoop benchmarks, synthetic workloads, and a bioinformatics application. The results show that the error rate is within 10% for the majority of test cases. The experiments prove that YARNsim can provide what-if analysis for system designers in a timely manner and at minimal cost compared with testing and evaluating on a real system.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Liu, Ning, Yang, Xi, Sun, Xian-He, Jenkins, John, Ross, Robert. 2015-01-01. YARNsim: Simulating Hadoop YARN. https://doi.org/10.1109/ccgrid.2015.61. https://searchnasa.com/record/osti%3A1335904/yarnsim-simulating-hadoop-yarn

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Search NASA by Indiciae: BibTeX · RIS.