Search NASA⌕ Search

DOE OSTI · 1827006

DYFLOW: A flexible framework for orchestrating scientific workflows on supercomputers

Abstract

Modern scientific workflows are increasing in complexity with growth in computation power, incorporation of non-traditional computation methods, and advances in technologies enabling data streaming to support on-the-fly computation. These workflows have unpredictable runtime behaviors, and a fixed, predetermined resource assignment on supercomputers can be inefficient for overall performance and throughput. Inability to change resource assignments further limits the scientists to avail of science-driven opportunities or respond to failures.We introduce DYFLOW, a flexible framework that orchestrates scientific workflows on supercomputers based on user-designed policies. DYFLOW compartmentalizes orchestration stages into simplified constructs, and end-users can program and reuse them according to their workflow requirements through an easy-to-use interface. These constructs hide the intricacies involved in runtime management from end-users, for instance, procurement of information to understand the workflow state, assessment, and supervision of the runtime changes. DYFLOW is designed to work alongside existing workflow management systems and reuse the available (static) support for workflow management. We have integrated DYFLOW with an existing workflow management tool as a demonstration. With experiments performed on use cases from three types of scientific workflows and two different parallel architectures, we show that DYFLOW achieves the desired orchestration incurring a small cost to carry out the runtime changes.

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Singhal, Swati, Sussman, Alan, Wolf, Matthew, Mehta, Kshitij, Choi, Jong Youl. 2021-08-01. DYFLOW: A flexible framework for orchestrating scientific workflows on supercomputers. https://searchnasa.com/record/osti%3A1827006/dyflow-a-flexible-framework-for-orchestrating-scientific-workflows-on-supercomputers

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Search NASA by Indiciae: BibTeX · RIS.