Making the case for reforming the I/O software stack of extreme-scale systems
Not Available
Isaila, Florin↗
Engineering topics
Isaila, Florin
Publications and source records attributed to Isaila, Florin.
Experimental evaluation of a flexible I/O architecture for accelerating workflow engines in ultrascale environments
The increasing volume of scientific data and the limited scalability and performance of storage systems are currently presenting a significant limitation for the productivity of the scientific workflows running on both high-performance computing (HPC) and cloud platforms. Clearly needed is better integration of storage systems and workflow engines to address this problem. This paper presents and evaluates a novel solution that leverages codesign principles for integrating Hercules—an in-memory data store—with a workflow management system. We consider four main aspects: workflow representation, task scheduling, task placement, and task termination. As a result, the experimental evaluation on both cloud and HPC systems demonstrates significant performance and scalability improvements over existing state-of-the-art approaches.
97 MATHEMATICS AND COMPUTING↗
Flexible Data-Aware Scheduling for Workflows over an In-Memory Object Store
This paper explores novel techniques for improving the performance of many-task workflows based on the Swift scripting language. We propose novel programmer options for automated distributed data placement and task scheduling. These options trigger a data placement mechanism used for distributing intermediate workflow data over the servers of Hercules, a distributed key-value store that can be used to cache file system data. We demonstrate that these new mechanisms can significantly improve the aggregated throughput of many-task workflows with up to 86x, reduce the contention on the shared file system, exploit the data locality, and trade off locality and load balance.
Duro, Francisco Rodrigo↗