DOE OSTI · 3374487
SWARM: Reimagining scientific workflow management systems in a distributed world
Abstract
Modern scientific workflows process massive amounts of data from diverse instruments and sensors, leveraging geographically distributed, heterogeneous compute and storage resources—from leadership-class systems to edge devices—connected by high-performance networks. The diversity of resources introduces challenges in harnessing their full potential, with resilience issues arising across applications, system software, networks, storage, and hardware. Today, workflow management systems (WMS) coordinate the execution of computation and data management tasks across target resources. However, WMS’s centralized nature makes them vulnerable to faults and scalability issues that may result in failures of entire computational campaigns. In conclusion, this paper introduces a novel agentic framework for workflow management, fully distributing and decentralizing the WMS functions and modeling them as swarm intelligence agents infused with advanced artificial intelligence solutions and traditional distributed computing algorithms that can make coordinated decisions in the presence of failures of the underlying cyberinfrastructure.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Balaprakash, Prasanna [Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)], Raghavan, Krishnan [Argonne National Laboratory (ANL), Argonne, IL (United States)], Cappello, Franck [Argonne National Laboratory (ANL), Argonne, IL (United States)] (ORCID:0000000278903934), Deelman, Ewa [University of Southern California, Marina del Rey, CA (United States)] (ORCID:000000015106503X), Mandal, Anirban [University of North Carolina, Chapel Hill, NC (United States)], Jin, Hongwei [Argonne National Laboratory (ANL), Argonne, IL (United States)] (ORCID:000000022851595X), Mahmud, Imtiaz [Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)], Thareja, Komal [University of North Carolina, Chapel Hill, NC (United States)], Wu, Shixun [University of California, Riverside, CA (United States)], Zuk, Pawel [University of Southern California, Marina del Rey, CA (United States)], Kiran, Mariam [Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)], Chen, Zizhong [University of Southern California, Marina del Rey, CA (United States)], Di, Sheng [Argonne National Laboratory (ANL), Argonne, IL (United States)] (ORCID:0000000273395256), Wu, Kesheng [Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)]. 2025-05-15. SWARM: Reimagining scientific workflow management systems in a distributed world. https://doi.org/10.1177/10943420251339317
Cite the original work for its findings. Save a collection to share your selection of sources.