Tovar, Benjamin | Search NASA

DOE OSTI2024

Workflows Community Summit 2024: Future Trends and Challenges in Scientific Workflows

The 2024 Workflows Community Summit report presents the outcomes of a three-day international gathering that brought together 109 experts from 18 countries to discuss future trends and challenges in scientific workflows. The summit focused on six key areas: time-sensitive workflows, convergence of AI and HPC workflows, multi-facility workflows, heterogeneous HPC environments, user experience and interfaces, and FAIR computational workflows. Discussions highlighted emerging challenges such as integrating AI with traditional HPC, managing workflows across diverse facilities, addressing heterogeneity in computing environments, and ensuring workflows are findable, accessible, interoperable, and reusable (FAIR). The report outlines recent advances, ongoing challenges, and provides recommendations for each topic area, emphasizing the need for standardization, improved interoperability, and the development of more sophisticated tools and frameworks to support the evolving landscape of scientific workflows in the era of exascale computing and AI integration.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Workflows Community Summit 2022: A Roadmap Revolution

Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from the execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing (often referred to as a computing continuum) and the evolving needs of emerging scientific applications, it is paramount that the development of novel scientific workflows and system functionalities seek to increase the efficiency, resilience, and pervasiveness of existing systems and applications. Specifically, the proliferation of machine learning/artificial intelligence (ML/AI) workflows, need for processing large-scale datasets produced by instruments at the edge, intensification of near real-time data processing, support for long-term experiment campaigns, and emergence of quantum computing as an adjunct to HPC, have significantly changed the functional and operational requirements of workflow systems. Workflow systems now need to, for example, support data streams from the edge-to-cloud-to-HPC, enable the management of many small-sized files, allow data reduction while ensuring high accuracy, orchestrate distributed services (workflows, instruments, data movement, provenance, publication, etc.) across computing and user facilities, among others. Further, to accelerate science, it is also necessary that these systems implement specifications/standards and APIs for seamless (horizontal and vertical) integration between systems and applications, as well as enable the publication of workflows and their associated products according to the FAIR principles.

97 MATHEMATICS AND COMPUTING↗