Search NASA⌕ Search

SEARCH · Search NASA

Results for “Workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

Exascale workflow applications and middleware: An ExaWorks retrospective

Exascale computers offer transformative capabilities to combine data-driven and learning-based approaches with traditional simulation applications to accelerate scientific discovery and insight. However, these software combinations and integrations are difficult to achieve due to the challenges of coordinating and deploying heterogeneous software components on diverse and massive platforms. Here, we present the ExaWorks project, which addresses many of these challenges. We developed a workflow Software Development Toolkit (SDK), a curated collection of workflow technologies that can be composed and interoperated through a common interface, engineered following current best practices, and specifically designed to work on HPC platforms. ExaWorks also developed PSI/J, a job management abstraction API, to simplify the construction of portable software components and applications that can be used over various HPC schedulers. The PSI/J API is a minimal interface for submitting and monitoring jobs and their execution state across multiple and commonly used HPC schedulers. We also describe several leading and innovative workflow examples of ExaWorks tools used on DOE leadership platforms. Furthermore, we discuss how our project is working with the workflow community, large computing facilities, and HPC platform vendors to address the requirements of workflows sustainably at the exascale.

97 MATHEMATICS AND COMPUTING↗

MaDaTS: Managing Data on Tiered Storage for Scientific Workflows

Scientific workflows are processing large amounts of data through complex simulation and analysis tasks. Meanwhile, the need to minimize I/O costs on next generation systems and the evolution of new technologies (NVRAMs, SSDs etc.) is resulting in deeper storage hierarchies on High Performance Computing (HPC) systems. A multi-tiered storage hierarchy introduces complexities in workflow and data management. There is need for simple and flexible data abstractions that can allow users to seamlessly manage workflow data and tasks on HPC systems with multiple storage tiers. MaDaTS (Managing Data on Tiered Storage for Scientific Workflows) provides an API and a command-line tool that allows users to manage their workflows and data on tiered storage (Ghoshal & Ramakrishnan (2017)).

97 MATHEMATICS AND COMPUTING↗

STNS01-21 BEE - FY21 P6-2: Archive, clone, and re-run workflows [Slide]

BEE will give ECP a tool that great simplifies the deployment of containerized workflows on the next generation of pre-exascale and exascale systems, as well as public and private clouds. BEE allows scientists to describe their workflow using the Common Workflow Language and then deploy that workflow across the entire spectrum of systems without having to learn the specifics of each container runtime, HPC resource manager, or cloud API. BEE also streamlines the curation and sharing of common workflows among the scientific community.

97 MATHEMATICS AND COMPUTING↗

Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows (IPPD) (Final Report)

This report details the accomplishments from the ASCR funded project “Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows” under the award numbers FWP-66406 and DE-SC0012630, with a focus on the UC San Diego (Award No. DE-SC0012630) part of the accomplishments. We refer to the project as IPPD. The main activities of IPPD were centered on the development and integration of provenance information to capture empirically workflow information and identify the sources of bottlenecks as well as variability, and modeling and simulation to provide insights and predictively explore multiple scenarios for development of advanced techniques for optimization of resources and workflow execution. After a Phase I of the project, a Phase II effort focused on three major aspects: a) observe how data is generated, distributed, and used; b) analyze how data is (repeatedly) consumed with a focus both on repeated patterns and anomalies; and c) explore how to optimize data motion. The project leveraged and extended our existing tools with new research and demonstrated our work on the Belle II workflow suite as well as on workflows from NSLS-II.

97 MATHEMATICS AND COMPUTING↗

Predictive Capability Maturity Model Demonstration for Cylindrical Cavity Coupling Using Gemma in the Next Generation Workflow

The predictive capability maturity model (PCMM) uses the expert elicitation process to generate credibility evidence for a particular analysis. To ensure Gemma has the capability to efficiently produce this credibility evidence, next generation workflows (NGW) are created for the solution verification, calibration/validation, and input uncertainty quantification portions of the PCMM assessment. These workflows are then used on the Higgins cylinder problem, which is representative of applications involving external-to-internal electromagnetic field coupling through a slot. The uncertainties calculated using these workflows are then used to calculate the validation comparison error and the validation uncertainty for the model following the American Society of Mechanical Engineers (ASME) verification and validation (V&V) 20 standard. These workflows will enable analysts to iterate each element of PCMM more efficiently than if completed without using a NGW workflow. An example of this iterative process is shown in Section 7.2.

42 ENGINEERING↗

Lambda-PFLOTRAN 1.0: a workflow for incorporating organic matter chemistry informed by ultra high resolution mass spectrometry into biogeochemical modeling

Abstract. Organic matter (OM) composition plays a central role in microbial respiration of dissolved organic matter and subsequent biogeochemical reactions. Here, a direct connection of organic matter chemistry and thermodynamics to reactive transport simulators has been achieved through the newly developed Lambda-PFLOTRAN workflow tool that succinctly incorporates carbon chemistry data generated from Fourier transform ion cyclotron resonance mass spectrometry (FTICR-MS) into reaction networks to simulate organic matter degradation and the resulting biogeochemistry. Lambda-PFLOTRAN is a Python-based workflow, executed through a Jupyter notebook interface, that digests raw FTICR-MS data, develops a representative reaction network based on substrate-explicit thermodynamic modeling (also termed lambda modeling due to its key thermodynamic parameter λ used therein), and completes a biogeochemical simulation with the open source, reactive flow and transport code PFLOTRAN. The workflow consists of the following five steps: configuration, thermodynamic (lambda) analysis, sensitivity analysis, parameter estimation, and simulation output and visualization. Two test cases are provided to demonstrate the functionality of the Lambda-PFLOTRAN workflow. The first test case uses laboratory incubation data of temporal oxygen depletion to fit lambda parameters (i.e., maximum utilization rate and microbial carrying capacity). A slightly more complex second test case fits multiple lambda formulation and soil organic matter release parameters to temporal greenhouse gas generation measured during a soil incubation. Overall, the Lambda-PFLOTRAN workflow facilitates upscaling by using molecular-scale characterization to inform biogeochemical processes occurring at larger scales.

58 GEOSCIENCES↗

Domain-Specific Languages For Developing and Deploying Signature Discovery Workflows

Domain-agnostic Signature Discovery entails scientific investigation across multiple domains through the re-use of existing algorithms into workflows. The existing algorithms may be written in any programming language for various hardware architectures (e.g., desktops, commodity clusters, and specialized parallel hardware platforms). This raises an engineering issue in generating Web services for heterogeneous algorithms so that they can be composed into a scientific workflow environment (e.g., Taverna). In this paper, we present our software tool that defines two simple Domain-Specific Languages (DSLs) to automate these processes: SDL and WDL. Our Service Description Language (SDL) describes key elements of a signature discovery algorithm and generates the service code. The Workflow Description Language (WDL) describes the pipeline of services and generates deployable artifacts for the Taverna workflow management system. We demonstrate our tool with a landscape classification example that is represented by BLAST workflows composed of services that wrap original scripts.

Jacob, Ferosh↗

FireWorks: a Dynamic Workflow System Designed for High-Throughput Applications

This work introduces FireWorks, a workflow software for running high-throughput calculation workflows at supercomputing centers. FireWorks has been used to complete over 50 million CPU-hours worth of computational chemistry and materials science calculations at the National Energy Research Supercomputing Center. It has been designed to serve the demanding high-throughput computing needs of these applications, with extensive support for (i) concurrent execution through job packing, (ii) failure detection and correction, (iii) provenance and reporting for long-running projects, (iv) automated duplicate detection, and (v) dynamic workflows (i.e., modifying the workflow graph during runtime). We have found that these features are highly relevant to enabling modern data-driven and high-throughput science applications, and we discuss our implementation strategy that rests on Python and NoSQL databases (MongoDB). Finally, we present performance data and limitations of our approach along with planned future work.

97 MATHEMATICS AND COMPUTING↗

Watershed Workflow: A toolset for parameterizing data-intensive, integrated hydrologic models

Integrated, distributed hydrologic models leverage advances in computational power and data accessibility to improve predictive understanding of the water cycle. While impressive advances in this area of environmental modeling have been accomplished, such models are still rarely used, partially because of difficulty integrating model and data. This research describes the release of Watershed Workflow version 1.2, a new library aiming to automate and enable complex workflows defining inputs to high resolution, integrated, distributed hydrologic models. Watershed Workflow provides tools enabling the discovery, acquisition, mapping, and coordination of watershed geometry, land cover, soil properties, and meteorological data. It enables the construction of unstructured meshes that incorporate this data, and provides tools for automating a “first” simulation on any watershed in the United States. We present the design of the workflow tool, and describe best practices for its usage, culminating in a final example from watershed specification to simulation at the Coweeta Hydrologic Laboratory.

Integrated hydrologic modeling↗

MPI jobs within MPI jobs: A practical way of enabling task-level fault-tolerance in HPC workflows

While the use of workflows for HPC is growing, MPI interoperability remains a challenge for workflow management systems. The MPI standard and/or its implementations provide a number of ways to build multiple-programs-multiple-data (MPMD) applications. These methods present limitations related to fault tolerance, and are not easy to use. In this paper, we advocate for a novel MPI_Comm_launch function acting as the parallel counterpart of a system(3) call. MPI_Comm_launch allows a child MPI application to be launched inside the resources originally held by processes of a parent MPI application. Two important aspects of MPI_Comm_launch is that it pauses the calling process, and runs the child processes on the parent’s CPU cores, but in an isolated manner with respect to memory. This function makes it easier to build MPMD applications with well-decoupled subtasks. We show how this feature can provide better flexibility and better fault tolerance in ensemble simulations and HPC workflows. Here, we report results showing 2x throughput improvement for application workflows with faults, and scaling results for challenging workloads up to 256 nodes.

97 MATHEMATICS AND COMPUTING↗

A workflow to assess the efficacy of brine extraction for managing injection-induced seismicity potential using data from a CO 2 injection site near Decatur, Illinois

Injection of CO 2 for storage in a geologic formation increases pore pressure and alters in situ stresses. Depending on the orientation of any existing fault and fracture planes, such as critically stressed planes, this stress alteration will modify normal stresses acting on planes and could result in frictional sliding and release stored energy in the form of seismicity. Brine extraction (BE) is a technique that can be applied prior to, or during, CO 2 injection to reduce pore pressure for increasing storage capacity and, potentially, for reducing the likelihood of frictional sliding. Here a workflow is described to assess the efficacy of BE for mitigating frictional sliding (i.e., seismicity) during injection and entails: site characterization, stress calculations and failure assessment, static and dynamic modeling, and BE operational planning. Site characterization describes the stress field used to calculate the Coulomb Failure Function (CFF) that constrains allowable pore pressure changes and injection rates in the numerical simulation of CO 2 injection scenarios. The inclusion of BE in the workflow allows for determination of the potential need for pressure reduction, and evaluation of the effectiveness of this operation. Example application of the workflow using an injection field dataset near Decatur, IL, provides insight on fracture planes and stresses at the site, formation properties and the impact of variable CO 2 injection-rate targets on whether BE plans are required. The study workflow indicates that BE could enhance CO 2 injection rate by 39% and correspondingly reduce the potential for injection-induced seismicity as indicated by a reduction in CFF.

58 GEOSCIENCES↗

Experimental evaluation of a flexible I/O architecture for accelerating workflow engines in ultrascale environments

The increasing volume of scientific data and the limited scalability and performance of storage systems are currently presenting a significant limitation for the productivity of the scientific workflows running on both high-performance computing (HPC) and cloud platforms. Clearly needed is better integration of storage systems and workflow engines to address this problem. This paper presents and evaluates a novel solution that leverages codesign principles for integrating Hercules—an in-memory data store—with a workflow management system. We consider four main aspects: workflow representation, task scheduling, task placement, and task termination. As a result, the experimental evaluation on both cloud and HPC systems demonstrates significant performance and scalability improvements over existing state-of-the-art approaches.

97 MATHEMATICS AND COMPUTING↗

CoREC: Scalable and Resilient In-memory Data Staging for In-situ Workflows

The dramatic increase in the scale of current and planned high-end HPC systems is leading new challenges, such as the growing costs of data movement and IO, and the reduced mean time between failures (MTBF) of system components. In-situ workflows, i.e., executing the entire application workflows on the HPC system, have emerged as an attractive approach to address data-related challenges by moving computations closer to the data, and staging-based frameworks have been effectively used to support in-situ workflows at scale. However, the resilience of these staging-based solutions has not been addressed, and they remain susceptible to expensive data failures. Furthermore, naive use of data resilience techniques such as n-way replication and erasure codes can impact latency and/or result in significant storage overheads. In this article, we present CoREC, a scalable and resilient in-memory data staging runtime for large-scale in-situ workflows. CoREC uses a novel hybrid approach that combines dynamic replication with erasure coding based on data access patterns. It also leverages multiple levels of replications and erasure coding to support diverse data resiliency requirements. Furthermore, the article presents optimizations for load balancing and conflict-avoiding encoding, and a low overhead, lazy data recovery scheme. We have implemented the CoREC runtime and have deployed with the DataSpaces staging service on leadership class computing machines and present an experimental evaluation in the article. Here, the experiments demonstrate that CoREC can tolerate in-memory data failures while maintaining low latency and sustaining high overall storage efficiency at large scales.

97 MATHEMATICS AND COMPUTING↗

Managing Dynamic Workflows in BEE

BEE is a powerful tool for: Managing and visualizing scientific workflows; Simplifying workflow execution on HPC and cloud platforms. BEE supports much of the CWL specification. Did not support execution of complex ”scattering” workflows. By introducing the PseudoTask: Can generate tasks to run on variable number of inputs; BEE is another step closer to supporting the entire CWL specification; BEE can now support parallelized workflows with scattering tasks.

97 MATHEMATICS AND COMPUTING↗

Deploying Machine Learning Workflows into HPC environment

Outline: Workflows Overview; Common Workflow Language (CWL), Example of a CWL; BEE Overview; Machine Learning Components; Machine Learning Scientific Workflow using CWL → A new test case for BEE; Discussion: Benefits and Caveats of current ML workflow; Conclusion.

97 MATHEMATICS AND COMPUTING↗

SMART Mobility. Modeling Workflow Development, Implementation, and Results Capstone Report

The U.S. Department of Energy’s Systems and Modeling for Accelerated Research in Transportation (SMART) Mobility Consortium is a multiyear, multi-laboratory collaborative, managed by the Energy Efficient Mobility Systems Program of the Office of Energy Efficiency and Renewable Energy, Vehicle Technologies Office, dedicated to further understanding the energy implications and opportunities of advanced mobility technologies and services. The first three-year research phase of SMART Mobility occurred from 2017 through 2019, and included five research pillars: Connected and Automated Vehicles, Mobility Decision Science, Multi-Modal Freight, Urban Science, and Advanced Fueling Infrastructure. A sixth research thrust integrated aspects of all five pillars to develop a SMART Mobility Modeling Workflow to evaluate new transportation technologies and services at scale. This report summarizes the work of the SMART Mobility Modeling Workflow effort. The SMART Mobility Modeling Workflow was developed to evaluate new transportation technologies such as connectivity, automation, sharing, and electrification through multi-level systems analysis that captures the dynamic interactions between technologies. By integrating multiple models across different levels of fidelity and scale, the Workflow yields insights about the influence of new mobility and vehicle technologies at the system level. For information about the other Pillars, please refer to the relevant pillar’s Capstone Report.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Wilkins: HPC in situ workflows made easy

In situ approaches can accelerate the pace of scientific discoveries by allowing scientists to perform data analysis at simulation time. Current in situ workflow systems, however, face challenges in handling the growing complexity and diverse computational requirements of scientific tasks. In this work, we present Wilkins, an in situ workflow system that is designed for ease-of-use while providing scalable and efficient execution of workflow tasks. Wilkins provides a flexible workflow description interface, employs a high-performance data transport layer based on HDF5, and supports tasks with disparate data rates by providing a flow control mechanism. Wilkins seamlessly couples scientific tasks that already use HDF5, without requiring task code modifications. We demonstrate the above features using both synthetic benchmarks and two science use cases in materials science and cosmology.

HPC↗

Performance Visualization for TAU Instrumented Scientific Workflows

In exascale scientific computing, it is essential to efficiently monitor, evaluate and improve performance. Visualization and especially visual analytics are useful and inevitable techniques in the exascale computing era to enable such a human-centered experience. In this ongoing work, we present a visual analytics framework for performance evaluation of scientific workflows. Ultimately, we aim to solve two current challenges: the capability to deal with workflows, and the scalability toward exascale scenario. On the way to achieve these goals, in this work, we first incorporate TAU (Tuning and Analysis Utilities) instrumentation tool and improve it to accommodate workflow measurements. Then we establish a web-based visualization framework, whose back end handles data storage, query and aggregation, while front end presents the visualization and takes user interaction. In order to support the scalability, a few level-of-detail mechanisms are developed. Finally, a chemistry workflow use case is adopted to verify our methods

97 MATHEMATICS AND COMPUTING↗