SEARCH · Search NASA

Results for “computational workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23

All sources NASA NTRS DOE OSTI

DOE OSTI2026

Tachyon: Intelligent Multi-Scale Modeling of Distributed Resilient Infrastructure and Workflows for Data Intensive HEP Analyses

The DOE High Energy Physics (HEP) program in Neutrino and Collider science drives data-intensive science and simulation on extreme-scale platforms. Modeling and optimizing the complex distributed components from experimental to leadership computing facilities are essential for HEP workflows to achieve required response times and resilience under various conditions. Tachyon proposes a framework for scalable modeling, simulation, and validation of key performance characteristics for the distributed infrastructure between FNAL and ALCF, along with associated HEP workflows.

Carothers, Chris [Rensselaer Poly.]↗

DOE OSTI2022

An introduction to developing GitLab/Jacamar runner analyst centric workflows at Sandia

This document provides very basic background information and initial enabling guidance for computational analysts to develop and utilize GitOps practices within the Common Engineering Environment (CEE) and High Performance Computing (HPC) computational environment at Sandia National Laboratories through GitLab/Jacamar runner based workflows.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Nanometer Scale Imaging to Develop Quantitative Descriptors of Bipolar Membrane Junction Structure

Swings in pH can be achieved by electrically polarizing a bipolar membrane (BPM) to drive water dissociation at the BPM junction for electrochemical conversion and separation processes. BPM junction design is critical to tailor performance for specific applications; however, characterization techniques capable of resolving the nanometer scale physical structure of the junction are limited. We present sample preparation, imaging, and analysis workflows that are adaptable to a variety of BPM junction architectures. Atomic force microscopy produces BPM junction images with nanometer scale lateral resolution for samples with and without a graphene oxide water dissociation catalyst in the junction. Subsequent image segmentation and analysis quantify line edge roughness and catalyst layer thickness as descriptors of junction structure. Comparison of pre- and post-electrodialysis junctions suggests electric field-induced alignment of catalyst particles during electrodialysis. This characterization workflow can inform manufacturing protocols, computational modeling, and failure mode analysis for next-generation BPMs.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2015

Porting Ordinary Applications to Blue Gene/Q Supercomputers

Efficiently porting ordinary applications to Blue Gene/Q supercomputers is a significant challenge. Codes are often originally developed without considering advanced architectures and related tool chains. Science needs frequently lead users to want to run large numbers of relatively small jobs (often called many-task computing, an ensemble, or a workflow), which can conflict with supercomputer configurations. In this paper, we discuss techniques developed to execute ordinary applications over leadership class supercomputers. We use the high-performance Swift parallel scripting framework and build two workflow execution techniques-sub-jobs and main-wrap. The sub-jobs technique, built on top of the IBM Blue Gene/Q resource manager Cobalt's sub-block jobs, lets users submit multiple, independent, repeated smaller jobs within a single larger resource block. The main-wrap technique is a scheme that enables C/C++ programs to be defined as functions that are wrapped by a high-performance Swift wrapper and that are invoked as a Swift script. We discuss the needs, benefits, technicalities, and current limitations of these techniques. We further discuss the real-world science enabled by these techniques and the results obtained.

Maheshwari, Ketan C.↗

DOE OSTI2017

Quantum Virtual Machine (QVM)

There is a lack of state-of-the-art HPC simulation tools for simulating general quantum computing. Furthermore, there are no real software tools that integrate current quantum computers into existing classical HPC workflows. This product, the Quantum Virtual Machine (QVM), solves this problem by providing an extensible framework for pluggable virtual, or physical, quantum processing units (QPUs). It enables the execution of low level quantum assembly codes and returns the results of such executions.

McCaskey, AlexanderJ.↗

DOE OSTI2025

torc (Torc Workflow Management System) [SWR-24-127]

This software package orchestrates execution of a workflow of jobs on distributed computing resources. It is optimized for use on HPCs with Slurm, but also can be used in the cloud and on local computers. Please refer to the documentation at https://nrel.github.io/torc

Thom, Daniel [National Renewable Energy Laboratory↗

DOE OSTI2016

Data Storage and sharing for the long tail of science

Research data infrastructure such as storage must now accommodate new requirements resulting from trends in research data management that require researchers to store their data for the long term and make it available to other researchers. We propose Data Depot, a system and service that provides capabilities for shared space within a group, shared applications, flexible access patterns and ease of transfer at Purdue University. We evaluate Depot as a solution for storing and sharing multiterabytes of data produced in the long tail of science with a use case in soundscape ecology studies from the Human- Environment Modeling and Analysis Laboratory. We observe that with the capabilities enabled by Data Depot, researchers can easily deploy fine-grained data access control, manage data transfer and sharing, as well as integrate their workflows into a High Performance Computing environment.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2024

Enabling Efficient Surveillance, Control, and Automation of Geothermal Operations with Advanced Predictive Analytics

Automation and control of geothermal energy production and operations require reliable and efficient predictive tools. While physics-based simulation offers a comprehensive tool for predicting energy production performance in geothermal systems, predicting the behavior of geothermal reservoirs involves complex multi-physics processes with coupling effects, highly uncertain input parameters and subsurface descriptions. Moreover, building, running, and integrating simulation models into standard model calibration and optimization workflows entail significant technical and computational efforts. An emerging alternative to physics-based simulation is data-driven predictive analytics models that have gained popularity in energy industry. In this report, we develop novel predictive models for integration into real-time fault diagnosis and model predictive control algorithms to improve the efficiency of energy production operations in geothermal reservoirs. The report includes two major research Thrust Areas, that is, the surface power plant and the subsurface reservoir.

15 GEOTHERMAL ENERGY↗

DOE OSTI2026

Advancing SiC clad fuel performance model: bridging micro- and macro-scale models and experimental validation

This report presents a workflow for advancing fuel performance modeling of SiC composite cladding for light-water reactors by linking microscale, experimental data-informed finite element analysis with rod-scale fuel performance codes such as BISON. The workflow uses X-ray computed tomography (XCT) to capture the actual geometry and processing-induced defects of as-fabricated SiC composite tube specimens, particularly porosity and wall-thickness variations, and converts the segmented XCT volumes into image-based finite element meshes for high fidelity structural analysis.

Koyanagi, Takaaki [Oak Ridge National Laboratory (↗

DOE OSTI2016

Quantum Virtual Machine (QVM)

McCaskey, Alexander J.↗

DOE OSTI2020

Data Federation Challenges in Remote Near-Real-Time Fusion Experiment Data Processing

Fusion energy experiments and simulations provide critical information needed to plan future fusion reactors. As next-generation devices like ITER move toward long-pulse experiments, analyses, including AI and ML, should be performed in a wide range of time and computing constraints, from near-real-time constraints, between-shot analysis, and to campaign-wide long-term analysis. However, the data volume, velocity, and variety make it extremely challenging for analyses using only local computational resources. Researchers need the ability to compose and execute workflows spanning edge resources to large-scale high-performance computing facilities.We present Delta, a system to address data analysis challenges, including AI/ML, in fusion science, by leveraging the ADIOS I/O library and middleware, to support executing science workflows over the wide area network for near-real-time streaming. We discuss the data federation challenges in performing remote workflows, focusing on on-going research work in (1) managing, reducing, and streaming data to minimize I/O and data movement overheads, (2) decompressing and reorganizing data for analysis, and (3) executing workflows for automated data analysis. We introduce examples for deep-learning based data analysis for the fusion domain and demonstrate how we use Delta to construct end-to-end workflows for a fusion device in Korea, connecting a remote DOE facility in the USA. The capability demonstrated by this project is the basis for improving the state of the art for near-real-time data federation amongst remote facilities.

Choi, Jong Youl↗

DOE OSTI2017

MaDaTS: Managing Data on Tiered Storage for Scientific Workflows

Scientific workflows are increasingly used in High Performance Computing (HPC) environments to manage complex simulation and analyses, often consuming and generating large amounts of data. However, workflow tools have limited support for managing the input, output and intermediate data. The data elements of a workflow are often managed by the user through scripts or other ad-hoc mechanisms. Technology advances for future HPC systems is redefining the memory and storage subsystem by introducing additional tiers to improve the I/O performance of data-intensive applications. These architectural changes introduce additional complexities to managing data for scientific workflows. Thus, we need to manage the scientific workflow data across the tiered storage system on HPC machines. In this paper, we present the design and implementation of MaDaTS (Managing Data on Tiered Storage for Scientific Workflows), a software architecture that manages data for scientific workflows. We introduce Virtual Data Space (VDS), an abstraction of the data in a workflow that hides the complexities of the underlying storage system while allowing users to control data management strategies. We evaluate the data management strategies with real scientific and synthetic workflows, and demonstrate the capabilities of MaDaTS. Our experiments demonstrate the flexibility, performance and scalability gains of MaDaTS as compared to the traditional approach of managing data in scientific workflows.

Ghoshal, Devarshi↗

DOE OSTI2016

TH-AB-202-03: A Novel Tool for Computing Deliverable Doses in Dynamic MLC Tracking Treatments

Purpose: In tracked dynamic multi-leaf collimator (MLC) treatments, segments are continuously adapted to the target centroid motion in beams-eye-view. On-the-fly segment adaptation, however, potentially induces dosimetric errors due to the finite MLC leaf width and non-rigid target motion. In this study, we outline a novel tool for computing the 4d dose of lung SBRT plans delivered with MLC tracking. Methods: The following automated workflow was developed: A) centroid tracking, where the initial segments are morphed to each 4dCT phase based on the beams-eye-view GTV shift (followed by a dose calculation on each phase); B) re-optimized tracking, in which all morphed initial plans from (A) are further optimised (“warm-started”) in each 4dCT phase using the initial optimisation parameters but phase-specific volume definitions. Finally, both dose sets are accumulated to the reference phase using deformable image registration. Initial plans were generated according to the RTOG-1021 guideline (54Gy, 3-Fx, equidistant 9-beam IMRT) on the peak-exhale (reference) phase of a phase-binned 4dCT. Treatment planning and delivery simulations were performed in RayStation (research v4.6) using our in-house segment-morphing algorithm, which directly links to RayStation through a native C++ interface. Results: Computing the tracking plans and 4d dose distributions via the in-house interface takes 5 and 8 minutes respectively for centroid and re-optimized tracking. For a sample lung SBRT patient with 14mm peak-to-peak motion in sup-inf direction, mainly perpendicular leaf motion (0-collimator) resulted in small dose changes for PTV-D95 (−13cGy) and GTV-D98 (+18cGy) for the centroid tracking case compared to the initial plan. Modest reductions of OAR doses (e.g. spinal cord D2: −11cGy) were achieved in the idealized tracking case. Conclusion: This study presents an automated “1-click” workflow for computing deliverable MLC tracking doses in RayStation. Adding a non-deliverable re-optimized tracking scenario is expected to help quantify plan robustness for more challenging patients with anatomy deformations. We acknowledge support of the MLC tracking research from Elekta AB. MFF is supported by Cancer Research UK under Programme C33589/A19908. Research at ICR is also supported by Cancer Research UK under Programme C33589/A19727 and NHS funding to the NIHR Biomedical Research Centre at RMH and ICR.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2019

Risk-based monitoring designs for detecting CO 2 leakage through abandoned wellbores: An application of NRAP’s WLAT and DREAM tools

We report that one suggested approach to managing the risk of geologic CO 2 storage projects is to perform risk-based subsurface monitoring targeting early leak detection. The challenge in risk-based monitoring is that leakage impacts are expected to be small in comparison with the footprint of the stored CO 2 plume and could be at depths ranging from the depth of storage (often ~1 km bgs) up to surficial aquifers. Addressing this challenge requires a multidisciplinary effort capable of characterizing potential subsurface leakage in terms of likelihood, location, magnitude, impacts to overlying aquifers, and uncertainty as well as innovative monitoring technology design and implementation. The National Risk Assessment Partnership (NRAP) was established to develop related risk assessment tools to provide guidance on such assessments. Here we demonstrate the application workflow of two of the NRAP computational tools, WLAT (Wellbore Leakage Analysis Tool) and DREAM (Designs for Risk Evaluation and Management), to a hypothetical CO 2 storage site based on a study area in the Midwestern United States. This workflow shows how CO 2 leakage may be estimated, evaluated, and monitored in terms of risk. Results provide guidance on monitoring configurations and technologies which minimize the time to leak detection, maximize the probability detection, and provide cost considerations.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

Streaming Data in HPC Workflows Using ADIOS

The “IO Wall” problem, in which the gap between computation rate and data access rate grows continuously, poses significant problems to scientific workflows which have traditionally relied upon using the filesystem for intermediate storage between workflow stages. One way to avoid this problem in scientific workflows is to stream data directly from producers to consumers and avoiding storage entirely. However, the manner in which this is accomplished is key to both performance and usability. This paper presents the Sustainable Staging Transport, an approach which allows direct streaming between traditional file writers and readers with few application changes. SST is an ADIOS “engine”, accessible via standard ADIOS APIs, and because ADIOS allows engines to be chosen at run-time, many existing file-oriented ADIOS workflows can utilize SST for direct application-to-application communication without any source code changes. This paper describes the design of SST and presents performance results from various applications that use SST, for feeding model training with simulation data with substantially higher bandwidth than the theoretical limits of Frontier’s file system, for strong coupling of separately developed applications for multiphysics multiscale simulation, or for in situ analysis and visualization of data to complete all data processing shortly after the simulation finishes.

Podhorszki, Norbert [ORNL] (ORCID:000000019647542X↗

DOE OSTI2020

Deploying Machine Learning Workflows into HPC environment

Outline: Workflows Overview; Common Workflow Language (CWL), Example of a CWL; BEE Overview; Machine Learning Components; Machine Learning Scientific Workflow using CWL → A new test case for BEE; Discussion: Benefits and Caveats of current ML workflow; Conclusion.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Predictive Modeling and Diagnostic Monitoring of Extreme Science Workflows (Final Report)

This proposal addresses a critical issue of performance prediction identified in the report from the ASCR \Computational Modeling of Big Networks (COMBINE)" workshop: "end-to-end performance is not predictable due to a variety of factors. Even when some performance forecasts or predictions can be made, they often cannot explain the reasons why some predictions fail." We will develop new analytical models to predict the end-to-end performance of scientific workflows on DOE computing infrastructures, and use simulations and experimentation to validate and refine these models, as well as to pinpoint the sources of model inaccuracy. We will also use these models to help diagnose application and infrastructure problems, and to adapt the system based on this diagnosis. This section provides background in the areas relevant to the proposed work. RPI’s specific tasks within the Panorama project are as follows: (1) Develop Aspen-Simulation interface for Workflow Model Driven Simulation. (2) Validate manual performance models of two target workflow scenarios with empirical measurement and simulation. (3) Extend ROSS-Aspen API to simulate workflow descriptions when required. (4) Validate Aspen performance models of two target workflow scenarios with automatic performance model empirical measurement and simulation. (5) Design and implement final system to automatically generate Aspen performance models from workflow descriptions (including methods to compensate for limitations of Aspen analytical models). (6) Validate improved Aspen performance models with target workflow on production infrastructure. To date, all the project milestones assigned to us where reached within the best of our abilities over the course of the project performance period. Below describes the key outcome from our collaborative research in a system named, Durango .

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Ad Hoc File Systems for High-Performance Computing

Storage backends of parallel compute clusters are still based mostly on magnetic disks, while newer and faster storage technologies such as flash-based SSDs or non-volatile random access memory (NVRAM) are deployed within compute nodes. Including these new storage technologies into scientific workflows is unfortunately today a mostly manual task, and most scientists therefore do not take advantage of the faster storage media. One approach to systematically include nodelocal SSDs or NVRAMs into scientific workflows is to deploy ad hoc file systems over a set of compute nodes, which serve as temporary storage systems for single applications or longer-running campaigns. This paper presents results from the Dagstuhl Seminar 17202 “Challenges and Opportunities of User-Level File Systems for HPC” and discusses application scenarios as well as design strategies for ad hoc file systems using node-local storage media. The discussion includes open research questions, such as how to couple ad hoc file systems with the batch scheduling environment and how to schedule stage-in and stage-out processes of data between the storage backend and the ad hoc file systems. Also presented are strategies to build ad hoc file systems by using reusable components for networking and how to improve storage device compatibility. Various interfaces and semantics are presented, for example those used by the three ad hoc file systems BeeOND, GekkoFS, and BurstFS. Their presentation covers a range from file systems running in production to cutting-edge research focusing on reaching the performance limits of the underlying devices.

97 MATHEMATICS AND COMPUTING↗