SEARCH · Search NASA

Results for “Workflows”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,279 records · Page 71

All sources NASA NTRS DOE OSTI

DOE OSTI2019

Software Defined Architectures for Data Analytics

Data analytics applications increasingly are complex workflows composed of phases with very different program behaviors (e.g., graph algorithms and machine learning, algorithms operating on sparse an dense data structures, etc). To reach the levels of efficiency required to process these workflows in real time, upcoming architectures will need to leverage even more workload specialization. If, at one end, we may find even more heterogenous processors composed by a myriad of specialized processing elements, at the other end we may see novel reconfigurable architectures, composed of sets of functional units and memories interconnected with (re)configurable on-chip networks, able to adapt dynamically to adapt the workload characteristics. Field Programmable Gate Arrays are more and more used for accelerating various workloads and, in particular, inferencing in machine learning, providing higher efficiency than other solutions. However, their fine grained nature still leads to issues for the design software and still makes dynamic reconfiguration impractical. Future, more coarse-grained architectures could offer the features to execute diverse workloads at high efficiency while providing better reconfiguration mechanisms for dynamic adaptability. Nevertheless, we argue that the challenges for reconfigurable computing remain in the software. In this position paper, we describe a possible toolchain for reconfigurable architectures targeted at data analytics.

FPGA, Dataflow, Reconfigurable Architectures, Soft↗

DOE OSTI2021

Extending C++ for Heterogeneous Quantum-Classical Computing

In this report we present qcor - a language extension to C++ and compiler implementation that enables heterogeneous quantum-classical programming, compilation, and execution in a single-source context. Our work provides a first-of-its-kind C++ compiler enabling high-level quantum kernel (function) expression in a quantum-language agnostic manner, as well as a hardware-agnostic, retargetable compiler workflow targeting a number of physical and virtual quantum computing backends. qcor leverages novel Clang plugin interfaces and builds upon the XACC system-level quantum programming framework to provide a state-of-the-art integration mechanism for quantum-classical compilation that leverages the best from the community at-large. qcor translates quantum kernels ultimately to the XACC intermediate representation, and provides user-extensible hooks for quantum compilation routines like circuit optimization, analysis, and placement. This work details the overall architecture and compiler workflow for qcor, and provides a number of illuminating programming examples demonstrating its utility for near-term variational tasks, quantum algorithm expression, and feed-forward error correction schemes.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Understanding the Computing and Analysis Needs for Resiliency of Power Systems from Severe Weather Impacts

As the frequency and intensity of severe weather has increased, its effect on the electric grid has manifested in the form of significantly more and larger outages in the United States. This has become especially true for regions that were previously isolated from weather extremes. In this paper, we analyze the weather impacts on the electric power grid across a variety of weather conditions, draw correlations, and provide practical insights into the operational state of these systems. High resolution computational modeling of specific meteorological variables, computational approaches to solving power system models under these conditions, and the types of resiliency needs are highlighted as goal-oriented computing approaches are being built to address grid resiliency needs. An example analysis correlating outages to 1km day-ahead weather from two historical winter storms, calculated on a large cluster using a combination of interpolated and extrapolated inputs from multiple instrumented sites to workflows that produce primary meteorological outputs, is shown as initial proof of concept.

analysis↗

DOE OSTI2023

Uncertainty Quantification of Metal Additive Manufacturing Processing Conditions Through the use of Exascale Computing

Metal additive manufacturing (AM) is a disruptive manufacturing technology that opens the design space for parts outside those possible from traditional manufacturing methods. In order to accelerate industry and R&D needs to certify AM parts, the Exascale Additive Manufacturing project (ExaAM) has developed a suite of exascale-ready computational tools to model the process-to-structure-to-properties (PSP) relationship for additively manufactured metal components. One such tool is an uncertainty quantification (UQ) pipeline to quantify the effect that uncertainty in processing conditions has on local mechanical responses. We present an overview of this pipeline and its required simulation and workflow codes. Using the Oak Ridge National Laboratory’s (ORNL) exascale computer, Frontier, we utilize this pipeline to cross multiple length and time scales to predict the local mechanical response of a location within a complex AM bridge part, AMB2018-01 produced by the National Institute of Standards and Technology (NIST) as part of their 2018 AM-Bench test series. Our results are then compared to experimental mechanical tests of parts from the NIST build to quantify the error in the ExaAM UQ workflow.

Carson, Robert↗

DOE OSTI2023

Empowering Scientific Discovery Through Computing at the Advanced Photon Source

This paper explores the challenges and solutions for managing and processing the vast amount of data generated by the Advanced Photon Source (APS), a synchrotron light source facility producing ultra-bright x-rays for diverse scientific domains. With 68 experimental beamlines covering materials research, biology, and more, the APS serves a wide user base across academia, government, and industry. The ongoing upgrade of the APS storage ring and installation of new instruments will amplify data generation and processing demands. This paper discusses the approach to address these demands through automated data processing using standardized workflows that produce faster scientific insights. The APS Data Management System coordinates various data related tasks to manage storage, data transfer, metadata cataloging, data processing, and interfaces with tools provided by Globus. Through integration with the Argonne Leadership Computing Facility (ALCF), APS users can efficiently access high-performance computing resources. Standardized workflows have led to reduced computational burdens on scientists and greater accessibility of high performance computing resources. We demonstrate how standardization and collaboration enable scientists to rapidly convert raw data into meaningful scientific results, establishing a streamlined path from data collection to analysis and ultimately to publication.

Parraga, Hannah↗

DOE OSTI2025

From Edge to HPC: Investigating Cross-Facility Data Streaming Architectures

In this paper, we investigate three cross-facility data streaming architectures, Direct Streaming (DTS), Proxied Streaming (PRS), and Managed Service Streaming (MSS). We examine their architectural variations in data flow paths and deployment feasibility, and detail their implementation using the Data Streaming to HPC (DS2HPC) architectural framework and the SciStream memory-to-memory streaming toolkit on the production-grade Advanced Computing Ecosystem (ACE) infrastructure at Oak Ridge Leadership Computing Facility (OLCF). We present a workflow-specific evaluation of these architectures using three synthetic workloads derived from the streaming characteristics of scientific workflows. Through simulated experiments, we measure streaming throughput, round-trip time, and overhead under work sharing, work sharing with feedback, and broadcast and gather messaging patterns commonly found in AI-HPC communication motifs. Our study shows that DTS offers a minimal-hop path, resulting in higher throughput and lower latency, whereas MSS provides greater deployment feasibility and scalability across multiple users but incurs significant overhead. PRS lies in between, offering a scalable architecture whose performance matches DTS in most cases.

George, Anjus [ORNL] (ORCID:0000000179737061)↗

DOE OSTI2018

Cinema Toolkit

Cinema is a workflow and toolkit for capturing, storing and exploring extreme scale scientific data. It is a highly interactive image-based approach to data analysis and exploration that promotes investigation of large scientific datasets, and is easily integrated into existing workflows through extensions to widely used open source tools. In short, many different images of the dataset are captured and stored in a database, along with metadata describing the images. This provides a method of exploration that is unlike other current methods. This novel approach supports interactive exploration of a wide range of results, while still significantly reducing data movement and storage. The Cinema toolkit includes database specifications and software such as database writers, database viewers, algorithms and other support software that enable a wide variety of applications to write, operate on, and view Cinema databases.

ECP↗

DOE OSTI2020

HERON

HERON is a modeling toolset and plugin for RAVEN to accelerate stochastic technoeconomic assessment of the economic viability of various grid-energy system configurations, especially with application to electrical grids and integrated energy systems (IES). The frameworks enabling this toolset are INL's RAVEN and its TEAL plugin. The toolset includes automated RAVEN workflow generation tools for performing stochastic analysis and optimization driven by technoeconomic factors. The workflows generated employ RAVEN for parametric or optimizing study of grid component characteristics, and again for stochastic sampling of regulated and deregulated markets, load, and weather histories. The toolset further provides methods for economically-optimal dispatch of system components given the stochastic samples of market, load, and weather histories. The toolset allows assembling of existing and new models through the RAVEN interface to the Modelica process modeling language.

Talbot, PaulW↗

DOE OSTI2021

Scoreboard

Emerging HPC machines have given rise to enhanced compute power that far outstrips the machine's ability to save large scale results for post-processing. To combat this, in situ data analysis techniques are slowly being adopted. With in situ data management favoring workflows composed of multiple simulations and analyses connected in transit on heterogeneous machines, scientists and engineers need a tool that enables them to create data extracts, visualizations, and interactively monitor and steer their simulations. Scoreboard Phase II is a next generation analysis software that supports composite in transit workflows on heterogeneous architectures and restores interactivity to in situ data analysis through simulation monitoring and computational steering. Scoreboard provides a simulation dashboard with graphs of metrics over time, controls for setting custom simulation steering parameters, controls for managing the set of data extracts being produced in the simulation, as well as the ability to explore data extracts, all from a web browser. Realizing the vision outlined in this project required research into making a system that integrates end to end from simulations all the way to the user. In situ tools generally suffer from complexity and excessive software dependencies. Scoreboard, by contrast, is easy to build and integrate into simulation codes and it provides first class FORTRAN support. The Scoreboard library is capable of in situ and in transit data analysis that can produce data extracts commonly needed for Computational Fluid Dynamics (CFD) analysis. Simulations can transparently stage data in transit to a Scoreboard Endpoint program, which can accept their data and produce the requested data extracts. This lets simulations return to their work while the Endpoint works on the analysis. Efficiently staging the data at scale was a topic of this research. Scoreboard provides the means to let the user manage data extracts and monitor/steer many simulations from a web browser. This area of the research focused on discovery of in transit network components to expose and control their steering parameters within an interactive browser-based user interface that includes: system topology, gathered metrics, notifications, dynamically-generated steering controls, and exploration of visualization data products.

Whitlock, BradJoseph [Intelligent Light] (00000001↗

DOE OSTI2020

Mesoscale Science Data Analytics

This is software that will be used to do data analytics in experimental workflows for x-ray mesoscale science. This tool set will provide a mechanism for supporting experiments in many ways, from collecting calibration information and raw data, to managing and viewing data to extracting crystallographic and physical parameters. It will eventually include development of a fully automated workflow that will include statistical information and prediction capabilities to support the scientists in decision-making and replanning their experiments when necessary.

Sweeney, Christine↗

DOE OSTI2023

Auto_PDI

Protein-DNA Interaction Workflow (PDI Workflow), a pipeline that focuses on generating high-quality docking and molecular dynamics simulations for Protein-DNA complexes. This allows us to take DNA sequences with unknown tertiary structures, accurately predict their structure, dock them with the desired target protein, and then simulate their interactions using molecular dynamics simulations

Kumar, Neeraj↗

DOE OSTI2023

pnnl/PTMPSI

PTM-Psi is a Python 3 package that combines several capabilities to streamline the workflow to interrogate the impact of PTMs on proteins using well-established software packages. The workflow of the PTM-Psi software package includes input files and launch instances from standard packages such as AlphaFold, NWChem, GROMACS, and the Autodock Suite

Mejia-Rodriguez, Daniel↗

DOE OSTI2023

LLNL Automized Surface Titration Model

The LLNL Automized Surface Titration Model (L-ASTM) is a community data-driven surface complexation modeling workflow for simulating potentiometric titration of mineral surfaces. The model accepts raw experimental potentiometric titration data formatted in a findable, accessible, interoperable, and reusable (FAIR) structure. The workflow was coded in Python and coupled to PHREEQC for surface complexation modeling and PEST for data fitting and parameter estimation.

Solchan, Han↗

DOE OSTI2024

Clearance Tracker

The manual process of managing security clearance was cumbersome and inefficient, involving the emailing of a fillable PDF from person to person. There was no visibility into the progress of a clearance submission. Clearance Tracker provides an automated workflow with data validation and notifications to manage the clearance process. The automated workflow routes notifications to individuals when their input is required. All participants can track the progress of clearance requests.

Short, RickyD↗

DOE OSTI2024

Synthetic Biology PacBio/JAWS QC Analysis (PBJ) v3.0

This software was designed as a sequence validation tool for the assembly of synthetic constructs. It analyzes FASTQ files against a list of reference sequences, combining the results from eight sequencing libraries to generate a summary, and the files needed to view the results in the Integrative Genomics Viewer (IGV) application for manual verification. This was developed for FASTQ files generated by PacBio sequencing, but could be used on any FASTQ files that do not have paired end reads. It can be used to analyze one - eight libraries at a time, and assumes that each construct sequence in the reference will be in each pool, however, this is not a requirement. This is used to identify which libraries of pooled sequences contains a perfect match, or fixable match to the reference file. This pipeline uses many freely available open source libraries, the value added is that in our application the steps of the pipeline are defined in Workflow Description Language (WDL) and run through the Cromwell workflow engine in Docker containers, for easy distribution and set up, as well as the user friendly html summary that is generated.

Simirenko, Lisa↗

DOE OSTI2024

SynBio QC Dual Barcode QC (DBC) v1.0

This software was designed as a sequence validation tool for the assembly of synthetic constructs, where the constructs have a high degree of similarity and thus are barcoded prior to the sequencing library prep. It demultiplexes each FASTQ file for each barcode, then analyzes the resulting FASTQ files against a list of reference sequences for that barcode/library, combining the results from eight sequencing libraries to generate a summary, and the files needed to view the results in the Integrative Genomics Viewer (IGV) application for manual verification. This was developed for FASTQ files generated by PacBio sequencing, but could be used on any FASTQ files that do not have paired end reads. It can be used to analyze one - eight libraries at a time. Each construct is independently analyzed with only the sequences with the same barcode, in the same pooled library. Then the results are combined into a user friendly summary. This is used to identify which libraries of pooled sequences contains a perfect match, or fixable match to the reference file. This pipeline uses many freely available open source libraries, the value added is that in our application the steps of the pipeline are defined in Workflow Description Language (WDL) and run through the Cromwell workflow engine in Docker containers, for easy distribution and set up, as well as the user friendly html summary that is generated.

Simirenko, Lisa↗

DOE OSTI2024

Tardigrade-examples V0.1.0

Tardigrade-examples is a repository of computational workflows that exercise the Tardigrade software package. The Tardigrade software package is an implementation of Eringen’s micromorphic continuum theory with capabilities to support multiscale material modeling. These capabilities include homogenization through the Micromorphic Filter, calibration of micromorphic material models, and macroscale simulation in Tardigrade-MOOSE. This repository investigates continuum upscaling of various direct numerical simulations (DNS) conducted in Abaqus finite element (FE), Ratel FE, and GEOS material point method (MPM) software. Verification of the upscaling workflow is first investigated by considering DNS of trivial stress states for homogeneous materials, results of which indicate that classical continuum behavior is recovered as expected. DNS of heterogeneous materials are then considered.

Allard, Thomas↗

DOE OSTI2023

LowFive v1.0

LowFive is a new data transport layer based on the HDF5 data model, for in situ workflows. Executables using LowFive can communicate in situ (using in-memory data and MPI message passing), reading and writing traditional HDF5 files to physical storage, and combining the two modes. Minimal and often no source-code modification is needed for programs that already use HDF5. LowFive maintains deep copies or shallow references of datasets, configurable by the user. More than one task can produce (write) data, and more than one task can consume (read) data, accommodating fan-in and fan-out in the workflow task graph. LowFive supports data redistribution from n producer processes to m consumer processes.

Morozov, Dmitriy↗