Search NASA⌕ Search

SEARCH · Search NASA

Results for “data pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Data Format and Descriptions for the Alabama Carbon Storage: Data Sharing and Engagement Project

The Alabama Carbon Storage: Data Sharing and Engagement (ACS-DSE) project seeks to develop publicly accessible geologic carbon storage models and data across the southern Gulf Coastal Plain of Alabama. The public online platform developed for this project will include geologic, geophysical, infrastructure, and other relevant datasets and geologic models of the study area. Datasets, model surfaces (e.g. structural contour maps, isolith maps, porosity maps), and infrastructure data (e.g. offshore pipelines, field boundaries) will be downloadable in commonly used file formats. The anticipated primary geologic datasets are well headers, formation tops, average reservoir properties, and core analyses; these will be available as commaseparated values (CSV) text files and MS Excel workbooks. Geophysical logs will be available in Log ASCII Standard (LAS) file format. Modeled surfaces, such as structure contour maps, will be available in ArcGIS formats and text files. Infrastructure data will be available as ArcGIS shapefiles. This document provides information on the data sources and attributes of the datasets.

01 COAL, LIGNITE, AND PEAT↗
DOE OSTI2025

Large Language Model Integration for Knowledge Retrieval and Interaction for the DUNE Experiment

The Deep Underground Neutrino Experiment (DUNE) is a next-generation neutrino experiment that will generate an unprecedented volume of heterogeneous information-from documentation and technical notes to experimental data and reconstruction pipelines. Efficient knowledge retrieval and contextual understanding are increasingly critical for collaboration-wide productivity and onboarding. In this work, we present DUNE-GPT, a prototype framework that leverages large language models (LLMs) and retrieval-augmented generation (RAG) to enable natural-language querying of DUNE's internal documentation and technical resources. The system provides an intelligent interface for DUNE collaborators to interact with experiment-specific knowledge while maintaining data privacy and infrastructure compliance within Fermilab computing resources.

Rafique, A. [Argonne (main)]↗
DOE OSTI2024

CONSTRAINT-INDEPENDENT CONSTANT CTOA DETERMINATION FOR DUCTILE STABLE CRACK GROWTH

Crack tip opening angle (CTOA) has been used as a reliable fracture toughness parameter for decades to characterize stable ductile crack growth for thin-walled aerospace structures in the low-constraint conditions. Recently, the CTOA parameter was also applied to the pipeline industry, and a CTOA test standard ASTM E3039 was thus developed for testing a critical constant CTOA. Research showed that the constant CTOA can reasonably describe fracture toughness required to arrest a dynamic crack propagation for a modern gas pipeline. However, the CTOA fracture criterion requires constraint-independent CTOA toughness against stable ductile crack growth. ASTM E3039 recommends a drop weight tearing test (DWTT) specimen for CTOA testing. Since a shallow crack is used, DWTT measured CTOA may depend on constraint level at the crack tip. To understand if it is the case, this paper evaluates the critical CTOA for a set of fracture toughness tests on single edge notched bend (SENB) specimens with shallow and deep cracks based on four CTOA estimation models. In which, the Ln(P)-LLD linear fit model is similar to that used by ASTM E3039 in the CTOA calculation. Fracture test data for X80 pipeline steel and HY80 structural steel are considered in the CTOA evaluation. The results show that the four CTOA models can determine a crack size-independent constant CTOA over stable ductile crack growth for the SENB specimens. As a result, CTOA determined by ASTM E3039 is constraintindependent and transferable to use for an actual crack propagating in a gas pipeline.

Zhu, Xian-Kui↗
DOE OSTI2025

White Paper: Scalable Digital Twin Capabilities for Aging and Surveillance of Engineered Systems

This white paper presents a multi-year initiative to develop practical, secure, and scalable digital twin capabilities for engineered systems in aging and surveillance contexts—an approach pioneered at the National Nuclear Security Administration (NNSA) Lawrence Livermore National Laboratory (LLNL) that maps directly onto the needs and ambitions of the Navy for ship- and fleet-level digital twins. LLNL’s work in building part- and process-level digital twins for advanced manufacturing, with a vision to scale up to entire factory floors and, ultimately, enterprise-wide digital twins, offers an adaptable pathway for the Navy as it seeks to modernize lifecycle management, readiness, and predictive maintenance across ships and fleets. For our application, we integrate physics-based modeling with automated data ingestion, processing, and AI-driven calibration, creating hybrid models that are both interpretable and data responsive. We modernized legacy workflows, established centralized data infrastructure, automated experimental pipelines, and demonstrated end-to-end coupling of accelerated aging data with finite element simulations via optimization and surrogate modeling. The result is a generalizable framework that supports part-level digital twins today and lays the groundwork for future system-level twins suitable for Navy applications.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Accelerate microstructure evolution simulation using graph neural networks with adaptive spatiotemporal resolution

Abstract Surrogate models driven by sizeable datasets and scientific machine-learning methods have emerged as an attractive microstructure simulation tool with the potential to deliver predictive microstructure evolution dynamics with huge savings in computational costs. Taking 2D and 3D grain growth simulations as an example, we present a completely overhauled computational framework based on graph neural networks with not only excellent agreement to both the ground truth phase-field methods and theoretical predictions, but enhanced accuracy and efficiency compared to previous works based on convolutional neural networks. These improvements can be attributed to the graph representation, both improved predictive power and a more flexible data structure amenable to adaptive mesh refinement. As the simulated microstructures coarsen, our method can adaptively adopt remeshed grids and larger timesteps to achieve further speedup. The data-to-model pipeline with training procedures together with the source codes are provided.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Cryptic cycling by electroactive bacterioplankton in Trout Bog Lake

The potential for extracellular electron transfer (EET) is a prevailing genomic feature of humic lake bacterioplankton. However, there has been little evidence for the substantial ecological contribution predicted by genetics. We hypothesized that anoxygenic phototrophic electrotrophs and accompanying heterotrophic electrogens cycle dissolved organic matter (DOM) between oxidized and reduced states. We predicted that such bacterioplankton would exhibit diel-scale oscillations due to the light dependency of photosynthesis. Using Trout Bog Lake in Wisconsin, USA, as our model ecosystem, we profiled the water column with depth-discrete metagenomic, physiochemical, and electrochemical analyses. We observed variation in oxidation reduction potential (ORP) in response to sunlight, initiating at depths populated by anoxygenic phototrophs with EET genes. We developed an automated buoy to measure electric current flow between many pairs of electrodes simultaneously, observing correlation in electron consumption to sunlight. Our results, combined with published metatranscriptomic analysis, indicate the occurrence of electron cycling between phototrophic oxidation (electrotrophic metabolism) by Chlorobium and anaerobic respiration (electrogenic metabolism) by Geothrix, involving DOM. We also repeatedly observed gradual seasonal increases in hypolimnion ORP throughout summer. These diel and seasonal patterns imply that electroactive DOM mediates the ecology of electroactive bacteria in lakes, controlling humic lake methane emissions.IMPORTANCEWe investigated the physical, chemical, and redox characteristics of a bog lake and electrodes hung therein to test the hypothesis that dissolved organic matter is being cycled between oxidized and reduced states by electroactive bacterioplankton powered by phototrophy. To do so, we performed field-based analyses on multiple timescales using both established and novel instrumentation. We paired these analyses with recently developed bioinformatics pipelines for metagenomics data to investigate genes that enable electroactive metabolism and accompanying metabolisms. Our results are consistent with our hypothesis and yet upend some of our other expectations. Our findings have implications for understanding greenhouse gas emissions from lakes, including electroactivity as an integral part of lake metabolism throughout more of the anoxic parts of lakes and for a longer portion of the summer than expected. Our results also give a sense of what electroactivity occurs at given depths and provide a strong basis for future studies.

carbon emissions↗
DOE OSTI2026

ML–Enabled FPGA Framework for Fast Quantum State Discrimination in Mid-Circuit Measurement Regimes

Accurate and low-latency quantum state discrimination is essential for protocols involving mid-circuit measurement (MCM) and conditional feed-forward. In superconducting quantum systems, conventional readout pipelines transfer measurement data to host processors for post-processing, introducing millisecond-scale delays that far exceed qubit coherence times. To overcome this bottleneck, we present an in-situ machine learning (ML) inference engine implemented on an FPGA for real-time quantum state discrimination. Our design performs inference directly on digitized readout signals with 40 ns latency, supports both qubit and qutrit readout, and enables conditional operations without host-side intervention. This capability is critical for MCM and for feedback-driven protocols such as quantum error correction. We validate the system on superconducting transmon hardware, demonstrating robust discrimination fidelity across multiple qubit and qutrit channels. We further demonstrate conditional qutrit logic driven by FPGA-resident classification, highlighting the potential of low-latency ML-on-FPGA control for NISQ applications and scalable fault-tolerant quantum computing.

Vora, Neel [Lawrence Berkeley National Laboratory ↗
DOE OSTI2024

Field Testing of a Mixed Potential IoT Sensor Platform for Methane Quantification

Emissions of CH 4 from natural gas infrastructure must urgently be addressed to mitigate its effect on global climate. With hundreds of thousands of miles of pipeline in the US used to transport natural gas, current methods of surveying for leaks are inadequate. Mixed potential sensors are a low cost, field deployable technology for remote and continuous monitoring of natural gas infrastructure. We demonstrate for the first time a field trial of a mixed potential sensor device coupled with machine learning and internet-of-things platform at Colorado State University’s Methane Emissions Technology Evaluation Center (METEC). Emissions were detected from a simulated buried underground pipeline source. Sensor data was acquired and transmitted from the field test site to a remote cloud server. Quantification of concentration as a function of vertical distance is consistent with previously reported transport modelling efforts and experimental surveys of methane emissions by more sophisticated CH 4 analyzers.

03 NATURAL GAS↗
DOE OSTI2020

Adapting DESGW Single Epoch Image Differencing to the LSST Software Stack

As gravitational wave detections become an ever-increasing part of cosmological research, the Gravitational Waves Group of the Dark Energy Survey (DESGW) seeks to match and interpret optical observations for these short-lived, transient events.[1]To provide timely analysis for each follow up, DESGW has developed a custom Single Epoch Differencing Pipeline for processing optical data obtained from the Dark Energy Camera (DECam).[2]While effective, the pipeline is a custom design and thus requires extra maintenance and specialized knowledge to operate and modify at a time when new projects are taking precedence.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Advancing Additive Manufacturing Through Artificial Intelligence–Powered, High-Throughput, Nondestructive Characterization and Process Optimization

This Cooperative Research and Development Agreement (CRADA) between Oak Ridge National Laboratory (ORNL) and ZEISS Industrial Metrology has demonstrated the transformative potential of artificial intelligence (AI)-enabled x-ray computed tomography (XCT) to accelerate the qualification and certification of additively manufactured (AM) parts. At the core of this effort is Simurgh, an AI-powered XCT reconstruction framework jointly advanced by ORNL and ZEISS that integrates computer-aided design (CAD) models, physics-based simulations, and deep learning to overcome the long-standing challenges of metal artifact correction, long scan durations, and limited flaw detectability in dense and geometrically complex components. Simurgh enables high-throughput, high-quality 3D reconstruction from sparse and fast scans, which reduces XCT acquisition times by more than an order of magnitude and simultaneously improves defect detection limits by up to fourfold compared with industry-standard approaches. This capability reduces scan costs by more than 50%, lowers labor overhead, and makes XCT characterization economically viable for routine industrial use. By enabling reliable flaw detection in minutes rather than hours, Simurgh facilitates real-time feedback loops for process parameter optimization, which was highlighted in a recent npj Computational Materials (a Nature journal) issue. In the published study, more than 100 alloy coupons were characterized within a single day. This work represents a tenfold acceleration in the development of novel AM alloys and processes compared with conventional workflows. The ZEISS collaboration has also demonstrated the scalability of Simurgh to diverse application domains, including aerospace, nuclear, automotive, and biomedical components; in these applications, ensuring structural integrity is paramount. By drastically reducing barriers to XCT adoption, this partnership has laid the foundation for digital twins and data-driven certification pipelines and directly addressed bottlenecks in qualifying new materials and designs. Together, ORNL and ZEISS have shown that Simurgh advances the state of the art in nondestructive evaluation and aligns with the broader mission of enabling Industry 4.0 manufacturing ecosystems, in which intelligent, cost-effective, rapid quality assurance is integral to accelerating innovation and ensuring safety in critical applications.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Advancing Additive Manufacturing Through Artificial Intelligence–Powered, High-Throughput, Nondestructive Characterization and Process Optimization

This Cooperative Research and Development Agreement (CRADA) between Oak Ridge National Laboratory (ORNL) and ZEISS Industrial Metrology has demonstrated the transformative potential of artificial intelligence (AI)-enabled x-ray computed tomography (XCT) to accelerate the qualification and certification of additively manufactured (AM) parts. At the core of this effort is Simurgh, an AI-powered XCT reconstruction framework jointly advanced by ORNL and ZEISS that integrates computer-aided design (CAD) models, physics-based simulations, and deep learning to overcome the long-standing challenges of metal artifact correction, long scan durations, and limited flaw detectability in dense and geometrically complex components. Simurgh enables high-throughput, high-quality 3D reconstruction from sparse and fast scans, which reduces XCT acquisition times by more than an order of magnitude and simultaneously improves defect detection limits by up to fourfold compared with industry-standard approaches. This capability reduces scan costs by more than 50%, lowers labor overhead, and makes XCT characterization economically viable for routine industrial use. By enabling reliable flaw detection in minutes rather than hours, Simurgh facilitates real-time feedback loops for process parameter optimization, which was highlighted in a recent npj Computational Materials (a Nature journal) issue. In the published study, more than 100 alloy coupons were characterized within a single day. This work represents a tenfold acceleration in the development of novel AM alloys and processes compared with conventional workflows. The ZEISS collaboration has also demonstrated the scalability of Simurgh to diverse application domains, including aerospace, nuclear, automotive, and biomedical components; in these applications, ensuring structural integrity is paramount. By drastically reducing barriers to XCT adoption, this partnership has laid the foundation for digital twins and data-driven certification pipelines and directly addressed bottlenecks in qualifying new materials and designs. Together, ORNL and ZEISS have shown that Simurgh advances the state of the art in nondestructive evaluation and aligns with the broader mission of enabling Industry 4.0 manufacturing ecosystems, in which intelligent, cost-effective, rapid quality assurance is integral to accelerating innovation and ensuring safety in critical applications.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Computationally Guided and Experimentally Validated Design of Custom Chelators for Critical Mineral Recovery

Selective, high throughput separation of target critical metals from complex environments such as fly ash leachates and mining process streams presents a significant challenge for economical production. Custom chelators and sorbents are an attractive technology for selective metal extraction, however it can be difficult to predict their performance, and significant experimental efforts are often required to develop chelating technologies. Here, we present a computational strategy focused on modelling chelator-metal binding interactions and benchmark these results versus experimental data. A computational pipeline combining forcefield, semiempirical, and meta-GGA methods with a thermodynamic framework optimized for error cancellation has been developed to predict binding energies of chelator complexes towards critical mineral recovery applications. This approach, originally validated on [2.2.2] cryptates binding mono- and divalent cations, demonstrated robust predictive capabilities with an R2 of 0.850 against experimental aqueous binding energies. The workflow includes metadynamics for exploring high-dimensional potential energy surfaces and a cluster-continuum model for accurate yet computationally efficient solvation modeling. Error cancellation between solvation energies of free and chelator-coordinated ions enables faster convergence, even with finite cluster sizes. Initial studies on the cryptates revealed consistent metal-ligand coordination patterns, with systematic variations influenced by ion size and charge, highlighting key structural features linked to binding selectivity. Further studies of a proprietary chelator have resulted in identification of previously unreported selectivity towards economically significant metals, which in-house experiments have confirmed, demonstrating the feasibility of this approach. By applying this methodology to new chelators targeting critical minerals such as lithium, cobalt, nickel and other strategic metals, we aim to accelerate the discovery of next-generation chelators for efficient recovery, recycling, and separation processes. This computational framework serves as the backbone of a high-throughput design pipeline tailored for sustainable resource utilization and may be applied to a wide range of systems to meet experimental needs.

computational materials↗
DOE OSTI2024

BRAKER3: Fully automated genome annotation using RNA-seq and protein evidence with GeneMark-ETP, AUGUSTUS, and TSEBRA

Gene prediction has remained an active area of bioinformatics research for a long time. Still, gene prediction in large eukaryotic genomes presents a challenge that must be addressed by new algorithms. The amount and significance of the evidence available from transcriptomes and proteomes vary across genomes, between genes, and even along a single gene. User-friendly and accurate annotation pipelines that can cope with such data heterogeneity are needed. The previously developed annotation pipelines BRAKER1 and BRAKER2 use RNA-seq or protein data, respectively, but not both. A further significant performance improvement integrating all three data types was made by the recently released GeneMark-ETP. We here present the BRAKER3 pipeline that builds on GeneMark-ETP and AUGUSTUS, and further improves accuracy using the TSEBRA combiner. BRAKER3 annotates protein-coding genes in eukaryotic genomes using both short-read RNA-seq and a large protein database, along with statistical models learned iteratively and specifically for the target genome. We benchmarked the new pipeline on genomes of 11 species under an assumed level of relatedness of the target species proteome to available proteomes. BRAKER3 outperforms BRAKER1 and BRAKER2. The average transcript-level F1-score is increased by about 20 percentage points on average, whereas the difference is most pronounced for species with large and complex genomes. BRAKER3 also outperforms other existing tools, MAKER2, Funannotate, and FINDER. The code of BRAKER3 is available on GitHub and as a ready-to-run Docker container for execution with Docker or Singularity. Overall, BRAKER3 is an accurate, easy-to-use tool for eukaryotic genome annotation.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Open Reproducible Electron Microscopy Data Analysis

Electron microscopy (EM) is a cornerstone technique in the materials and biological sciences capable of imaging structures at nano- to atomic-scale resolution. Advances in technologies mean that one acquires datasets at increasing data rates and sizes. These advancements present enormous opportunities for researchers to understand complex systems. However, processing the resulting large-scale, complex data in a reproducible and shareable way is a real challenge for researchers. The building, managing, and maintaining complex workflows in a reproducible manner requires extensive knowledge in several areas outside the researchers’ core skill sets, such as software engineering, data science, and high-performance computing (HPC). Our work demonstrates an innovative approach to solving these problems, enabling reproducible EM data analysis through container encapsulated pipelines. Using modern container technologies, we encapsulate processing elements and connect them using shared memory. We expose user-friendly, advanced algorithms and tools to allow end users to utilize without expert programming skills. The platform enables reproducible, scalable, shareable pipelines for the analysis and visualization of EM data. Focusing on interoperability, we leverage the DOE and other agencies’ existing investments to provide a powerful software platform for EM data analysis.

Harris, Christopher↗
DOE OSTI2025

UBW (USLCI-Brightway2) [SWR-25-169]

Life cycle inventory (LCI) data are critical for robust life cycle assessment (LCA), yet many widely used datasets such as the U.S. Life Cycle Inventory (USLCI) are not natively compatible with advanced modeling frameworks like Brightway2. This work presents an automated pipeline to transform USLCI data into a fully functional Brightway2 project. The workflow performs systematic data cleaning, resolves duplicate process and exchange identifiers, and applies allocation to multi-output processes. Technosphere and biosphere flows are harmonized through unit conversions and a bridge mapping to the biosphere3 database, with comprehensive logging of missing flows and cutoff issues. The resulting Brightway2 database is validated using matrix diagnostics to ensure consistency of the technosphere, and is benchmarked via life cycle impact assessment (LCIA) methods such as ReCiPe and IPCC GWP. Outputs include reproducible CSV exports of corrected processes, elementary flows, characterization factors, and LCIA results, alongside backup utilities for project sharing. This pipeline lowers barriers for integrating USLCI data into open-source LCA workflows, enabling reproducible, validated LCA inventories within the Brightway 2 framework.

Ghosh, Tapajyoti [National Laboratory of the Rocki↗
DOE OSTI2022

SDSS-IV MaNGA: pyPipe3D Analysis Release for 10,000 Galaxies

We present here the analysis performed using the pyPipe3D pipeline for the final MaNGA data set included in the Sloan Digital Sky Survey data release 17. This data set comprises more than 10,000 individual data cubes, being the integral field spectroscopic (IFS) galaxy survey with the largest number of galaxies. pyPipe3D processes the IFS data cubes to extract spatially resolved spectroscopic properties of both the stellar population and the ionized gas emission lines. A brief summary of the properties of the sample and the characteristics of the analyzed data are included. The article provides details of: (i) the analysis performed; (ii) a description of the pipeline; (iii) the adopted stellar population library; (iv) the morphological and photometric analysis; (v) the adopted data model for the spatially resolved properties derived; and (vi) the individual integrated and characteristic galaxy properties included in the final catalog. Comparisons with the results from a previous version of the pipeline for earlier data releases and from other tools using this data set are included. A practical example of how to use the full data set and the final catalog illustrates how to handle the delivered product. Our full analysis can be accessed and downloaded from our web page.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

mphys-surrogate-model

This repository contains python scripts for building and studying reduced-order-modeling representations of droplet coalescence for eventual use in atmospheric models. The included data are generated from high-fidelity superdroplet methods and are utilized by machine learning pipelines to build data-driven models of droplet size distributions that evolve under coalescence. This repository further includes scripts to determine prediction (uncertainty) intervals on the data-driven model products based on conformal prediction.

Katona, JonasE [Lawrence Livermore National Labora↗
DOE OSTI2024

ICED: An Integrated CGRA Framework Enabling DFVS-Aware Acceleration

oarse-grained reconfigurable arrays (CGRAs) are a promising solution to enable energy-efficient acceleration of applications from different domains. By leveraging reconfiguration at the functional level, they can adapt to significantly different computational patterns. Existing CGRA mapping approaches extract instruction-level parallelism, exploit loop-pipelining opportunities, guarantee the data dependency, and target high throughput of a given loop. However, the recurrence data-dependency in the DFG and the mismatch between required and available computing/communication resources complicate the mapping, and might lead to significant unbalances in the utilization of the CGRA's tiles. This results in wasted power for tiles with low utilization. Applying dynamic voltage and frequency scaling (DVFS) can potentially solve this challenge and improve energy efficiency by adjusting voltage and frequency of different tiles independently. CGRAs have also been successful in accelerating data-dependent streaming applications. However, in these applications, the execution time of each kernel in the pipeline might dynamically vary depending on the characteristics of the input. This also leads to under-utilization of resources for the dynamically changing kernels that do not limit the application throughput. DVFS can also improve energy efficiency for these applications by dynamically changing the voltage and frequency levels of tiles that host non performance-constraining kernels. This paper proposes ICEDTEA -- an integrated DVFS-aware framework to map applications on CGRAs that support power islands. ICEDTEA proposes a CGRA architecture supporting DVFS islands at varying granularity (from a single tile to a group of tiles) and the related DVFS-aware compilation and mapping toolchain. ICEDTEA is the first work that introduces DVFS support for spatio-temporal CGRAs at power-island levels. The experimental evaluation shows that ICEDTEA improves average utilization by 2.3$\times$ and energy-efficiency by 1.32$\times$ over a conventional CGRA. With streaming applications, ICEDTEA improves energy efficiency by 1.12$\times$ over a state-of-the-art CGRA that introduces partial dynamic reconfiguration to adapt to variations in kernels' throughput.

Tan, Cheng↗