Search NASA⌕ Search

SEARCH · Search NASA

Results for “data pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Image processing pipeline for AI-driven nanoparticle megalibrary characterization

Recent innovations have made it possible to produce megalibraries, millions of structurally and compositionally distinct nanoparticles on a chip. These megalibraries yield vast volumes of data that are impossible to analyze manually, necessitating the development of automated tools. In previous work, we created a binary classification machine learning model to select quality nanoparticle images for downstream analysis. In this work, we show that adding a custom image processing step before training can produce significantly higher-performing models in a fraction of the time and make them more robust to different image noise levels and microscope acquisition settings. The image processing pipeline proposed here effectively cleans raw nanoparticle images, enhances key features, and allows us to use much lower resolution images and simpler neural network model architectures. These features result in higher performance and significant cost savings. Experiments demonstrate superior performance relative to baseline, including an 18.2% improvement in recall and a 13.1% increase in accuracy. Given the high cost of downstream analysis, it is critical to minimize false positives, and our best-performing model reaches a precision of 95.9% and a weighted F-score of 95.1% on an unseen test set. Additionally, model training time is reduced from hours to less than a minute. We also show that, using this custom image processing pipeline, model performance is significantly improved at lower pixel resolutions compared to downsizing alone. We expect that adopting this pipeline for AI-driven automated nanoparticle characterization will allow researchers to rapidly and accurately analyze much greater volumes of data, thereby accelerating materials discovery.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
DOE OSTI2022

In‐Silico Device Performance Prediction of Cosensitizer Dye Pairs for Dye‐Sensitized Solar Cells

Abstract Endeavors in the field of dye‐sensitized solar cells (DSCs) have shown great promise when adopting a data‐driven approach to materials discovery, such as successful molecular‐scale predictions of light‐harvesting chromophores. However, predictions of DSC dyes would become much more sophisticated if a molecular‐to‐macroscopic DSC device prediction methodology existed. Thereby, a fully computational pipeline is presented that predicts device‐performance parameters of DSCs which contain varying dye combinations. Optimal pairing of complementary dyes is identified via a data‐driven workflow that affords cosensitized DSCs with maximum power‐conversion efficiencies. Six high‐performing DSC dyes are paired with partner dyes that are screened from a database of 8488 compounds using sequential heuristic filters. Existing models that predict short‐circuit‐current density ( J SC ) and open‐circuit voltage ( V OC ) parameters are adapted to predict singly sensitized and cosensitized DSC performance. The predictions for J sc values of singly sensitized devices match experimental literature values with comparable accuracy to more computationally costly methods. Five out of six dye pairings are predicted to have greater J SC values when cosensitized compared to their corresponding singly sensitized devices, including two pairs that show strong J sc boosts of +13% and +12% when cosensitized. Thus, the prospect of an entirely in‐silico prediction pipeline for DSC performance that can be used to realize the fully automated design of optimized cosensitized DSCs is demonstrated.

14 SOLAR ENERGY↗
DOE OSTI2023

Topological Analysis of Temporal Hypergraphs

In this work we study the topological properties of temporal hypergraphs. Hypergraphs provide a higher dimensional generalization of a graph that is capable of capturing multi-way connections. As such, they have become an integral part of network science. A common use of hypergraphs is to model events as hyperedges in which the event can involve many elements as nodes. This provides a more complete picture of the event in comparison to the standard dyadic connection limitation of a graph. However, a common attribution to events is temporal information as an interval for when the event occurred. Consequently, a temporal hypergraph is born which accurately captures both the temporal information of events as well as their multi-way connections. Common tools for studying these temporal hypergraphs typically use summary statistics of snapshots from a sliding window procedure to capture changes in the underlying dynamics. However, these do not provide insight into how the changing structure of the hypergraph evolves and which components of the temporal hypergraph persist and are influential to the underlying system. To alleviate this need we leverage zigzag persistence from the field of Topological Data Analysis (TDA) to study the change in topological structure of time-evolving hypergraphs. We apply our pipeline to both a cyber security and social network dataset and show how the topological structure of their temporal hypergraphs change and can be used to understand the underlying dynamics.

hypergraphs, topological data analysis, zigzag per↗
DOE OSTI2025

Quality-Controlled Meteorological Data from the Flood Control District of Maricopa County (FCDMC) Network, Phoenix, Arizona (1987-2024)

This dataset contains 15- or 30-minute interval meteorological data from the Flood Control District of Maricopa County (FCDMC), Arizona, USA, covering eight key variables across multiple sensor stations between 1987 and 2024. Each variable is stored as a separate CSV file, containing time-series data that have undergone rigorous quality control (QC) procedures and, where appropriate, short-gap interpolation for consistency. The quality control (QC) pipeline consisted of four sequential tests: (1) a range test to ensure all values fall within physically realistic limits, (2) a step test to identify abrupt and implausible changes between consecutive records, (3) a proximity test that validates flagged values from step test using data from nearby stations and exceedance probability thresholds, and (4) a persistence test to detect and remove periods of unrealistically constant readings. These thresholds were calibrated to Arizona’s environmental conditions and sensor specifications. After QC, short gaps (≤2 hours) were linearly interpolated to ensure consistent temporal resolution, except for wind variables. Due to a major upgrade in FCDMC’s data transmission system, only ALERT-2 protocol data (2016–2024) for wind variables are included; earlier ALERT-1 data were excluded because of irregular sampling and high missing rates. This dataset supports regional climate and infrastructure resilience studies by providing standardized, high-resolution meteorological data for the greater Phoenix metropolitan area.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Bridging microscopy with molecular dynamics and quantum simulations: an atomAI based pipeline

Recent advances in (scanning) transmission electron microscopy have enabled a routine generation of large volumes of high-veracity structural data on 2D and 3D materials, naturally offering the challenge of using these as starting inputs for atomistic simulations. In this fashion, the theory will address experimentally emerging structures, as opposed to the full range of theoretically possible atomic configurations. However, this challenge is highly nontrivial due to the extreme disparity between intrinsic timescales accessible to modern simulations and microscopy, as well as latencies of microscopy and simulations per se. Addressing this issue requires as a first step bridging the instrumental data flow and physics-based simulation environment, to enable the selection of regions of interest and exploring them using physical simulations. Here we report the development of the machine learning workflow that directly bridges the instrument data stream into Python-based molecular dynamics and density functional theory environments using pre-trained neural networks to convert imaging data to physical descriptors. Additionally, the pathways to ensure structural stability and compensate for the observational biases universally present in the data are identified in the workflow. This approach is used for a graphene system to reconstruct optimized geometry and simulate temperature-dependent dynamics including adsorption of Cr as an ad-atom and graphene healing effects. However, it is universal and can be used for other material systems.

36 MATERIALS SCIENCE↗
DOE OSTI2021

ML Clustering to Identify Natural Gas Pipeline Infrastructure Vulnerabilities

The network of more than 2.5 million miles of natural gas pipelines the U.S. are exposed to a range of vulnerabilities, ranging from extreme weather, to human behavior, and increasingly, to cyber threats. Advanced, data-driven analytics, including the use of machine learning and artificial intelligence, afford an opportunity to identify potential vulnerabilities, better understand risks, and help mitigate vulnerabilities in the network. At NETL, machine learning is being used to explore pipeline vulnerabilities and identify significant clusters of failure events, including failures due to weather, human behavior, and materials. These findings are being used to support the development of new technologies, including sensors, but can also be used to evaluate and inform mitigation strategies to reduce risks and vulnerabilities throughout the network.

Bauer, Jennifer↗
DOE OSTI2025

Pilot-Scale Validation of Distributed Optical Fiber Sensors for Underground Pipeline Monitoring

Monitoring parameters such as hoop strain, pressure, and acoustic vibrations is key to detecting potential leaks, intrusions, or structural issues. Distributed optical fiber sensor (DOFS) systems provide a compelling solution for continuous, real-time monitoring over long distances. This paper details the development and pilot-scale implementation of DOFS systems for underground pipeline monitoring, evolving from a proof-of-concept stage. Multiple custom-designed DOFS interrogator units—such as optical frequency-domain reflectometry (OFDR), Brillouin optical time-domain analysis (BOTDA), and multimodal interferometer-based fiber acoustic sensor systems were tested to measure the key parameters, such as hoop strain, pipe pressure, surrounding soil temperature, and acoustic vibrations. The underground product pipeline’s outer diameter is 30 inches, the wall thickness is 1.28 inches, and 3 feet deep from the surface. The fiber deployment strategies and sensing data acquisition methods for these systems are discussed. The results demonstrate the effectiveness of DOFS in detecting hoop strain, temperature changes, and acoustic vibrations, showcasing their potential for real-time monitoring and enhancing pipeline safety. These findings from pilot-scale testing offer valuable insights into advancing pipeline monitoring technologies and improving the reliability of underground pipeline systems.

fiber optic sensors↗
DOE OSTI2025

Validation of the DESI 2024 Lyα forest BAO analysis using synthetic datasets

The first year of data from the Dark Energy Spectroscopic Instrument (DESI) contains the largest set of Lyman-α (Lyα) forest spectra ever observed. This data, collected in the DESI Data Release 1 (DR1) sample, has been used to measure the Baryon Acoustic Oscillation (BAO) feature at redshift z = 2.33. In this work, we use a set of 150 synthetic realizations of DESI DR1 to validate the DESI 2024 Lyα forest BAO measurement presented in [1]. The synthetic data sets are based on Gaussian random fields using the log-normal approximation. We produce realistic synthetic DESI spectra that include all major contaminants affecting the Lyα forest. The synthetic data sets span a redshift range 1.8 < z < 3.8, and are analyzed using the same framework and pipeline used for the DESI 2024 Lyα forest BAO measurement. To measure BAO, we use both the Lyα auto-correlation and its cross-correlation with quasar positions. We use the mean of correlation functions from the set of DESI DR1 realizations to show that our model is able to recover unbiased measurements of the BAO position. We also fit each mock individually and study the population of BAO fits in order to validate BAO uncertainties and test our method for estimating the covariance matrix of the Lyα forest correlation functions. Finally, we discuss the implications of our results and identify the needs for the next generation of Lyα forest synthetic data sets, with the top priority being to simulate the effect of BAO broadening due to non-linear evolution.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Technoeconomic Analysis of Infrastructure Buildout Scenarios

The success of CCUS deployment in the southeast will depend on the optimized pipeline network from CO 2 point sources to geologic sinks for CO 2 storage. An optimal pipeline framework will reduce both environmental impacts and pipeline building cost. However, finding an optimal pipeline/transport infrastructure design is a non-trivial task and requires simultaneous usage of large volumes of data, computational resources, and a state-of-the-art simulator. Such a transport infrastructure model must consider point sources for CO 2 capture and associated volumes, sinks for CO 2 storage, and transportation from source to sink via pipeline networks. These considerations must be addressed simultaneously and systemically in an optimization process, in which a defined objective function (i.e., capital, variable CO 2 capture, transport, and storage cost function) is required to be minimized with the consideration of practical constraints (e.g., CO 2 flow through pipelines is less than the maximum capacity). Although optimization has been widely used in subsurface resources production and CO 2 storage, its application in large scale CCUS infrastructure design is rarely reported in the literature. SimCCS, developed by Los Alamos National Laboratory, is an open-source CCUS pipeline infrastructure design toolset that facilitates the optimization of pipeline infrastructure networks.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2023

MFA-DVR: direct volume rendering of MFA models

3D volume rendering is widely used to reveal insightful intrinsic patterns of volumetric datasets across many domains. However, the complex structures and varying scales of volumetric data can make efficiently generating high-quality volume rendering results a challenging task. Multivariate functional approximation (MFA) is a new data model that addresses some of the critical challenges: high-order evaluation of both value and derivative anywhere in the spatial domain, compact representation for large-scale volumetric data, and uniform representation of both structured and unstructured data. Here, in this paper, we present MFA-DVR, the first direct volume rendering pipeline utilizing the MFA model, for both structured and unstructured volumetric datasets. We demonstrate improved rendering quality using MFA-DVR on both synthetic and real datasets through a comparative study. We show that MFA-DVR not only generates more faithful volume rendering than using local filters but also performs faster on high-order interpolations on structured and unstructured datasets. MFA-DVR is implemented in the existing volume rendering pipeline of the Visualization Toolkit (VTK) to be accessible by the scientific visualization community.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Semi-automatic image annotation using 3D LiDAR projections and depth camera data

Efficient image annotation is necessary to utilize deep learning object recognition neural networks in nuclear safeguards, such as for the detection and localization of target objects like nuclear material containers (NMCs). This capability can help automate the inventory accounting of different types of NMCs within nuclear storage facilities. The conventional manual annotation process is labor-intensive and time-consuming, hindering the rapid deployment of deep learning models for NMC identifications. This paper introduces a novel semi-automatic method for annotating 2D images of nuclear material containers (NMCs) by combining 3D light detection and ranging (LiDAR) data with color and depth camera images collected from a handheld scan system. The annotation pipeline involves an operator manually marking new target objects on a LiDAR-generated map, and projecting these 3D locations to images, thereby automatically creating annotations from the projections. The semi-automatic approach significantly reduces manual efforts and the expertise in image annotation that is required to perform the task, allowing deep learning models to be trained on-site within a few hours. The paper compares the performance of models trained on datasets annotated through various methods, including semi-automatic, manual, and commercial annotation services. The evaluation demonstrates that the semi-automatic annotation method achieves comparable or superior results, with a mean average precision (mAP) above 0.9, showcasing its efficiency in training object recognition models. Additionally, the paper explores the application of the proposed method to instance segmentation, achieving promising results in detecting multiple types of NMCs in various formations.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
DOE OSTI2021

Cost of a potential hydrogen-refueling network for heavy-duty vehicles with long-haul application in Germany 2050

Long-distance road-freight transport emits a large share of Germany's greenhouse gas (GHG) emissions. A potential solution for reducing GHG emissions in this sector is to use green hydrogen in fuel cell electric vehicles (FC-HDV) and establish an accompanying hydrogen refueling station (HRS) network. In this paper, we apply an existing refueling network design model to a HDV-HRS network for Germany until 2050 based on German traffic data for heavy-duty trucks and estimate its costs. Comparing different fuel supply scenarios (pipeline vs. on-site), the on-site scenario results show a network consisting of 137 stations at a cost of 8.38 billion € per year in 2050 (0.40 € per vehicle km), while the centralized scenario with the same amount of stations shows a cheaper cost with 7.25 billion euros per year (0.35 € per vehicle km). Finally, the hydrogen cost (LCOH) varies from 5.59 €/kg (pipeline) to 6.47 €/kg (on-site) in 2050.

08 HYDROGEN↗
DOE OSTI2021

Design-time performance modeling of compositional parallel programs

Performance models are powerful instruments for understanding the performance of parallel systems and uncovering their bottlenecks. Already during system design, performance models can help ponder alternative development options. However, creating a performance model – whether theoretically or empirically – for an entire application that does not exist yet is challenging. In this paper, we propose to generate performance models of full programs from performance models of their components using formal composition operators derived from parallel design patterns. As long as the design of the overall system follows such a pattern, its performance model can be predicted with reasonable accuracy without an actual implementation. In conclusion, we demonstrate our approach with design patterns of varying complexity, including pipeline, task pool, and eventually MapReduce, which is representative of a broad class of data-analytics applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

PoolDilutionR : An R package for easy optimization of isotope pool dilution calculations

Abstract Isotopic pool dilution is a powerful approach to quantify gross biogeochemical transformation rates, but remains seldom used despite its potential. To facilitate broader implementation of pool dilution methods, we present a user‐friendly R package that optimizes gross production and consumption rates (and optionally fractionation constants as well) based on standard pool dilution time series data. This package features extensive documentation and example analyses, and is easily integrated into analytical pipelines. With this open‐source tool, the biogeochemistry community will be able to readily apply isotope pool dilution to a wide range of processes.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Mitigating the noise of DESI mocks using analytic control variates

In order to address fundamental questions related to the expansion history of the Universe and its primordial nature with the next generation of galaxy experiments, we need to model reliably large-scale structure observables such as the correlation function and the power spectrum. Cosmological N-body simulations provide a reference through which we can test our models, but their output suffers from sample variance on large scales. Fortunately, this is the regime where accurate analytic approximations exist. To reduce the variance, which is key to making optimal use of these simulations, we can leverage the accuracy and precision of such analytic descriptions using Control Variates (CV). The power of control variates stems from utilizing inexpensive but highly correlated surrogates of the statistics one wishes to measure. The stronger the correlation between the surrogate and the statistic of interest, the larger the variance reduction delivered by the method. We apply two control variate formulations to mock catalogs generated in anticipation of upcoming data from the Dark Energy Spectroscopic Instrument (DESI) to test the robustness of its analysis pipeline. Our CV-reduced measurements offer a factor of 5-10 improvement in the measurement error compared with the raw measurements. We explore the relevant properties of the galaxy samples that dictate this reduction and comment on the improvements we find on some of the derived quantities relevant to Baryon Acoustic Oscillation (BAO) analysis.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

Semi-supervised Bayesian Low-shot Learning

Deep neural networks (NNs) typically outperform traditional machine learning (ML) approaches for complicated, non-linear tasks. It is expected that deep learning (DL) should offer superior performance for the important non-proliferation task of predicting explosive device configuration based upon observed optical signature, a task which human experts struggle with. However, supervised machine learning is difficult to apply in this mission space because most recorded signatures are not associated with the corresponding device description, or “truth labels.” This is challenging for NNs, which traditionally require many samples for strong performance. Semi-supervised learning (SSL), low-shot learning (LSL), and uncertainty quantification (UQ) for NNs are emerging approaches that could bridge the mission gaps of few labels and rare samples of importance. NN explainability techniques are important in gaining insight into the inferential feature importance of such a complex model. In this work, SSL, LSL, and UQ are merged into a single framework, a significant technical hurdle not previously demonstrated. Exponential Average Adversarial Training (EAAT) and Pairwise Neural Networks (PNNs) are chosen as the SSL and LSL methods of choice. Permutation feature importance (PFI) for functional data is used to provide explainability via the Variable importance Explainable Elastic Shape Analysis (VEESA) pipeline. A variety of uncertainty quantification approaches are explored: Bayesian Neural Networks (BNNs), ensemble methods, concrete dropout, and evidential deep learning. Two final approaches, one utilizing ensemble methods and one utilizing evidential learning, are constructed and compared using a well-quantified synthetic 2D dataset along with the DIRSIG Megascene.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Integrative SP3 Workflow for Multi-PTM Proteomics Profiling (TZ-DP0)

The goal of the experiment was to demonstrate that the optimized multiplexed multi-PTM profiling workflow can comprehensively and quantitatively capture dynamic changes in protein abundance, cysteine oxidation, phosphorylation, and acetylation in cytokine-induced inflammatory stress in mouse pancreatic ß-cells. Global proteomic, redox proteomic, phosphoproteomic, and acetylomic were data collected from mouse Beta-TC-6 pancreatic Beta-cells, untreated (mock) and cytokine-treated Beta-cells at 4, 8, and 24 hours with 4 biological replicates. Samples were digested with trypsin and Lys-C, then analyzed by LC-MS/MS. Data were searched with MS-GF+, MASIC, and MaxQuant using PNNL's DMS processing pipeline.

59 BASIC BIOLOGICAL SCIENCES↗