Search NASA⌕ Search

SEARCH · Search NASA

Results for “data pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 559 records · Page 31
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

pygwb: A Python-based Library for Gravitational-wave Background Searches

The collection of gravitational waves (GWs) that are either too weak or too numerous to be individually resolved is commonly referred to as the gravitational-wave background (GWB). A confident detection and model-driven characterization of such a signal will provide invaluable information about the evolution of the universe and the population of GW sources within it. We present a new, user-friendly, Python-based package for GW data analysis to search for an isotropic GWB in ground-based interferometer data. We employ cross-correlation spectra of GW detector pairs to construct an optimal estimator of the Gaussian and isotropic GWB, and Bayesian parameter estimation to constrain GWB models. The modularity and clarity of the code allow for both a shallow learning curve and flexibility in adjusting the analysis to one’s own needs. We describe the individual modules that make up pygwb, following the traditional steps of stochastic analyses carried out within the LIGO, Virgo, and KAGRA Collaboration. We then describe the built-in pipeline that combines the different modules and validate it with both mock data and real GW data from the O3 Advanced LIGO and Virgo observing run. We successfully recover all mock data injections and reproduce published results.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

ORBIT-2 Dataset for Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling

This dataset release corresponds to the work conducted in ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling, where large-scale AI methods were applied to improve climate and weather resolution. The collection integrates four widely used, publicly available datasets: ERA5, PRISM, DAYMET, and IMERG. To prepare the data for ORBIT-2 model training and evaluation, we applied a preprocessing pipeline that generates paired low-resolution and high-resolution samples, enabling supervised downscaling experiments. The transformation from coarse to fine scales was performed using bilinear regridding, consistent with the procedures described in WeatherBench2, a community benchmark for weather and climate AI models. This dataset supports the development and evaluation of foundation models designed for weather and climate downscaling at exascale. Additional details on methodology and applications can be found in Wang et al., ORBIT-2 (arXiv:2505.04802, 2025).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

SNIa Cosmology Analysis Results from Simulated LSST Images: From Difference Imaging to Constraints on Dark Energy

Abstract The Vera Rubin Observatory Legacy Survey of Space and Time (LSST) is expected to process ∼10 6 transient detections per night. For precision measurements of cosmological parameters and rates, it is critical to understand the detection efficiency, magnitude limits, artifact contamination levels, and biases in the selection and photometry. Here we rigorously test the LSST Difference Image Analysis (DIA) pipeline using simulated images from the Rubin Observatory LSST Dark Energy Science Collaboration Data Challenge (DC2) simulation for the Wide-Fast-Deep survey area. DC2 is the first large-scale (300 deg 2 ) image simulation of a transient survey that includes realistic cadence, variable observing conditions, and CCD image artifacts. We analyze ∼15 deg 2 of DC2 over a 5 yr time span in which artificial point sources from Type Ia supernova (SNIa) light curves have been overlaid onto the images. The magnitude limits per filter are u = 23.66 mag, g = 24.69 mag, r = 24.06 mag, i = 23.45 mag, z = 22.54 mag, and y = 21.62 mag. The artifact contamination levels are ∼90% of all detections, corresponding to ∼1000 artifacts deg –2 in g band, and falling to 300 deg –2 in y band. The photometry has biases <1% for magnitudes 19.5 < m < 23. Our DIA performance on simulated images is similar to that of the Dark Energy Survey difference-imaging pipeline on real images. We also characterize DC2 image properties to produce catalog-level simulations needed for distance bias corrections. We find good agreement between DC2 data and simulations for distributions of signal-to-noise ratio, redshift, and fitted light-curve properties. Applying a realistic SNIa cosmology analysis for redshifts z < 1, we recover the input cosmology parameters to within statistical uncertainties.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Visualization of Noisy and Less Noisy Computational Basis States in Quantum Computing

Quantum computing technology holds substantial promise as a reliable computational paradigm. However, current noisy intermediate scale quantum (NISQ) systems, are significantly impacted by noise originating from hardware inconsistencies. This noise causes errors and lowers output fidelity. So we must find which basis states cause errors. However, there are two main challenges in analyzing noise corresponding to basis states. First, the noise distribution data is high dimensional in nature, thereby making its analysis challenging. Second, although functional box plots have been used in the state of the art research to understand such a high dimensional data, they suffer from clutter and occlusion issues because of overplotting. In this study, we introduce an innovative visualization pipeline to address the aforementioned challenges to provide a clear depiction of noisy and less-noisy basis states. Specifically, our proposed visualization pipeline comprises three stages namely, low dimensional embedding, clustering, and violin plot visualization, to reduce visual clutter and effectively analyze high-dimensional noise distribution data. Our analysis uses quantum machine learning (QML) circuits as case study for drawing a distinction between noisy and less noisy basis states.

Senapati, Priyabrata [Kent State University]↗
DOE OSTI2026

Persistent DynAMICS

Persistent DynAMICS is a lightweight software platform that enables distributed sensing systems to operate as a unified, responsive network. It integrates with existing sensors, data systems, and processing tools as an overlay, without requiring hardware replacement or increasing user burden. As data is collected, the system identifies situations that require additional attention and directs selected sensors to gather more information. For example, if a sensor detects an unusual condition along a pipeline, the system can task nearby sensors or different sensing modalities to confirm the event and focus data collection where it is needed. To support this, the platform forms temporary groups of sensors that work together to address specific objectives. Once complete, these groups are dissolved and resources are reassigned as needed. Communication is handled through an integrated messaging framework with client and server components, enabling reliable operation and data continuity even in low-bandwidth or intermittent connectivity environments.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Snowmass2021 Cosmic Frontier White Paper: Primordial black hole dark matter

Primordial Black Holes (PBHs) are a viable candidate to comprise some or all of the dark matter and provide a unique window into the high-energy physics of the early universe. This white paper discusses the scientific motivation, current status, and future reach of observational searches for PBHs. Future observational facilities supported by DOE, NSF, and NASA will provide unprecedented sensitivity to PBHs. However, devoted analysis pipelines and theoretical modeling are required to fully leverage these novel data. The search for PBHs constitutes a low-cost, high-reward science case with significant impact on the high energy physics community.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2022

A systematic method for selecting molecular descriptors as features when training models for predicting physiochemical properties

Machine learning has proven to be a powerful tool for accelerating biofuel development. Although numerous models are available to predict a range of properties using chemical descriptors, there is a trade-off between interpretability and performance. Neural networks provide predictive models with high accuracy at the expense of some interpretability, while simpler models such as linear regression often lack in accuracy. In addition to model architecture, feature selection is also critical for developing interpretable and accurate predictive models. We present a method for systematically selecting molecular descriptor features and developing interpretable machine learning models without sacrificing accuracy. Our method simplifies the process of selecting features by reducing feature multicollinearity and enables discoveries of new relationships between global properties and molecular descriptors. To demonstrate our approach, we developed models for predicting melting point, boiling point, flash point, yield sooting index, and net heat of combustion with the help of the Tree-based Pipeline Optimization Tool (TPOT). For training, we used publicly available experimental data for up to 8351 molecules. Our models accurately predict various molecular properties for organic molecules (mean absolute percent error (MAPE) ranges from 3.3% to 10.5%) and provide a set of features that are well-correlated to the property. This method enables researchers to explore sets of features that significantly contribute to the prediction of the property, offering new scientific insights. To help accelerate early stage biofuel research and development, we also integrated the data and models into a open-source, interactive web tool.

09 BIOMASS FUELS↗
DOE OSTI2024

Automated Gold Nanorod Spectral Morphology Analysis Pipeline

The development of a colloidal synthesis procedure to produce nanomaterials with high shape and size purity is often a time-consuming, iterative process. This is often due to quantitative uncertainties in the required reaction conditions and the time, resources, and expertise intensive characterization methods required for quantitative determination of nanomaterial size and shape. Absorption spectroscopy is often the easiest method for colloidal nanomaterial characterization. However, due to the lack of a reliable method to extract nanoparticle shapes from absorption spectroscopy, it is generally treated as a more qualitative measure for metal nanoparticles. This work demonstrates a gold nanorod (AuNR) spectral morphology analysis tool, called AuNR-SMA, which is a fast and accurate method to extract quantitative structural information from colloidal AuNR absorption spectra. To demonstrate the practical utility of this model, we apply it to three distinct applications. First, we demonstrate this model's utility as an automated analysis tool in a high-throughput AuNR synthesis procedure by generating quantitative size information from optical spectra. Second, we use the predictions generated by this model to train a machine learning model to predict the resulting AuNR size distributions under specified reaction conditions. Third, we apply this model to spectra extracted from the literature where no size distributions are reported and impute unreported quantitative information on AuNR synthesis. This approach can potentially be extended to any other nanocrystal system where absorption spectra are size dependent, and accurate numerical simulation of absorption spectra is possible. In addition, this pipeline could be integrated into automated synthesis apparatuses to provide interpretable data from simple measurements, help explore the synthesis science of nanoparticles in a rational manner, or facilitate closed-loop workflows.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Optimal 1D Ly α forest power spectrum estimation – III. DESI early data

ABSTRACT The 1D power spectrum P1D of the Ly α forest provides important information about cosmological and astrophysical parameters, including constraints on warm dark matter models, the sum of the masses of the three neutrino species, and the thermal state of the intergalactic medium. We present the first measurement of P1D with the quadratic maximum likelihood estimator (QMLE) from the Dark Energy Spectroscopic Instrument (DESI) survey early data sample. This early sample of 54 600 quasars is already comparable in size to the largest previous studies, and we conduct a thorough investigation of numerous instrumental and analysis systematic errors to evaluate their impact on DESI data with QMLE. We demonstrate the excellent performance of the spectroscopic pipeline noise estimation and the impressive accuracy of the spectrograph resolution matrix with 2D image simulations of raw DESI images that we processed with the DESI spectroscopic pipeline. We also study metal line contamination and noise calibration systematics with quasar spectra on the red side of the Ly α emission line. In a companion paper, we present a similar analysis based on the Fast Fourier Transform estimate of the power spectrum. We conclude with a comparison of these two approaches and discuss the key sources of systematic error that we need to address with the upcoming DESI Year 1 analysis.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

A consensus protocol for the recovery of mercury methylation genes from metagenomes

Abstract Mercury (Hg) methylation genes ( hgcAB ) mediate the formation of the toxic methylmercury and have been identified from diverse environments, including freshwater and marine ecosystems, Arctic permafrost, forest and paddy soils, coal‐ash amended sediments, chlor‐alkali plants discharges and geothermal springs. Here we present the first attempt at a standardized protocol for the detection, identification and quantification of hgc genes from metagenomes. Our Hg‐cycling microorganisms in aquatic and terrestrial ecosystems (Hg‐MATE) database, a catalogue of hgc genes, provides the most accurate information to date on the taxonomic identity and functional/metabolic attributes of microorganisms responsible for Hg methylation in the environment. Furthermore, we introduce “marky‐coco”, a ready‐to‐use bioinformatic pipeline based on de novo single‐metagenome assembly, for easy and accurate characterization of hgc genes from environmental samples. We compared the recovery of hgc genes from environmental metagenomes using the marky‐coco pipeline with an approach based on coassembly of multiple metagenomes. Our data show similar efficiency in both approaches for most environments except those with high diversity (i.e., paddy soils) for which a coassembly approach was preferred. Finally, we discuss the definition of true hgc genes and methods to normalize hgc gene counts from metagenomes.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Agent-Based Simulation Model for Analyzing Multimodal Transportation of CO2 for CCUS

The overarching goal of this work is to determine how the United States can avoid being constrained by transportation limitations as the CCUS process evolves to a full gigaton-level system. The model is data-driven, open to CO2 supply and demand points defined as input data, and designed to allow for transportation on waterways, rail, or pipeline.

Clark, RobinJ↗
DOE OSTI2022

A meta-analysis of semantic classification of citations

The aim of this literature review is to examine the current state of the art in the area of citation classification. In particular, we investigate the approaches for characterizing citations based on their semantic type. We conduct this literature review as a meta-analysis covering 60 scholarly articles in this domain. Although we included some of the manual pioneering works in this review, more emphasis is placed on the later automated methods, which use Machine Learning and Natural Language Processing (NLP) for analyzing the fine-grained linguistic features in the surrounding text of citations. The sections are organized based on the steps involved in the pipeline for citation classification. Specifically, we explore the existing classification schemes, data sets, pre-processing methods, extraction of contextual and non-contextual features, and the different types of classifiers and evaluation approaches. The review highlights the importance of identifying the citation types for research evaluation, the challenges faced by the researchers in the process, and the existing research gaps in this field.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2023

Reverse engineering environmental metatranscriptomes clarifies best practices for eukaryotic assembly

Abstract Background Diverse communities of microbial eukaryotes in the global ocean provide a variety of essential ecosystem services, from primary production and carbon flow through trophic transfer to cooperation via symbioses. Increasingly, these communities are being understood through the lens of omics tools, which enable high-throughput processing of diverse communities. Metatranscriptomics offers an understanding of near real-time gene expression in microbial eukaryotic communities, providing a window into community metabolic activity. Results Here we present a workflow for eukaryotic metatranscriptome assembly, and validate the ability of the pipeline to recapitulate real and manufactured eukaryotic community-level expression data. We also include an open-source tool for simulating environmental metatranscriptomes for testing and validation purposes. We reanalyze previously published metatranscriptomic datasets using our metatranscriptome analysis approach. Conclusion We determined that a multi-assembler approach improves eukaryotic metatranscriptome assembly based on recapitulated taxonomic and functional annotations from an in-silico mock community. The systematic validation of metatranscriptome assembly and annotation methods provided here is a necessary step to assess the fidelity of our community composition measurements and functional content assignments from eukaryotic metatranscriptomes.

Krinos, Arianna I. (ORCID:0000000197678392)↗
DOE OSTI2024

Evaluating Machine Learning-Based MRI Reconstruction Using Digital Image Quality Phantoms

Quantitative and objective evaluation tools are essential for assessing the performance of machine learning (ML)-based magnetic resonance imaging (MRI) reconstruction methods. However, the commonly used fidelity metrics, such as mean squared error (MSE), structural similarity (SSIM), and peak signal-to-noise ratio (PSNR), often fail to capture fundamental and clinically relevant MR image quality aspects. To address this, we propose evaluation of ML-based MRI reconstruction using digital image quality phantoms and automated evaluation methods. Our phantoms are based upon the American College of Radiology (ACR) large physical phantom but created in k-space to simulate their MR images, and they can vary in object size, signal-to-noise ratio, resolution, and image contrast. Our evaluation pipeline incorporates evaluation metrics of geometric accuracy, intensity uniformity, percentage ghosting, sharpness, signal-to-noise ratio, resolution, and low-contrast detectability. We demonstrate the utility of our proposed pipeline by assessing an example ML-based reconstruction model across various training and testing scenarios. The performance results indicate that training data acquired with a lower undersampling factor and coils of larger anatomical coverage yield a better performing model. The comprehensive and standardized pipeline introduced in this study can help to facilitate a better understanding of the performance and guide future development and advancement of ML-based reconstruction algorithms.

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

End-to-end deep learning pipeline for real-time Bragg peak segmentation: from training to large-scale deployment

X-ray crystallography reconstruction, which transforms discrete X-ray diffraction patterns into three-dimensional molecular structures, relies critically on accurate Bragg peak finding for structure determination. As X-ray free electron laser (XFEL) facilities advance toward MHz data rates (1 million images per second), traditional peak finding algorithms that require manual parameter tuning or exhaustive grid searches across multiple experiments become increasingly impractical. While deep learning approaches offer promising solutions, their deployment in high-throughput environments presents significant challenges in automated dataset labeling, model scalability, edge deployment efficiency, and distributed inference capabilities. We present an end-to-end deep learning pipeline with three key components: (1) a data engine that combines traditional algorithms with our peak matching algorithm to generate high-quality training data at scale, (2) a modular architecture that scales from a few million to hundreds of million parameters, enabling us to train large expert-level models offline while deploying smaller, distilled models at the edge, and (3) a decoupled producer-consumer architecture that separates specialized data source layer from model inference, enabling flexible deployment across diverse computing environments. Using this integrated approach, our pipeline achieves accuracy comparable to traditional methods tuned by human experts while eliminating the need for experiment-specific parameter tuning. Although current throughput requires optimization for MHz facilities, our system's scalable architecture and demonstrated model compression capabilities provide a foundation for future high-throughput XFEL deployments.

Wang, Cong↗
DOE OSTI2020

LyaCoLoRe : synthetic datasets for current and future Lyman-α forest BAO surveys

The statistical power of Lyman-α forest Baryon Acoustic Oscillation (BAO) measurements is set to increase significantly in the coming years as new instruments such as the Dark Energy Spectroscopic Instrument deliver progressively more constraining data. Generating mock datasets for such measurements will be important for validating analysis pipelines and evaluating the effects of systematics. With such studies in mind, here we present LyaCoLoRe: a package for producing synthetic Lyman-α forest survey datasets for BAO analyses. LyaCoLoRe transforms initial Gaussian random field skewers into skewers of transmitted flux fraction via a number of fast approximations. In this work we explain the methods of producing mock datasets used in LyaCoLoRe, and then measure correlation functions on a suite of realisations of such data. We demonstrate that we are able to recover the correct BAO signal, as well as large-scale bias parameters similar to literature values. Finally, we briefly describe methods to add further astrophysical effects to our skewers — high column density systems and metal absorbers — which act as potential complications for BAO analyses.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Streaming Large-Scale Microscopy Data to a Supercomputing Facility

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy generates data at a nominal rate of 480 Gbit s -1 (87,000 frames s -1 ⁠), producing a 700 GB dataset in 15 s. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera’s data acquisition system to supercomputing nodes at the National Energy Research Scientific Computing Center, bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a postmortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.

4D-STEM↗
DOE OSTI2021

A consensus-based ensemble approach to improve transcriptome assembly

Systems-level analyses, such as differential gene expression analysis, co-expression analysis, and metabolic pathway reconstruction, depend on the accuracy of the transcriptome. Multiple tools exist to perform transcriptome assembly from RNAseq data. However, assembling high quality transcriptomes is still not a trivial problem. This is especially the case for non-model organisms where adequate reference genomes are often not available. Different methods produce different transcriptome models and there is no easy way to determine which are more accurate. Furthermore, having alternative-splicing events exacerbates such difficult assembly problems. While benchmarking transcriptome assemblies is critical, this is also not trivial due to the general lack of true reference transcriptomes. In this study, we first provide a pipeline to generate a set of the simulated benchmark transcriptome and corresponding RNAseq data. Using the simulated benchmarking datasets, we compared the performance of various transcriptome assembly approaches including both de novo and genome-guided methods. The results showed that the assembly performance deteriorates significantly when alternative transcripts (isoforms) exist or for genome-guided methods when the reference is not available from the same genome. To improve the transcriptome assembly performance, leveraging the overlapping predictions between different assemblies, we present a new consensus-based ensemble transcriptome assembly approach, ConSemble. Without using a reference genome, ConSemble using four de novo assemblers achieved an accuracy up to twice as high as any de novo assemblers we compared. When a reference genome is available, ConSemble using four genome-guided assemblies removed many incorrectly assembled contigs with minimal impact on correctly assembled contigs, achieving higher precision and accuracy than individual genome-guided methods. Furthermore, ConSemble using de novo assemblers matched or exceeded the best performing genome-guided assemblers even when the transcriptomes included isoforms. We thus demonstrated that the ConSemble consensus strategy both for de novo and genome-guided assemblers can improve transcriptome assembly. The RNAseq simulation pipeline, the benchmark transcriptome datasets, and the script to perform the ConSemble assembly are all freely available from: http://bioinfolab.unl.edu/emlab/consemble/.

59 BASIC BIOLOGICAL SCIENCES↗