Search NASA⌕ Search

SEARCH · Search NASA

Results for “data pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Guided construction of single cell reference for human and mouse lung

Accurate cell type identification is a key and rate-limiting step in single-cell data analysis. Single-cell references with comprehensive cell types, reproducible and functionally validated cell identities, and common nomenclatures are much needed by the research community for automated cell type annotation, data integration, and data sharing. Here, we develop a computational pipeline utilizing the LungMAP CellCards as a dictionary to consolidate single-cell transcriptomic datasets of 104 human lungs and 17 mouse lung samples to construct LungMAP single-cell reference (CellRef) for both normal human and mouse lungs. CellRefs define 48 human and 40 mouse lung cell types catalogued from diverse anatomic locations and developmental time points. We demonstrate the accuracy and stability of LungMAP CellRefs and their utility for automated cell type annotation of both normal and diseased lungs using multiple independent methods and testing data. We develop user-friendly web interfaces for easy access and maximal utilization of the LungMAP CellRefs.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Clustering with general photo- z uncertainties: application to Baryon Acoustic Oscillations

ABSTRACT Photometric data can be analysed using the 3D correlation function ξp to extract cosmological information via e.g. measurement of the Baryon Acoustic Oscillations (BAO). Previous studies modeled ξp assuming a Gaussian photo-z approximation. In this work we improve the modeling by incorporating realistic photo-z distribution. We show that the position of the BAO scale in ξp is determined by the photo-z distribution and the Jacobian of the transformation. The latter diverges at the transverse scale of the separation s⊥, and it explains why ξp traces the underlying correlation function at s⊥, rather than s, when the photo-z uncertainty σz/(1+ z) ≳ 0.02. We also obtain the Gaussian covariance for ξp. Due to photo-z mixing, the covariance of ξp shows strong off-diagonal elements. The high correlation of the data causes some issues to the data fitting. None the less, we find that either it can be solved by suppressing the largest eigenvalues of the covariance or it is not directly related to the BAO. We test our BAO fitting pipeline using a set of mock catalogs. The data set is dedicated for Dark Energy Survey Year 3 (DES Y3) BAO analyses and includes realistic photo-z distributions. The theory template is in good agreement with mock measurement. Based on the DES Y3 mocks, ξp statistic is forecast to constrain the BAO shift parameter α to be 1.001 ± 0.023, which is well consistent with the corresponding constraint derived from the angular correlation function measurements. Thus, ξp offers a competitive alternative for the photometric data analyses.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Personalized and uncertainty-aware coronary hemodynamics simulations: From Bayesian estimation to improved multi-fidelity uncertainty quantification

Non-invasive simulations of coronary hemodynamics have improved clinical risk stratification and treatment outcomes for coronary artery disease, compared to relying on anatomical imaging alone. However, simulations typically use empirical approaches to distribute total coronary flow amongst the arteries in the coronary tree, which ignores patient variability, the presence of disease, and other clinical factors. Further, uncertainty in the clinical data often remains unaccounted for in the modeling pipeline. We present an end-to-end uncertainty-aware pipeline to (1) personalize coronary flow simulations by incorporating vessel-specific coronary flows as well as cardiac function; and (2) predict clinical and biomechanical quantities of interest with improved precision, while accounting for uncertainty in the clinical data. We assimilate patient-specific measurements of myocardial blood flow from clinical CT myocardial perfusion imaging to estimate branch-specific coronary artery flows. Simulated noise in the clinical data is used to estimate the joint posterior distributions of the model parameters using adaptive Markov Chain Monte Carlo sampling. Additionally, the posterior predictive distribution for the relevant quantities of interest is determined using a new approach combining multi-fidelity Monte Carlo estimation with non-linear, data-driven dimensionality reduction. This leads to improved correlations between high- and low-fidelity model outputs. Our framework accurately recapitulates clinically measured cardiac function as well as branch-specific coronary flows under measurement noise uncertainty. We observe substantial reductions in confidence intervals for estimated quantities of interest compared to single-fidelity Monte Carlo estimation and state-of-the-art multi-fidelity Monte Carlo methods. This holds especially true for quantities of interest that showed limited correlation between the low- and high-fidelity model predictions. In addition, the proposed multi-fidelity Monte Carlo estimators are significantly cheaper to compute than traditional estimators, under a specified confidence level or variance. The proposed pipeline for personalized and uncertainty-aware predictions of coronary hemodynamics is based on routine clinical measurements and recently developed techniques for CT myocardial perfusion imaging. The proposed pipeline offers significant improvements in precision and reduction in computational cost.

Bayesian parameter estimation↗
DOE OSTI2023

pmartR 2.0 : A Quality Control, Visualization, and Statistics Pipeline for Multiple Omics Datatypes

The pmartR (https://github.com/pmartR/pmartR) package was designed for the quality control (QC) and analysis of mass spectrometry (MS) data, tailored to specific characteristics of proteomic (isobaric or labelled), metabolomic, and lipidomic datasets. Since its initial release, the tool has been expanded to address the needs of its growing userbase and now includes QC and statistics for nuclear magnetic resonance (NMR) metabolomic data, and leverages the DESeq2, edgeR, and limma-voom R packages for some transcriptomic data analyses. These improvements have made progress towards a unified omics processing pipeline for ease of reporting and streamlined statistical purposes. The package’s statistics and visualization capabilities have also been expanded by adding support for paired data and by integrating pmartR with the trelliscopejs R package for the quick creation of trellis displays (https://github.com/hafen/trelliscopejs). Here, we present relevant examples of each of these enhancements to pmartR and highlight how each new feature benefits the omics community.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

The persistence of large scale structures. Part I. Primordial non-Gaussianity

Abstract We develop an analysis pipeline for characterizing the topology of large scale structure and extracting cosmological constraints based on persistent homology . Persistent homology is a technique from topological data analysis that quantifies the multiscale topology of a data set, in our context unifying the contributions of clusters, filament loops, and cosmic voids to cosmological constraints. We describe how this method captures the imprint of primordial local non-Gaussianity on the late-time distribution of dark matter halos, using a set of N-body simulations as a proxy for real data analysis. For our best single statistic, running the pipeline on several cubic volumes of size 40 (Gpc/h) 3 , we detect f NL loc =10 at 97.5% confidence on ~ 85% of the volumes. Additionally wetest our ability to resolve degeneracies betweenthe topological signature of f NL loc and variation of σ 8 and argue that correctly identifying nonzero f NL loc in this case is possible via an optimal template method. Our method relies on information living at $\mathcal{O}$(10) Mpc/h, a complementary scale with respect to commonly used methods such as the scale-dependent bias in the halo/galaxy power spectrum. Therefore, while still requiring a large volume, our method does not require sampling long-wavelength modes to constrain primordial non-Gaussianity. Moreover, our statistics are interpretable: we are able to reproduce previous results in certain limits and we make new predictions for unexplored observables, such as filament loops formed by dark matter halos in a simulation box.

Astronomy & Astrophysics↗
DOE OSTI2023

The Hidden Flexibility of the Natural Gas Network for Electric Power Operations: A Case Study of a Near-Miss Winter Event

The U.S. power sector has become increasingly reliant on gas pipeline networks to deliver fuel to natural gas power plants. In addition to supplying relatively low-cost fuel, gas networks offer generators flexibility in their operations through the ability to deliver fuel when needed by using gas storage facilities or linepack if the gas network is at an operating point below its design capacity. However, disruptions or stress events on the gas network - like those occurring in the Northeast and Texas in recent years - can result in limitations on gas availability to generators at times when generation is in short supply. Here we examine a period of stress that occurred in the winter of 2022 in the Western United States. Using data on the region's natural gas pipeline network and electric generators, we build an integrated gas and electric model that closely replicates the actual dispatch of the period. We then evaluate the implications of removing flexibility employed by the gas network operator, which during that period curtailed scheduled gas deliveries to other parties to increase deliveries to natural gas power plants, which requested more gas than initially forecasted. We find that without the flexibility supplied by the gas network operator, there would have been curtailment of gas generation due to gas offtake constraints, requiring the power system operator to redispatch relying on more expensive generation or to potentially shed load. A sensitivity exploring a wind drought further exacerbates the strain, illustrating the potential challenge of managing gas and grid interactions as systems move to higher shares of variable renewable electricity. Based on this example, we discuss potential coordination strategies between the two system operators to ensure that the power system can successfully utilize and rely on the flexibility offered by natural gas networks.

03 NATURAL GAS↗
DOE OSTI2026

Large-scale real-time signal processing in physics experiments: the ALICE TPC FPGA pipeline

For LHC Run 3, the ALICE Time Projection Chamber was upgraded to operate in continuous readout mode. Interaction rates of up to 50 kHz in Pb-Pb collisions require real-time processing of more than 3 TB s -1 of raw detector data. This requirement is met by a custom FPGA-based processing pipeline that performs the complete front-end data treatment fully in-stream, including common-mode correction, pedestal subtraction, ion-tail filtering, zero suppression, and dense data packing. A central element of the design is a highly parallel common-mode correction algorithm operating directly on the streaming data. It robustly identifies signal-free readout channels on a time-bin basis and applies pad-dependent scaling to compensate for local variations in capacitive coupling in the GEM readout. In combination with pedestal subtraction and ion-tail filtering, this enables accurate baseline restoration under extreme high-occupancy conditions, preventing signal loss while efficiently suppressing noise prior to zero suppression. The pipeline operates continuously at the full detector bandwidth and reduces the raw input rate of approximately 3 TB s -1 to about 900 GBps for Pb-Pb collisions at the target interaction rate. Overall, it represents a large-scale FPGA-based real-time signal-processing implementation for high-energy physics detector readout.

Digital signal processing (DSP)↗
DOE OSTI2026

Integration of ultra-low coverage whole-genome sequences for reconstructing the evolutionary history of Galapagos giant tortoises

Genomic data from contemporary and historical samples often need to be coupled for evolutionary reconstructions of multitaxon complexes. However, the genetic data recovered from historical samples may result only in ultra-low coverage whole-genome sequences (ulcWGS; <0.15× depth), leading to inaccurate evolutionary inferences given a preponderance of missing data. Using the Galapagos giant tortoise radiation as a study system (Chelonoidis spp., composed of 13 extant and four extinct lineages), we assembled a novel methodological pipeline that removes potential noise introduced by the missing data and enhances the evolutionary signal from ulcWGS samples. We leveraged existing tools for phylogenomic placement (EPA-ng), population genomic structure (smartsnp) and admixture (Admixfrog, NGSadmix) to demonstrate that the evolutionary history of samples can be uncovered with sequencing depths as low as 0.008–0.139×. Importantly, these approaches do not use genotype imputation of the ulcWGS samples, which would require extensive reference datasets. Our application to two cases of extinct lineages of Galapagos giant tortoises, with and without references from the same lineage, demonstrates the general value of the approach. We confirm where the extinct lineages from San Cristóbal and Santa Fe islands fit into the Galapagos giant tortoise radiation, and that these lineages were evolutionarily distinct entities.

ancient DNA↗
DOE OSTI2025

VA EDH Advanced Software Pipeline Framework Report: Enhancing Automation and Scalability

The VA Environmental Determinants of Health (EDH) Advanced Software Pipeline Framework is designed to enhance the efficiency, scalability, and security of geospatial data processing workflows. This framework integrates modern data orchestration and containerization technologies, including Prefect for workflow automation, Docker for containerization, and PostgreSQL/PostGIS for geospatial data storage and analysis. It ensures standardized, reproducible, and automated data processing, supporting VA objectives related to substance use risk assessment and recovery research. The pipeline addresses key scalability and performance challenges through horizontal and vertical scaling, high-performance computing (HPC) integration, parallel processing, task caching, and dynamic resource allocation. These optimizations improve throughput and reduce latency, allowing the system to efficiently manage large and complex datasets. Additionally, security and compliance measures—such as data encryption (SSL), Role-Based Access Control (RBAC), and adherence to GDPR and HIPAA standards—safeguard sensitive information throughout data transmission and storage. A key implementation of this framework includes the automation of shelter list geolocation workflows, ensuring that up-to-date data is readily available for VA decision-making. Lessons learned from this project include the transition from in-memory processing to incremental storage writes, improving resource management and reliability. Future enhancements aim to expand automation, integrate AI-driven anomaly detection, and incorporate high-performance computing resources. This framework provides a scalable, secure, and adaptable solution for managing geospatial datasets, reinforcing the VA’s ability to support clinical and strategic initiatives through data-driven decision-making.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

The "PVLib" of Degradation: PVDeg

The Photovoltaic (PV) industry constantly aims for lower costs through higher-efficiency cells, improved module designs, and improvements in durability. This leads to the use of new materials, designs, and manufacturing processes, and not always with a sufficient amount of durability testing. To help drive down costs there is a desire to create modules that will last for up to 50 years of service life. To accomplish this, every degradation mode and mechanism must be identified and either eliminated or otherwise mitigated. This involves the extrapolation of laboratory results to the field conditions. There is a need to organize the existing degradation data into an accessible format and to provide industry relevant tools for extrapolation from laboratory to field conditions. While the basic equations used to model degradation are sometimes very simple, the full analysis involves calculations are cumbersome but ubiquitous for many degradation processes. A simplified, modeling framework to accomplish these repetitive processes will facilitate the analysis to help researchers keep up with the rapid pace of technological changes. In this talk, we will describe our progress creating the open-source tool PVDeg. This tool can be used to search for and analyze degradation information and extrapolate PV module performance and durability to field exposure. PVDeg simplifies many of the common foundational computational operations for obtaining meteorological data and using it to generate a model of the PV deployment. This prediction tool repository also contains various degradation models as well as a library of material parameters suitable for estimating the durability assessment of materials and components. We use an integration pipeline approach that allows us to leverage weather data from the National Solar Radiation Database, and other weather sources, to perform geospatial degradation analysis in the US and worldwide. We hope to become a repository that can be used for weathering and degradation analysis for various applications beyond the PV industry. During the talk, we will provide the PVPMC attendees the opportunity to interact with the tool via a Google Collab tutorial they can run on their phones or laptops.

durability↗
DOE OSTI2021

DESC DC2 Data Release Note

In preparation for cosmological analyses of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST), the LSST Dark Energy Science Collaboration (LSST DESC) has created a 300 deg$^2$ simulated survey as part of an effort called Data Challenge 2 (DC2). The DC2 simulated sky survey, in six optical bands with observations following a reference LSST observing cadence, was processed with the LSST Science Pipelines (19.0.0). In this Note, we describe the public data release of the resulting object catalogs for the coadded images of five years of simulated observations along with associated truth catalogs. We include a brief description of the major features of the available data sets. To enable convenient access to the data products, we have developed a web portal connected to Globus data services. We describe how to access the data and provide example Jupyter Notebooks in Python to aid first interactions with the data. We welcome feedback and questions about the data release via a GitHub repository.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2026

nf-core/proteinfamilies: a scalable pipeline for the generation of protein families

The growth of metagenomics-derived amino acid sequence data has transformed our understanding of protein function, microbial diversity, and evolutionary relationships. However, the vast majority of these proteins remain functionally uncharacterized. Grouping the millions of such uncharacterized sequences with the few experimentally characterized ones allows the transfer of annotations, while the inspection of conserved residues with multiple sequence alignments can provide clues to function, even in the absence of existing functional information. To address the challenges associated with this data surge and the need to group sequences, we present a scalable, open-source, parametrizable Nextflow pipeline (nf-core/proteinfamilies) that generates nascent protein families or assigns new proteins to existing families. The computational benchmarks demonstrated that resource usage scales approximately linearly with input size, and the biological benchmarks showed that the generated protein families closely resemble manually curated families in widely used databases.

Nextflow↗
DOE OSTI2026

FC Site 4.0 - NLR Thermodynamic profiler (ASSIST II-11) Thermodynamic Retrievals TROPoe

This dataset contains daily files with thermodynamic profiles retrieved with the optimal estimation physical retrieval TROPoe v0.19 (Turner and Löhnert 2014; Turner and Blumberg 2019; Turner and Löhnert 2021). The profiles are retrieved every 10 minutes from instantaneous observations from the NLR ASSIST II infrared spectrometer. Observations are noise-filtered but not averaged in time to minimize errors due to non-uniform clouds. Additional input data in TROPoe are cloud base height (CBH) from co-located scanning lidar. The full pipeline for running the retrieval is available at https://github.com/StefanoWind/TROPoe_processor. Met data was not ingested. In addition to these temporally resolved input data, TROPoe requires an a priori dataset (prior) that provides mean climatological estimates of thermodynamic profiles and specifies how temperature and humidity covary with height as an input (for details see, e.g., Djalalova et al. 2022). The prior is a key component of the retrieval and provides a constraint on the ill-posed inversion problem. A monthly prior was computed from operational radiosonde launches in Denver, CO.

17 WIND ENERGY↗
DOE OSTI2025

Constraining the phase shift of relativistic species in DESI BAOs

In the early Universe, neutrinos decouple quickly from the primordial plasma and propagate without further interactions. The impact of free-streaming neutrinos is to create a temporal shift in the gravitational potential that impacts the acoustic waves known as baryon acoustic oscillations (BAOs), resulting in a non-linear spatial shift in the Fourier-space BAO signal. In this work, we make use of and extend upon an existing methodology to measure the phase shift amplitude $\beta _{\phi }$ and apply it to the Dark Energy Spectroscopic Instrument (DESI) Data Release 1 (DR1) BAOs with an anisotropic BAO fitting pipeline. We validate the fitting methodology by testing the pipeline with two publicly available fitting codes applied to highly precise cubic box simulations and realistic simulations representative of the DESI DR1 data. We find further study towards the methods used in fitting the BAO signal will be necessary to ensure accurate constraints on $\beta _{\phi }$ in future DESI data releases. Using DESI DR1, we present individual measurements of the anisotropic BAO distortion parameters and the $\beta _{\phi }$ for the different tracers, and additionally a combined fit to $\beta _{\phi }$ resulting in $\beta _{\phi } = 2.7 \pm 1.7$. After including a prior on the distortion parameters from constraints using Planck we find $\beta _{\phi } = 2.7^{+0.60}_{-0.67}$ suggesting $\beta _{\phi } > 0$ at 4.3$\sigma$ significance. This result may hint at a phase shift that is not purely sourced from the standard model expectation for $N_{\rm {eff}}$ or could be a upwards statistical fluctuation in the measured $\beta _{\phi }$; this result relaxes in models with additional freedom beyond Lambda-cold dark matter.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

A Public Data Set of Auto-Generated Geotagged PV Site Equipment, Generated via Deep Learning

In this research, we present a data set over 100 photovoltaic (PV) sites in TX, which have been automatically geotagged via a fully autonomous deep learning (DL) pipeline. Specifically, locations of inverters, tracker/fixed tilt rows, batteries, and substations are labeled algorithmically. To ensure high data quality, all systems have been reviewed manually and any deep learning errors have been corrected. This public data set, as well as the open-sourced pipeline used to generate it, is valuable for site planning, modelling, and insurance purposes. Given time and resources, we hope to extend the data set to additional states/regions in the US.

14 SOLAR ENERGY↗
DOE OSTI2025

A simulation pipeline for fast neutron imaging and spectroscopy using quantified detector attributes

Radiation imaging capabilities, essential in the nuclear nonproliferation regime, facilitate source localization and, in certain cases, spectroscopy. Scatter-based neutron cameras, which can measure the neutron signatures from special nuclear material, hold particular interest. Systems incorporating organic scintillators can extract neutron energy spectra, potentially distinguishing fission neutron sources from others, such as alpha-neutron sources. The development and testing of a scatter-based neutron imager, however, can be challenging without having an accurate simulation model or first constructing a prototype. This work describes a simulation pipeline that takes output from MCNPX-PoliMi simulations and creates the expected back-projection neutron images and neutron energy spectra. This pipeline was developed to improve the modeling of fast neutron imagers and bridge the current gap in literature, which predominantly focuses on gamma-ray Compton imager models. This work also reports on the significance of various real-world system considerations and their effects on the simulated detector responses. The pipeline was verified and validated with experimental data collected using a 252 Cf spontaneous fission source using a fast neutron scattering imager developed at the University of Michigan.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2022

Optical emissivity dataset of multi-material heterogeneous designs generated with automated figure extraction

Optical device design is typically an iterative optimization process based on a good initial guess from prior reports. Optical properties databases are useful in this process but difficult to compile because their parsing requires finding relevant papers and manually converting graphical emissivity curves to data tables. Here, we present two contributions: one is a dataset of thermal emissivity records with design-related parameters, and the other is a software tool for automated colored curve data extraction from scientific plots. We manually collected 64 papers with 176 figures reporting thermal emissivity and automatically retrieved 153 colored curve data records. The automated figure analysis software pipeline uses Faster R-CNN for axes and legend object detection, EasyOCR for axes numbering recognition, and k-means clustering for colored curve retrieval. Additionally, we manually extracted geometry, materials, and method information from the text to add necessary metadata to each emissivity curve. Finally, we analyzed the dataset to determine the dominant classes of emissivity curves and determine the underlying design parameters leading to a type of emissivity profile.

47 OTHER INSTRUMENTATION↗
DOE OSTI2020

X-ray properties of the X-CLASS-redMaPPer galaxy cluster sample: the luminosity–temperature relation

ABSTRACT This article presents the results of a spectroscopic analysis of the X-CLASS-redMaPPer (XC1-RM) galaxy cluster sample. X-CLASS is a serendipitous search for clusters in X-ray wavebands based on the XMM–Newton archive, whereas redMaPPer is an optical cluster catalogue derived from the Sloan Digital Sky Survey (SDSS). The present sample comprises 92 X-ray extended sources identified in optical images within 1 arcmin separation. The area covered by the cluster sample is ∼ 27 deg2. The clusters span a wide redshift range (0.05 < z < 0.6) and 88 clusters benefit from spectrosopically confirmed redshifts using data from SDSS Data Release 14. We present an automated pipeline to derive the X-ray properties of the clusters in three distinct apertures: R500 (at fixed mass overdensity), Rfit (at fixed signal-to-noise ratio) and ${R}_{300\, {\rm kpc}}$ (fixed physical radius). The sample extends over wide temperature and luminosity ranges: from 1–10 keV and from 6 × 1042 to 11 × 1044 erg s−1, respectively. We investigate the luminosity–temperature (L–T) relation of the XC1-RM sample and find a slope equal to 3.03 ± 0.26. It is steeper than predicted by self-similar assumptions, in agreement with independent studies. A simplified approach is developed to estimate the amount and impact of selection biases that might be affecting our recoveredL–Tparameters. The result of this simulation process suggests that the measuredL–Trelation is biased to a steeper slope and higher normalization.

Molham, Mona↗