SEARCH · Search NASA

Results for “data pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 703 records · Page 39

All sources NASA NTRS DOE OSTI

DOE OSTI2021

A Review of Machine Learning Classification Using Quantum Annealing for Real-World Applications

Optimizing the training of a machine learning pipeline helps in reducing training costs and improving model performance. One such optimizing strategy is quantum annealing, which is an emerging computing paradigm that has shown potential in optimizing the training of a machine learning model. The implementation of a physical quantum annealer has been realized by D-wave systems and is available to the research community for experiments. Recent experimental results on a variety of machine learning applications using quantum annealing have shown interesting results where the performance of classical machine learning techniques is limited by limited training data and high dimensional features. This article explores the application of D-wave’s quantum annealer for optimizing machine learning pipelines for real-world classification problems. Finally, we review the application domains on which a physical quantum annealer has been used to train machine learning classifiers. We discuss and analyze the experiments performed on the D-Wave quantum annealer for applications such as image recognition, remote sensing imagery, computational biology, and particle physics. We discuss the possible advantages and the problems for which quantum annealing is likely to be advantageous over classical computation.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

21 cm Power Spectrum Analysis of North Celestial Pole Observations with the Tianlai Dish Pathfinder Array

The Tianlai Dish Pathfinder Array (TDPA) is a radio interferometer designed to test techniques for 21 cm intensity mapping in the post-reionization Universe as a means of measuring large-scale cosmic structure. Using nine nights of observations targeting the North Celestial Pole field, totaling approximately 107 hr of integration time, we analyze data in the frequency range 700–800 MHz (corresponding to redshift z ∼ 0.9). We do the data format conversion, radio frequency interference flagging, calibration, imaging and point source subtraction, and foreground removal via Singular Value Decomposition. The spherically averaged power spectrum Δ 2 (k) is obtained. Furthermore, this work successfully establishes and validates a comprehensive data analysis framework for the TDPA. We identify key improvements including sky model refinement, increased integration time, and pipeline optimization that will enable future detection of the 21 cm signal through auto-correlation and cross-correlation with optical galaxy surveys.

cosmology: large-scale structure of universe↗

DOE OSTI2024

Quantum Annealing for Real-World Machine Learning Applications

Optimizing the training of a machine learning pipeline is important for reducing training costs and improving model performance. One such optimizing strategy is quantum annealing, which is an emerging computing paradigm that has shown potential in optimizing the training of a machine learning model. The implementation of a physical quantum annealer has been realized by D-Wave systems and is available to the research community for experiments. Recent experimental results on a variety of machine learning applications have shown interesting results especially under the conditions where the performance of classical machine learning techniques are limited such as limited training data and high dimensional features. This chapter explores the application of D-Wave’s quantum annealer for optimizing machine learning pipelines for real-world classification problems. We review the application domains on which a physical quantum annealer has been used to train machine learning classifiers. We discuss and analyze the experiments performed on the D-Wave quantum annealer for applications such as image recognition, remote sensing imagery, security, computational biology, biomedical sciences, and physics. We discuss the possible advantages and the problems for which quantum annealing is likely to be advantageous over classical computation.

Kumar nath, Rajdeep↗

DOE OSTI2021

The clustering of the SDSS-IV extended Baryon Oscillation Spectroscopic Survey quasar sample: testing observational systematics on the Baryon Acoustic Oscillation measurement

ABSTRACT Baryon Acoustic Oscillations (BAO) are considered to be a very robust standard ruler against various systematics. This premise has been tested against observational systematics, but not to the level required for the next generation of galaxy surveys such as the Dark Energy Spectroscopic Instrument (DESI) and Euclid. In this paper, we investigate the effect of observational systematics on the BAO measurement of the final sample of quasars from the extended Baryon Oscillation Spectroscopic Survey Data Release 16 in order to prepare and hone a similar analysis for upcoming surveys. We employ catalogues with various treatments of imaging systematic effects using linear and neural network-based non-linear approaches and consider how the BAO measurement changes. We also test how the variations to the BAO fitting model respond to the observational systematics. As expected, we confirm that the BAO measurements obtained from the DR16 quasar sample are robust against imaging systematics well within the statistical error, while reporting slightly modified constraints that shift the line-of-sight BAO signal by less than 1.1 per cent. We use realistic simulations with similar redshift and angular distributions as the DR16 sample to conduct statistical tests for validating the pipeline, quantifying the significance of differences, and estimating the expected bias on the BAO scale in future high-precision data sets. Although we find a marginal impact for the eBOSS QSO data, the work presented here is of vital importance for constraining the nature of dark energy with the BAO feature in the new era of big data cosmology.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

Containerization in ATLAS Software Development and Data Production

The ATLAS experiment's software production and distribution on the grid benefits from a semi-automated infrastructure that provides up-to-date information about software usability and availability through the CVMFS dis-tribution service for all relevant systems. The software development process uses a Continuous Integration pipeline involving testing, validation, packag-ing and installation steps. For opportunistic sites that can not access CVMFS, containerized releases are needed. These standalone containers are currently created manually to support Monte-Carlo data production at such sites. In this paper we will describe an automated procedure for the containerization of AT-LAS software releases in the existing software development infrastructure, its motivation, integration and testing in the distributed computing system.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

A scalable, open-source implementation of a large-scale mechanistic model for single cell proliferation and death signaling

Mechanistic models of how single cells respond to different perturbations can help integrate disparate big data sets or predict response to varied drug combinations. However, the construction and simulation of such models have proved challenging. Here, we developed a python-based model creation and simulation pipeline that converts a few structured text files into an SBML standard and is high-performance- and cloud-computing ready. We applied this pipeline to our large-scale, mechanistic pan-cancer signaling model (named SPARCED) and demonstrate it by adding an IFNγ pathway submodel. We then investigated whether a putative crosstalk mechanism could be consistent with experimental observations from the LINCS MCF10A Data Cube that IFNγ acts as an anti-proliferative factor. The analyses suggested this observation can be explained by IFNγ-induced SOCS1 sequestering activated EGF receptors. This work forms a foundational recipe for increased mechanistic model-based data integration on a single-cell level, an important building block for clinically-predictive mechanistic models.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2021

Challenges in solving structures from radiation-damaged tomograms of protein nanocrystals assessed by simulation

Structure-determination methods are needed to resolve the atomic details that underlie protein function. X-ray crystallography has provided most of our knowledge of protein structure, but is constrained by the need for large, well ordered crystals and the loss of phase information. The rapidly developing methods of serial femtosecond crystallography, micro-electron diffraction and single-particle reconstruction circumvent the first of these limitations by enabling data collection from nanocrystals or purified proteins. However, the first two methods also suffer from the phase problem, while many proteins fall below the molecular-weight threshold required for single-particle reconstruction. Cryo-electron tomography of protein nanocrystals has the potential to overcome these obstacles of mainstream structure-determination methods. Here, a data-processing scheme is presented that combines routines from X-ray crystallography and new algorithms that have been developed to solve structures from tomograms of nanocrystals. This pipeline handles image-processing challenges specific to tomographic sampling of periodic specimens and is validated using simulated crystals. The tolerance of this workflow to the effects of radiation damage is also assessed. The simulations indicate a trade-off between a wider tilt range to facilitate merging data from multiple tomograms and a smaller tilt increment to improve phase accuracy. Since phase errors, but not merging errors, can be overcome with additional data sets, these results recommend distributing the dose over a wide angular range rather than using a finer sampling interval to solve the protein structure.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2023

The Target-selection Pipeline for the Dark Energy Spectroscopic Instrument

Abstract In 2021 May, the Dark Energy Spectroscopic Instrument (DESI) began a 5 yr survey of approximately 50 million total extragalactic and Galactic targets. The primary DESI dark-time targets are emission line galaxies, luminous red galaxies, and quasars. In bright time, DESI will focus on two surveys known as the Bright Galaxy Survey and the Milky Way Survey. DESI also observes a selection of “secondary” targets for bespoke science goals. This paper gives an overview of the publicly available pipeline ( desitarget ) used to process targets for DESI observations. Highlights include details of the different DESI survey targeting phases, the targeting ID ( TARGETID ) used to define unique targets, the bitmasks used to indicate a particular type of target, the data model and structure of DESI targeting files, and examples of how to access and use the desitarget code base. This paper will also describe “supporting” DESI target classes, such as standard stars, sky locations, and random catalogs that mimic the angular selection function of DESI targets. The DESI target-selection pipeline is complex and sizable; this paper attempts to summarize the most salient information required to understand and work with DESI targeting data.

47 OTHER INSTRUMENTATION↗

DOE OSTI2025

CO2 hydrate crystal thickening, morphology, and Raman spectroscopy in a microfluidic device

Gas hydrates are a solid, crystalline form of water that often form at low temperatures and high pressures. Carbon dioxide (CO2) hydrates may form during carbon dioxide capture and storage (CCS) processes. These solid compounds may form in CO2 pipelines, potentially leading to a full blockage and process shutdown for plug removal. On the other hand, formation of CO2 hydrates may be desired for CO2 capture and separation. In either case, understanding the growth behavior and nature of the hydrates is vital to managing these CCS processes. Using a high-pressure, transparent microfluidic reactor, the crystalline film thickening of CO2 hydrates was observed and measured through visual microscopy and Raman spectroscopy. The impact of subcooling, pressure, and CO2 flow rate was investigated, and only CO2 flow rate was found to have a significant impact on the overall thickness of the film. Visual observations and Raman spectroscopy measurements confirmed that two distinct hydrate layers formed during thickening, one which was more porous than the other. The capillary-like channels in the porous layer indicated a mechanism for mass transfer of water through the hydrate layer. A model was developed based on this observation, and it was fit to the thickening data in order to obtain mass transfer coefficients. Results of this study can be applied to CO2 hydrate formation in pipelines and near porous media used for CO2 capture.

Wadsworth, Lindsey [Colorado School of Mines, Gold↗

DOE OSTI2025

A Radiation-Hard 8-Channel 15-Bit 40-MSPS ADC for the ATLAS Liquid Argon Calorimeter Readout

The custom design of a radiation-hardened, 8-channel, 40-MSPS, 15-bit resolution, 14.2-bit dynamic range, 11.4-ENOB ADC data acquisition ASIC fabricated in a commercial 65-nm triple-well CMOS technology is presented. The ADC is developed for and integrates seamlessly into the readout system for the ATLAS liquid argon (LAr) calorimeter in the high-luminosity large hadron collider (HLLHC) upgrade at CERN, which will require a total of 364 936 ADC channels. A three-stage MDAC+SAR pipelined ADC architecture was designed to meet the physics requirements and scientific goals of the ATLAS experiment. The ADC is a fully self-contained data acquisition system that includes foreground calibration, digital data processing, digital control, and supporting circuitry. The measured performance shows the ADC achieves a competitive dynamic range and SNDR, and it meets or exceeds the ATLAS analog requirements. Radiation tolerance and scalability design considerations were implemented at the device-, circuit-, and system-level. Radiation-hardening-by-design techniques used include redundancy for digital circuits, the use of MiM capacitors, and a hybrid RC-DAC for the ADC core. The ADC ASIC was demonstrated to be robust against the effects of the intense radiation expected in the HL-LHC experimental environment.

DAQ↗

DOE OSTI2026

Enhancing transfer learning in angle-resolved photoemission spectroscopy (ARPES) with spatially-aware representations via graph convolution

A recent application of machine learning has been to spatially-resolved angle-resolved photoemission spectroscopy (ARPES). Here we advance the state-of-the-art by applying representational learning to transform ARPES data into an embedding space of a pre-trained self-supervised learning model, thus enhancing the pipeline that improves the bandstructure classification and domain assignment/segmentation performance compared to a k-means clustering method. In the current iteration, the real-space information is entered into the domain assignment through the graph convolution method, which improves the transfer learning performance of the original self-supervised model. Lastly, an unsupervised automated tool is developed that incorporates these techniques to enable automatic domain assignment.

ARPES↗

DOE OSTI2026

pySimpleMask

SF-26-118 pySimpleMask is a tool for creating masks and Q-partition maps for X-ray scattering patterns, supporting SAXS, WAXS, and XPCS data reduction. It ships both a desktop GUI and a headless Python API that can drive the full pipeline from scripts.

Chu, Miaoqi [Argonne National Laboratory (ANL), Ar↗

DOE OSTI2025

Building Datasets and Training Methods for ML Based Magnet Quench Detection

Detecting quenches in superconducting (SC) magnets during training is a challenging process that involves capturing physical events that occur at different frequencies and appear as various signal features. These events may be correlated across instrumentation type, thermal cycle, and ramp. These events together build a more complete picture of continuous processes occurring in the magnet, and may allow us to flag potential precursors for quench detection. We present our work on building an automatic machine learning (ML) based quench detection system. We build upon our existing work on unsupervised auto-encoders for acoustic sensors and quench antenna (QA) by first establishing a supervised ML training pipeline. We show the results of an event tagging, analysis, and simulation framework on our QA and acoustic data which are used concurrently to build a training dataset for a supervised implementation. We then show how this supervised training can be used as a prior in a semi-supervised framework and compare this to the unsupervised neural network auto-encoder performance.This allows us to have a more concrete understanding of the performance of our algorithms relative to physical events occurring in the magnet, and also provides a baseline software tool to generically evaluate our quench prediction autoencoders under completely unsupervised, supervised, and semi-supervised training conditions.

Khan, Maira [Fermilab]↗

DOE OSTI2022

Panel-Segmentation: A Python Package for Automated Solar Array Metadata Extraction Using Satellite Imagery

The NREL Python Panel-Segmentation package is a toolkit that automates the process of extracting accurate and valuable metadata related to solar array installations, using publicly available Google Maps satellite imagery. Previously published work includes automated azimuth estimation for individual solar installations in satellite images. Our continued research focuses on automated detection and classification of solar installation mounting configuration (tracking or fixed-tilt; rooftop, ground, or carport). Specifically, a Faster-RCNN Resnet-50 feature pyramid network (FPN) model was trained and validated on 862 manually labeled satellite images. This model was used to perform object detection on satellite imagery, locating and classifying individual solar installations' mounting configuration and type. Model results showed a mean average precision score (mAP) of 77.79%, with the model strongest at detecting fixed-tilt ground mount and fixed-tilt carport installations. The object detection model and its outputs have been incorporated into the Panel-Segmentation package's automated metadata extraction pipeline, which returns the mounting configuration and azimuth for individual solar arrays in satellite imagery. The complete image data set with labels has been released on the U.S. Department of Energy (DOE) DuraMAT DataHub, to encourage further research in this area.

deep learning↗

DOE OSTI2023

Improving Protein–Ligand Interaction Modeling with cryo-EM Data, Templates, and Deep Learning in 2021 Ligand Model Challenge

Elucidating protein–ligand interaction is crucial for studying the function of proteins and compounds in an organism and critical for drug discovery and design. The problem of protein–ligand interaction is traditionally tackled by molecular docking and simulation, which is based on physical forces and statistical potentials and cannot effectively leverage cryo-EM data and existing protein structural information in the protein–ligand modeling process. In this work, we developed a deep learning bioinformatics pipeline (DeepProLigand) to predict protein–ligand interactions from cryo-EM density maps of proteins and ligands. DeepProLigand first uses a deep learning method to predict the structure of proteins from cryo-EM maps, which is averaged with a reference (template) structure of the proteins to produce a combined structure to add ligands. The ligands are then identified and added into the structure to generate a protein–ligand complex structure, which is further refined. The method based on the deep learning prediction and template-based modeling was blindly tested in the 2021 EMDataResource Ligand Challenge and was ranked first in fitting ligands to cryo-EM density maps. These results demonstrate that the deep learning bioinformatics approach is a promising direction for modeling protein–ligand interactions on cryo-EM data using prior structural information.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2026

Differential Privacy in Grid Kitchen: Implementation & Software Documentation

Sharing of power grid feeder models faces significant challenges due to the potential risk of exposing sensitive operational information. Traditional anonymization techniques have shown notable limitations in other sensitive domains, as evidenced by documented re-identification attacks that combine supposedly anonymized datasets with auxiliary information, raising concerns that similar vulnerabilities could affect power grid data. Consequently, there is a pressing need for a more rigorous privacy protection strategy that not only delivers formal mathematical guarantees but also preserves the analytical value of the shared models. To address this challenge, we have enhanced the Grid Kitchen framework by implementing differential privacy mechanisms within the distribution model dehydration pipeline. This implementation carefully calibrates and applies noise to sensitive attributes in feeder models according to configurable privacy levels—low, moderate, and high—each offering different balances between data utility and privacy protection. Our approach uses established noise functions (Gaussian for continuous data and Discrete Laplace for integer values) with parameters carefully calibrated so that the impact of individual data points is effectively masked in the final output. The integration leverages our Noise Catalog, which we developed to categorize feeder model properties by component type, data type, and sensitivity. This catalog guides the application of appropriate noise functions and privacy parameters ($\varepsilon$ and $\delta$) to each attribute, ensuring consistent privacy protection across the model while maintaining its structural integrity and analytical usefulness. This implementation also includes evaluation tools that allow model owners to assess the impact of privacy-preserving transformations before sharing data with external parties. This report provides documentation for the differential privacy capabilities added to the Grid Kitchen project. It includes a primer on differential privacy concepts and their importance in modern data sharing, details the architecture of our implementation, explains the privacy modes and parameter configurations, and offers practical guidance on using the code for applying differential privacy to grid feeder models. Through examples and code snippets, we demonstrate the effective application of these privacy-enhancing technologies, enabling utility operators and researchers to confidently share grid data while protecting sensitive information.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2023

A common resequencing‐based genetic marker data set for global maize diversity

SUMMARY Maize ( Zea mays ssp. mays ) populations exhibit vast ranges of genetic and phenotypic diversity. As sequencing costs have declined, an increasing number of projects have sought to measure genetic differences between and within maize populations using whole‐genome resequencing strategies, identifying millions of segregating single‐nucleotide polymorphisms (SNPs) and insertions/deletions (InDels). Unlike older genotyping strategies like microarrays and genotyping by sequencing, resequencing should, in principle, frequently identify and score common genetic variants. However, in practice, different projects frequently employ different analytical pipelines, often employ different reference genome assemblies and consistently filter for minor allele frequency within the study population. This constrains the potential to reuse and remix data on genetic diversity generated from different projects to address new biological questions in new ways. Here, we employ resequencing data from 1276 previously published maize samples and 239 newly resequenced maize samples to generate a single unified marker set of approximately 366 million segregating variants and approximately 46 million high‐confidence variants scored across crop wild relatives, landraces as well as tropical and temperate lines from different breeding eras. We demonstrate that the new variant set provides increased power to identify known causal flowering‐time genes using previously published trait data sets, as well as the potential to track changes in the frequency of functionally distinct alleles across the global distribution of modern maize.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2024

Power Profile Monitoring and Tracking Evolution of System-Wide HPC Workloads

The power & energy demands of HPC machines have grown significantly. Modern exascale HPC systems require tens of megawatts of combined power for computing resources and cooling facilities at full capacity. The current energy trend is not sustainable for future HPC systems, and there is a need to work toward the energy efficiency aspect of HPC performance. Energy awareness of the HPC applications at the job level is essential for running an efficient HPC system. This work aims to develop a pipeline to provide a production-level system-wide overview of the HPC workloads' power profile while handling evolving workloads exhibiting new power trends. We developed an open-set classification model for HPC jobs based on the properties of power profiles to continuously provide a system-wide holistic view of recently completed jobs. The pipeline helps continuously monitor the job-level power usage pattern of HPC and enables us to capture the new trends in applications' power behavior. We employed a comprehensive set of techniques to generate job-level data, custom-designed feature extraction methods to extract critical features from jobs' power profiles, clustering techniques powered by generative modeling, and open-set classification for identifying job profiles into known classes or an unknown set. With extensive evaluations, we demonstrate the effectiveness of each component in our pipeline. We provide an analysis of the resulting clusters that characterize the power profile landscape of the Summit supercomputer from more than 60K jobs executed in a year. The open-set classification classifies the known data sets into known classes with high accuracy and identifies unknown data noints with over 85% accuracy.

Karimi, Ahmad Maroof↗