Search NASA⌕ Search

SEARCH · Search NASA

Results for “data challenges”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Polarized and unpolarized gluon PDFs: Generative machine learning applications for lattice QCD matrix elements at short distance and large momentum

Lattice quantum chromodynamics (QCD) calculations share a defining challenge by requiring a small finite range of spatial separation z between quark/gluon bilinears for controllable power corrections in the perturbative QCD factorization, and a large hadron boost p z for a successful determination of collinear parton distribution functions (PDFs). However, these two requirements make the determination of PDFs from lattice data very challenging. We present the application of generative machine learning algorithms to estimate the polarized and unpolarized gluon correlation functions utilizing short-distance data and extending the correlation up to z p z ≲ 14 , surpassing the current capabilities of lattice QCD calculations. We train physics-informed machine learning algorithms to learn from the short-distance correlation at z ≲ 0.36 fm and take the limit, p z → ∞ , thereby minimizing possible contamination from the higher-twist effects for a successful reconstruction of the polarized gluon PDF. We also expose the bias and problems with underestimating uncertainties associated with the use of model-dependent and overly constrained functional forms, such as x α ( 1 − x ) β and its variants to extract PDFs from the lattice data. We propose the use of generative machine learning algorithms to mitigate these issues and present our determination of the polarized and unpolarized gluon PDFs in the nucleon. Published by the American Physical Society 2025

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

Error-controlled, progressive, and adaptable retrieval of scientific data with multilevel decomposition

Extreme-scale simulations and high-resolution instruments have been generating an increasing amount of data, which poses significant challenges to not only data storage during the run, but also post-processing where data will be repeatedly retrieved and analyzed for a long period of time. The challenges in satisfying a wide range of post-hoc analysis needs while minimizing the I/O overhead caused by inappropriate and/or excessive data retrieval should never be left unmanaged. In this paper, we propose a data refactoring, compressing, and retrieval framework capable of 1) fine-grained data refactoring with regard to precision; 2) incrementally retrieving and recomposing the data in terms of various error bounds; and 3) adaptively retrieving data in multi-precision and multi-resolution with respect to different analysis. With the progressive data re-composition and the adaptable retrieval algorithms, our framework significantly reduces the amount of data retrieved when multiple incremental precision are requested and/or the downstream analysis time when coarse resolution is used. Experiments show that the amount of data retrieved under the same progressively requested error bound using our framework is 64% less than that using state-of-the-art single-error-bounded approaches. Parallel experiments with up to 1, 024 cores and ~ 600 GB data in total show that our approach yields 1.36× and 2.52× performance over existing approaches in writing to and reading from persistent storage systems, respectively.

Liang, Xin↗
DOE OSTI2022

Toward CO 2 Multimeasurement Geophysical Monitoring in the North Dakota CarbonSAFE Project

Paper presented at 16th International Conference on Greenhouse Gas Control Technologies (GHGT-16), October 23–27, 2022, Lyon, France. Overcoming data acquisition challenges associated with a lake in the study area, electrical infrastructure around the plant, noise from mine activities, complex near-surface conditions associated with reclaimed mine land, high wind speed, and extremely cold temperatures, data processing, modeling, and inversion of the baseline geophysical data demonstrated the importance of high-quality data for a CO 2 geophysical monitoring program. Implementing more advanced data acquisition and processing techniques in future monitoring surveys can further improve confidence in interpretations of geophysical survey data.

20 FOSSIL-FUELED POWER PLANTS↗
DOE OSTI2023

Generating Synthetic Time Series Photovoltaic Data with Real-World Physical Challenges and Noise for Use in Algorithm Test and Validation

The PV Fleet Data Initiative and other projects seek the develop algorithms for automated analysis of PV time series data for extraction of statistical information and other parameters of the data such as degradation rates, soiling loss information, tracker performance, clipping or curtailment, system availability and other valuable information. While there is a vast body of PV data available for application of said extraction algorithms it is difficult to validate these algorithms because the true parameters to be extracted are not known. There has been a wide use of synthetic data in the literature for algorithm validation but this synthetic data is typically very bounded by the problem or topic at hand. The PV Fleet Data Initiative project has demonstrated that real time series PV data almost always includes a host of data quality and physical problems that, in reality, any automated PV abstraction algorithm must handle appropriately. For this reason, this work describes the development of a complex synthetic PV times series data set that includes data quality and physical problems that have been experienced in real world PV data. The various quality and physical problems are documented in the synthetic data so that users can test the validity of various PV extraction algorithms as well as develop new algorithms to solve problems this data set can support.

14 SOLAR ENERGY↗
DOE OSTI2020

Basic Research Needs for Transformative Manufacturing (Brochure)

Manufacturing is central to the nation’s prosperity and security. Manufacturing currently represents about 12% of the gross domestic product, provides nearly 13 million jobs, and accounts for about 25% of energy use. The nation’s economy relies heavily on wide-ranging manufacturing sectors - all of which share common challenges including data issues, lack of physics and chemistry-based models across scales, and resource constraints in a global environment. Furthermore, there are many hurdles that must be overcome to move basic science innovations to market. Addressing broad-ranging challenges demands a basic-science strategy that underpins applied research activities. This strategy would accelerate innovation and transform manufacturing. A Basic Research Needs workshop for Transformative Manufacturing was held in March 2020. The focus of the workshop was to identify the basic science research priorities that could accelerate innovation to transform manufacturing in the future. This was the first workshop of its kind to examine how basic energy science can drive manufacturing forward and innovate new ways to manufacture goods. Five Priority Research Directions were identified that address these science challenges: (1) innovative synthetic approaches to enable scalable assembly of matter, (2) computational methods and theoretical models to transform how manufacturing processes are controlled, (3) new characterization tools that can handle the necessary complexity, scales, and processing speeds to meet manufacturing needs, (4) new science to address opportunities relevant to sustainable and energy-efficient manufacturing, and (5) foundational approaches to co-design of materials, process, and products.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Impact of genotype‐calling methodologies on genome‐wide association and genomic prediction in polyploids

Abstract Discovery and analysis of genetic variants underlying agriculturally important traits are key to molecular breeding of crops. Reduced representation approaches have provided cost‐efficient genotyping using next‐generation sequencing. However, accurate genotype calling from next‐generation sequencing data is challenging, particularly in polyploid species due to their genome complexity. Recently developed Bayesian statistical methods implemented in available software packages, polyRAD, EBG, and updog, incorporate error rates and population parameters to accurately estimate allelic dosage across any ploidy. We used empirical and simulated data to evaluate the three Bayesian algorithms and demonstrated their impact on the power of genome‐wide association study (GWAS) analysis and the accuracy of genomic prediction. We further incorporated uncertainty in allelic dosage estimation by testing continuous genotype calls and comparing their performance to discrete genotypes in GWAS and genomic prediction. We tested the genotype‐calling methods using data from two autotetraploid species, Miscanthus sacchariflorus and Vaccinium corymbosum , and performed GWAS and genomic prediction. In the empirical study, the tested Bayesian genotype‐calling algorithms differed in their downstream effects on GWAS and genomic prediction, with some showing advantages over others. Through subsequent simulation studies, we observed that at low read depth, polyRAD was advantageous in its effect on GWAS power and limit of false positives. Additionally, we found that continuous genotypes increased the accuracy of genomic prediction, by reducing genotyping error, particularly at low sequencing depth. Our results indicate that by using the Bayesian algorithm implemented in polyRAD and continuous genotypes, we can accurately and cost‐efficiently implement GWAS and genomic prediction in polyploid crops.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2026

pathSQE : an automated workflow for single-crystal inelastic neutron scattering data processing and analysis

Inelastic neutron scattering (INS) experiments utilizing modern time-of-flight spectrometers enable the comprehensive mapping of the energy (E)- and momentum (Q)-resolved dynamical structure factor of single crystals, probing both the lattice and magnetic excitations. Yet, the large size and complexity of four-dimensional INS data are challenging current analysis workflows, often resulting in an underutilization of the measured information. To help address this issue, this paper introduces new software interfaced with the Mantid framework, pathSQE, designed to streamline the processing, analysis and interpretation of 4D single-crystal INS data. By automating key tasks such as 1D/2D slicing, symmetrization, Brillouin zone folding, data visualization, prioritization and filtering, and comparisons with simulations, pathSQE facilitates and accelerates INS data analysis workflows. Here, this paper outlines the features and implementation and provides several illustrations of the use of pathSQE on data collected on single crystals using direct-geometry time-of-flight spectrometers at the Spallation Neutron Source, including Ge, FeSi, MnO and SnS single-crystal measurements on the ARCS, HYSPEC and CNCS neutron spectrometers. Beyond streamlining post-experiment data processing, pathSQE establishes an automated and modular processing pipeline that could support future real-time experiment steering.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Simultaneous cross-evaluation of heterogeneous E. coli datasets via mechanistic simulation

The extensive heterogeneity of biological data poses challenges to analysis and interpretation. Construction of a large-scale mechanistic model of Escherichia coli enabled us to integrate and cross-evaluate a massive, heterogeneous dataset based on measurements reported by various groups over decades. We identified inconsistencies with functional consequences across the data, including that the total output of the ribosomes and RNA polymerases described by data are not sufficient for a cell to reproduce measured doubling times, that measured metabolic parameters are neither fully compatible with each other nor with overall growth, and that essential proteins are absent during the cell cycle—and the cell is robust to this absence. Finally, considering these data as a whole leads to successful predictions of new experimental outcomes, in this case protein half-lives.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Development of a Deep Learning Model for Predicting the Drag Coefficients of Spherical and Non-Spherical Particles,

There is yet to be a well-established drag model for non-spherical particles required in a particle-laden flow that could cover a wide range of sphericities. This talk will explore the development of a general drag model for non-spherical particles by applying deep learning using available experimental data available in the literature. The integration of several raw experimental measurements from different sources and research directions allows the training of robust Artificial Intelligence and Machine Learning (ML) models. Neural networks are an ML approach inspired by the inner biological workings of the brain. This work aims to develop a Deep Neural Network (DNN) that predicts drag coefficient values with the ability to adapt appropriately to unseen data. Given the limited number of data points available and the variance found within the data collected from various sources, challenges may arise when looking to train the model. Our study tests and implements various model regularization techniques and assesses different loss and activation functions for the proposed DNN. The proposed model considers a broader range of features other than sphericity and Reynold number. These features include density ratio, solid volume fraction, lengthwise and crosswise sphericity, and more. Furthermore, we present the features that play a significant role in predicting different drag coefficients through feature importance. Within the investigated parameter ranges in this study, the following conclusions can be achieved and summarized below: • An improved drag coefficient model can be developed by considering more features such as, aspect ratio, lengthwise sphericity, crosswise sphericity, and density ratio. • DNN model can predict better results compared to traditional methods using MAE metric. • The proposed model addresses data challenges such as limited data and extreme data points through expanded feature-set and regularization. • Three major features that mostly affect the drag coefficient were identified from a feature importance analysis.

Presa-Reyes, Maria↗
DOE OSTI2024

Detection of Isotopes in Urban Source Search Gamma Spectra Using Hopfield Neural Network

Source search campaigns involve the measurement of consecutive background gamma-ray spectra obtained with a mobile detector-spectrometer, such as tellurium-doped sodium iodide NaI(Tl) scintillator detector-spectrometer. Counts are measured with NaI(Tl) detector in short acquisition intervals, such as one second. Detection of isotopes in search data is challenging because of the low signal-to-noise ratio (S/N) of the spectral lines and highly varying background due to naturally occurring radioactive material (NORM). In this work, we investigate the performance of the detection of orphan Cs-137, Am-241, and I-131 isotopes in a gamma ray data set obtained in a search survey. Isotopic detection is performed with a Hopfield Neural Network (HNN), which implements an auto-associative memory by matching incoming measurements to the closest one of its memory-stored signature patterns. HNN is trained on examples of spectra of calibrated isotopic sources with well-resolved spectral lines. During testing, HNN processes spectra with partially complete or distorted isotopic lines. The performance of the HNN detection algorithm is benchmarked with Precision, Recall, and F1 scores. Test results demonstrate the potential of HNN to detect the isotope of interest with high accuracy (F1 > 90%) for all isotopes in this study.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2022

Introduction to Special Section: Machine Learning for Image-based Geologic Interpretation

Image-based geological interpretation has been a labor-intensive and time-consuming process because it requires well-trained geoscientists to identify geological structures, features, and textures from various types of images. These images include scanning electron microscopic images, optical microscopic images, optical photos, resistivity images, seismic volumes, remote-sensing images, etc. With fast-evolving machine learning (ML) technology and computing power in recent decades, computers can achieve nearhuman-level to super-human-level performance with scalable high efficiency in the computer vision field. These technological revolutions facilitated image-based geological interpretation in petroleum exploration and production. For example, a fault picking method applied to 3-D seismic volume data using deep learning can achieve superior performance in comparison to conventional auto-picking methods. In addition, under the new normal of low oil prices, the petroleum industry seeks cost-effective strategies such as automating traditionally labor-intensive processes. Nevertheless, the potential of applying ML to geological image interpretation is still facing a few key challenges including data scarcity, data distribution, poor data and/or label quality, data leakage, learning algorithms, model architecture, training methodologies, testing and evaluation metrics, hyper-parameters optimization, model drift, production deployment, and the like.

58 GEOSCIENCES↗
DOE OSTI2024

Development of spatially and temporally resolved electron density measurements for the assessment of radiation hydrodynamics simulations of laboratory X-ray photoionized plasmas

The photoionized plasma gas cell experiment is an established platform we use to make at-parameter (ξ > > 1 ergs cm s –1 ) measurements of plasma properties with application to high-energy astrophysical systems. We model the experiments with 1D radiation hydrodynamics simulations using the HELIOS-CR code to inform our understanding and assist in the interpretation of results. The simulations predict that the bulk of the plasma is in a quasi-uniform and hydrodynamically unperturbed state throughout the duration of the experiment. To evaluate this prediction, we introduced a photonic Doppler velocimetry (PDV) diagnostic to measure spatially and temporally resolved plasma electron density. The initial measurements were successful but had limitations that made model-data comparisons challenging. To address this, we re-designed the gas cell PDV diagnostic and doubled the number of measurement locations to sample across two thirds of the depth of the cell. We also present a comparison of the results from the upgraded PDV diagnostic to the HELIOS-CR simulations for the first time. As a result, the experimental data confirms the prediction of an unperturbed region in the bulk of the plasma but reveals discrepancies in the time evolution and spatial distribution of the simulated electron density.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Multi phenomena melt pool sensor data fusion for enhanced process monitoring of laser powder bed fusion additive manufacturing

Finding actionable trends in laser-based metal additive manufacturing process monitoring data is challenging owing to the diversity and complexity of the underlying physical interactions. A single monitoring solution that captures a particular process phenomenon, such as a photodiode that tracks melt pool intensity, is not alone capable of evaluating process stability or detecting flaw formation with sufficient precision for routine application in industry. In this work, to improve flaw detection performance, we adopted a data fusion approach that captures multiple process phenomena. To demonstrate this, we acquired data from laser powder bed fusion (LPBF) builds of cylindrical specimens produced with different laser spot sizes, emulating defocusing due to process faults such as thermal lensing. The resulting specimens had porosity of varying types and severity, quantified by post-build non-destructive X-ray computed tomography, Archimedes density measurements, and destructive metallographic characterization. During the build, the melt pool state was monitored with two coaxial high-speed video cameras and a temperature field imaging system. Physically intuitive low-level melt pool signatures, such as melt pool temperature, shape and size, and spatter intensity were extracted from this high-dimensional, image-based sensor data. These process signatures were subsequently used as input features in relatively simple machine learning models, such as a support vector machine, which were trained to detect laser defocusing, and in addition, predict porosity type and severity. The results show that the data fusion approach significantly enhanced system performance by reducing the overall false positive rate from ~ 0.1 to ~ 0.001 without sacrificing the true positive rate (~0.90). These results were at par with a black-box, deep machine learning approach (convolutional neural network).

36 MATERIALS SCIENCE↗
DOE OSTI2025

Energy-resolved neutron imaging and diffraction including grain orientation mapping using event camera technology

Time-of-flight neutron diffraction and energy-resolved imaging each provide unique perspectives into material properties. Neutron diffraction is useful for assessing microstructural parameters such as phase composition, texture, and dislocation densities, though it typically provides averaged data over the sampled volume. Energy-resolved imaging, on the other hand, offers both spatial and spectral information by detecting Bragg edges and neutron absorption resonances, which enables detailed mapping of microstructure and isotopic composition. When combined, these techniques have the potential to enrich our understanding of material behavior across different scales, enhancing our understanding of complex materials. Traditionally, these modalities are conducted on separate instruments, which is time-consuming and poses challenges for data integration. Here, we report the integration of the LumaCam, an event-mode energy-resolved neutron imaging camera with the HIPPO time-of-flight diffractometer at LANSCE. This integration enables simultaneous diffraction and imaging across the full spectrum, with analysis optimized for diffraction and Bragg-edge imaging in the thermal range (0.45–10 Å) and resonance imaging in the epithermal range (0.5–3000 eV), facilitating comprehensive multi-modal analysis. We demonstrate its capabilities through case studies, including spatial mapping of grain orientations in a steel sample and accurate thickness estimations for irregular samples including a depleted uranium cylinder and a natural silver-containing mineral specimen. The combined setup enhances real-time sample alignment and provides comprehensive data for crystal structure, texture, and isotopic composition analysis. This approach opens new possibilities for advanced applications in nuclear engineering, archaeology, and materials science.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Data efficiency assessment of generative adversarial networks in energy applications

This study investigates the data requirements of generative artificial intelligence (AI), particularly generative adversarial networks (GANs), for reliable data augmentation in energy applications. Generative AI, though seen as a solution to data limitations, requires substantial data to learn meaningful distributions—a challenge often overlooked. This study addresses the challenge through synthetic data generation for critical heat flux (CHF) and power grid demand, focusing on renewable and nuclear energy. Two variants of GAN employed are conditional GAN (cGAN) and Wasserstein GAN (wGAN). Our findings include the strong dependency of GAN on data size, with performance declining on smaller datasets and varying performance when generalizing to unseen experiments. Mass flux and heated length significantly influence CHF predictions. wGAN is more robust to feature exclusion, making it suitable for constrained synthetic data generation. In energy demand forecasting, wGAN performed well for solar, wind, and load predictions. Longer lookback hours and larger datasets improved predictions, especially for load power. Seasonal variations posed challenges, with wGAN achieving a relatively high error of Root Mean Squared Error (RMSE) of 0.32 for load power prediction, compared to RMSE of 0.07 under same-season conditions. Feature exclusions impacted cGAN the most, while wGAN showed greater robustness. This study concludes that, while generative AI is effective for data augmentation, it requires substantial data and careful training to generate realistic synthetic data and generalize to new experiments in engineering applications.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2021

Predicting Nugget Size of Resistance Spot Welds Using Infrared Thermal Videos With Image Segmentation and Convolutional Neural Network

Resistance spot welding (RSW) is a widely adopted joining technique in automotive industry. Recent advancement in sensing technology makes it possible to collect thermal videos of the weld nugget during RSW using an infrared (IR) camera. The effective and timely analysis of such thermal videos has the potential of enabling in situ nondestructive evaluation (NDE) of the weld nugget by predicting nugget thickness and diameter. Deep learning (DL) has demonstrated to be effective in analyzing imaging data in many applications. However, the thermal videos in RSW present unique data-level challenges that compromise the effectiveness of most pre-trained DL models. We propose a novel image segmentation method for handling the RSW thermal videos to improve the prediction performance of DL models in RSW. The proposed method transforms raw thermal videos into spatial-temporal instances in four steps: video-wise normalization, removal of uninformative images, watershed segmentation, and spatial-temporal instance construction. The extracted spatial-temporal instances serve as the input data for training a DL-based NDE model. The proposed method is able to extract high-quality data with spatial-temporal correlations in the thermal videos, while being robust to the impact of unknown surface emissivity. Overall, our case studies demonstrate that the proposed method achieves better prediction of nugget thickness and diameter than predicting without the transformation.

42 ENGINEERING↗
DOE OSTI2024

ARPA-E Grid Optimization (GO) Competition Challenge 3

Synthetic Input Data and Team Results for the GO Competition Challenge 3 for Events 1 - 4 and the Sandbox, along with problem and format descriptions and code to validate data and solutions, are available here. Data for industry scenarios will not be made public. The Grid Optimization (GO) Competition Challenge 3 focused on the security-constrained optimal power flow (SCOPF) problem. It is part of a continuing effort begun with Challenges 1 and 2, to successfully discover, develop, and test innovative and disruptive software solutions for critical energy challenges and to overcome existing barriers. The broader goal of the of the GO Competition is to accelerate the development of transformational and disruptive methods for solving problems related to the electric power grid and to provide a transparent, fair, and comprehensive evaluation of new solution methods. Challenge 3 used multiperiod dynamic markets, including advisory models for extreme weather events, day-ahead markets, and the real-time markets with an extended look-ahead. In Event 4, whose submission window was August 31-September 4, 2023, 14 teams solved for the objective values of 669 scenarios (39 scenarios required solutions both with and without line switching being allowed). The 591 synthetic scenarios from 9 network models (3.6 GB) are available here. Ten teams were funded to participate and 7 won prizes totaling $2,400,000. The largest prize ($550,000) went to Mississippi State University. An additional $600,000 was awarded in Event 3 (6/15-16/2023). No prizes were awarded in Events 1 (1/25-27/2023) or 2 (4/13-14/2023). For more information on the competition and challenge see the "GO Competition Challenge 3 Information" resource below.

ACOPF↗