Search NASA⌕ Search

SEARCH · Search NASA

Results for “data challenges”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

VizBrick

Brick (https://brickschema.org/) is a unified metadata schema to address the problem of building data standardization. Creating Brick models for building datasets means that the contents of the datasets are semantically described using the standard terms defined in the Brick ontology, and it will enable the benefits of data standardization, without having to recollect or reorganize the data. The challenge is that building brick models for building datasets leads to repeated manual trial and error processes, which can be time-consuming. VizBrick is a tool with a graphic/Web-based user interface that can assist users to create Brick models visually and interactively without having to understand the Resource Description Framework (RDF) syntax. VizBrick contains a web server that renders VizBrick web interface pages for browsers. The web server utilizes software components that (1) provide Brick ontology entity mapping to data column suggestions to users so that they can efficiently create their model; (2) provide keyword/Metadata-based search capability for easy find of relevant brick concepts and relations to their data columns

Lee, Sangkeun [Oak Ridge National Lab. (ORNL), Oak↗
DOE OSTI2022

Organizing Large Data Sets for Efficient Analyses on HPC Systems

Upcoming exascale applications could introduce significant data management challenges due to their large sizes, dynamic work distribution, and involvement of accelerators such as graphical processing units, GPUs. In this work, we explore the performance of reading and writing operations involving one such scientific application on two different supercomputers. Our tests showed that the Adaptable Input and Output System, ADIOS, was able to achieve speeds over 1TB/s, a significant fraction of the peak I/O performance on Summit. We also demonstrated the querying functionality in ADIOS could effectively support common selective data analysis operations, such as conditional histograms. In tests, this query mechanism was able to reduce the execution time by a factor of five. More importantly, ADIOS data management framework allows us to achieve these performance improvements with only a minimal amount of coding effort.

Gu, Junmin↗
DOE OSTI2021

Noise reduction in X-ray photon correlation spectroscopy with convolutional neural networks encoder–decoder models

Abstract Like other experimental techniques, X-ray photon correlation spectroscopy is subject to various kinds of noise. Random and correlated fluctuations and heterogeneities can be present in a two-time correlation function and obscure the information about the intrinsic dynamics of a sample. Simultaneously addressing the disparate origins of noise in the experimental data is challenging. We propose a computational approach for improving the signal-to-noise ratio in two-time correlation functions that is based on convolutional neural network encoder–decoder (CNN-ED) models. Such models extract features from an image via convolutional layers, project them to a low dimensional space and then reconstruct a clean image from this reduced representation via transposed convolutional layers. Not only are ED models a general tool for random noise removal, but their application to low signal-to-noise data can enhance the data’s quantitative usage since they are able to learn the functional form of the signal. We demonstrate that the CNN-ED models trained on real-world experimental data help to effectively extract equilibrium dynamics’ parameters from two-time correlation functions, containing statistical noise and dynamic heterogeneities. Strategies for optimizing the models’ performance and their applicability limits are discussed.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Data from Improved Annotation of Untargeted Metabolomics Data through Buffer Modifications That Shift Adduct Mass and Intensity

Annotation of untargeted high-resolution full-scan LC-MS metabolomics data remains challenging due to individual metabolites generating multiple LC-MS peaks arising from isotopes, adducts, and fragments. Adduct annotation is a particular challenge, as the same mass difference between peaks can arise from adduct formation, fragmentation, or different biological species. To address this, here we describe a buffer modification workflow (BMW) in which the same sample is run by LC-MS in both liquid chromatography solvent with 14NH3–acetate buffer and in solvent with the buffer modified with 15NH3–formate. Buffer switching results in characteristic mass and signal intensity changes for adduct peaks, facilitating their annotation. This relatively simple and convenient chromatography modification annotated yeast metabolomics data with similar effectiveness to growing the yeast in isotope-labeled media. Application to mouse liver data annotated both known metabolite and known adduct peaks with 95% accuracy. Overall, it identified 26% of ∼27 000 liver LC-MS features as putative metabolites, of which ∼2600 showed HMDB or KEGG database formula match. This workflow is well suited to biological samples that cannot be readily isotope labeled, including plants, mammalian tissues, and tumors.

Conversion↗
DOE OSTI2023

Particle size influences decay rates of environmental DNA in aquatic systems

Abstract Environmental DNA (eDNA) analysis is a powerful tool for remote detection of target organisms. However, obtaining quantitative and longitudinal information from eDNA data is challenging, requiring a deep understanding of eDNA ecology. Notably, if the various size components of eDNA decay at different rates, and we can separate them within a sample, their changing proportions could be used to obtain longitudinal dynamics information on targets. To test this possibility, we conducted an aquatic mesocosm experiment in which we separated fish‐derived eDNA components using sequential filtration to evaluate the decay rate and changing proportion of various eDNA particle sizes over time. We then fit four alternative mathematical decay models to the data, building towards a predictive framework to interpret eDNA data from various particle sizes. We found that medium‐sized particles (1–10 μm) decayed more slowly than other size classes (i.e., <1 and > 10 μm), and thus made up an increasing proportion of eDNA particles over time. We also observed distinct eDNA particle size distribution (PSD) between our Common carp and Rainbow trout samples, suggesting that target‐specific assays are required to determine starting eDNA PSDs. Additionally, we found evidence that different sizes of eDNA particles do not decay independently, with particle size conversion replenishing smaller particles over time. Nonetheless, a parsimonious mathematical model where particle sizes decay independently best explained the data. Given these results, we suggest a framework to discern target distance and abundance with eDNA data by applying sequential filtration, which theoretically has both metabarcoding and single‐target applications.

Brandão‐Dias, Pedro F. P.↗
DOE OSTI2024

Generalization error guaranteed auto-encoder-based nonlinear model reduction for operator learning

Many physical processes in science and engineering are naturally represented by operators between infinite-dimensional function spaces. The problem of operator learning, in this context, seeks to extract these physical processes from empirical data, which is challenging due to the infinite or high dimensionality of data. An integral component in addressing this challenge is model reduction, which reduces both the data dimensionality and problem size. In this paper, we utilize low-dimensional nonlinear structures in model reduction by investigating Auto-Encoder-based Neural Network (AENet). AENet first learns the latent variables of the input data and then learns the transformation from these latent variables to corresponding output data. Our numerical experiments validate the ability of AENet to accurately learn the solution operator of nonlinear partial differential equations. Furthermore, we establish a mathematical and statistical estimation theory that analyzes the generalization error of AENet. Finally, our theoretical framework shows that the sample complexity of training AENet is intricately tied to the intrinsic dimension of the modeled process, while also demonstrating the robustness of AENet to noise.

Auto-encoder↗
DOE OSTI2020

Improved Annotation of Untargeted Metabolomics Data through Buffer Modifications That Shift Adduct Mass and Intensity

Annotation of untargeted high-resolution full-scan LC-MS metabolomics data remains challenging due to individual metabolites generating multiple LC-MS peaks arising from isotopes, adducts and fragments. Adduct annotation is a particular challenge, as the same mass difference between peaks can arise from adduct formation, fragmentation, or different biological species. To address this, here we describe a Buffer Modification Workflow (BMW), in which the same sample is run by LC-MS in both liquid chromatography solvent with 14 NH 3 -acetate buffer, and in solvent with the buffer modified with 15 NH 3 -formate. Buffer switching results in characteristic mass and signal intensity changes for adduct peaks, facilitating their annotation. This relatively simple and convenient chromatography modification annotated yeast metabolomics data with similar effectiveness to growing the yeast in isotope-labeled media. Application to mouse liver data annotated both known metabolite and known adduct peaks with 95% accuracy. Altogether, it identified 26% of ~ 27,000 liver LC-MS features as putative metabolites, of which ~ 2600 showed HMDB or KEGG database formula match. This workflow is well-suited to biological samples that cannot be readily isotope labeled, including plants, mammalian tissues, and tumors.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Verifying the Computational Integrity of Power Grid Controls with Zero-Knowledge Proof

The control of future power grids is migrating from a centralized to a distributed/decentralized scheme to enable a massive penetration of distributed energy resources with rising dependence on communication infrastructure. A common assumption made for most existing distributed/decentralized controllers is that local controllers would faithfully follow the designated controller dynamics based on the data received from communication channels. However, with increased probability of cyberattacks on Operational Technology infrastructure, such an assumption could be risky because proper execution of the controller dynamics is then built on trust in secure communication and computation. In this work, we leverage a cryptography technology known as zero-knowledge scalable transparent arguments of knowledge (zk-STARK) to verify the computational integrity of power grid control algorithms, with projected linear dynamics-based control schemes as the initial proof-of-concept test case. The method presented here converts the cybersecurity challenge of data integrity for grid control into a subset of computational integrity.

computational integrity↗
DOE OSTI2021

Performance Portability in the Exascale Computing Project: Exploration Through a Panel Series

Performance portability is a critical issue for the Exascale Computing Project (ECP) because of nontrivial architectural differences between machines available today and those expected at exascale. Many ECP project teams are working toward performance portability, and would expect to benefit from sharing lessons learned, identifying gaps, and discovering opportunities for partnerships. To facilitate this communication, the IDEAS-ECP project partnered with the three focus areas of ECP (application development, software technology, and hardware and integration), and Department of Energy computing facilities, to lead a series of panel discussions on performance portability. The panels were organized around broadly common themes of algorithmic and data locality challenges. In this article, we describe the panel series, its objectives, and perspectives from the various areas of the project. Finally, we also discuss use cases that are distinctive, as well as conclusions drawn from the collective experience of the participants.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Black Swan Event Small-Sample Transfer Learning (BEST-L) and its Case Study on Electrical Power Prediction in COVID-19

The black swan event will usually cause a great impact on the normal operation of society. The scarcity of such events leads to a lack of relevant data and challenges in dealing with related problems. Different situations also make the traditional methods invalid. In this paper, a transfer learning framework and a convolutional neuron network are proposed to deal with the black swan small-sample events (BEST-L). Taking the COVID-19 as a typical black swan event, the BEST-L is utilized to achieve accurate mid-term load forecasting using the relationship between economy and electricity consumption. The experiment results show that the transfer learning model can effectively learn the basic knowledge about the relationship between the adopted input and output data and use a relatively small amount of data during the black swan event to improve the target areas' generalization. The approach and results can provide an effective approach to respond and react to sudden changes quickly and effectively in similar open problems.

black swan event↗
DOE OSTI2023

Testing shell-model interactions at high excitation energy and low spin: Nuclear resonance fluorescence in Ge 74

With their complex low-spin structure, the germanium isotopes have accrued a large number of experimental data which challenge nuclear models. The structure of the 74 Ge isotope is investigated here with nuclear resonance fluorescence as part of an extensive campaign of experimental tests of the shell model in the germanium isotopic chain using a number of complementary techniques. Levels were excited with 3.10 – 5.44 MeV photon beams provided by the High Intensity Gamma-Ray Source at TUNL. Many new levels were identified and their spins, parities, branching ratios, and associated scattering cross section values were determined. Large-scale shell-model calculations, which include jj44b and JUN45 effective interactions developed to describe nuclei in this mass region, predict the new data satisfactorily. Herein this study extends the validation of these two interactions, which are candidates for computation of matrix elements to be used in the interpretation of neutrinoless double-β decay experiments in 76 Ge, into the low-spin, high excitation energy domain.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2022

Topical Group on Application and Industry Community Engagement Frontier Snowmass 2021 (Summary Report)

HEP community leads and operates cutting-edge experiments for the DOE Office of Science which have challenging sensing, data processing, and computing requirements that far surpass typical industrial applications. To make necessary progress in the energy, material, and fundamental sciences, development of novel technologies is often required to enable these advanced detector and accelerator programs. Our capabilities include efficient co-design, which is a prerequisite to enable the deployment of advanced techniques in a scientific setting where development spans from rapid prototyping to robust and reliable production scale. This applies across the design spectrum from the low level fabrication techniques to the high level software development. It underpins the requirement for a holistic approach of innovation that accelerates the cycle of technology development and deployment. The challenges set by the next generation of experiments requires a collaborative approach between academia, industry and national labs. Just a single stakeholder will be unable to deliver the technologies required for the success of the scientific goals. Tools and techniques developed for High Energy Physics (HEP) research can accelerate scientific discovery more broadly across DOE Office of Science and other federal initiatives and also benefit industry applications.

43 PARTICLE ACCELERATORS↗
DOE OSTI2024

Unveiling Temporal Performance Deviation: Leveraging Clustering in Microservices Performance Analysis

As the market for cloud computing continues to grow, an increasing number of users are deploying applications as microservices. The shift introduces unique challenges in identifying and addressing performance issues, particularly within large and complex infrastructures. To address this challenge, we propose a methodology that unveils temporal performance deviations in microservices by clustering containers based on their performance characteristics at different time intervals. Showcasing our methodology on the Alibaba dataset, we found both stable and dynamic performance patterns, providing a valuable tool for enhancing overall performance and reliability in modern application landscapes.

Clustering↗
DOE OSTI2021

Strategic Energy Management Program Persistence and Cost Effectiveness An Analysis of the SEM Program Landscape

This study examines the relationship between strategic energy management (SEM) programs and their persistence and cost effectiveness, with analysis based on interview data from 24 SEM program administrators, SEM program evaluations, and other reports. The 80 interview questions focused on the topics of program design, energy savings, energy savings persistence, cost effectiveness, and customer SEM persistence. The generosity of interview respondents provided a wealth of data, resulting in a report of sufficient length to warrant inclusion of this brief guide of the report structure. The major sections are listed below with brief descriptions. Individual sections of this report are mainly stand-alone and do not require reading of other sections. As a result, there is some duplication between sections, but with differing levels of detail. Executive Summary: Presents three key conclusions of this work with a short description of potential actions to advance the understanding of each key finding. Brief Observations: Lists a large number of bulleted observations resulting from this research, arranged by the five major topic areas included in the interviews. Analysis details are provided in the Analysis of Interview Results section. Foundations for this Research: Provides an overview of SEM, SEM frameworks, SEM programs, the topics of persistence and cost effectiveness, and the focus of this research. Methodology: Details the approach and strategy of this research, providing background information relevant to the formulation of interview questions and the identification of which SEM programs to interview. Observations from Compiled Evaluations and Other Reports: Reports observations from the collection and analysis of program evaluations, annual reports, utility planning documents, and SEM-related white papers. This section, presented in bullet form, highlights challenges in data collection and ultimately a comparison of program practices as they pertain to persistence and cost effectiveness. Analysis of Interview Results: Presents detailed analysis of responses from SEM program administrators, arranged by the five major categories examined: program design, energy savings, energy savings persistence, cost effectiveness, and customer SEM persistence. Interview questions are generally grouped together into subsections when it makes sense to examine them together. SEM Programs Challenge Traditional Cost-Effectiveness Metrics: Details an analysis based on five key factors showing that applying traditional cost-effectiveness metrics to SEM programs is not straightforward. This invites the opportunity to consider whether traditional cost-effectiveness metrics are applicable to SEM programs, either individually or at large. Resolution of Research Hypotheses: Tabulates a set of hypotheses that were developed to address the fundamental nature of the research at hand. Analysis of responses to multiple questions informs an understanding of each hypothesis and can be used to better understand the SEM program environment at large.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

Predicting the von Neumann entanglement entropy using a graph neural network

Calculating the von Neumann entanglement entropy from experimental data is challenging due to its dependence on the complete wavefunction, forcing reliance on approximations such as classical mutual information (MI). We propose a machine learning approach using a graph neural network to predict the von Neumann entropy directly from experimentally accessible bitstrings. We test this approach on a Rydberg ladder system and achieve a mean absolute error of $3.6\,\times 10^{-3}$ when evaluating within the training range on a dataset with entropy values ranging from 0 to 1.9. The model achieves a mean absolute percentage error of 1.44% and outperforms MI-based bounds. When tested beyond the training range, the model maintains reasonable accuracy. Furthermore, we demonstrate that fine-tuning the model with small datasets significantly improves performance on data outside the original training range.

graph neural networks↗
DOE OSTI2025

Unsupervised anomaly clustering via offset alignment in multivariate grid sensing data

Modern industries increasingly rely on multi-sensor technologies to acquire complex, high-dimensional data streams, enabling advanced monitoring and control systems. One critical application is online anomaly detection in electrical smart grids, where multivariate and multimodal sensing technologies play a vital role. However, detecting anomalies in such time-series data is challenging due to their inherent temporal dependencies and stochastic behavior. Traditional approaches based on supervised and semi-supervised learning methods depend on labeled datasets, which are often unavailable in real-world scenarios. While unsupervised methods have emerged as promising alternatives, these methods are highly susceptible to noise and outliers commonly present in sensing applications. Furthermore, deep learning-based anomaly detection methods, despite their performance, are often criticized for their black-box nature, limiting their applicability in safety-critical and online environments where interpretability and explainability are paramount. In this work, we propose an unsupervised anomaly clustering method leveraging a cyclic alignment-based offset detection algorithm for multivariate time-series signals. The proposed method is applied to multivariate data collected from vibrational, voltage, and magnetic field sensors deployed in a local grid substation. Our results demonstrate the robustness of the algorithm in accurately clustering various anomalies/events across different sensing modalities. Additionally, we compare the effectiveness of the proposed approach against a simple pattern-based anomaly detection method, which performs well for univariate data but fails to generalize to multivariate and multimodal time-series data.

Mukherjee, Subrata [ORNL] (ORCID:0000000309930338)↗
DOE OSTI2025

Leveraging differentiable programming in the inverse problem of neutron stars

Neutron stars (NSs) probe the high-density regime of the nuclear equation of state (EOS). However, inferring the EOS from observations of NSs is a computationally challenging task. Here, in this work, we efficiently solve this inverse problem by leveraging differential programming in two ways. First, we enable full Bayesian inference in under one hour of wall time on a GPU by using gradient-based samplers, without requiring pretrained machine learning emulators. Moreover, we demonstrate efficient scaling to high-dimensional parameter spaces. Second, we introduce a novel gradient-based optimization scheme that recovers the EOS of a given NS mass-radius curve. We demonstrate how our framework can reveal consistencies or tensions between nuclear physics and astrophysics. First, we show how the breakdown density of a metamodel description of the EOS can be determined from NS observations. Second, we demonstrate how degeneracies in EOS modeling using nuclear empirical parameters can influence the inverse problem during gradient-based optimization. Looking ahead, our approach opens up new theoretical studies of the relation between NS properties and the EOS, while effectively tackling the data analysis challenges brought by future detectors.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2022

A laser powder bed fusion system for operando synchrotron x-ray imaging and correlative diagnostic experiments at the Stanford Synchrotron Radiation Lightsource

Laser powder bed fusion (LPBF) is a highly dynamic multi-physics process used for the additive manufacturing (AM) of metal components. Improving process understanding and validating predictive computational models require high-fidelity diagnostics capable of capturing data in challenging environments. Synchrotron x-ray techniques play a vital role in the validation process as they are the only in situ diagnostic capable of imaging sub-surface melt pool dynamics and microstructure evolution during LPBF-AM. In this article, a laboratory scale system designed to mimic LPBF process conditions while operating at a synchrotron facility is described. The system is implemented with process accurate atmospheric conditions, including an air knife for active vapor plume removal. Significantly, the chamber also incorporates a diagnostic sensor suite that monitors emitted optical, acoustic, and electronic signals during laser processing with coincident x-ray imaging. The addition of the sensor suite enables validation of these industrially compatible single point sensors by detecting pore formation and spatter events and directly correlating the events with changes in the detected signal. Experiments in the Ti–6Al–4V alloy performed at the Stanford Synchrotron Radiation Lightsource using the system are detailed with sufficient sampling rates to probe melt pool dynamics. X-ray imaging captures melt pool dynamics at frame rates of 20 kHz with a 2 µm pixel resolution, and the coincident diagnostic sensor data are recorded at 470 kHz. Furthermore, this work shows that the current system enables the in situ detection of defects during the LPBF process and permits direct correlation of diagnostic signatures at the exact time of defect formation.

36 MATERIALS SCIENCE↗