SEARCH · Search NASA

Results for “data challenges”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19

All sources NASA NTRS DOE OSTI

DOE OSTI2023

Conflation of Geospatial POI Data and Ground-level Imagery

The code solves the problem of conflating POI (Points of Interest) geospatial data and geo-tagged ground-level imagery data. The main challenges with fusion or conflation of these two types of data has been that these two data entities are represented not only in several ways in current state-of-the-art, but also there is mismatch of representation format of these two data types. This source code/software brings POI datapoints and ground-level imagery datapoints into same representation format, and then generates valuable Knowledge Graph combining POI and images data (so that it can be used for multitude of applications).

De, Debraj↗

DOE OSTI2025

High-Fidelity, Large-Scale, Realistic Dataset Development

The final report summarizes the work performed for supporting the ARPA-E Grid Optimization Competition (Challenge 2 and Challenge 3) within the stated period. Challenge 2 For the challenge period, the main responsibility of the team is to investigate, gen- erate, and deliver parts of the data sets for the competition, based on the competition model for Challenge 2, existing data sets from Challenge 1, and data source supplied by other data set teams. Challenge 3 For the challenge period, the main responsibility of the team is to propose, create, deliver, and maintain the data format during the competition period. The data format will specify how the benchmark data will be represented and communicated to competitors. It will also specify how competitors should report back the solutions. The data format will be closely aligned with the problem formulation (maintained by the formulation team) and the solution validation process (maintained by the validation team). Our team is also responsible in investigating, generating, and delivering parts of the data sets for the competition. The data sets will be created based on the competition model for Challenge 3, existing data sets from Challenge 1 and Challenge 2, and data source supplied by other data set teams.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2024

Simulating image coaddition with the Nancy Grace Roman Space Telescope – I. Simulation methodology and general results

The upcoming Nancy Grace Roman Space Telescope will carry out a wide-area survey in the near-infrared. A key science objective is the measurement of cosmic structure via weak gravitational lensing. Roman data will be undersampled, which introduces new challenges in the measurement of source galaxy shapes; a potential solution is to use linear algebra-based coaddition techniques such as imcom that combine multiple undersampled images to produce a single oversampled output mosaic with a desired ‘target’ point spread function (PSF). We present here an initial application of imcom to 0.64 square degrees of simulated Roman data, based on the Roman branch of the Legacy Survey of Space and Time (LSST) Dark Energy Science Collaboration (DESC) Data Challenge 2 (DC2) simulation. We show that imcom runs successfully on simulated data that includes features such as plate scale distortions, chip gaps, detector defects, and cosmic ray masks. We simultaneously propagate grids of injected sources and simulated noise fields as well as the full simulation. We quantify the residual deviations of the PSF from the target (the ‘leakage’), as well as noise properties of the output images; we discuss how the overall tiling pattern as well as Moiré patterns appear in the final leakage and noise maps. We include appendices on interpolation algorithms and the interaction of undersampling with image processing operations that may be of broader applicability. The companion paper (‘Paper II’) explores the implications for weak lensing analyses.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

The Challenges of Using Real-Time Detection Systems: From Data Gathering to Actionable Information [Slides]

This session will discuss challenges and real life examples associated with Real-Time Detection Systems (RTDS) used in exposure assessment strategies. This first presentation will provide an overview of RTDS as used as part of an overall exposure assessment strategy and to document regulatory compliance. Managing OH consequences has been problematic when it comes to actionable information gleaned from sensor data. The judgement of regulatory entities, can be at odds with IH practitioners implementing an exposure strategy that creates exposure profiles and judges workplace exposures.

61 RADIATION PROTECTION AND DOSIMETRY↗

DOE OSTI2021

On using droplet burning data to validate combustion kinetic mechanisms of transportation fuels: challenges and opportunities

This presentation introduces our research on using droplet burning data to validate combustion kinetic mechanisms of transportation fuels: challenges and opportunities.

Guo, Songtao↗

DOE OSTI2025

From Machine Learning to Machine Reasoning: A Model-based Approach to Analyze Equipment Reliability Data

In current nuclear power plants (NPPs) a large amount of condition-based data which can be used to assess and monitor component health and performance. Assessing component health from such data can be performed with a large variety of methods. While the analysis of numeric data can be performed with several methods, the extraction of information from textual data remains a challenge. Currently employed natural language processing (NLP) methods do not really provide quantitative information that might be contained in IRs. In addition, the integration of numeric and textual data to identify possible causal relationships between data elements is still an unresolved challenge. This paper presents an approach to extract information from textual (e.g., incident or maintenance reports) and numeric data that relies on model based system engineer (MBSE) models. MBSE are diagrams designed to represent system and component dependencies (from both a form and functional point of view). In our approach, MBSE models emulate system engineer knowledge about component/system architecture. NLP methods are employed to perform syntactic and semantic analyses. Syntactic analysis analyzes the grammatical structure of a sentence while semantic analysis is designed to analyze the logic structure of a sentence. An innovative element of our approach is that semantic analysis uses MBSE models to identify links between textual elements. Similarly, numeric data is directly linked to elements of the MBSE models in order to map which functions are being monitored.

97 - MATHEMATICS AND COMPUTING↗

DOE OSTI2025

ROOT RNTuple and EOS: The Next Generation of Event Data I/O

For several years, the ROOT team is developing the new RNTuple I/O subsystem in preparation of the next generation of collider experiments. Both HL-LHC and DUNE are expected to start data taking by the end of this decade. They pose unprecedented challenges to event data I/O in terms of data rates, event sizes, and event complexity. At the same time, the I/O landscape is becoming more diverse. HPC cluster file systems and object stores, NVMe disk cache layers in analysis facilities, and S3 storage on cloud resources are mixing with traditional XRootD-managed spinning disk pools.The ROOT team will finalize a first production version of the RNTuple binary format by the end of 2024. After this point, ROOT will provide backward compatibility for RNTuple data. This contribution provides an overview of the RNTuple feature set, the related R&D activities and the long-term vision for RNTuple. We report on performance, interface design, tooling, robustness, integration with experiment frameworks, and validation results, as well as recent R&D on parallel reading and writing and exploitation of modern hardware and storage systems. We will give an outlook on possible future features after a first production release.Collaboratively, the IT and EP departments at CERN have launched a formal project within the Research and Computing sector to evaluate the novel data format for physics analysis data utilized in LHC experiments and other fields. This part of the project focuses on validating the scalability of the EOS storage backend during the transition from the over 25 years old TTree production format to the newly developed RNTuple format, using both replicated and erasure-coded storage profiles.

Blomer, Jakob [CERN]↗

DOE OSTI2023

Improving Progressive Retrieval for HPC Scientific Data using Deep Neural Network

As the disparity between compute and I/O on high-performance computing systems has continued to widen, it has become increasingly difficult to perform post-hoc data analytics on full-resolution scientific simulation data due to the high I/O cost. Error-bounded data decomposition and progressive data retrieval framework has recently been developed to address such a challenge by performing data decomposition before storage and reading only part of the decomposed data when necessary. However, the performance of the progressive retrieval framework has been suffering from the over-pessimistic error control theory, such that the achieved maximum error of recomposed data is significantly lower than the required error. Therefore, more data than required is fetched for recomposition, incurring additional I/O overhead. In order to tackle this issue, we propose a DNN-based progressive retrieval framework that can better identify the minimum amount of data to be retrieved. Our contributions are as follows: 1) We provide an in-depth investigation of the recently developed progressive retrieval framework; 2) We propose two designs of prediction models (named D-MGARD and E-MGARD) to estimate the amount of retrieved data size based on error bounds. 3) We evaluate our proposed solutions using scientific datasets generated by real-world simulations from two domains. Evaluation results demonstrate the effectiveness of our solution in accurately predicting the amount of retrieval data size, as well as the advantages of our solution over the traditional approach to reducing the I/O overhead. Based on our evaluation, our solution is shown to read significantly less data (5% - 40% with D-MGARD, 20% - 80% with E-MGARD).

Wang, Jinzhen↗

DOE OSTI2021

Quality Assurance Program Plan for SFR Metallic Fuel Data Qualification

This document contains an evaluation of the applicability of the current Quality Assurance Standards from the American Society of Mechanical Engineers Standard NQA-1 (NQA-1) criteria and identifies and describes the quality assurance process(es) by which attributes of historical, analytical, and other data associated with sodium-cooled fast reactor [SFR] metallic fuel will be evaluated. This process is being instituted to facilitate validation of data to the extent that such data may be used to support future licensing efforts associated with advanced reactor designs. The initial data to be evaluated under this program were generated during the US Integral Fast Reactor program between 1984-1994, where the data include, but are not limited to, research and development data and associated documents, test plans and associated protocols, operations and test data, technical reports, and information associated with past United States Nuclear Regulatory Commission reviews of SFR designs. It is recognized that managing the data generated by large research and development projects presents a significant challenge for retaining data integrity and availability. American Society of Mechanical Engineers Standard NQA-1 (NQA-1) 2008/2009a provides appropriate requirements for this plan.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2025

Quality Assurance Program Plan for SFR Metallic Fuel Data Qualification

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2020

NDMAS System and Process Description

The U. S. Department of Energy (DOE) has made a significant investment in research to develop the next generation of reactor technologies as well as to improve the performance and lengthen the life cycle of existing nuclear reactors. Data collected to demonstrate new concepts may also be used in the future to support licensing of these technologies. Provenance of these data must be preserved. The Nuclear Data Management and Analysis System (NDMAS) was established to manage and preserve data collected by fuels and materials research conducted by the high-temperature, gas-cooled reactor program. The scope of NDMAS is expanding to include other nuclear research programs that have the shared need to preserve the provenance of research data. Nuclear research funded by DOE is conducted by Idaho National Laboratory (INL), universities, other national laboratories, foreign research partners, and private companies. This research will generate a large amount of data from a variety of sources over a period of many years. Managing the data generated by the research and development projects presents a significant challenge for retaining data integrity and availability.

+↗

DOE OSTI2025

Data Centers and Digital Assurance Workshop 2 – Prioritizing Digital Assurance Challenges, Session 2

The second session of the TADA (Technical Assistance for Digital Assurance) Data Centers Cohort, held on November 10, 2025, focused on prioritizing digital assurance challenges at the intersection of data centers and the electric grid. Building on the foundational concepts introduced in Workshop 1, this session deepened the application of the Threat–Vulnerability–Consequence (TVC) framework and emphasized the urgency of addressing cybersecurity, supply chain integrity, and operational reliability. Participants explored the growing convergence of digital and physical systems, the expanding attack surface due to global supply chain dependencies, and the implications of AI-driven load behavior. Real-world incidents—including the Volt Typhoon campaign and vulnerabilities in Solarman and Deye platforms—were analyzed to illustrate the risks of unpatched systems, insecure APIs, and inadequate vendor oversight. Key themes included architecture and interface weaknesses, governance gaps, and human and procedural shortcomings. The workshop also examined the evolving regulatory landscape, highlighting new federal mandates around Foreign Entity of Concern (FEOC) compliance and large-load reliability standards. Through interactive exercises, stakeholders ranked and mapped digital assurance risks from their respective perspectives—utilities, operators, and vendors—laying the groundwork for mitigation strategies and shared accountability models to be developed in Workshop 3. Session 2 of 3.

24 - POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2021

SIRIUS: Science-Driven Data Management for Multi-Tiered Storage

The data sets being generated by large applications on very large-scale systems are increasing in both size and complexity. At the same time, there are new ways available to store and access these data sets. The goal in this project is to develop software that applications can use to make use of new and existing storage technologies in more sophisticated ways. One challenge in scientific data management is handling ‘hot’ vs ‘cold’ data. Data that is hot is data that is needed (or will be needed soon) in order for the program to continue progressing, while cold data is either output (and so will not be need further during the life of the program) or will not be needed until significantly later in the program’s run. Hot data should be stored in a way that allows fast access. On most systems, economic factors lead to an inverse relationship between storage performance and storage capacity and so fast access storage is limited. This makes it important to correctly place hot and cold data and avoid cold data unnecessarily consuming precious resources. In this reporting period, we addressed this challenge in various ways and at various levels. Data management frameworks offer only limited control to applications in how data is stored. We have added software capabilities for seamlessly moving data between layers of the storage technology using promote and demote functions to existing software frameworks. This gives direct control to applications in deciding what priority data receives. Additionally, we integrated different storage layer management frameworks in order to allow data to be exchanged and moved between storage layers in a consistent way across the application. Further, applications are not always able to directly decide what storage level makes sense for a given piece of data without an understanding of the underlying storage technologies. Data storage frameworks are often positioned to make these sorts of decisions in service of the application. We have added machine-learning based capabilities to data staging frameworks in order to make intelligent decisions about where data should be stored given learning about patterns in previous usage of similar data.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Geospatial Data Platform for All

Spatiotemporal data has evolved in scale due to augmented use in cross-domain applications. Simultaneously, there is substantial growth in the availability of Geographic Information Systems (GIS) data provided by the United States Geological Survey (USGS) along with other federal, state, county, or local agencies through open-data portals and public access APIs. However, data availability does not equate with accessibility. Large-scale analyses and applications require robust, performant data management with co-location of data storage and computing. The insufficiency of data management infrastructure compels researchers to adopt ad hoc project- specific GIS data storage solutions (e.g., copying data to High-Performance computer file systems). As an ad hoc storage strategy does not scale, it hampers cross-domain analyses causing difficulty in data reuse and utilizing existing code bases. Furthermore, GIS data is complex and requires expertise to analyze and manipulate due to its intricate data structures and data-specific projection transformations. Despite the challenges, we recognize that derived GIS data products, e.g., satellite or LIDAR-based images, can be used in downstream applications such as AI by domain, but non-GIS experts. To address the data needs and overcome the challenges, we are working towards a GIS Data Platform focused on efficient data storage, data discovery and access, and an API to enable common workflows. We propose a knowledge-graph (KG) approach for data discovery, whereby datasets are semantically linked to higher- level constructs such as projects and research areas. The semantic data links enable researchers to explore datasets in a top-down approach by specifying relevant and meaningful terms (assists in finding hidden data). An advantage is that the nodes and edges in a knowledge graph create built-in semantic documentation. Deeper spatiotemporal connections between data sources can be encoded via Graph Neural Networks (GNN) (Zhang et al., 2021). The KG approach can be extended to integrate the data itself in a Virtual KG (VKG). Our work will derive inspiration from large-scale VKG efforts that have been undertaken or are currently underway as part of the OpenStreetMap project (Ding et al., 2021). For DOE Data Days, we share the proposed geospatial data platform hybrid (cloud/on-prem) architecture, our work-to-date on storing, retrieving, and transforming LiDAR and raster data relevant to two important NREL use-cases, including the Renewable Energy Potential (reV) Model, and present our proposal for a KG based data discovery engine.

data platform↗

DOE OSTI2023

A Machine Learning–Based Tire Life Prediction Framework for Increasing Life of Commercial Vehicle Tires

In the commercial freight industry, tire retreading decisions are often conservative due to limited knowledge of a tire’s remaining service life. This practice leads to increased costs and material waste. This paper proposes a machine learning–based approach for estimating tire casing life and retreadability, focusing on usage data rather than wear information. This approach could extend the tire’s lifespan and reduce landfill waste. Data integration from diverse tire casing measurement sources presents challenges, including imbalanced removal data. Our methodology addresses these challenges by using historical inspection, telematics, and finite element modeling (FEM) datasets. We introduce “Tire Casing Energy” as a comprehensive usage input and apply a Variance-Reduction Synthetic Minority Oversampling Technique (VR-SMOTE) for data imbalance rectification. A random forest model is used to estimate the state of the tire casing and the casing removal probability, with Bayesian optimization applied for hyperparameter tuning, enhancing model accuracy. Here, the proposed prediction framework is able to differentiate different truck fleets and tire locations based on their usage parameters. With the aid of this machine learning model, the importance and sensitivity of different tire usage parameters can be obtained, which is beneficial to maximize tire life.

Data balancing↗

DOE OSTI2020

Using Machine Learning to Predict Future Temperature Outputs in Geothermal Systems

Optimizing the power output, and economic value, of geothermal power plants over decades of operation is a major challenge in renewable energy. Optimizing the output requires the ability to predict the mass flow rates and the output temperatures of production wells based on the inputs of injection wells, as well as the time history of the system. Machine Learning (ML) that incorporates the known physics of geothermal systems is one possible solution to this challenge. In this work, we explore the ability of ML algorithms to predict future temperature outputs based on historical data. Considering the challenges with obtaining an empirical dataset from field data that is large enough to enable reliable ML, we propose an alternate approach: developing a high-fidelity reservoir model and using computational resources to build a dataset that enables ML. As a first step towards achieving this goal, we present preliminary results from applying ML to predict the temperature timeseries of simple modeled geothermal systems. We describe the application of relevant state-of-the-art ML approaches, such as the Long Short-Term Memory (LSTM) networks and Convolutional Neural Networks (CNN), to extract temporal structures in the model data. We assess the accuracy of the forecasts we obtain, compare the selected approaches, and share the lessons learned that would inform the process of training and utilizing ML algorithms for larger and more complex geothermal systems.

GEOTHERMAL ENERGY↗

DOE OSTI2022

Deep learning-accelerated 3D carbon storage reservoir pressure forecasting based on data assimilation using surface displacement from InSAR

Fast forecasting of the reservoir pressure distribution during geologic carbon storage (GCS) by assimilating monitoring data is a challenging problem. Due to high drilling cost, GCS projects usually have spatially sparse measurements from few wells, leading to high uncertainties in reservoir pressure prediction. To address this challenge, we use low-cost Interferometric Synthetic-Aperture Radar (InSAR) data as monitoring data to infer reservoir pressure build up. We develop a deep learning-accelerated workflow to assimilate surface displacement maps interpreted from InSAR and to forecast dynamic reservoir pressure. Employing an Ensemble Smoother Multiple Data Assimilation (ES-MDA) framework, the workflow updates three-dimensional (3D) geologic properties and predicts reservoir pressure with quantified uncertainties. We use a synthetic commercial-scale GCS model with bimodally distributed permeability and porosity to demonstrate the efficacy of the workflow. A two-step CNN-PCA approach is employed to parameterize the bimodal fields. The computational efficiency of the workflow is boosted by two residual U-Net based surrogate models for surface displacement and reservoir pressure predictions, respectively. The workflow can complete data assimilation and reservoir pressure forecasting in half an hour on a personal computer.

25 ENERGY STORAGE↗

DOE OSTI2020

Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.

Godoy, William↗