Results for “data challenges” — Page 30

NASA NTRS2017

GEONEX: Land Monitoring From a New Generation of Geostationary Satellite Sensors

The latest generation of geostationary satellites carry sensors such as ABI (Advanced Baseline Imager on GOES-16) and the AHI (Advanced Himawari Imager on Himawari) that closely mimic the spatial and spectral characteristics of Earth Observing System flagship MODIS for monitoring land surface conditions. More importantly they provide observations at 5-15 minute intervals. Such high frequency data offer exciting possibilities for producing robust estimates of land surface conditions by overcoming cloud cover, enabling studies of diurnally varying local-to-regional biosphere-atmosphere interactions, and operational decision-making in agriculture, forestry and disaster management. But the data come with challenges that need special attention. For instance, geostationary data feature changing sun angle at constant view for each pixel, which is reciprocal to sun-synchronous observations, and thus require careful adaptation of EOS algorithms. Our goal is to produce a set of land surface products from geostationary sensors by leveraging NASA's investments in EOS algorithms and in the data/compute facility NEX. The land surface variables of interest include atmospherically corrected surface reflectances, snow cover, vegetation indices and leaf area index (LAI)/fraction of photosynthetically absorbed radiation (FPAR), as well as land surface temperature and fires. In order to get ready to produce operational products over the US from GOES-16 starting 2018, we have utilized 18 months of data from Himawari AHI over Australia to test the production pipeline and the performance of various algorithms for our initial tests. The end-to-end processing pipeline consists of a suite of modules to (a) perform calibration and automatic georeference correction of the AHI L1b data, (b) adopt the Multi-Angle Implementation of Atmospheric Correction (MAIAC) algorithm to produce surface spectral reflectances along with compositing schemes and QA, and (c) modify relevant EOS retrieval algorithms (e.g., LAI and FPAR, GPP, etc.) for subsequent science product generation. Initial evaluation of Himawari AHI products against standard MODIS products indicate general agreement, suggesting that data from geostationary sensors can augment low earth orbit (LEO) satellite observations.

geostationary↗

NASA NTRS2022

Building a Land Data Assimilation Community to Tackle Technical Challenges in Quantifying and Reducing Uncertainty in Land Model Predictions

Explore the source record for details and available documents.

Land↗

DOE OSTI2024

GraphAide: Advanced Graph-Assisted Query and Reasoning System

Curating knowledge from multiple siloed sources that contain both structured and unstructured data is a major challenge in many real-world applications. Pattern matching and querying represent fundamental tasks in modern data analytics that leverage this curated knowledge. The development of such applications necessitates overcoming several research challenges, including data extraction, named entity recognition, data modeling, and designing query interfaces. Moreover, the explainability of these functionalities is critical for their broader adoption. The emergence of Large Language Models (LLMs) has accelerated the development lifecycle of new capabilities. Nonetheless, there is an ongoing need for domain-specific tools tailored to user activities. The creation of digital assistants has gained considerable traction in recent years, with LLMs offering a promising avenue to develop such assistants utilizing domain-specific knowledge and assumptions. In this context, we introduce an advanced query and reasoning system, GraphAide, which constructs a knowledge graph (KG) from diverse sources and allows to query and reason over the resulting KG. GraphAide harnesses both the KG and LLMs to rapidly develop domain-specific digital assistants. It integrates design patterns from retrieval augmented generation (RAG) and the semantic web to create an agentic LLM application. GraphAide underscores the potential for streamlined and efficient development of specialized digital assistants, thereby enhancing their applicability across various domains.

Purohit, Sumit [BATTELLE (PACIFIC NW LAB)] (ORCID:↗

NASA NTRS2019

Air Mobility Data & Reasoning Fabric

Introduce mobility challenge, how air mobility can address the mobility challenge, then how an Air Mobility Data & Reasoning Fabric can enable the envisioned future air mobility. Poses the question of what is an effective role for NASA in enabling an Air Mobility Data & Reasoning Fabric.

Van Dalsem, William R.↗

DOE OSTI2024

From Data to Knowledge: A Graph-Based Reliability Approach to Assess System Health

With the goal of maximizing plant reliability and availability, complex systems such as nuclear power plants continuously monitor and record the performance and the health status of many components, assets, and systems. Such data may take the form of online monitoring data, condition reports, and maintenance reports and it carries the potential to provide system engineers with insights into anomalous behaviors or degradation trends as well as the possible causes behind them and to predict their direct consequences. The analysis of such data poses however few challenges. While some of these challenges are technical in nature (i.e., data are often distributed over several physical servers or databases), others are conceptual in nature (i.e., data elements come in different formats, numeric or textual), and measured values have different scales (e.g., vibration spectra and oil temperature). This paper directly tackles these challenges, and it focuses on the integration of all these data elements in order to assist plant system engineers in analyzing component, assets, and systems performances and optimize maintenance activities. This is performed by 1) extracting knowledge from textual data via technical language processing methods, and 2) quantifying system, asset, and component health from numeric condition-based data. We rely on model-based system engineering (MBSE) models of systems and assets to identify their architecture and functional (i.e., cause and effect) relations. Numeric and textual data elements are then associated with an MBSE graph element, based on their nature. This bonding of MBSE models and data elements constitutes a first-of-its-kind knowledge graph of a nuclear power plants system, with data elements being organized in a structured manner that enables system engineers to identify cause-effect trends in data elements and carry out appropriate actions in response.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Challenges and Vision for Standardization of Biopolymer Data Sets for Machine Learning

Machine learning (ML) is transforming materials research, yet potential for biopolymer discovery remains constrained by fragmented data and nonstandardized reporting. Biopolymers differ significantly from synthetic polymers, requiring specialized approaches to represent their biosynthetic origins, hierarchical structures, and application-specific metrics. In this Perspective, we identify three core challenges limiting biopolymer representation: information encoding, data quality, and data sharing. We describe the most pressing issues and propose commensurate approaches to address each key challenge. Recommendations include the design and adoption of biopolymer-specific fingerprinting and representation frameworks, development of hybrid human-large language model (LLM) data extraction strategies, and expanding Findable, Accessible, Interoperable, Reusable (FAIR)-compliant repositories. We propose a robust foundation to define interoperable, high-quality data sets that capture the full context of biopolymer materials. Standardized metadata, shared ontologies, and community-driven infrastructure would enable scalable, reproducible workflows and accelerate the ML-driven development of biopolymers.

36 MATERIALS SCIENCE↗

DOE OSTI2021

Rapid failure mode classification and quantification in batteries: A deep learning modeling framework

Unique, rapid identification and quantification of the dominant aging modes in lithium-ion batteries (LiBs) with early and non-specialized test data is a significant scientific challenge. Leveraging synthetic-data, deep-learning (DL) techniques have great potential to enable fast and robust classification and quantification of battery aging modes that produce different patterns of cell aging. This study, for the first time, presents a synthetic–data-based DL modeling framework for rapid and automatic classification and quantification of battery-aging modes and resultant aging with experimental validation. Availing synthetic dQ.dV -1 curves for ~26000 initial conditions and aging modes, the framework classified the dominant aging modes, for cells undergoing fast charge, in fewer than 100 cycles. Upon classification, the framework quantified the evolution of the aging modes, which were often nonuniform with cycling, for 22 gr/NMC532 pouch cells tested up to 600 cycles at different charging rates (1C–9C).

25 ENERGY STORAGE↗

DOE OSTI2025

Predicting Large‐Scale Systematic Missing Pipe Attributes in Water Distribution Networks

Water distribution network (WDN) models are an essential tool used by water utilities for hydraulic analysis. Unfortunately, missing data and insufficient resources often make creating and maintaining these models unfeasible. Existing methods to address missing pipe properties, like sequential imputation for missing values and reconstruction using graph metrics, are designed to accommodate random patterns of missing information and require a significant percentage of the system's attributes to be known. However, these data completeness assumptions do not always align with real‐world scenarios where large sections of the WDN model have missing data. To address this challenge, this study proposes a data‐driven approach for estimating pipe diameter when considering different spatial patterns and degrees of data completeness (i.e., 0%–90%). Using data from 16 WDNs in Kentucky, this study compares the use of machine learning (ML) using topological and geospatial features against an existing deterministic approach. Results demonstrate that WDN models with pipe diameters predicted by the proposed ML method had comparable hydraulic performance to the ground truth models. Moreover, results showed that ML method performance varies between WDNs of differing topological classification. Insights from this study help advance the ability to leverage partial data to create and maintain WDN models amid uncertainty and inadequate resources.

Poff, Jason W. [Oregon State Univ., Corvallis, OR ↗

DOE OSTI2025

Developing an oxidation materials ontology for data-driven materials design

Materials data is complex, and managing and storing materials data for use and reuse is a common challenge. An ontology-based data management framework can address these challenges through encoding data attributes and relationships in a flexible way. This presentation discusses the creation of an ontology for alloy oxidation test data and reviews the logic, structure and interoperability of the ontology.

advanced alloy development↗

NASA NTRS1991

Microgravity science requirements and the need for data compression

The Microgravity Science and Applications Div. (MSAD) of the NASA Office of Space Science and Applications (OSSA) is responsible for encouraging and directing the research of a wide range of physical phenomena in reduced gravity. Under MSAD's direction, NASA-Lewis is presently developing the concept of a multiuser facility which will perform combustion science experiments in space. This facility, the Combustion Experiment Module (CEM), will be located in either the Shuttle Spacelab or the Space Station Freedom lab and will be operational by mid-1997. In addition to standard instrumentation to measure temperature, pressure, and acceleration, CEM shall use a variety of imaging and optical diagnostic techniques. Images shall be the primary source of experimental data. These images create an enormous amount of data which must be archived on orbit for later analysis. Also, ground based investigators will require enough data from the orbiting facility to determine if the experimental parameters need to be changed before proceeding with the next run. The storage and transmission of this data present a major challenge to the CEM design. Data compression will play a major role in the design of the CEM diagnostics system.

Hartz, William G.↗

NASA NTRS2000

Assimilation of IASI and AIRS Data: Information Content and Quality Control

The Infrared Atmospheric Sounding Interferometer (IASI) and Atmospheric Infrared Sounder (AIRS) instruments have two orders of magnitude more channels that the current operational infrared sounder (High Resolution Infra-Red Sounder (HIRS)). This data volume presents a technological challenge for using the data in a data assimilation system. Data reduction will be a necessary for assimilation. It is important to understand the information content of the radiance measurements for data reduction purposes. In this talk, I will discuss issues relating to information content and quality control for assimilation of the AIRS and IASI data.

Joiner, J.↗

NASA NTRS2004

Application of Bayesian Classification to Content-Based Data Management

The high volume of Earth Observing System data has proven to be challenging to manage for data centers and users alike. At the Goddard Earth Sciences Distributed Active Archive Center (GES DAAC), about 1 TB of new data are archived each day. Distribution to users is also about 1 TB/day. A substantial portion of this distribution is MODIS calibrated radiance data, which has a wide variety of uses. However, much of the data is not useful for a particular user's needs: for example, ocean color users typically need oceanic pixels that are free of cloud and sun-glint. The GES DAAC is using a simple Bayesian classification scheme to rapidly classify each pixel in the scene in order to support several experimental content-based data services for near-real-time MODIS calibrated radiance products (from Direct Readout stations). Content-based subsetting would allow distribution of, say, only clear pixels to the user if desired. Content-based subscriptions would distribute data to users only when they fit the user's usability criteria in their area of interest within the scene. Content-based cache management would retain more useful data on disk for easy online access. The classification may even be exploited in an automated quality assessment of the geolocation product. Though initially to be demonstrated at the GES DAAC, these techniques have applicability in other resource-limited environments, such as spaceborne data systems.

Lynnes, Christopher↗

NASA NTRS2017

Role of Forcing Uncertainty and Background Model Error Characterization in Snow Data Assimilation

Accurate specification of the model error covariances in data assimilation systems is a challenging issue. Ensemble land data assimilation methods rely on stochastic perturbations of input forcing and model prognostic fields for developing representations of input model error covariances. This article examines the limitations of using a single forcing dataset for specifying forcing uncertainty inputs for assimilating snow depth retrievals. Using an idealized data assimilation experiment, the article demonstrates that the use of hybrid forcing input strategies (either through the use of an ensemble of forcing products or through the added use of the forcing climatology) provide a better characterization of the background model error, which leads to improved data assimilation results, especially during the snow accumulation and melt-time periods. The use of hybrid forcing ensembles is then employed for assimilating snow depth retrievals from the AMSR2 (Advanced Microwave Scanning Radiometer 2) instrument over two domains in the continental USA with different snow evolution characteristics. Over a region near the Great Lakes, where the snow evolution tends to be ephemeral, the use of hybrid forcing ensembles provides significant improvements relative to the use of a single forcing dataset. Over the Colorado headwaters characterized by large snow accumulation, the impact of using the forcing ensemble is less prominent and is largely limited to the snow transition time periods. The results of the article demonstrate that improving the background model error through the use of a forcing ensemble enables the assimilation system to better incorporate the observational information.

assimilation↗

NASA NTRS2018

Human in the Loop Tests: Experiences in Space Suit Testing

What is a space suit? Space suit testing; How and why we test them; hardware design vs. user functionality; Data collection lessons learned; Challenges of objective and subjective data; Personal experience from 2 perspectives: test director and test subject

Ross, Amy J.↗

NASA NTRS2024

Bridging the Gap: Enhancing Prominence and Provenance of NASA Datasets in Research Publications

Attribution of datasets that were used to generate research results described in peer-reviewed publications to the original source of these datasets (which are often archived at NASA Earth Science data centers) has been very challenging. Even though the data citation standard of citing datasets as research artifacts and citing them with Digital Object Identifiers (DOIs) was introduced over a decade ago, most authors do not properly reference the data used in their studies and merely mention them in the text. The lack of proper citations of datasets makes the peer-reviewed publication less transparent, imperils reproducibility, and impedes open science. We offer an open-source publication management methodology and a tool that can help to enhance usage-based data discovery, prominence, and provenance of the data; reproducibility of the research results; and potentially increase the return on investment on NASA-funded research.

open-source↗

DOE OSTI2021

Streamflow simulation in data-scarce basins using Bayesian and physics-informed machine learning models

Hydrologic predictions at rural watersheds are important but also challenging due to data shortage. Long short-term memory (LSTM) networks are a promising machine learning approach and have demonstrated good performance in streamflow predictions. However, due to its data-hungry nature, most LSTM applications focus on well-monitored catchments with abundant and high-quality observations. In this work, we investigate predictive capabilities of LSTM in poorly monitored watersheds with short observation records. To address three main challenges of LSTM applications in data-scarce locations, i.e., overfitting, uncertainty quantification (UQ), and out-of-distribution prediction, we evaluate different regularization techniques to prevent overfitting, apply a Bayesian LSTM for UQ, and introduce a physics-informed hybrid LSTM to enhance out-of-distribution prediction. Through case studies in two diverse sets of catchments with and without snow influence, we demonstrate that 1) when hydrologic variability in the prediction period is similar to the calibration period, LSTM models can reasonably predict daily streamflow with Nash–Sutcliffe efficiency above 0.8, even with only 2 years of calibration data; 2) when the hydrologic variability in the prediction and calibration periods is dramatically different, LSTM alone does not predict well, but the hybrid model can improve the out-of-distribution prediction with acceptable generalization accuracy; 3) L2 norm penalty and dropout can mitigate overfitting, and Bayesian and hybrid LSTM have no overfitting; and 4) Bayesian LSTM provides useful uncertainty information to improve prediction understanding and credibility. In conclusion, these insights have vital implications for streamflow simulation in watersheds where data quality and availability are a critical issue.

54 ENVIRONMENTAL SCIENCES↗

NASA NTRSDate not supplied

Expanding Repository Data Available For Sharing and Knowledge Discovery

Some of the hardest space biology and space health challenges require data-intensive, bioinformatic, meta-analytical, and computer-assisted research approaches. These challenges include examining interdisciplinary space life science research across experiments and across interacting spaceflight hazards (radiation, altered gravity, confinement, hostile-closed environments, distance-duration from Earth). The approaches to confront these challenges involve mining multiple datasets simultaneously from various hierarchical organizations of biological complexity, all while concurrently evaluating how experimental design factors affect endpoints of standard assays. To enable this field, it is essential that principal investigators (PIs) submit data in a structure so it can be maximally re-used. The purpose of the NASA Ames Life Sciences Data Archive (ALSDA) is to collect, curate, and make publicly available all non-human space-relevant biological data. ALSDA must also ensure data are open-access, and maximally findable, accessible, interoperable, and reusable (FAIR). The scope of ALSDA data collected and submitted by PIs include subject and study design metadata, assay metadata parameters, raw and processed assay data, assay imagery/video, and subject-experienced mission data telemetry (radiation, temperature, humidity, acoustics, vibrations, etc.). ALSDA recently integrated into a collaborative group of Open Science projects to facilitate a suite of new tools and workflows that will improve data submission, accessibility, and reusability by implementing digital data submission agreements, and adopting the data management system originally developed by NASA GeneLab. ALSDA intends to bring current biological repository data and all future collected data into this new scientific data reuse reality. This new suite of tools will enable ALSDA to deploy a science curation system using scientific assay configurations for the data submission portal. It will capture essential assay parameters according to established standards in each sub-field within biology. The submission portal expedites data collection by enhancing ease of PI data submission, providing a user interface and specificity for which data is to be submitted. Data submissions can be brought into cutting-edge informatic analysis portals to enable mining of physiological, behavioral, biochemical, and imaging datasets in conjunction with ‘omics-level datasets. As ALSDA datasets are submitted, curated, and published (e.g., micro-computed tomography, histology, pulse oximetry, serum metabolites, magnetic resonance imaging, intraocular pressure, novel object recognition, etc.), the merging together of spaceflight data along this multi-hierarchical complexity of biology will enable informatics and data-intensive approaches resulting in knowledge discoveries across missions, space hazards, and biological disciplines.

Biology↗

NASA NTRSDate not supplied

Expanding Repository Data Available For Sharing And Knowledge Discovery

Some of the hardest space biology and space health challenges require data-intensive, bioinformatic, meta-analytical, and computer-assisted research approaches. These challenges include examining interdisciplinary space life science research across experiments and across interacting spaceflight hazards (radiation, altered gravity, confinement, hostile-closed environments, distance-duration from Earth). The approaches to confront these challenges involve mining multiple datasets simultaneously from various hierarchical organizations of biological complexity, all while concurrently evaluating how experimental design factors affect endpoints of standard assays. To enable this field, it is essential that principal investigators (PIs) submit data in a structure so it can be maximally re-used. The purpose of the NASA Ames Life Sciences Data Archive (ALSDA) is to collect, curate, and make publicly available all non-human space-relevant biological data. ALSDA must also ensure data are open-access, and maximally findable, accessible, interoperable, and reusable (FAIR). The scope of ALSDA data collected and submitted by PIs include subject and study design metadata, assay metadata parameters, raw and processed assay data, assay imagery/video, and subject-experienced mission data telemetry (radiation, temperature, humidity, acoustics, vibrations, etc.). ALSDA recently integrated into a collaborative group of Open Science projects to facilitate a suite of new tools and workflows that will improve data submission, accessibility, and reusability by implementing digital data submission agreements, and adopting the data management system originally developed by NASA GeneLab. ALSDA intends to bring current biological repository data and all future collected data into this new scientific data reuse reality. This new suite of tools will enable ALSDA to deploy a science curation system using scientific assay configurations for the data submission portal. It will capture essential assay parameters according to established standards in each sub-field within biology. The submission portal expedites data collection by enhancing ease of PI data submission, providing a user interface and specificity for which data is to be submitted. Data submissions can be brought into cutting-edge informatic analysis portals to enable mining of physiological, behavioral, biochemical, and imaging datasets in conjunction with ‘omics-level datasets. As ALSDA datasets are submitted, curated, and published (e.g., micro-computed tomography, histology, pulse oximetry, serum metabolites, magnetic resonance imaging, intraocular pressure, novel object recognition, etc.), the merging together of spaceflight data along this multi-hierarchical complexity of biology will enable informatics and data-intensive approaches resulting in knowledge discoveries across missions, space hazards, and biological disciplines.

life science↗