Search NASA⌕ Search

SEARCH · Search NASA

Results for “ensemble data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

Delineating Facies Spatial Distribution by Integrating Ensemble Data Assimilation and Indicator Geostatistics with Level-Set Transformation

The facies-based approach has been widely adopted to delineate an aquifer into distinct geological units with unique distributions of hydraulic, physical, and/or chemical properties. The recent development in ensemble-based data assimilation methods allows both the direct and indirect data to be used to improve facies delineation. A major difficulty in those applications is to honor the spatial continuity and avoid overfitting after data assimilation. We introduce a new facies delineation framework to integrate ensemble data assimilation with traditional transition probability-based geostatistics. A level-set concept is used to parametrize discrete facies indicators and for updating facies shape. During the iterative data assimilation process, we impose spatial continuity by conditioning facies field generation on points selected adaptively based on their sensitivity to observation data. This reconditioning step is a key step to maintain spatial continuity and overcome over-fitting problems in inversion. We selected two examples to evaluate the performance of the new framework in estimating facies-based permeability field. The first example is a two-dimensional synthetic system with transient head data induced by pumping tests used for delineating two facies. The second example is a three-dimensional case with three facies, conceptualized from a field tracer experiment within the Columbia River corridor in Washington State, USA. Both examples demonstrate that the new method can adequately capture the spatial pattern of hydrofacies with reconditioning, which leads to the improved prediction of system behaviors.

Song, Xuehang↗
NASA NTRS2013

Ensemble Data Assimilation Without Ensembles: Methodology and Application to Ocean Data Assimilation

Two methods to estimate background error covariances for data assimilation are introduced. While both share properties with the ensemble Kalman filter (EnKF), they differ from it in that they do not require the integration of multiple model trajectories. Instead, all the necessary covariance information is obtained from a single model integration. The first method is referred-to as SAFE (Space Adaptive Forecast error Estimation) because it estimates error covariances from the spatial distribution of model variables within a single state vector. It can thus be thought of as sampling an ensemble in space. The second method, named FAST (Flow Adaptive error Statistics from a Time series), constructs an ensemble sampled from a moving window along a model trajectory. The underlying assumption in these methods is that forecast errors in data assimilation are primarily phase errors in space and/or time.

Data Assimilation↗
NASA NTRS2004

Ensemble Data Mining Methods

Ensemble Data Mining Methods, also known as Committee Methods or Model Combiners, are machine learning methods that leverage the power of multiple models to achieve better prediction accuracy than any of the individual models could on their own. The basic goal when designing an ensemble is the same as when establishing a committee of people: each member of the committee should be as competent as possible, but the members should be complementary to one another. If the members are not complementary, Le., if they always agree, then the committee is unnecessary---any one member is sufficient. If the members are complementary, then when one or a few members make an error, the probability is high that the remaining members can correct this error. Research in ensemble methods has largely revolved around designing ensembles consisting of competent yet complementary models.

Oza, Nikunj C.↗
NASA NTRS2014

Device and Method for Gathering Ensemble Data Sets

An ensemble detector uses calibrated noise references to produce ensemble sets of data from which properties of non-stationary processes may be extracted. The ensemble detector comprising: a receiver; a switching device coupled to the receiver, the switching device configured to selectively connect each of a plurality of reference noise signals to the receiver; and a gain modulation circuit coupled to the receiver and configured to vary a gain of the receiver based on a forcing signal; whereby the switching device selectively connects each of the plurality of reference noise signals to the receiver to produce an output signal derived from the plurality of reference noise signals and the forcing signal.

Racette, Paul E.↗
DOE OSTI2024

Uncertainty Visualization Challenges in Decision Systems with Ensemble Data & Surrogate Models: Preprint

Uncertainty visualization is a key component in translating important insights from ensemble data into actionable decision-making by visually conveying various aspects of uncertainty within a system. With the recent advent of fast surrogate models for computationally expensive simulations, users can interact with more aspects of data spaces than ever before. However, the integration of ensemble data with surrogate models in a decision-making tool brings up new challenges for uncertainty visualization, namely how to reconcile and communicate the new and different types of uncertainties brought in by surrogates and how to utilize these new data estimates in actionable ways. In this work, we examine these issues as they relate to high-dimensional data visualization, the integration of discrete datasets and the continuous representations of those datasets, and the unique difficulties associated with systems that allow users to iterate between input and output spaces. We assess the role of uncertainty visualization in facilitating intuitive and actionable interaction with ensemble data and surrogate models, and highlight key challenges in this new frontier of computational simulation.

ensemble visualization↗
DOE OSTI2024

Uncertainty Visualization Challenges in Decision Systems with Ensemble Data & Surrogate Models

Uncertainty visualization is a key component in translating important insights from ensemble simulation data into actionable decision-making by visually conveying various aspects of uncertainty within a system. With the recent advent of fast surrogate models trained on ensemble data, we can substitute computationally expensive simulations, which allows users to interact with more aspects of data spaces than ever before. However, the use of ensemble data with surrogate models in a decision-making tool brings up new challenges for uncertainty visualization, namely how to reconcile and communicate the new and different types of uncertainties brought in by surrogates and how to utilize these new data estimates in actionable ways. In this work, we examine these issues as they relate to high-dimensional data visualization, the integration of discrete datasets and the continuous representations of those datasets, and the unique difficulties associated with systems that allow users to iterate between input and output spaces. We assess the role of uncertainty visualization in facilitating intuitive and actionable interaction with ensemble data and surrogate models, and highlight key challenges in this new frontier of computational simulation.

ensemble data↗
DOE OSTI2026

Repository of HydroSMADE: Hydropower Site-level Monthly Availability Data Ensemble for 1950-2100 at Existing and Potential Global Sites

This repository presents HydroSMADE—Hydropower Site-level Monthly Availability Data Ensemble, a new open dataset that provides monthly hydropower availability for 1,593 existing and 124,333 potential sites worldwide over the period 1950–2100. The dataset is generated by using a global hydrologic model (Xanthos) with explicit representation of hydropower operation. Specifically, HydroSMADE distinguishes between storage and diversion sites, applies optimized operating rules, and incorporates site-specific characteristics such as generation capacity, maximum turbine flow, and reservoir storage. Driven by bias-corrected meteorological inputs, the data is provided for 30 alternative future scenarios. The scenarios consist of the full factorial combination of three standard CMIP6 atmospheric forcing pathways (SSP1-2.6, SSP3-7.0, and SSP5-8.5) and ten CMIP6 General Circulation Models (GCMs): GFDL-ESM4, IPSL-CM6A-LR, MPI-ESM1-2-HR, MRI-ESM2-0, EC-Earth3, CanESM5, MIROC6, CNRM-ESM2-1, UKESM1-0-LL, and CNRM-CM6-1. The repository contains a total of 122 files: a text file (readme.txt) containing a brief description of the included data, a CSV file containing site attributes, and the remaining 120 files (in CSV) containing site-level monthly hydropower availability. Example Jupyter Notebooks to explore the HydroSMADE dataset are available on GitHub at https://github.com/kamal0013/HydroSMADE More details on the methods and technical validation of HydroSMADE are available in the following paper by the same authors: Chowdhury, A. K., Abeshu, G. W., Zhao, M., Wild, T. B., Hassan, N., Ying, Z., Kim, G. J., Matthew, B., Jonathan, L., & Li, H.-Y. (Submitted). Hydropower Site-level Monthly Availability Data Ensemble for 1950-2100 at Existing and Potential Global Sites.

Existing and Potential Sites↗
DOE OSTI2015

Delineating Hydrofacies Spatial Distribution by Integrating Ensemble Data Assimilation and Indicator Geostatistics

This study develops a new framework of facies-based data assimilation for characterizing spatial distribution of hydrofacies and estimating their associated hydraulic properties. This framework couples ensemble data assimilation with transition probability-based geostatistical model via a parameterization based on a level set function. The nature of ensemble data assimilation makes the framework efficient and flexible to be integrated with various types of observation data. The transition probability-based geostatistical model keeps the updated hydrofacies distributions under geological constrains. The framework is illustrated by using a two-dimensional synthetic study that estimates hydrofacies spatial distribution and permeability in each hydrofacies from transient head data. Our results show that the proposed framework can characterize hydrofacies distribution and associated permeability with adequate accuracy even with limited direct measurements of hydrofacies. Our study provides a promising starting point for hydrofacies delineation in complex real problems.

Song, Xuehang↗
DOE OSTI2025

Navigating Uncertainty: Challenges in Visualizing Ensemble Data and Surrogate Models for Decision Systems

Uncertainty visualization plays a critical role in transforming ensemble simulation data into actionable insights by effectively communicating various dimensions of uncertainty within a system. The emergence of artificial intelligence-driven surrogate models trained on multirun ensemble data offers a transformative opportunity to replace computationally intensive simulations with fast estimates, enabling users to explore data spaces with unprecedented depth and interactivity. However, integrating ensemble data and surrogate models into decision-making workflows and tools introduces novel challenges for uncertainty visualization. These include reconciling and clearly communicating the unique uncertainties associated with ensembles and their surrogate model estimates, and leveraging these approximations to inform actionable decisions. This work explores these challenges in the context of high-dimensional data visualization, bridging discrete datasets with their continuous representations and addressing the complexities of systems that support iterative navigation between input and output spaces. We evaluate the role of uncertainty visualization in fostering intuitive, actionable interactions and identify critical hurdles in advancing this frontier of computational simulation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities

We present algorithmic improvements to the loading operations of certain reduced data ensembles produced from neutron scattering experiments at Oak Ridge National Laboratory (ORNL) facilities. Ensembles from multiple measurements are required to cover a wide range of the phase space of a sample material of interest. They are stored using the standard NeXus schema on individual HDF5 files. This makes it a scalability challenge, as the number of experiments stored increases in a single ensemble file. The present work follows up on our previous efforts on data management algorithms, to address identified input output (I/O) bottlenecks in Mantid, an open-source data analysis framework used across several neutron science facilities around the world. We reuse an in-memory binary-tree metadata index that resembles data access patterns, to provide a scalable search and extraction mechanism. In addition, several memory operations are refactored and optimized for the current common use cases, ranging most frequently from 10 to 180, and up to 360 separate measurement configurations. Results from this work show consistent speed ups in wall-clock time on the Mantid LoadMD routine, ranging from 19% to 23% on average, on ORNL production computing systems. The latter depends on the complexity of the targeted instrument-specific data and the system I/O and compute variability for the shared computational resources available to users of ORNL’s Spallation Neutron Source (SNS) and the High Flux Isotope Reactor (HFIR) instruments. Nevertheless, we continue to highlight the need for more research to address reduction challenges as experimental data volumes, user time and processing costs increase.

Godoy, William↗
NASA NTRS2013

Assessing the Impact of Pre-gpm Microwave Precipitation Observations in the Goddard WRF Ensemble Data Assimilation System

The forthcoming Global Precipitation Measurement (GPM) Mission will provide next generation precipitation observations from a constellation of satellites. Since precipitation by nature has large variability and low predictability at cloud-resolving scales, the impact of precipitation data on the skills of mesoscale numerical weather prediction (NWP) is largely affected by the characterization of background and observation errors and the representation of nonlinear cloud/precipitation physics in an NWP data assimilation system. We present a data impact study on the assimilation of precipitation-affected microwave (MW) radiances from a pre-GPM satellite constellation using the Goddard WRF Ensemble Data Assimilation System (Goddard WRF-EDAS). A series of assimilation experiments are carried out in a Weather Research Forecast (WRF) model domain of 9 km resolution in western Europe. Sensitivities to observation error specifications, background error covariance estimated from ensemble forecasts with different ensemble sizes, and MW channel selections are examined through single-observation assimilation experiments. An empirical bias correction for precipitation-affected MW radiances is developed based on the statistics of radiance innovations in rainy areas. The data impact is assessed by full data assimilation cycling experiments for a storm event that occurred in France in September 2010. Results show that the assimilation of MW precipitation observations from a satellite constellation mimicking GPM has a positive impact on the accumulated rain forecasts verified with surface radar rain estimates. The case-study on a convective storm also reveals that the accuracy of ensemble-based background error covariance is limited by sampling errors and model errors such as precipitation displacement and unresolved convective scale instability.

microwave observations↗
NASA NTRS2017

Assimilation of AERONET and MODIS AOT Observations Using Variational and Ensemble Data Assimilation Methods and Its Impact on Aerosol Forecasting Skill

Data assimilation of Aerosol Robotic Network (AERONET) and Moderate Resolution Imaging Spectroradiometer (MODIS) aerosol optical thickness (AOT) for aerosol forecasting was tested within the Navy Aerosol Analysis Prediction System (NAAPS) framework, using variational and ensemble data assimilation methods. Navy aerosol forecasting currently makes use of a deterministic NAAPS simulation coupled to Navy Variational Data Assimilation System for aerosol optical depth, a two-dimensional variational data assimilation system, for MODIS AOT assimilation. An ensemble version of NAAPS (ENAAPS) coupled to an ensemble adjustment Kalman filter (EAKF) from the Data Assimilation Research Testbed was recently developed, allowing for a range of data assimilation and forecasting experiments to be run with deterministic NAAPS and ENAAPS. The main findings are that the EAKF, with its flow-dependent error covariances, makes better use of sparse observations such as AERONET AOT. Assimilating individual AERONET observations in the two-dimensional variational system can increase the analysis errors when observations are located in high AOT gradient regions. By including AERONET with MODIS AOT assimilation, the magnitudes of peak aerosol events (AOT> 1) were better captured with improved temporal variability, especially in India and Asia where aerosol prediction is a challenge. Assimilating AERONET AOT with MODIS had little impact on the 24 h forecast skill compared to MODIS assimilation only, but differences were found downwind of AERONET sites. The 24 h forecast skill was approximately the same for forecasts initialized with analyses from AERONET AOT assimilation alone compared to MODIS assimilation, particularly in regions where the AERONET network is dense; including the United States and Europe, indicating that AERONET could serve as a backup observation network for over-land synoptic-scale aerosol events.

Rubin, Juli I.↗
DOE OSTI2022

An ensemble data assimilation modeling system for operational outdoor microalgae growth forecasting

Microalgae have received increasing attention as a potential feedstock for biofuel or biobased products. Forecasting the microalgae growth is beneficial for managers in planning pond operations and harvesting decisions. This study proposed a biomass forecasting system comprised of the Huesemann Algae Biomass Growth Model (BGM), the Modular Aquatic Simulation System in Two Dimensions (MASS2), ensemble data assimilation (DA), and numerical weather prediction Global Ensemble Forecast System (GEFS) ensemble meteorological forecasts. The novelty of this study is to seek the use of ensemble DA to improve both BGM and MASS2 model initial conditions with the assimilation of biomass and water temperature measurements and consequently improve short-term biomass forecasting skills. This study introduces the theory behind the proposed integrated biomass forecasting system, with an application undertaken in pseudo-real-time in three outdoor ponds cultured with Chlorella sorokiniana in Delhi, California, United States. Results from all three case studies demonstrate that the biomass forecasting system improved the short-term (i.e., 7-day) biomass forecasting skills by about 60% on average, comparing to forecasts without using the ensemble DA method. Given the satisfactory performances achieved in this study, it is probable that the integrated BGM-MASS2-DA forecasting system can be used operationally to inform managers in making pond operation and harvesting planning decisions.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2017

Delineating Facies Spatial Distribution by Integrating Ensemble Data Assimilation and Indicator Geostatistics with Level Set Transformation.

A new approach is developed to delineate the spatial distribution of discrete facies (geological units that have unique distributions of hydraulic, physical, and/or chemical properties) conditioned not only on direct data (measurements directly related to facies properties, e.g., grain size distribution obtained from borehole samples) but also on indirect data (observations indirectly related to facies distribution, e.g., hydraulic head and tracer concentration). Our method integrates for the first time ensemble data assimilation with traditional transition probability-based geostatistics. The concept of level set is introduced to build shape parameterization that allows transformation between discrete facies indicators and continuous random variables. The spatial structure of different facies is simulated by indicator models using conditioning points selected adaptively during the iterative process of data assimilation. To evaluate the new method, a two-dimensional semi-synthetic example is designed to estimate the spatial distribution and permeability of two distinct facies from transient head data induced by pumping tests. The example demonstrates that our new method adequately captures the spatial pattern of facies distribution by imposing spatial continuity through conditioning points. The new method also reproduces the overall response in hydraulic head field with better accuracy compared to data assimilation with no constraints on spatial continuity on facies.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

A global coupled ensemble data assimilation system using the Community Earth System Model and the Data Assimilation Research Testbed

In this paper, we present a description of the CESM/DART ensemble coupled data assimilation (DA) system based on the Community Earth System Model (CESM) and the Data Assimilation Research Testbed (DART) assimilation software. The CESM/DART should be viewed as a flexible system to support the DA needs of the CESM research community and not as a static reanalysis product. In this implementation of the CESM/DART, conventional insitu observations of the ocean and atmosphere are assimilated into the respective component models of the CESM using a 30-member ensemble adjustment Kalman filter (EAKF). CESM/DART is run in a “weakly coupled” configuration wherein observations native to each climate system component only directly impact the state vector for that component. Information is passed between components indirectly through the short-term coupled model forecasts that provide the EAKF background ensemble. This system leverages previous ensemble DA development for the Community Atmosphere Model and Parallel Ocean Program models using the DART EAKF. The CESM/DART project is a step towards providing increasingly useful DA capabilities for the CESM research community. Results are presented for our prototype 12-year reanalysis, run from 1970 to mid 1982. Multiple lines of evidence demonstrate that the system is capable of constraining the CESM coupled model to simulate the historical variability of the climate system in the well-observed Northern Hemisphere. A collection of monthly average variables, climate mode indices, observation diagnostics and snapshots of synoptic weather in the ocean and atmosphere are compared to established datasets, showing especially good agreement in the Northern Hemisphere. A discussion of the CESM/DART as a modular, community facility and the benefits and challenges associated with this vision is also included.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

A Pattern-Recognition-Based Ensemble Data Imputation Framework for Sensors from Building Energy Systems

Building operation data are important for monitoring, analysis, modeling, and control of building energy systems. However, missing data is one of the major data quality issues, making data imputation techniques become increasingly important. There are two key research gaps for missing sensor data imputation in buildings: the lack of customized and automated imputation methodology, and the difficulty of the validation of data imputation methods. In this paper, a framework is developed to address these two gaps. First, a validation data generation module is developed based on pattern recognition to create a validation dataset to quantify the performance of data imputation methods. Second, a pool of data imputation methods is tested under the validation dataset to find an optimal single imputation method for each sensor, which is termed as an ensemble method. The method can reflect the specific mechanism and randomness of missing data from each sensor. The effectiveness of the framework is demonstrated by 18 sensors from a real campus building. The overall accuracy of data imputation for those sensors improves by 18.2% on average compared with the best single data imputation method.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2020

A Multiyear Gridded Data Ensemble of Surface Biogenic Carbon Fluxes for North America: Evaluation and Analysis of Results

Accurate and fine-scale estimates of biogenic carbon fluxes are critical for measuring and monitoring the biosphere's responses and feedback to the climate system. Currently available data products from flux towers and model-intercomparison projects struggle to adequately represent spatiotemporal dynamics of surface biogenic carbon fluxes, and to quantify their uncertainties, which also are crucial to atmospheric inversion systems. To address these gaps, we introduce a new perturbed-parameter model ensemble with the CASA model to estimate surface biogenic carbon fluxes at monthly and 3-hourly scales for North America at ~500-m and 5-km resolutions. We first use the Extended Fourier Amplitude Sensitivity Testing to choose the three most sensitive parameters to be perturbed, maximum light use efficiency (E max ), optimal temperature of photosynthesis (T opt ), and temperature response of respiration (Q 10 ). The initial range for each parameter is broadly sampled for the L1 ensemble, but then we pruned E max with site-level primary productivity to derive an L2 ensemble with narrower uncertainty ranges. Ensembles are strongly correlated with site-level results at both monthly and 3-hourly scales, and the spread across L1/L2 ensemble members encompasses the range of AmeriFlux observations. Monthly variability in the L2 ensemble mean is 85% of the observed variability. The L2 ensemble outperforms diverse data products with the highest Taylor skill scores at diurnal to annual scales. Finally, the ensemble's seasonality agrees well with other models for most biome types and in high and middle latitudes, but inconsistencies are found in subtropical and tropical ecoregions and for annual totals over North America.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Using Ensemble Data Assimilation to Estimate Transient Hydrologic Exchange Flow Under Highly Dynamic Flow Conditions

Abstract Quantifying dynamic hydrologic exchange flows (HEFs) within river corridors that experience high‐frequency flow variations caused by dam regulations is important for understanding the biogeochemical processes at the river water and groundwater interfaces. Heat has been widely used as a tracer to infer steady‐state flow velocities through analytical solutions of heat transport defined by the diurnal temperature signals. Under sub‐daily dynamic flow conditions, however, such analytical solutions are not applicable due to the violation of their fundamental assumptions. In this study, we developed a data assimilation‐based approach to estimate the sub‐daily flux under highly dynamic flow conditions using multi‐depth temperature observations at a 5‐min resolution. If the hydraulic gradient is measured, Darcy's law was used to calculate the flux with permeability estimated from temperature responses below the riverbed. Otherwise, flux was estimated directly by assimilating multi‐depth temperature data at 1‐ or 2‐hr time intervals assuming one‐dimensional flow and heat transport governing equation. By comparing estimated fluxes with model‐generated synthetic truth, we demonstrated that both schemes have robust performance in estimating fluxes under highly dynamic flow conditions. This data assimilation‐based flux estimation method was able to capture the vertical sub‐daily fluxes using multi‐depth high‐resolution temperature data alone, even in the presence of multi‐dimensional flow. This approach has been successfully applied to real field temperature data collected at the Hanford site, which experiences highly dynamic HEFs. Our study shows the promise of adopting distributed 1‐D temperature monitoring to capture spatial and temporal exchange dynamics in river corridors at a watershed scale or beyond.

54 ENVIRONMENTAL SCIENCES↗