Search NASA⌕ Search

SEARCH · Search NASA

Results for “ensemble data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

Evaluating lossy data compression on climate simulation data within a large ensemble

High-resolution Earth system model simulations generate enormous data volumes, and retaining the data from these simulations often strains institutional storage resources. Further, these exceedingly large storage requirements negatively impact science objectives, for example, by forcing reductions in data output frequency, simulation length, or ensemble size. To lessen data volumes from the Community Earth System Model (CESM), we advocate the use of lossy data compression techniques. While lossy data compression does not exactly preserve the original data (as lossless compression does), lossy techniques have an advantage in terms of smaller storage requirements. To preserve the integrity of the scientific simulation data, the effects of lossy data compression on the original data should, at a minimum, not be statistically distinguishable from the natural variability of the climate system, and previous preliminary work with data from CESM has shown this goal to be attainable. However, to ultimately convince climate scientists that it is acceptable to use lossy data compression, we provide climate scientists with access to publicly available climate data that have undergone lossy data compression. In particular, we report on the results of a lossy data compression experiment with output from the CESM Large Ensemble (CESM-LE) Community Project, in which we challenge climate scientists to examine features of the data relevant to their interests, and attempt to identify which of the ensemble members have been compressed and reconstructed. We find that while detecting distinguishing features is certainly possible, the compression effects noticeable in these features are often unimportant or disappear in post-processing analyses. In addition, we perform several analyses that directly compare the original data to the reconstructed data to investigate the preservation, or lack thereof, of specific features critical to climate science. Overall, we conclude that applying lossy data compression to climate simulation data is both advantageous in terms of data reduction and generally acceptable in terms of effects on scientific results.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2014

Impact of I/O and Data Management in Ensemble Large Scale Climate Forecasting Using EC-Earth3

The EC-Earth climate model is a seamless Earth System Model (ESM) used to carry out climate research in 24 academic institutions and meteorological services from 11 countries in Europe. This model couples several components and it is continuously under development. In this work we present a study regarding the impact of the I/O and data management when using EC-Earth in well-known supercomputing environments. Most large-scale and long-term climate simulators have been developed bearing in mind the paramount importance of its scalability. However, the computational capabilities of the High Performance Computing (HPC) environments increase at so great speed that it is almost impossible to re-implement the whole models so that they are able to exploit efficiently the new features. Therefore, it is necessary to design different strategies to take advantage of them. In this work we present an operational framework to run ensemble simulations in HPC platforms. A set of experiments are presented in order to validate the suitability of this technique. Moreover, the derived impact regarding the I/O and data management aspects is analyzed.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Approximate CFTs and random tensor models

Abstract A key issue in both the field of quantum chaos and quantum gravity is an effective description of chaotic conformal field theories (CFTs), that is CFTs that have a quantum ergodic limit. We develop a framework incorporating the constraints of conformal symmetry and locality, allowing the definition of ensembles of ‘CFT data’. These ensembles take on the same role as the ensembles of random Hamiltonians in more conventional quantum ergodic phases of many-body quantum systems. To describe individual members of the ensembles, we introduce the notion of approximate CFT, defined as a collection of ‘CFT data’ satisfying the usual CFT constraints approximately, i.e. up to small deviations. We show that they generically exist by providing concrete examples. Ensembles of approximate CFTs are very natural in holography, as every member of the ensemble is indistinguishable from a true CFT for low-energy probes that only have access to information from semi-classical gravity. To specify these ensembles, we impose successively higher moments of the CFT constraints. Lastly, we propose a theory of pure gravity in AdS 3 as a random matrix/tensor model implementing approximate CFT constraints. This tensor model is the maximum ignorance ensemble compatible with conformal symmetry, crossing invariance, and a primary gap to the black-hole threshold. The resulting theory is a random matrix/tensor model governed by the Virasoro 6j-symbol.

Physics↗
DOE OSTI2018

Improved Reanalysis and Prediction of Atmospheric Fields Over the Southern Ocean Using Campaign‐Based Radiosonde Observations

Abstract This study investigated the impact of radiosonde observations from the Southern Ocean obtained by the Australian R/V Aurora Australis on the ALERA2 experimental ensemble reanalysis data set and ensemble forecast experiment. An observing system experiment (OSE) that included additional ship‐launched radiosonde data captured the atmospheric structure over the Southern Ocean. ALERA2 without additional radiosondes had positive temperature biases exceeding 7 °C in the upper troposphere when low‐pressure cyclonic systems passed over the ship. The spread in the upper level was reduced by 15% in the OSE, which propagated downstream from the ship's position because of the sparse observing network over southern high latitudes. Comparison of two 63‐member ensemble forecast experiments initialized by ALERA2 and the OSE revealed that prediction of midlatitude cyclone tracks was improved by the realistic representation of upper‐level troughs in the OSE forecast. This confirms that additional radiosondes over the Southern Ocean reduce uncertainty and error in midlatitude cyclone forecasts.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Real-time ensemble microalgae growth forecasting with data assimilation

Accurate short-range (e.g., 7-day) microalgae growth forecasts will be beneficial for both production and harvesting of microalgae. This study developed an operational microalgae growth forecasting system with ensemble data assimilation (DA). The forecasting system was validated against observed Monoraphidium minutum 26B-AM growth in two outdoor pond cultures located in Mesa, Arizona, U.S. We first examined the relative roles of uncertainty in the meteorological forecast and initial conditions (i.e., algal concentration at the time of forecast) in the microalgae 7-day forecast and found initial conditions dominated the microalgae forecasting skill, suggesting the importance of implementing DA to improve initial condition characterization. To correct the systematic bias in biomass simulations, we developed a particle filter with bias estimation (PFBE) DA method to estimate biases and correct the model forecast. We found the DA forecasting system could improve the 7-day microalgae forecasting skill by about 85% on average compared to model forecasts without DA. These results suggest the potential accuracy of biomass growth forecasts may be sufficient to inform real-time operational decisions, such as harvesting planning, for commercial-scale microalgae production.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Data Science Infrastructure SOFTWARE

LANL science workflows generate complex data sets and ensembles of data requiring significant compute and storage resources. The Data Science Infrastructure (DSI) project focuses on data-driven approaches to make data more readily available to LANL projects. DSI workflows leverage metadata stored in data-agnostic databases, supported by an abstraction layer API to simplify searching and accessing data across simulation runs, experimental runs, filesystems and environments. The abstraction layer API allows the user to query a range of data types: raw output, processed data, configuration data, machine learning models, performance data, etc. In addition to the abstraction backend API, the DSI project is developing client-driven query APIs and UIs to support specific user workflows.

Turton, Terece↗
DOE OSTI2026

Challenges and alternatives to empirical orthogonal functions for earth system data

Empirical orthogonal functions (EOFs) applied to gridded Earth system data enables users to diagnose modes of variability with relative ease. Yet, many challenges to interpretation exist such that they must be used with awareness and intention when applied to gridded climate data, especially with large ensembles. Utilizing data from two different Earth system modelling large ensemble frameworks, the Energy Exoscale Earth System Model and the Community Earth System Model, as well as reanalysis data, common EOF pitfalls are summarized and discussed. Challenges include erroneous mode swapping, sign flipping, and the temporal variability of the centers of action. For modes of variability with similar contribution to variance, mode swapping is not uncommon. Sign flipping can occur with almost any mode where the pattern is correct, but the sign is arbitrary. Although the variability of the center of action is not necessarily problematic, it potentially complicates interpretation over multi-century timescales. A wide variety of alternative methods to EOFs exist, but fitness-for-purpose must be evaluated. Additionally, illustrations of alternative methods and examples of proper use are provided. Alternative methods fit into three categories: EOF variants, linear methods, and multilinear methods.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Finite-temperature many-body perturbation theory in the canonical ensemble

Benchmark data are presented for the zeroth- through third-order many-body perturbation corrections to the electronic Helmholtz energy, internal energy, and entropy in the canonical ensemble in a wide range of temperature. They are determined as numerical λ-derivatives of the respective quantities computed by thermal full configuration interaction with a perturbation-scaled Hamiltonian, $\hat{H} = \hat{H}_0 + λ\hat{V}$. Sum-over-states analytical formulas for up to the third-order corrections to these properties are also derived as analytical λ-derivatives. These formulas, which are verified by exact numerical agreement with the benchmark data, are given in terms of the Hirschfelder–Certain degenerate perturbation energies and should be valid for both degenerate and nondegenerate reference states at any temperature down to zero. Further, the results in the canonical ensemble are compared with the same in the grand canonical ensemble.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2023

Estimating Ocean Observation Impacts on Coupled Atmosphere‐Ocean Models Using Ensemble Forecast Sensitivity to Observation (EFSO)

Ensemble Forecast Sensitivity to Observation (EFSO) is a technique that can efficiently identify the beneficial/detrimental impacts of every observation in ensemble-based data assimilation (DA). While EFSO has been successfully employed on atmospheric DA, it has never been applied to ocean or coupled DA due to the lack of a suitable error norm for oceanic variables. This study introduces a new density-based error norm incorporating sea temperature and salinity forecast errors, making EFSO applicable to ocean DA for the first time. We implemented the oceanic EFSO on the CFSv2-LETKF and investigated the impact of ocean observations under a weakly coupled DA framework. By removing the detrimental ocean observations detected by EFSO, the CFSv2 forecasts were significantly improved, showing the validation of impact estimation and the great potential of EFSO to be extended as a data selection criterion.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2013

Ensemble Feature Selection in Scientific Data Analysis

Feature selection has emerged as an important method of both examining and predicting results from scientific experiments. However, current feature selection methods cannot consistently provide meaningful descriptions of feature importance in the complex, high-dimensional, natural datasets encountered in scientific data analysis. Here, an ensemble feature selection method is demonstrated to collectively reduce the bias associated with individual feature selection algorithms and provide uncertainty estimates on feature importance ranks for both synthetic and natural datasets.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Using feature importance as an exploratory data analysis tool on Earth system models

Abstract. Machine learning (ML) models are commonly used to generate predictions, but these models can also support the discovery of new science. Generating accurate predictions necessitates that a model captures the structure of the underlying data. If the structure is properly extracted, ML could be a useful exploratory and evidential tool. In this paper, we present a case study that demonstrates the use of ML for exploratory data analysis (EDA) in the climate space. We apply the ML explainability method of spatiotemporal zeroed feature importance (stZFI) to understand how climate-variable associations evolve over space and time. Our analyses focus on data from ensembles of Earth system models (ESMs) which provide data on different climate states and conditions. We elect to work with ESM ensembles since they allow us to compare feature importance across alternative scenarios not available with observed data. The ensembles also account for natural variability so that we can distinguish between signal and noise due to natural climate variability when computing feature importance. The use of perturbed initial condition ensembles introduces variability mimicking the natural variability in the atmosphere; thus the signals emerging using feature importance (FI) can be evaluated against the natural variability in the climate system. For our analyses, we consider the 1991 volcanic eruption of Mount Pinatubo, which was a large stratospheric aerosol injection. We explore the climate pathway associated with the eruption from aerosols to radiation to temperature at both the near-surface and stratospheric levels. In addition to applying the method to data generated from two different ESMs, we apply stZFI to reanalysis data to compare the associations identified by stZFI. We show how stZFI tracks the importance of aerosol optical depth over time on forecasting temperatures. This case study illustrates usefulness of an ML tool (stZFI) for EDA on a well-studied climate exemplar.

Ries, Daniel (ORCID:0000000250294647)↗
DOE OSTI2016

An ensemble constrained variational analysis of atmospheric forcing data and its application to evaluate clouds in CAM5: Ensemble 3DCVA and Its Application

Large-scale atmospheric forcing data can greatly impact the simulations of atmospheric process models including Large Eddy Simulations (LES), Cloud Resolving Models (CRMs) and Single-Column Models (SCMs), and impact the development of physical parameterizations in global climate models. This study describes the development of an ensemble variationally constrained objective analysis of atmospheric large-scale forcing data and its application to evaluate the cloud biases in the Community Atmospheric Model (CAM5). Sensitivities of the variational objective analysis to background data, error covariance matrix and constraint variables are described and used to quantify the uncertainties in the large-scale forcing data. Application of the ensemble forcing in the CAM5 SCM during March 2000 intensive operational period (IOP) at the Southern Great Plains (SGP) of the Atmospheric Radiation Measurement (ARM) program shows systematic biases in the model simulations that cannot be explained by the uncertainty of large-scale forcing data, which points to the deficiencies of physical parameterizations. The SCM is shown to overestimate high clouds and underestimate low clouds. These biases are found to also exist in the global simulation of CAM5 when it is compared with satellite data.

58 GEOSCIENCES↗
DOE OSTI2022

Autodifferentiable Ensemble Kalman Filters

Data assimilation is concerned with sequentially estimating a temporally evolving state. This task, which arises in a wide range of scientific and engineering applications, is particularly challenging when the state is high-dimensional and the state-space dynamics are unknown. This paper introduces a machine learning framework for learning dynamical systems in data assimilation. Here, our auto-differentiable ensemble Kalman filters (AD-EnKFs) blend ensemble Kalman filters for state recovery with machine learning tools for learning the dynamics. In doing so, AD-EnKFs leverage the ability of ensemble Kalman filters to scale to high-dimensional states and the power of automatic differentiation to train high-dimensional surrogate models for the dynamics. Numerical results using the Lorenz-96 model show that AD-EnKFs outperform existing methods that use expectation-maximization or particle filters to merge data assimilation and machine learning. In addition, AD-EnKFs are easy to implement and require minimal tuning.

autodifferentiation↗
DOE OSTI2026

Ensemble Kalman filter for data assimilation coupled with low-resolution computations techniques applied in fluid dynamics

This paper presents an innovative Reduced-order model (ROM) for merging experimental and simulation data using data assimilation (DA) to estimate the "True" state of a fluid dynamics system, leading to more accurate predictions. Our methodology introduces a novel approach by implementing the ensemble Kalman filter (EnKF) within a reduced-dimensional framework, grounded in a robust theoretical foundation and applied to fluid dynamics. To address the substantial computational demands of DA, the proposed ROM employs low-resolution (LR) techniques to drastically reduce computational costs. This innovative approach involves downsampling datasets for DA computations, followed by an advanced reconstruction technique based on low-cost singular value decomposition (lcSVD). The lcSVD method, a key innovation in this paper, has never been applied to DA before and offers a highly efficient way to enhance resolution with minimal computational resources. Our results demonstrate significant reductions in both computation time and RAM usage through these LR techniques without compromising the accuracy of the estimations. For instance, in a turbulent test case, for a data compression rate of 15.9, the LR approach can achieve a speed-up of 13.7 and a RAM compression of 90.9% while maintaining a low relative root mean square error (RRMSE) of 2.6%, compared to 0.8% in the high-resolution (HR) reference. Furthermore, we highlight the effectiveness of the EnKF in estimating and predicting the state of fluid flow systems based on limited observations and given low-fidelity numerical data. This paper highlights the potential of the proposed DA method in fluid dynamics applications, particularly for improving computational efficiency in CFD and related fields. Its ability to balance accuracy with low computational and memory costs makes it especially suitable for large-scale and real-time applications, such as environmental monitoring or engineering design. This method will be incorporated into ModelFLOWs-app.

Data Assimilation↗
DOE OSTI2020

eFESTA: Ensemble Feature Exploration with Surface Density Estimates

We propose surface density estimate (SDE) to model the spatial distribution of surface features-isosurfaces, ridge surfaces, and streamsurfaces-in 3D ensemble simulation data. The inputs of SDE computation are surface features represented as polygon meshes, and no field datasets are required (e.g., scalar fields or vector fields). The SDE is defined as the kernel density estimate of the infinite set of points on the input surfaces and is approximated by accumulating the surface densities of triangular patches. We also propose an algorithm to guide the selection of a proper kernel bandwidth for SDE computation. An ensemble Feature Exploration method based on Surface densiTy EstimAtes (eFESTA) is then proposed to extract and visualize the major trends of ensemble surface features. For an ensemble of surface features, each surface is first transformed into a density field based on its contribution to the SDE, and the resulting density fields are organized into a hierarchical representation based on the pairwise distances between them. The hierarchical representation is then used to guide visual exploration of the density fields as well as the underlying surface features. We demonstrate the application of our method using isosurface in ensemble scalar fields, Lagrangian coherent structures in uncertain unsteady flows, and streamsurfaces in ensemble fluid flows.

Computational modeling↗