Search NASA⌕ Search

SEARCH · Search NASA

Results for “Model Uncertainty”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,153 records · Page 64

Random Predictor Models for Rigorous Uncertainty Quantification: Part 2

This and a companion paper propose techniques for constructing parametric mathematical models describing key features of the distribution of an output variable given input-output data. By contrast to standard models, which yield a single output value at each value of the input, Random Predictors Models (RPMs) yield a random variable at each value of the input. Optimization-based strategies for calculating RPMs having a polynomial dependency on the input and a linear dependency on the parameters are proposed. These formulations yield RPMs having various levels of fidelity in which the mean, the variance, and the range of the model's parameter, thus of the output, are prescribed. As such they encompass all RPMs conforming to these prescriptions. The RPMs are optimal in the sense that they yield the tightest predictions for which all (or, depending on the formulation, most) of the observations are less than a fixed number of standard deviations from the mean prediction. When the data satisfies mild stochastic assumptions, and the optimization problem(s) used to calculate the RPM is convex (or, when its solution coincides with the solution to an auxiliary convex problem), the model's reliability, which is the probability that a future observation would be within the predicted ranges, is bounded rigorously.

Crespo, Luis G.↗

Random Predictor Models for Rigorous Uncertainty Quantification: Part 1

This and a companion paper propose techniques for constructing parametric mathematical models describing key features of the distribution of an output variable given input-output data. By contrast to standard models, which yield a single output value at each value of the input, Random Predictors Models (RPMs) yield a random variable at each value of the input. Optimization-based strategies for calculating RPMs having a polynomial dependency on the input and a linear dependency on the parameters are proposed. These formulations yield RPMs having various levels of fidelity in which the mean and the variance of the model's parameters, thus of the predicted output, are prescribed. As such they encompass all RPMs conforming to these prescriptions. The RPMs are optimal in the sense that they yield the tightest predictions for which all (or, depending on the formulation, most) of the observations are less than a fixed number of standard deviations from the mean prediction. When the data satisfies mild stochastic assumptions, and the optimization problem(s) used to calculate the RPM is convex (or, when its solution coincides with the solution to an auxiliary convex problem), the model's reliability, which is the probability that a future observation would be within the predicted ranges, can be bounded tightly and rigorously.

Crespo, Luis G.↗

Simulating Atmospheric Processes in Earth System Models and Quantifying Uncertainties With Deep Learning Multi‐Member and Stochastic Parameterizations

Abstract Deep learning is a powerful tool to represent subgrid processes in climate models, but many application cases have so far used idealized settings and deterministic approaches. Here, we develop stochastic parameterizations with calibrated uncertainty quantification to learn subgrid convective and turbulent processes and surface radiative fluxes of a superparameterization embedded in an Earth System Model (ESM). We explore three methods to construct stochastic parameterizations: (a) a single Deep Neural Network (DNN) with Monte Carlo Dropout; (b) a multi‐member parameterization; and (c) a Variational Encoder Decoder with latent space perturbation. We show that the multi‐member parameterization improves the representation of convective processes, especially in the planetary boundary layer, compared to individual DNNs. The respective uncertainty quantification illustrates that methods (b) and (c) are advantageous compared to a dropout‐based DNN parameterization regarding the spread of convective processes. Hybrid simulations with our best‐performing multi‐member parameterizations remained challenging and crash within the first days. Therefore, we develop a pragmatic partial coupling strategy relying on the superparameterization for condensate emulation. Partial coupling reduces the computational efficiency of hybrid Earth‐like simulations but enables model stability over 5 months with our multi‐member parameterizations. However, our hybrid simulations exhibit biases in thermodynamic fields and differences in precipitation patterns. Despite this, the multi‐member parameterizations enable improvements in reproducing tropical extreme precipitation compared to a traditional convection parameterization. Despite these challenges, our results indicate the potential of a new generation of multi‐member machine learning parameterizations leveraging uncertainty quantification to improve the representation of stochasticity of subgrid effects.

Behrens, Gunnar [Deutsches Zentrum für Luft‐ und R↗

Kinetic model development and Bayesian uncertainty quantification for the complete reduction of Fe-based oxygen carriers with CH 4 , CO, and H 2 for chemical looping combustion

In this work, three kinetic models are developed and calibrated for the complete multi-step reduction of an Fe-based oxygen carrier (OC) particle with CH 4 , CO, and H 2 , using data from thermogravimetric analysis. The complete reduction rate profiles exhibit complex dynamics whose trajectory is significantly different depending on the reducing gas. A Bayesian model building and parameter estimation framework is applied for simultaneous parameter and model structure uncertainty quantification. The final models show excellent agreement between model predictions and calibration data, as well as new data not used for calibration (for the reduction of the OC with CH 4 ). Parameter uncertainty is quantified by determining joint posterior distribution, and model structure uncertainty is addressed by incorporating Gaussian process stochastic functions (represented by Bayesian smoothing splines) into the kinetic models. The final kinetic models with discrepancy functions are readily employable in equation-oriented simulation and optimization platforms.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

A New Perspective for the Calibration of Computational Predictor Models.

This paper presents a framework for calibrating computational models using data from sev- eral and possibly dissimilar validation experiments. The offset between model predictions and observations, which might be caused by measurement noise, model-form uncertainty, and numerical error, drives the process by which uncertainty in the models parameters is characterized. The resulting description of uncertainty along with the computational model constitute a predictor model. Two types of predictor models are studied: Interval Predictor Models (IPMs) and Random Predictor Models (RPMs). IPMs use sets to characterize uncer- tainty, whereas RPMs use random vectors. The propagation of a set through a model makes the response an interval valued function of the state, whereas the propagation of a random vector yields a random process. Optimization-based strategies for calculating both types of predictor models are proposed. Whereas the formulations used to calculate IPMs target solutions leading to the interval value function of minimal spread containing all observations, those for RPMs seek to maximize the models' ability to reproduce the distribution of obser- vations. Regarding RPMs, we choose a structure for the random vector (i.e., the assignment of probability to points in the parameter space) solely dependent on the prediction error. As such, the probabilistic description of uncertainty is not a subjective assignment of belief, nor is it expected to asymptotically converge to a fixed value, but instead it is a description of the model's ability to reproduce the experimental data. This framework enables evaluating the spread and distribution of the predicted response of target applications depending on the same parameters beyond the validation domain (i.e., roll-up and extrapolation).

Crespo, Luis Guillermo↗

CO 2 plume evolution in a depleted natural gas reservoir: Modeling of conformance uncertainty reduction over time

Uncertainty in the long-term fate of CO 2 injected for geologic carbon sequestration (GCS) is a significant barrier to the adoption of GCS as a greenhouse-gas emission-mitigation for industry and regulatory agencies alike. Here, we present a modeling study that demonstrates that the uncertainty in forecasts of GCS site performance decreases over time as monitoring data are used to update operational models. We consider a case study of GCS in a depleted natural gas reservoir, with CO 2 injection occurring over 20 years, with a 50-year post-injection site care period. We constructed a detailed model to generate the actual model output, which is considered synthetic observation data. A series of simpler operational models based on limited data and assumptions about how an operator would model such a site are then run and compared against actual model output at specific monitoring points after one year, two years, etc. The operational model is updated and improved using the synthetic observation data from the actual model at the same time intervals. Model parameter values and model features needed to be updated over time to improve matches to the actual model. These kinds of model adjustments would be a normal part of reservoir engineering and site management at GCS sites. Uncertainty in two key measures related to site performance decreases with time: extent of the CO 2 plume up-dip migration, and radial extent of the pressure pulse. This conclusion should help allay the concerns of industry and regulators about uncertainty in long-term fate of CO 2 at GCS sites.

54 ENVIRONMENTAL SCIENCES↗

Gap-filling eddy covariance methane fluxes: Comparison of machine learning model predictions and uncertainties at FLUXNET-CH4 wetlands

Time series of methane fluxes measured by eddy-covariance require gap-filling to estimate annual emissions. Gap-filling methane fluxes is challenging because of high variability and complex responses to multiple drivers. To date, there is no widely established gap-filling standard for methane, with regards both to the best model algorithms and predictors. In this study, we address the need for standardization by synthesizing results of gap-filling methods applied at 17 wetland sites spanning boreal to tropical regions including all major wetlands classes and two rice paddies. We introduce new procedures for: 1) creating realistic artificial gap scenarios, 2) training and evaluating gap-filling models without overstating performance, and 3) predicting half-hourly methane fluxes and annual emissions with robust uncertainty estimates. We tested a conventional method (marginal distribution sampling) and four machine learning algorithms - penalized linear regression, artificial neural networks, random forests, and boosted decision trees - and four predictor sets, including temporal, meteorological, ecosystem carbon and energy flux, and soil predictors. We find that the conventional method can achieve similar median performance to the machine learning models but is worse than the best machine learning models and relatively insensitive to predictor choices. Of the machine learning models, decision tree algorithms performed the best in cross-validation experiments, even with a baseline predictor set, and artificial neural networks showed comparable performance when using all predictors. Soil temperature was frequently the most important predictor whilst water table depth was important at sites with substantial water table fluctuations, highlighting the value of data on soil conditions. Raw gap-filling uncertainties from the machine learning models were underestimated and we propose a method to calibrate uncertainties to observations. Finally, we gap-fill and provide summary evaluation metrics for all 81 sites in the FLUXNET-CH4 community dataset and publicly release the python code for model development, evaluation, and uncertainty estimation.

42 ENGINEERING↗

Carbon Dioxide Plume Evolution Following Injection into a Depleted Natural Gas Reservoir: Modeling of Conformance Uncertainty Reduction Over Time

The uncertainty in the long-term fate of CO 2 injected for geologic carbon sequestration (GCS) is a significant barrier to the adoption of GCS as a greenhouse gas emission mitigation approach for industry and regulatory agencies alike. Here we present a modeling study that demonstrates that the uncertainty in forecasts of GCS site performance decreases over time as monitoring data are used to inform and update operational models. The approach we take is to consider a case study consisting of a depleted natural gas reservoir that is used for GCS with CO 2 injection occurring over 20 years, with a 50-year post-injection site care (PISC) period. We constructed a detailed model of the system and ran this model out to 200 years to generate the actual site data. A series of simpler operational models based on limited data and assumptions about how an actual operator would model such a site are then run and compared against the actual model output at various specific monitoring points after one year, two years, etc. The operational model is then updated and improved using the observations (synthetic data from the actual model) at the same time intervals. We found that both model parameter values and model features needed to be added over time to improve matches to the actual system. These kinds of model adjustments are expected to be a normal part of reservoir engineering and site management at GCS sites. We found that the uncertainty in two key measures related to site performance at various locations decreases with time. This overall conclusion should help allay the concerns of industry and regulators about the uncertainty in GCS operations.

03 NATURAL GAS↗

Sparing Healthy Tissue and Increasing Tumor Dose Using Bayesian Modeling of Geometric Uncertainties for Planning Target Volume Personalization

Objective: To develop a mathematical tool that can update a patient's planning target volume (PTV) partway through a course of radiation therapy to more precisely target the tumor for the remainder of treatment and reduce dose to surrounding healthy tissue. Methods and Materials: Daily on-board imaging was used to collect large datasets of displacements for patients undergoing external beam radiation therapy for solid tumors. Bayesian statistical modeling of these geometric uncertainties was used to optimally trade off between displacement data collected from previously treated patients and the progressively accumulating data from a patient currently partway through treatment, to optimally predict future displacements for that patient. These predictions were used to update the PTV position and margin width for the remainder of treatment, such that the clinical target volume (CTV) was more precisely targeted. Results: Software simulation of dose to CTV and normal tissue for 2 real prostate displacement datasets consisting of 146 and 290 patients treated with a minimum of 30 fractions each showed that re-evaluating the PTV position and margin width after 8 treatment fractions reduced healthy tissue dose by 19% and 17%, respectively, while maintaining CTV dose. Conclusion: Incorporating patient-specific displacement patterns from early in a course of treatment allows PTV adaptation for the remainder of treatment. This substantially reduces the dose to healthy tissues and thus can reduce radiation therapy–induced toxicities, improving patient outcomes.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

What Limits Predictive Certainty of Long-Term Carbon Uptake?

Terrestrial biosphere models can help identify physical processes that control carbon dynamics, including land-atmosphere CO 2 fluxes, and have the potential to project the terrestrial ecosystem response to changing climate. It is important to identify ecosystem processes most responsible for model predictive uncertainty and design improved model representation and observational system studies to reduce that uncertainty. Here we identified model parameters that contribute the most uncertainty to long-term (~100 years) projections of net ecosystem exchange, net primary production, and aboveground biomass within a mechanistic terrestrial biosphere model (Ecosystem Demography, version 2.1) ED2. An uncertainty analysis identified parameters that represent the quantum efficiency of light to photosynthetic conversion, leaf respiration and soil-plant water transfer as the highest contributors to model uncertainty regardless of time frame (annual, decadal, and centennial) and output (e.g., net ecosystem exchange, net primary production, aboveground biomass). Contrary to expectations, the contribution of successional processes related to reproduction, competition, and mortality did not increase as the time scale increased. These findings suggest that uncertainty in the parameters governing short-term ecosystem processes remains the most significant bottleneck to reducing predictive uncertainty. Key actions to reduce parameter uncertainty include more leaf-level trait measurements across multiple sites for quantum efficiency and leaf respiration rate. Further, the empirical representation of soil-plant water transfer should be replaced with a mechanistic, hydraulic representation of water flow, which can be constrained with direct measurements. This analysis focused on aboveground ecosystem processes. The impact of belowground carbon cycling, initial conditions, and meteorological forcing should be addressed in future studies.

54 ENVIRONMENTAL SCIENCES↗

Identifying the sources of uncertainty in climate model simulations of solar radiation modification with the G6sulfur and G6solar Geoengineering Model Intercomparison Project (GeoMIP) simulations

We present here results from the Geoengineering Model Intercomparison Project (GeoMIP) simulations for the experiments G6sulfur and G6solar for six Earth system models participating in the Climate Model Intercomparison Project (CMIP) Phase 6. The aim of the experiments is to reduce the warming that results from a high-tier emission scenario (Shared Socioeconomic Pathways SSP5-8.5) to that resulting from a medium-tier emission scenario (SSP2-4.5). These simulations aim to analyze the response of climate models to a reduction in incoming surface radiation as a means to reduce global surface temperatures, and they do so either by simulating a stratospheric sulfate aerosol layer or, in a more idealized way, through a uniform reduction in the solar constant in the model. We find that over the final two decades of this century there are considerable inter-model spreads in the needed injection amounts of sulfate (29 ± 9 Tg-SO 2 /yr between 2081 and 2100), in the latitudinal distribution of the aerosol cloud and in the stratospheric temperature changes resulting from the added aerosol layer. Even in the simpler G6solar experiment, there is a spread in the needed solar dimming to achieve the same global temperature target (1.91 ± 0.44 %). The analyzed models already show significant differences in the response to the increasing CO 2 concentrations for global mean temperatures and global mean precipitation (2.05 K ± 0.42 K and 2.28 ± 0.80 %, respectively, for SSP5-8.5 minus SSP2-4.5 averaged over 2081–2100). With aerosol injection, the differences in how the aerosols spread further change some of the underlying uncertainties, such as the global mean precipitation response (–3.79 ± 0.76 % for G6sulfur compared to –2.07 ± 0.40 % for G6solar against SSP2-4.5 between 2081 and 2100). These differences in the behavior of the aerosols also result in a larger uncertainty in the regional surface temperature response among models in the case of the G6sulfur simulations, suggesting the need to devise various, more specific experiments to single out and resolve particular sources of uncertainty. The spread in the modeled response suggests that a degree of caution is necessary when using these results for assessing specific impacts of geoengineering in various aspects of the Earth system. However, all models agree that compared to a scenario with unmitigated warming, stratospheric aerosol geoengineering has the potential to both globally and locally reduce the increase in surface temperatures.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

CFD Validation Study of a Hypersonic Cone-Slice-Flap Variable Geometry Configuration

Model validation is the process of determining the degree of accuracy between physical reality and the model. The result of model validation can either be used to improve the model through calibration or quantify the model-form uncertainty. This work focuses on providing the model-form uncertainty through an area metric for a hypersonic cone-slice-flap variable geometry configuration given uncertainty in both the simulation and experimental data. For a variable geometry, performing uncertainty quantification to capture the model-form uncertainty on every configuration is computationally challenging. This work lays out a procedure that can give an accurate representation of the model-form uncertainty using a small number of high-fidelity runs and many low-fidelity runs on multiple configurations. Running this comparison provides a quantifiable measurement for the accuracy of each turbulence model for this type of design. The high-fidelity CFD solver used was VULCAN-CFD and the low-fidelity results came from Cart3D. The experimental data came from the 20-Inch Mach 6 Tunnel located at NASA Langley Research Center. The present work showed that the turbulence simulation overpredicted the drag and lift coefficient, while underpredicting the pitching moment coefficient. The model-form uncertainty estimate resulted up to a 13.6% change in the total uncertainty for the drag coefficient, up to a 57.4% change in total uncertainty for the lift coefficient, and up to a 100% change in total uncertainty for the pitching moment coefficient.

Laura White↗

Interactive Photochemistry in Earth System Models to Assess Uncertainty in Ozone and Greenhouse Gases. Final report

Atmospheric chemistry controls the abundances and hence climate forcing of important greenhouse gases including N 2 O, CH 4 , HFCs, CFCs, and O 3 . Attributing climate change to human activities requires, at a minimum, accurate models of the chemistry and circulation of the atmosphere that relate emissions to abundances. This DOE-funded research provided realistic, yet computationally optimized and affordable, photochemical modules to the Community Earth System Model (CESM) that augment the CESM capability to explore the uncertainty in future stratospheric-tropospheric ozone, stratospheric circulation, and thus the lifetimes of chemically controlled greenhouse gases from climate simulations. To this end, we have successfully implemented Fast-J (radiation algorithm determining key chemical photolysis rates) and Linoz v3.0 (linearized photochemistry for interactive O 3 , N 2 O, NO y and CH 4 ) packages in LLNL-CESM and for the first time demonstrated how change in O2 photolysis rate within its uncertainty range can significantly impact on the stratospheric climate and ozone abundances. From the UCI side, this proposal also helped LLNL develop a CAM-Superfast Chemistry model that was implemented for the IPCC AR5 and contributed chemical-climate simulations to CMIP5.

54 ENVIRONMENTAL SCIENCES↗

Strategies for Automation of Model Tuning in Multifidelity Trajectory Uncertainty Propagation

Multi-model Monte Carlo methods are efficient strategies to perform forward uncertainty quantification studies in entry, descent, and landing (EDL) applications. These multi-model methods are based on the classical Monte Carlo estimator, but fuse predictions from several low-fidelity models to obtain estimators with greater precision given a prescribed computational budget. The effectiveness of these approaches relies on the magnitudes of correlations between the low-fidelity models and the high-fidelity model, as well as the relative computational costs of all models. Identifying and exploiting the best trade-off between correlation and cost, which ultimately depends on the selection of hyperparameters in the low-fidelity models, is a task often performed by hand or simply inspired by the deterministic understanding available for a specific application. This work extends a preliminary effort,

Marten Thompson↗