Search NASA⌕ Search

SEARCH · Search NASA

Results for “Model Uncertainty”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,315 records · Page 73

Uncertainty propagation in feed-forward neural network models

We develop new uncertainty propagation methods for feed-forward neural network architectures with leaky ReLU activation functions subject to random perturbations in the input vectors. In particular, we derive analytical expressions for the probability density function (PDF) of the neural network output and its statistical moments as a function of the input uncertainty and the parameters of the network, i.e., weights and biases. A key finding is that an appropriate linearization of the leaky ReLU activation function yields accurate statistical results even for large perturbations in the input vectors. This can be attributed to the way information propagates through the network. We also propose new analytically tractable Gaussian copula surrogate models to approximate the full joint PDF of the neural network output. To validate our theoretical results, we conduct Monte Carlo simulations and a thorough error analysis on a multi-layer neural network representing a nonlinear integro-differential operator between two polynomial function spaces. Our findings demonstrate excellent agreement between the theoretical predictions and Monte Carlo simulations.

MLP networks↗

Estimation of Uncertainties of Wall Temperature Accomplished by Model Calibration Technique Using Small to Large Scale Reflooding Test Data

The fidelity of the prediction can be evaluated by performing validation and uncertainty quantification with suitable test data. Nuclear power plant design and operation must accommodate the uncertainties in predicting system performance, where those uncertainties originating due to initial and boundary conditions, numerics, and modeling uncertainties. To improve the prediction of the reactor system performance, these various sources of uncertainties can be refined through model calibration technique. The objective of this work is to refine the model parameters and to estimate the uncertainties on parameters/responses during reflood phase. To accomplish this experiments should be performed, data collected, and the data utilized to complete model calibration for the thermal-hydraulic input parameters. Simulations were performed using Safety and Performance Analysis Code (SPACE) developed by multiple research institutes in South Korea to predict the thermal hydraulic system behaviors of nuclear power plants. Since existing experimental data were obtained from several tests performed at the small scale, scaling considerations must be addressed during the uncertainty quantification process. (authors)

42 ENGINEERING↗

Boron Coordination in Multicomponent Glasses: Analytical Models and Machine Learning With Uncertainty

Borosilicate glasses are extensively used in a variety of applications from kitchenware to nuclear waste immobilization due to the strong network formed by the Si-O-B bond that makes it resistant to chemical corrosion and gives it a low thermal expansion. Boron, however, exists in both trigonal BO3 and tetrahedral BO4 bonds in glass systems, which impacts the chemical durability and thermal resistance of the glass, amongst other properties. Boron coordination (N4), or the ratio of the amount of BO4 to BO3 within a glass, may aid in predicting these properties but is difficult to derive without experimental data due to the complexity of impacts from varied glass compositions and processing factors. For this reason, compositional models have been developed to predict boron coordination, but the models typically include a limited number of glass components. To help fill this gap in the models, in this work, a diverse multicomponent glass dataset of 809 glasses is compiled from a literature search, and then a number of analytical and machine learning (ML) models are trained on the dataset. Previously developed modified Bernstein and modified Du Stebbins analytical models were fitted to update parameters with the new dataset. Then, partially Bayesian neural networks, Gaussian process regressor, and heteroskedastic deterministic neural networks were evaluated. The ML models examined all have different strategies to overcome the potential for overfitting as a result of a limited training dataset, and return results that account for model uncertainty, which can be valuable for understanding model reliability. For the first time, cooling rate is introduced as an input parameter for ML models, showing consistent improvements in performance and solidifying the importance of including parameters outside of composition alone for N4 prediction. The machine learning models examined here show promise in accurate predictions of boron coordination in borosilicate glasses, all achieving R2 values of 0.91.

boron coordination↗

Long-term uncertainty quantification in WRF-modeled offshore wind resource off the US Atlantic coast

Uncertainty quantification of long-term modeled wind speed is essential to ensure stakeholders can best leverage wind resource numerical data sets. Offshore, this need is even stronger given the limited availability of observations of wind speed at heights relevant for wind energy purposes and the resulting heavier relative weight of numerical data sets for wind energy planning and operational projects. In this analysis, we consider the National Renewable Energy Laboratory's 21-year updated numerical offshore data set for the US East Coast and provide a methodological framework to leverage both floating lidar and near-surface buoy observations in the region to quantify uncertainty in the modeled hub-height wind resource. We first show how using a numerical ensemble to quantify the uncertainty in modeled wind speed is insufficient to fully capture the model deviation from real-world observations. Next, we train and validate a random forest to vertically extrapolate near-surface wind speed to hub height using the available short-term lidar data sets in the region. We then apply this model to vertically extrapolate the long-term near-surface buoy wind speed observations to hub height so that they can be directly compared to the long-term numerical data set. We find that the mean 21-year uncertainty in 140 m hourly average wind speed is slightly lower than 3 m s -1 (roughly 30m% of the mean observed wind speed) across the considered region. Atmospheric stability is strictly connected to the modeled wind speed uncertainty, with stable conditions associated with an uncertainty which is, on average, about 20 % larger than the overall mean uncertainty.

17 WIND ENERGY↗

Uncertainty quantification for Multiphase-CFD simulations of bubbly flows: a machine learning-based Bayesian approach supported by high-resolution experiments

In this paper, we developed a machine learning-based Bayesian approach to inversely quantify and reduce the uncertainties of multiphase computational fluid dynamics (MCFD) simulations for bubbly flows. The proposed approach is supported by high-resolution two-phase flow measurements, including those by double-sensor conductivity probes, high-speed imaging, and particle image velocimetry. Local distributions of key physical quantities of interest (QoIs), including the void fraction and phasic velocities, are obtained to support the Bayesian inference. In the process, the epistemic uncertainties of the closure relations are inversely quantified while the aleatory uncertainties from stochastic fluctuations of the system are evaluated based on experimental uncertainty analysis. The combined uncertainties are then propagated through the MCFD solver to obtain uncertainties of the QoIs, based on which probability-boxes are constructed for validation. The proposed approach relies on three machine learning methods: feedforward neural networks and principal component analysis for surrogate modeling, and Gaussian processes for model form uncertainty modeling. The whole process is implemented within the framework of an open-source deep learning library PyTorch with graphics processing unit (GPU) acceleration, thus ensuring the efficiency of the computation. The results demonstrate that with the support of high-resolution data, the uncertainties of MCFD simulations can be significantly reduced. The proposed approach has the potential for other applications that involve numerical models with empirical parameters.

42 ENGINEERING↗

Statistically Rigorous Uncertainty Quantification for Physical Parameter Model Calibration with Functional Output

In experiments conducted on the Z-machine at Sandia National Laboratories, dynamic material properties cannot be analyzed using traditional analytic methods, necessitating solving an inverse problem. Bayesian model calibration is a statistical framework for solving an inverse problem to estimate parameters input into a computational model in the presence of multiple uncertainties. Disentangling input parameter uncertainty and model misspecification is often poorly identified problem. When using computational models for physical parameter estimation, the issue of parameter identifiability must be carefully considered to obtain accurate and precise estimates of physical parameters. Additionally, in dynamic material properties applications, the experimental output is a function, velocity over time. While we can sample an arbitrarily large number of points from the measured velocity, these curves only contain a finite amount of information about the calibration parameters. In this report, we propose modifications to the Bayesian model calibration framework to simplify and improve the estimation of physical parameters with functional outputs. Specifically, we propose scaling the likelihood function by an effective sample size rather than modeling the discrepancy function; and modularizing input nuisance parameters with weakly identified parameters. We evaluate the performance of these proposed methods using a statistical simulation study and then apply these methods to estimate parameters of the tantalum equation of state. We conclude that these proposed methods can provide simple, fast, and statistically valid alternatives to the full Bayesian model calibration procedure; and that these methods can be used to estimate parameters of the equation of state for tantalum.

97 MATHEMATICS AND COMPUTING↗

A Single-column Model Ensemble Approach Applied to the TWP-ICE Experiment

Single-column models (SCM) are useful test beds for investigating the parameterization schemes of numerical weather prediction and climate models. The usefulness of SCM simulations are limited, however, by the accuracy of the best estimate large-scale observations prescribed. Errors estimating the observations will result in uncertainty in modeled simulations. One method to address the modeled uncertainty is to simulate an ensemble where the ensemble members span observational uncertainty. This study first derives an ensemble of large-scale data for the Tropical Warm Pool International Cloud Experiment (TWP-ICE) based on an estimate of a possible source of error in the best estimate product. These data are then used to carry out simulations with 11 SCM and two cloud-resolving models (CRM). Best estimate simulations are also performed. All models show that moisture-related variables are close to observations and there are limited differences between the best estimate and ensemble mean values. The models, however, show different sensitivities to changes in the forcing particularly when weakly forced. The ensemble simulations highlight important differences in the surface evaporation term of the moisture budget between the SCM and CRM. Differences are also apparent between the models in the ensemble mean vertical structure of cloud variables, while for each model, cloud properties are relatively insensitive to forcing. The ensemble is further used to investigate cloud variables and precipitation and identifies differences between CRM and SCM particularly for relationships involving ice. This study highlights the additional analysis that can be performed using ensemble simulations and hence enables a more complete model investigation compared to using the more traditional single best estimate simulation only.

numerical weather forecasting↗

Cutting out the middleman: calibrating and validating a dynamic vegetation model (ED2-PROSPECT5) using remotely sensed surface reflectance

Canopy radiative transfer is the primary mechanism by which models relate vegetation composition and state to the surface energy balance, which is important to light- and temperature-sensitive plant processes as well as understanding land–atmosphere feedbacks. In addition, certain parameters (e.g., specific leaf area, SLA) that have an outsized influence on vegetation model behavior can be constrained by observations of shortwave reflectance, thus reducing model predictive uncertainty. Importantly, calibrating against radiative transfer outputs allows models to directly use remote sensing reflectance products without relying on highly derived products (such as MODIS leaf area index) whose assumptions may be incompatible with the target vegetation model and whose uncertainties are usually not well quantified. Here, we created the EDR model by coupling the two-stream representation of canopy radiative transfer in the Ecosystem Demography model version 2 (ED2) with a leaf radiative transfer model (PROSPECT-5) and a simple soil reflectance model to predict full-range, high-spectral-resolution surface reflectance that is dependent on the underlying ED2 model state. We then calibrated this model against estimates of hemispherical reflectance (corrected for directional effects) from the NASA Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) and survey data from 54 temperate forest plots in the northeastern United States. The calibration significantly reduced uncertainty in model parameters related to leaf biochemistry and morphology and canopy structure for five plant functional types.

54 ENVIRONMENTAL SCIENCES↗

Cutting out the Middleman: Calibrating and Validating a Dynamic Vegetation Model (ED2-PROSPECT5) Using Remotely Sensed Surface Reflectance

Canopy radiative transfer is the primary mechanism by which models relate vegetation composition and state to the surface energy balance, which is important to light- and temperature-sensitive plant processes as well as understanding land–atmosphere feedbacks. In addition, certain parameters (e.g., specific leaf area, SLA) that have an outsized influence on vegetation model behavior can be constrained by observations of shortwave reflectance, thus reducing model predictive uncertainty. Importantly, calibrating against radiative transfer outputs allows models to directly use remote sensing reflectance products without relying on highly derived products (such as MODIS leaf area index) whose assumptions may be incompatible with the target vegetation model and whose uncertainties are usually not well quantified. Here, we created the EDR model by coupling the two-stream representation of canopy radiative transfer in the Ecosystem Demography model version 2 (ED2) with a leaf radiative transfer model (PROSPECT-5) and a simple soil reflectance model to predict full-range, high-spectral-resolution surface reflectance that is dependent on the underlying ED2 model state. We then calibrated this model against estimates of hemispherical reflectance (corrected for directional effects) from the NASA Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) and survey data from 54 temperate forest plots in the northeastern United States. The calibration significantly reduced uncertainty in model parameters related to leaf biochemistry and morphology and canopy structure for five plant functional types. Using a single common set of parameters across all sites, the calibrated model was able to accurately reproduce surface reflectance for sites with highly varied forest composition and structure. However, the calibrated model's predictions of leaf area index (LAI) were less robust, capturing only 46 % of the variability in the observations. Comparing the ED2 radiative transfer model with another two-stream soil–leaf–canopy radiative transfer model commonly used in remote sensing studies (PRO4SAIL) illustrated structural errors in the ED2 representation of direct radiation backscatter that resulted in systematic underestimation of reflectance. In addition, we also highlight that, to directly compare with a two-stream radiative transfer model like EDR, we had to perform an additional processing step to convert the directional reflectance estimates of AVIRIS to hemispherical reflectance (also known as “albedo”). In future work, we recommend that vegetation models add the capability to predict directional reflectance, to allow them to more directly assimilate a wide range of airborne and satellite reflectance products. We ultimately conclude that despite these challenges, using dynamic vegetation models to predict surface reflectance is a promising avenue for model calibration and validation using remote sensing data.

Alexey N Shiklomanov↗

TRISO fuel performance analysis: Uncertainty quantification toward optimization

Tri-structural isotropic (TRISO) fuel particles are a fuel form being considered for potential use in next-generation nuclear reactors (i.e., high-temperature gas-cooled reactors). Though the TRISO fuel manufacturing process has continually advanced in recent years, particle comparisons still reveal statistical variations and uncertainties in terms of geometric configurations and material properties. Given that the physical processes ongoing in TRISO fuel particles during reactor operation are highly correlated with each other, a small degree of uncertainty in one model may lead to significant uncertainty in another. This makes appropriate uncertainty quantification of TRISO fuel particles essential. However, one may wonder about the extent to which the current version of TRISO particles has been optimized, and whether any room remains for further improvements. This paper quantifies TRISO fuel performance model uncertainties that stem from geometric and material data. For this analysis, the BISON code was used, and the Advanced Gas Reactor (AGR)-2 experiment served as a reference case. A total of 10 5 calculations was performed for the uncertainty and optimization analysis, altering the geometric and material data within their uncertainty range. Lastly, the optimization potential of TRISO particles is evaluated from a fuel performance perspective.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Model Structure and Climate Data Uncertainty in Historical Simulations of the Terrestrial Carbon Cycle (1850–2014)

Abstract The divergence among Earth system models in the terrestrial carbon cycle has prompted interest in how to reduce uncertainty. Previous studies have identified model structural uncertainty arising from process parameterizations and parameter values. The current study highlights the importance of climate forcing in generating carbon cycle uncertainty. We use simulations in which three models (Community Land Model version 4 (CLM4), CLM4.5, CLM5) with substantially different carbon cycles are forced with two climate reconstructions (CRUNCEPv7, Global Soil Wetness Project 3 version 1 (GSWP3v1)) to examine the contributions of model structure and climate to uncertainty in the carbon cycle over the period 1850–2014. Climate uncertainty for global annual net biome production exceeds one third of total uncertainty (defined as the sum of climate and model structure uncertainty) in the first half of the twentieth century, but declines after the 1950s. Global annual gross primary productivity, net primary productivity, heterotrophic respiration, and vegetation and soil carbon stocks have substantial climate uncertainty (relative to total uncertainty) throughout the simulation period. Climate forcing contributes more than one half of total uncertainty for these carbon cycle fluxes and stocks throughout boreal North America and Eurasia, some midlatitude regions, and in eastern Amazonia and western equatorial Africa during the decade 2000–2009. Comparison with observationally based data sets of the carbon cycle using model benchmarking methods provides insight into strengths and deficiencies among models and climate forcings, but we caution against overreliance on benchmarking to discriminate among models. The conceptualization of uncertainty arising from this study implies embracing multiple feasible model simulations rather than focusing on which model or simulation is best.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Ensemble Cancer Risk Model for Astronaut Risk Assessment

A new approach to NASA space radiation risk modeling has successfully extended the current NASA probabilistic cancer risk model to an ensemble framework able to consider sub-model parameter uncertainty (e.g. uncertainty in a radiation quality parameter) as well as model-form uncertainty associated with differing theoretical or empirical formalisms (e.g. combined dose-rate and radiation quality effects). Ensemble methodologies are already widely used in weather prediction, modeling of infectious disease outbreaks, and certain terrestrial radiation protection applications to better understand how uncertainty may influence risk decision-making. Applying ensemble methodologies to space radiation risk projections offers the potential to efficiently incorporate emerging research results, allow for the incorporation of future (including international) models, improve uncertainty quantification for underlying sub-models developed against sparse experimental data, and reduce the impact of subjective bias on risk projections. Moreover, risk forecasting across an ensemble of multiple predictive models can provide stakeholders additional information on risk acceptance if current health/medical standards cannot be met or the level of knowledge doesn’t permit a specific risk or exposure limit to be developed for future space exploration missions. In this work, ensemble risk projections implementing multiple sub-models of radiation quality, dose and dose-rate effectiveness factors, excess risk, and latency as ensemble members are presented. Initial consensus methods for ensemble model weights and correlations to account for individual model bias are discussed. In these analyses, the ensemble forecast compares well to results from NASA's current operational cancer risk projection model used to assess permissible exposure limits and permissible mission durations for astronauts. However, a large range of projected risk values are obtained at the upper 95th confidence level where models must extrapolate beyond available biological data sets; closer agreement is seen at the median + one sigma due to the inherent similarities in available models. Future work, including the addition of new models and methods for statistical correlation between predictive members are discussed to define alternate ways of thinking about risk and ‘acceptable’ uncertainty with respect to NASA’s current permissible exposure limits.

Lisa C Simonsen↗

“Scrutinizing entrainment and mass flux closures in shallow cumulus parameterizations using cloud-radar observations and large-eddy simulation” (Final report for DOE grant)

Warm, shallow clouds are responsible for a substantial portion of Earth System Model (ESM) uncertainty and divergence among model climate projections. A substantial amount of this uncertainty is tied to the representation of convection in ESMs, in particular assumptions underlying the convective parameterizations. This study focuses on entrainment and entrainment closures in ESM parameterizations. The completed project was a joint observational and modeling study to observationally derive estimates of entrainment and to use high-resolution modeling to scrutinize entrainment and mass flux closures in shallow convection parameterizations.

58 GEOSCIENCES↗

Development of a framework for sequential Bayesian design of experiments: Application to a pilot-scale solvent-based CO 2 capture process

In this paper, a methodology is developed for sequential design of experiments (SDoE) for process systems and applied to a solvent-based CO 2 capture system. In this approach, the prior knowledge of the system is used to prioritize process data collection at specific operating conditions. These data are then incorporated into a Bayesian inference methodology for updating a stochastic model by refining estimations of its underlying parameters, and the updated model is then used to generate the next set of test runs. Thus, the new knowledge obtained from the data is used to guide subsequent iterations of the experimental runs, ensuring that the overall data collection is maximally informative given that most experimental campaigns, especially at pilot or higher-scale plants, are costly, time-consuming, and resource-limited. The test run objective for this work was to minimize the maximum model prediction uncertainty for key output variables, but the methodology is generic and can be readily applied to other test run objectives. This methodology is applied to an aqueous monoethanolamine (MEA) pilot plant campaign at the National Carbon Capture Center (NCCC) in Wilsonville, Alabama, USA. The SDoE framework was utilized for two iterations, while collecting 18 sets of data representing different process conditions, and this resulted in an overall average reduction in uncertainty of approximately 50% in the prediction of CO 2 capture percentage. Moreover, 11 additional data sets were obtained with variation of absorber packing height for further model validation. This work shows the capability of the SDoE framework to maximize learning given limited resources, allowing for the reduction of model uncertainty, which is of great importance for many applications including reduction of technical risk associated with scale-up and economic analysis.

20 FOSSIL-FUELED POWER PLANTS↗

Modular hybrid modeling to increase efficiency, explore structural uncertainty, and allow multidimensional complexity scaling in land surface models

Land surface models (LSMs) are indispensable tools for predicting hydrologic extremes, as well as a particularly uncertain component of Earth system models that has stubbornly resisted the convergence in projections over several successive generations of model intercomparisons. This uncertainty in LSMs is poorly quantified and poorly attributed to specific processes, which has hampered efforts to focus research in reducing uncertainty. This has resulted from sparse sampling of the possible uncertainty space—which is high-dimensional and has contributions from parametric, structural, initial, and boundary condition uncertainties—as an artifact of CMIP-type ensembles of opportunity and limitations inherent in observational benchmarks. A new approach is needed to understand and reduce this uncertainty, based around individual LSMs that can represent the breadth of assumptions represented in current CMIP-type efforts, while at the same time exploring that uncertainty in a systematic way, confronting multiple types of observations, and where justified, replacing process representations with ML-driven emulators. We propose an approach of modular hybrid modeling to address these challenges.

58 GEOSCIENCES↗

Collaborative Project: The problem of bias in defining uncertainty in computationally enabled strategies for data-driven climate model development

The objective of the project is to develop strategies for better representing scientific sensibilities within statistical measures of model skill that then can be used within a Bayesian statistical framework for data-driven climate model development and improved measures of model scientific uncertainty. One of the thorny issues in model evaluation is quantifying the effect of biases on climate projections. While any bias is not desirable, only those biases that affect feedbacks affect scatter in climate projections. The effort at the University of Texas is to analyze previously calculated ensembles of CAM3.1 with perturbed parameters to discover how biases affect projections of global warming. The hypothesis is that compensating errors in the control model can be identified by their effect on a combination of processes and that developing metrics that are sensitive to dependencies among state variables would provide a way to select version of climate models that may reduce scatter in climate projections. Gabriel Huerta was responsible for leading the effort to develop statistical methods for evaluating these field dependencies. The UT effort will incorporate these developments into MECS, which is a set of python scripts being developed at the University of Texas for managing the workflow associated with data-driven climate model development over HPC resources. We regularly consult with Peter Gleckler (LLNL), Mark Taylor (SNL), and Rich Neale (NCAR) to provide additional insight into issues concerning CAM5 model development and multi-model evaluation of the AR5 archive. This grant was originally assigned to Indiana University where both the PI (Huerta) and the Postdoc (Nosedal) worked on the project from March 1, 2012-March 31, 2013.

97 MATHEMATICS AND COMPUTING↗

Collaborative Project: The problem of bias in defining uncertainty in computationally enabled strategies for data-driven climate model development. Final Technical Report.

The objective of the project is to develop strategies for better representing scientific sensibilities within statistical measures of model skill that then can be used within a Bayesian statistical framework for data-driven climate model development and improved measures of model scientific uncertainty. One of the thorny issues in model evaluation is quantifying the effect of biases on climate projections. While any bias is not desirable, only those biases that affect feedbacks affect scatter in climate projections. The effort at the University of Texas is to analyze previously calculated ensembles of CAM3.1 with perturbed parameters to discover how biases affect projections of global warming. The hypothesis is that compensating errors in the control model can be identified by their effect on a combination of processes and that developing metrics that are sensitive to dependencies among state variables would provide a way to select version of climate models that may reduce scatter in climate projections. Gabriel Huerta at the University of New Mexico is responsible for developing statistical methods for evaluating these field dependencies. The UT effort will incorporate these developments into MECS, which is a set of python scripts being developed at the University of Texas for managing the workflow associated with data-driven climate model development over HPC resources. This report reflects the main activities at the University of New Mexico where the PI (Huerta) and the Postdocs (Nosedal, Hattab and Karki) worked on the project.

58 GEOSCIENCES↗