Search NASA⌕ Search

SEARCH · Search NASA

Results for “model credibility”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

Magnetohydrodynamic Simulation of a Streamer Beside a Realistic Coronal Hole

Existing models of coronal streamers establish their credibility and act as the initial state for transients. The models have produced satisfactory streamer simulations, but unsatisfactory coronal hole simulations. This is a consequence of the character of the models and the boundary conditions. The models all have higher densities in the magnetically open regions than occur in coronal holes (Noci, et al., 1993).

Suess, S. T.↗

The Roles of Verification, Validation and Uncertainty Quantification in the NASA Standard for Models and Simulations

The National Aeronautics and Space Administration (NASA) recently issued an interim version of the Standard for Models and Simulations (M&S Standard) [1]. The action to develop the M&S Standard was identified in an internal assessment [2] of agency-wide changes needed in the wake of the Columbia Accident [3]. The primary goal of this standard is to ensure that the credibility of M&S results is properly conveyed to those making decisions affecting human safety or mission success criteria. The secondary goal is to assure that the credibility of the results from models and simulations meets the project requirements (for credibility). This presentation explains the motivation and key aspects of the M&S Standard, with a special focus on the requirements for verification, validation and uncertainty quantification. Some pilot applications of this standard to computational fluid dynamics applications will be provided as illustrations. The authors of this paper are the members of the team that developed the initial three drafts of the standard, the last of which benefited from extensive comments from most of the NASA Centers. The current version (number 4) incorporates modifications made by a team representing 9 of the 10 NASA Centers. A permanent version of the M&S Standard is expected by December 2007. The scope of the M&S Standard is confined to those uses of M&S that support program and project decisions that may affect human safety or mission success criteria. Such decisions occur, in decreasing order of importance, in the operations, the test & evaluation, and the design & analysis phases. Requirements are placed on (1) program and project management, (2) models, (3) simulations and analyses, (4) verification, validation and uncertainty quantification (VV&UQ), (5) recommended practices, (6) training, (7) credibility assessment, and (8) reporting results to decision makers. A key component of (7) and (8) is the use of a Credibility Assessment Scale, some of the details of which were developed in consultation with William Oberkampf, David Peercy and Timothy Trocano of Sandia National Laboratories. The focus of most of the requirements, including those for VV&UQ, is on the documentation of what was done and the reporting, using the Credibility Assessment Scale, of the level of rigor that was followed. The aspects of one option for the Credibilty Assessment Scale are (1) code verification, (2) solution verification, (3) validation, (4) predictive capability, (5) technical review, (6) process control, and (7) operator and analyst qualification.

Zang, Thomas A.↗

SWAT model application for evaluating agricultural conservation practice effectiveness in reducing phosphorous loss from the Western Lake Erie Basin

Lake Erie is threatened by eutrophication and harmful algal blooms due to excess nutrient loading from agricultural sources. Agricultural conservation practices (ACPs) have been developed and implemented to reduce nutrient losses but estimating ACP effectiveness is challenging. The Soil and Water Assessment Tool (SWAT) has been used to investigate ACP effectiveness for water quality improvement. Many SWAT applications have been developed by different investigators to evaluate ACP effectiveness for reducing nutrient, particularly phosphorus (P), loading in the agriculturally-dominated Western Lake Erie Basin (WLEB). Our objective is to establish what has been achieved by past modeling research and make suggestions for future applications and improvements. We synthesized the findings of 28 SWAT modeling studies within the WLEB. Models generally performed satisfactorily against accepted criteria for streamflow and sediment, but performance for P loads, like soluble reactive P, was mostly “unsatisfactory”. The “unsatisfactory” performance maybe due to imperfections and idealizations in model formulations and/or parameterization. Thus, simulations of P transport and transformation processes need improvement. In addition, model parameter selection is the key part of model set-up. Most SWAT modeling studies used default values during initial set-up, then performed calibration and validation. It was found that the calibrated P related parameter values varied widely across different studies, even within the same watershed with some values unrealistic for the study areas. The phenomena of different combinations of model parameters producing similar outputs indicates equifinality. Equifinality in the baseline model may impact results when ACPs are incorporated. Furthermore, the unrealistic values used in ACP assessment undermine the credibility of ACP effectiveness. Future model applications should try to re-examine the calibrated P parameters and make sure they are realistic for the study area as well as reduce equifinality by constraining the model with characterization of watershed conditions, better understanding of hydrologic processes, and parameter values based on real-world observations. As a result, future model applications should focus on improving P transport and transformation processes, using measured watershed characteristics for parameterization, and improving reflections of climate change, which could result in more accurate assessments of ACP effectiveness to meet targeted goals.

54 ENVIRONMENTAL SCIENCES↗

Verification Testing: Meet User Needs Figure of Merit

Verification is the process through which Modeling and Simulation(M&S) software goes to ensure that it has been rigorously tested and debugged for its intended use. Validation confirms that said software accurately models and represents the real world system. Credibility gives an assessment of the development and testing effort that the software has gone through as well as how accurate and reliable test results are. Together, these three components form Verification, Validation, and Credibility(VV&C), the process by which all NASA modeling software is to be tested to ensure that it is ready for implementation. NASA created this process following the CAIB (Columbia Accident Investigation Board) report seeking to understand the reasons the Columbia space shuttle failed during reentry. The reports conclusion was that the accident was fully avoidable, however, among other issues, the necessary data to make an informed decision was not there and the result was complete loss of the shuttle and crew. In an effort to mitigate this problem, NASA put out their Standard for Models and Simulations, currently in version NASA-STD-7009A, in which they detailed their recommendations, requirements and rationale for the different components of VV&C. They did this with the intention that it would allow for people receiving MS software to clearly understand and have data from the past development effort. This in turn would allow the people who had not worked with the MS software before to move forward with greater confidence and efficiency in their work. This particular project looks to perform Verification on several MATLAB (Registered Trademark)(The MathWorks, Inc.) scripts that will be later implemented in a website interface. It seeks to take note and define the limits of operation, the units and significance, and the expected datatype and format of the inputs and outputs of each of the scripts. This is intended to prevent the code from attempting to make incorrect or impossible calculations. Additionally, this project will look at the coding generally and note inconsistencies, redundancies, and other aspects that may become problematic or slow down the codes run time. Certain scripts lacking in documentation also will be commented and cataloged.

validation↗

Validation of implosion modeling through direct-drive shock timing experiments at the National Ignition Facility

Precise modeling of shocks in inertial confinement fusion implosions is critical for obtaining the desired compression in experiments. Shock velocities and post-shock conditions are determined by laser-energy deposition, heat conduction, and equations of state. This paper describes experiments at the National Ignition Facility (NIF) where multiple shocks are launched into a cone-in-shell target made of polystyrene, using laser pulse shapes with two or three pickets and varying on-target intensities. Shocks are diagnosed using the Velocity Interferometric System for Any Reflector diagnostic. Simulated and inferred shock velocities agree well for the range of intensities studied in this work. These results are the first shock-velocity measurements using direct drive on the NIF and provide a good measure of early-time, directly driven laser-energy coupling. Furthermore, the validated models add to the credibility of direct-drive-ignition designs at the megajoule scale.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Computationally efficient subglacial drainage modelling using Gaussian process emulators: GlaDS-GP v1.0

Subglacial drainage models represent water flow at the ice–bed interface through coupled distributed and channelized systems to determine water pressure, discharge, and drainage system geometry. While they are used to understand processes such as the relationship between surface melt and ice flow, the number of uncertain model parameters and the computational cost of running models makes it difficult to adequately explore the high-dimensional parameter space and evaluate uncertainty in model predictions. Here, we develop Gaussian process (GP) emulators that make fast predictions with associated uncertainty of subglacial drainage model outputs. Using a truncated principal component (PC) basis representation, we construct a GP emulator for diurnally averaged subglacial water pressure. We also explore emulation of scalar variables describing drainage efficiency and configuration. We train the emulators using ensembles of up to 512 simulations varying eight parameters of the Glacier Drainage System (GlaDS) model on a synthetic domain intended to represent an ice-sheet margin. The emulators make predictions ∼ 1000 times faster than GlaDS simulations, with errors <3 % for the water pressure field and ∼ 5 %–9 % for drainage efficiency and configuration. We apply the emulators to explore the eight-dimensional parameter space by computing variance-based parameter sensitivity indices, finding that three parameters (ice flow coefficient, bed bump aspect ratio, and the subglacial cavity system conductivity) explain 90 % of the variance in modelled water pressure in response to parameter changes. The GP emulator approach described here is well suited to integrating observational data with models to make calibrated, credible predictions of subglacial drainage.

58 GEOSCIENCES↗

Integrated avionics reliability

The integrated avionics reliability task is an effort to build credible reliability and/or performability models for multisensor integrated navigation and flight control. The research was initiated by the reliability analysis of a multisensor navigation system consisting of the Global Positioning System (GPS), the Long Range Navigation system (Loran C), and an inertial measurement unit (IMU). Markov reliability models were developed based on system failure rates and mission time.

Alikiotis, Dimitri↗

Cosmological inference from an emulator based halo model. II. Joint analysis of galaxy-galaxy weak lensing and galaxy clustering from HSC-Y1 and SDSS

Here, we present high-fidelity cosmology results from a blinded joint analysis of galaxy-galaxy weak lensing (ΔΣ) and projected galaxy clustering (w p ) measured from the Hyper Suprime-Cam Year-1 (HSC-Y1) data and spectroscopic Sloan Digital Sky Survey (SDSS) galaxy catalogs in the redshift range 0.15 < z < 0.7. We define luminosity-limited samples of SDSS galaxies to serve as the tracers of w p in three spectroscopic redshift bins, and as the lens samples for ΔΣ. For the ΔΣ measurements, we select a single sample of 4×10 6 source galaxies over 140 deg 2 from HSC-Y1 with photometric redshifts (photo z) greater than 0.75, enabling a better handle of photo- z errors by comparing the ΔΣ amplitudes for the three lens redshift bins. The deep, high-quality HSC-Y1 data enable significant detections of the ΔΣ signals, with integrated signal-to-noise ratio S/N ~ 15 in the range 3 ≤ R/[h –1 Mpc] ≤ 30 for the three lens samples, despite the small area coverage. For cosmological parameter inference, we use an input galaxy-halo connection model built on the dark emulator package (which uses an ensemble set of high-resolution N-body simulations and enables fast, accurate computation of the clustering observables) with a halo occupation distribution that includes nuisance parameters to marginalize over modeling uncertainties. We model the ΔΣ and wp measurements on scales from R≃3 and 2h –1 Mpc , respectively, up to 30 h –1 Mpc (therefore excluding the baryon acoustic oscillations information) assuming a flat Λ CDM cosmology, marginalizing over about 20 nuisance parameters and demonstrating the robustness of our results to them. With various tests using mock catalogs described in Miyatake et al. [preceding paper, Phys. Rev. D 106, 083519 (2022)], we show that any bias in the clustering amplitude S 8 ≡ σ 8 (Ω m /0.3) 0.5 due to uncertainties in the galaxy-halo connection is less than ~ 50 % of the statistical uncertainty of S 8 , unless the assembly biaseffect is unexpectedly large. Our best-fit models have S 8 = 0.795$_{-0.042}^{+0.049}$ (mode and 68% credible interval) for the flat Λ CDM model; we find tighter constraints on the quantity S 8 (α = 0.17)≡σ 8 (Ω m /0.3) 0.17 = 0.745$_{-0.031}^{+0.039}$.

79 ASTRONOMY AND ASTROPHYSICS↗

Single-size and cluster dynamics modeling of intra-granular fission gas bubbles in UO 2

For this work, we perform simulations of intra-granular fission gas bubble evolution in UO 2 using both a relatively simple, computationally inexpensive single-size model and a detailed cluster dynamics model. Simulations encompass 36 experimental cases from 4 different databases, covering various temperature and burnup levels. We systematically compare results from the two models to each other and to post-irradiation experimental data of bubble average size and number density. Overall, the model-to-model comparisons reveal an excellent agreement across the set of simulations. This outcome indicates that, in spite of the underlying assumptions, the single-size model provides a good approximation of the complex physical behavior that is more rigorously described by the cluster dynamics model. Qualitatively, both models reproduce the trends of the experimental data with temperature and burnup correctly. Quantitatively, calculated results are either in good agreement with the data or within errors that appear consistent with the inherent uncertainties. Moreover, for the single-size model, we demonstrate and assess a multiscale approach whereby values for the fission gas atom diffusion coefficient from separate atomistic calculations are used. Systematic comparisons to experimental data point out a credible accuracy of the multiscale model.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

A Comparison of Metamodeling Techniques via Numerical Experiments

This paper presents a comparative analysis of a few metamodeling techniques using numerical experiments for the single input-single output case. These experiments enable comparing the models' predictions with the phenomenon they are aiming to describe as more data is made available. These techniques include (i) prediction intervals associated with a least squares parameter estimate, (ii) Bayesian credible intervals, (iii) Gaussian process models, and (iv) interval predictor models. Aspects being compared are computational complexity, accuracy (i.e., the degree to which the resulting prediction conforms to the actual Data Generating Mechanism), reliability (i.e., the probability that new observations will fall inside the predicted interval), sensitivity to outliers, extrapolation properties, ease of use, and asymptotic behavior. The numerical experiments describe typical application scenarios that challenge the underlying assumptions supporting most metamodeling techniques.

Crespo, Luis G.↗

Supporting hierarchical soil biogeochemical modeling: version 2 of the Biogeochemical Transport and Reaction model (BeTR-v2)

Abstract. Reliable soil biogeochemical modeling is a prerequisite for credible projections of climate change and associated ecosystem feedbacks. This recognition has called for frameworks that can support flexible and efficient development and application of new or alternative soil biogeochemical modules in Earth system models (ESMs). The the Biogeochemical Transport and Reaction model version 1 (BeTR-v1) code (i.e., CLM4-BeTR) is one such framework designed to accelerate the development and integration of new soil biogeochemistry formulations into ESMs and to analyze structural uncertainty in ESM simulations. With a generic reactive transport capability, BeTR-v1 can represent multiphase (e.g., gaseous, aqueous, and solid), multi-tracer (e.g., nitrate and organic carbon), and multi-organism (e.g., plants, bacteria, and fungi) dynamics. Here, we describe the new version, Biogeochemical Transport and Reaction model version 2 (BeTR-v2), which adopts more robust numerical solvers for multiphase diffusion and advection and coupling between biogeochemical reactions and improves code modularization over BeTR-v1. BeTR-v2 better supports different mathematical formulations in a hierarchical manner by allowing the resultant model be run for a single topsoil layer or a vertically resolved soil column, and it allows the model to be fully coupled with the land component of the Energy Exascale Earth System Model (E3SM). We demonstrate the capability of BeTR-v2 with benchmark cases and example soil biogeochemical (BGC) implementations. By taking advantage of BeTR-v2's generic structure integrated in E3SM, we then found that calibration could not resolve biases introduced by different numerical coupling strategies of plant–soil biogeochemistry. These results highlight the importance of numerically robust implementation of soil biogeochemistry and coupling with hydrology, thermal dynamics, and plants – capabilities that the open-source BeTR-v2 provides. We contend that Earth system models should strive to minimize this uncertainty by applying better numerical solvers.

58 GEOSCIENCES↗

Method for Real-Time Salt Aerosol Concentration and Size Measurements for Molten Salt Reactor Safety Assessments

Licensing a new nuclear reactor involves evaluating credible accident scenarios by developing models that simulate accident progression to predict possible outcomes and their impact on safety. The formation of radionuclide-bearing aerosols in the respirable size range can significantly affect dose consequence and threaten human health and is therefore a focus of nuclear reactor safety assessments. Recent reviews of the literature identified a lack of existing experimental data describing the mechanisms of formation and properties (size, concentration, and composition) of radionuclide-bearing aerosols that may be produced from molten salt reactor (MSR) facilities during postulated accident scenarios. Experiments on aerosol formation from molten salt systems are a high priority need that will support MSR licensing by indicating the radiological consequences of aerosol formation and providing the data required for model development and validation. The evolving conditions that occur during a MSR accident, such as a spill of molten fuel salt, may affect aerosol formation mechanisms and aerosol properties over time. There is a need to conduct experiments that simulate credible MSR accidents in a laboratory to generate aerosols with realistic characteristics and behaviors and a need to measure these aerosols in real time under accident-relevant conditions. This report describes the development of a method that can be used to quantify the size and concentration of salt aerosol particles that form from molten salt systems in real time. This method will be employed in future integral effects tests that are conducted at an engineering scale to simulate realistic MSR accidents and in future separate effects tests that will provide mechanistic insight into aerosol formation and properties to support process model development.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Evaluating long-term model-based scenarios of the energy system

Energy-economic models are used to provide science-based decision support in a variety of contexts. Analyses using these tools often involve defining a “reference” scenario, which serves as a counterfactual against which alternative scenarios are compared. Evaluating scenarios, including reference scenarios, is critical for establishing the credibility of the analyses these models support. We propose a framework for evaluating energy system scenarios which consists of three parts – a qualitative storyline, quantitative metrics, and evaluation criteria. We apply this framework to the reference scenario for GCAM-USA, a version of the global human-Earth system model GCAM (Global Change Assessment Model) with state-level detail in the United States, focusing on the evolution of the electric power sector. We develop new visual analytic tools to facilitate the evaluation of model outcomes in 51 sub-national regions, and demonstrate how scenario performance can be tracked and compared across four quantifications of the GCAM-USA reference scenario.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Facilitating better and faster simulations of aerosol-cloud interactions in Earth system models

Focal Area(s): 1. Predictive modeling through the use of AI techniques and AI-derived model components; the use of AI and other tools to design a prediction system comprising a hierarchy of models. 2. Insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI. Science Challenge: One major challenge that Earth system models (ESMs) face in providing credible prediction of the Earth system and its water cycle characteristics (e.g., mean state, variability, and extreme events) is to accurately simulate aerosol-cloud interactions (ACI). The physical, chemical, and dynamical processes affecting ACI are extremely complex and they range from nanoscale to planetary scale. In each model development cycle, scientists spend significant efforts investigating model deficiencies and uncertainties associated with aerosols (e.g., emissions, chemical processes, aerosol microphysics, and transport) and clouds (e.g., macrophysics, microphysics, turbulence, and large-scale circulation) in order to develop improved treatments. However, despite decades of active research, ACI is still a major source of uncertainty in climate projections, even though great progress has been made. Specific scientific challenges include: (i) Parameterizations are developed based on limited data; (ii) The complexity of a parameterization required for accurate predictions is not understood; (iii) Incomplete and unknown physics leads to errors in the fully coupled Earth system; and (iv) Complex physics is computationally too expensive to employ in ESMs.

54 ENVIRONMENTAL SCIENCES↗

Dynamic logistic regression and variable selection: Forecasting and contextualizing civil unrest

Civil unrest can range from peaceful protest to violent furor, and researchers are working to monitor, forecast, and assess such events to allocate resources better. Twitter has become a real-time data source for forecasting civil unrest because millions of people use the platform as a social outlet. Additionally, daily word counts are used as model features, and predictive terms contextualize the reasons for the protest. To forecast civil unrest and infer the reasons for the protest, we consider the problem of Bayesian variable selection for the dynamic logistic regression model and propose using penalized credible regions to select parameters of the updated state vector. This method avoids the need for shrinkage priors, is scalable to high-dimensional dynamic data, and allows the importance of variables to vary in time as new information becomes available. A substantial improvement in both precision and F1-score using this approach is demonstrated through simulation. Finally, we apply the proposed model fitting and variable selection methodology to the problem of forecasting civil unrest in Latin America. Our dynamic logistic regression approach shows improved accuracy compared to the static approach currently used in event prediction and feature selection.

97 MATHEMATICS AND COMPUTING↗

Understanding and Predictability of Integrated Mountain Hydroclimate (Workshop Report)

Mountainous systems cover approximately 23% of Earth’s land and are distributed across all continents. They can capture and store atmospheric moisture that is then cycled through the terrestrial surface and subsurface system, released to downstream communities, and cycled back to the atmosphere. Mountain hydroclimate—characterized by steep gradients, geological, ecological, and biogeochemical diversity—is influenced by topographic forcing and elevated warming and susceptible to large subseasonal to multidecadal variability and rapid changes. Terrestrial hydrological and biogeochemical cycles also experience cascading effects from global warming impacts, such as multidecadal declines in mountain snowpack, longer growing seasons, and increased frequency and severity of extreme events like droughts and wildfires. However, little is known about the effects of these impacts and their feedbacks on climate systems and surface-subsurface compartments. Also unknown are the full implications of changing hydroclimate and extreme events on hydro biogeochemical cycles across atmosphere, terrestrial, and human systems in mountain regions and beyond. This knowledge gap is critical, given human reliance on mountain systems for stable water supply and quality. Mountain systems’ increasing vulnerability to climate change and human perturbations motivates the need to improve understanding of integrated mountain hydroclimate (IMHC) systems and their feedbacks and impacts on humans across scales. However, due to large heterogeneity and strong gradients, coupled natural-human processes in mountain regions present significant challenges for observations, modeling, predictions, and projections. Motivated by gaps in mountain hydroclimate understanding, observations, and modeling and the need for credible projections of future changes, the U.S. Department of Energy’s (DOE) Biological and Environmental Research (BER) program organized a virtual workshop on “Understanding and Predictability of Integrated Mountain Hydroclimate.” Sponsored by BER’s Earth and Environmental Systems Sciences Division (EESSD), the workshop aimed to inform and catalyze EESSD’s growing interests in enhancing predictive understanding of IMHC. Organizers structured the workshop to identify (1) knowledge gaps, (2) observational and modeling challenges, (3) short-term (1 to 3 years) and long-term (10 years and beyond) research opportunities, and (4) strategies for fostering collaboration and coordination. To address the outstanding challenges of IMHC, the workshop included two sessions organized by disciplinary, cross-disciplinary, and crosscutting science topics. The disciplinary and cross-disciplinary topics focused on essential IMHC elements: atmosphere, terrestrial, and human systems and their interactions. Breakout sessions on disciplinary and cross-disciplinary topics facilitated identification of crosscutting topics and central emerging themes. Session 1 focused on connecting existing DOE investments to accelerate progress related to scientific challenges in understanding mountain hydroclimate. In Session 2, participants further explored key Session 1 takeaways through the lens of multiagency collaborations and coordination.

54 ENVIRONMENTAL SCIENCES↗

Aerodynamic Analysis of Multistage Turbomachinery Flows in Support of Aerodynamic Design

This paper summarizes the state of 3D CFD based models of the time average flow field within axial flow multistage turbomachines. Emphasis is placed on models which are compatible with the industrial design environment and those models which offer the potential of providing credible results at both design and off-design operating conditions. The need to develop models which are free of aerodynamic input from semi-empirical design systems is stressed. The accuracy of such models is shown to be dependent upon their ability to account for the unsteady flow environment in multistage turbomachinery. The relevant flow physics associated with some of the unsteady flow processes present in axial flow multistage machinery are presented along with procedures which can be used to account for them in 3D CFD simulations. Sample results are presented for both axial flow compressors and axial flow turbines which help to illustrate the enhanced predictive capabilities afforded by including these procedures in 3D CFD simulations. Finally, suggestions are given for future work on the development of time average flow models.

Adamczyk, John J.↗

Standard for Models and Simulations

This NASA Technical Standard establishes uniform practices in modeling and simulation to ensure essential requirements are applied to the design, development, and use of models and simulations (MS), while ensuring acceptance criteria are defined by the program project and approved by the responsible Technical Authority. It also provides an approved set of requirements, recommendations, and criteria with which MS may be developed, accepted, and used in support of NASA activities. As the MS disciplines employed and application areas involved are broad, the common aspects of MS across all NASA activities are addressed. The discipline-specific details of a given MS should be obtained from relevant recommended practices. The primary purpose is to reduce the risks associated with MS-influenced decisions by ensuring the complete communication of the credibility of MS results.

verification↗