The 2nd Law of Modeling & Simulation, Inequality Statements, Engineering UQ, and Credibility Assessment
Explore the source record for details and available documents.
Romero, Vicente Jose [Sandia National Laboratories↗
SEARCH · Search NASA
Results for “Simulation Credibility”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Exploitation of a Validation Hierarchy for Modeling and Simulation
Across engineering there is an evolving need to increase reliance on physics-based simulation to develop, design and optimize engineering systems. This increased reliance on modeling and simulation has highlighted a growing need to transform the confidence that modeling and simulation analysts have in their results into credibility for systems engineers to design and field systems more quickly and with less physical testing. For isolated components of a complex system, where a single discipline may drive product design, this is less of a concern as the relationship is often straightforward and easy to explain. However, when these isolated components are integrated, and are expected to operate in a multi-disciplinary context in which safety critical systems are involved, new concepts and model assurance standards are required. In this paper we address this challenge by showing how a model validation hierarchy can be exploited to identify those model validation experiments that will contribute most to increasing confidence and credibility of modeling and simulation predictions. The approach that is adopted contains four main steps. The first step is the construction of a model validation hierarchy that links subsystems, assemblies, and components to a hierarchy of physical experiments that can be used support model validation. This hierarchy connects the concerns of systems engineers to those of the modeling and simulation analyst in a clear and logical way. The structure and content of this hierarchy is then used in a second step to establish which physical phenomena have the greatest impact on overall system performance metrics. A gap analysis technique, based upon modeling and simulation concerns, is then used to prioritize the important physical phenomenon. Unfortunately, a common outcome of such gap analyses is the identification of many important gaps and so, in the final step of our process, we advocate the use of a global sensitivity analysis as a means to complete the prioritization.
Verification and Validation↗
Exploitation of a Validation Hierarchy for Modeling and Simulation
Across engineering there is an evolving need to increase reliance on physics-based simulation to develop, design and optimize engineering systems. This increased reliance on modeling and simulation has highlighted a growing need to transform the confidence that modeling and simulation analysts have in their results into credibility for systems engineers to design and field systems more quickly and with less physical testing. For isolated components of a complex system, where a single discipline may drive product design, this is less of a concern as the relationship is often straightforward and easy to explain. However, when these isolated components are integrated, and are expected to operate in a multi-disciplinary context in which safety critical systems are involved, new concepts and model assurance standards are required. In this paper we address this challenge by showing how a model validation hierarchy can be exploited to identify those model validation experiments that will contribute most to increasing confidence and credibility of modeling and simulation predictions. The approach that is adopted contains four main steps. The first step is the construction of a model validation hierarchy that links subsystems, assemblies, and components to a hierarchy of physical experiments that can be used support model validation. This hierarchy connects the concerns of systems engineers to those of the modeling and simulation analyst in a clear and logical way. The structure and content of this hierarchy is then used in a second step to establish which physical phenomena have the greatest impact on overall system performance metrics. A gap analysis technique, based upon modeling and simulation concerns, is then used to prioritize the important physical phenomenon. Unfortunately, a common outcome of such gap analyses is the identification of many important gaps and so, in the final step of our process, we advocate the use of a global sensitivity analysis as a means to complete the prioritization.
Verification and Validation↗
NASA-STD-7009 Guidance Document for Human Health and Performance Models and Simulations
Rigorous verification, validation, and credibility (VVC) processes are imperative to ensure that models and simulations (MS) are sufficiently reliable to address issues within their intended scope. The NASA standard for MS, NASA-STD-7009 (7009) [1] was a resultant outcome of the Columbia Accident Investigation Board (CAIB) to ensure MS are developed, applied, and interpreted appropriately for making decisions that may impact crew or mission safety. Because the 7009 focus is engineering systems, a NASA-STD-7009 Guidance Document is being developed to augment the 7009 and provide information, tools, and techniques applicable to the probabilistic and deterministic biological MS more prevalent in human health and performance (HHP) and space biomedical research and operations.
numerical analysis↗
Improving thermodynamic nudging in the E3SM Atmosphere Model version 2 (EAMv2): strategy and hindcast skills on weather systems
Nudging techniques are commonly employed to constrain atmospheric simulations toward observed states, facilitating model evaluation and sensitivity studies. However, if applied improperly – particularly to thermodynamic variables such as temperature and humidity – nudging can distort physical processes and introduce spurious biases, undermining the credibility of the simulations. This study presents an improved nudging implementation that applies vertically modulated tendencies to reduce adverse impacts on model physics. The framework is tested in version 2 of the Energy Exascale Earth System Model (EAMv2) using a suite of hindcast simulations nudged toward ERA5 reanalysis. We systematically evaluate the individual and combined effects of nudging wind, temperature, and humidity fields on the model's ability to represent large-scale atmospheric states and high-impact weather systems. Results show that the revised strategy – particularly when nudging temperature and humidity at selected levels – enhances hindcast skill by improving agreement with ERA5 without degrading the hydrological cycle or precipitation processes. Additional improvements in surface temperature, outgoing longwave radiation, and precipitation biases are achieved through targeted nudging of land-surface variables. The proposed approach strengthens the representation of large-scale conditions relevant to tropical cyclones, atmospheric rivers, and extratropical cyclones in the low-resolution EAMv2. These findings demonstrate that carefully designed thermodynamic nudging, especially of temperature and humidity, improves the realism of constrained simulations and broadens the utility of nudged EAMv2 for atmospheric modeling, machine learning, and high-impact weather research.
Atmospheric river↗
The Cloud Feedback Model Intercomparison Project (CFMIP) contribution to CMIP6.
The primary objective of CFMIP is to inform future assessments of cloud feedbacks through improved understanding of cloud-climate feedback mechanisms and better evaluation of cloud processes and cloud feedbacks in climate models. However, the CFMIP approach is also increasingly being used to understand other aspects of climate change, and so a second objective has now been introduced, to improve understanding of circulation, regional-scale precipitation, and non-linear changes. CFMIP is supporting ongoing model inter-comparison activities by coordinating a hierarchy of targeted experiments for CMIP6, along with a set of cloud-related output diagnostics. CFMIP contributes primarily to addressing the CMIP6 questions 'How does the Earth system respond to forcing?' and 'What are the origins and consequences of systematic model biases?' and supports the activities of the WCRP Grand Challenge on Clouds, Circulation and Climate Sensitivity. A compact set of Tier 1 experiments is proposed for CMIP6 to address this question: (1) what are the physical mechanisms underlying the range of cloud feedbacks and cloud adjustments predicted by climate models, and which models have the most credible cloud feedbacks? Additional Tier 2 experiments are proposed to address the following questions. (2) Are cloud feedbacks consistent for climate cooling and warming, and if not, why? (3) How do cloud-radiative effects impact the structure, the strength and the variability of the general atmospheric circulation in present and future climates? (4) How do responses in the climate system due to changes in solar forcing differ from changes due to CO2, and is the response sensitive to the sign of the forcing? (5) To what extent is regional climate change per CO2 doubling state-dependent (non-linear), and why? (6) Are climate feedbacks during the 20th century different to those acting on long-term climate change and climate sensitivity? (7) How do regional climate responses (e.g. in precipitation) and their uncertainties in coupled models arise from the combination of different aspects of CO2 forcing and sea surface warming? CFMIP also proposes a number of additional model outputs in the CMIP DECK, CMIP6 Historical and CMIP6 CFMIP experiments, including COSP simulator outputs and process diagnostics to address the following questions. 1. How well do clouds and other relevant variables simulated by models agree with observations? 2. What physical processes and mechanisms are important for a credible simulation of clouds, cloud feedbacks and cloud adjustments in climate models? 3. Which models have the most credible representations of processes relevant to the simulation of clouds? 4. How do clouds and their changes interact with other elements of the climate system?
carbon dioxide↗
A Framework for Deep Learning Emulation of Numerical Models With a Case Study in Satellite Remote Sensing
Numerical models based on physics represent the state of the art in Earth system modeling and comprise our best tools for generating insights and predictions. Despite rapid growth in computational power, the perceived need for higher model resolutions overwhelms the latest generation computers, reducing the ability of modelers to generate simulations for understanding parameter sensitivities and characterizing variability and uncertainty. Thus, surrogate models are often developed to capture the essential attributes of the full-blown numerical models. Recent successes of machine learning methods, especially deep learning (DL), across many disciplines offer the possibility that complex nonlinear connectionist representations may be able to capture the underlying complex structures and nonlinear processes in Earth systems. A difficult test for DL-based emulation, which refers to function approximation of numerical models, is to understand whether they can be comparable to traditional forms of surrogate models in terms of computational efficiency while simultaneously reproducing model results in a credible manner. A DL emulation that passes this test may be expected to perform even better than simple models with respect to capturing complex processes and spatiotemporal dependencies. Here, we examine, with a case study in satellite-based remote sensing, the hypothesis that DL approaches can credibly represent the simulations from a surrogate model with comparable computational efficiency. Our results are encouraging in that the DL emulation reproduces the results with acceptable accuracy and often even faster performance. We discuss the broader implications of our results in light of the pace of improvements in high-performance implementations of DL and the growing desire for higher resolution simulations in the Earth sciences.
Bayesian Deep Learning↗
Propagating synthetic populations with dynamic Bayesian networks: a framework for long-horizon demographic forecasting
This study presents a dynamic demographic microsimulator using dynamic Bayesian networks to forecast long–term changes in household and individual life events. Leveraging longitudinal Panel Study of Income Dynamics (PSID) data, two networks for individuals and households were modeled to simulate transitions in employment, income, education, marriage, childbirth, leaving the parental home, home ownership, mortality, and household formation or dissolution. Across 1,000 simulation runs spanning 24 years, household–level outcomes remain highly accurate and individual–level predictions reasonable. Although accuracy naturally declines with projection horizon, performance remains promising at both levels. This study addresses a key limitation of existing population synthesis models, which typically generate only a single static snapshot of the population. In conclusion, by introducing a framework that propagates cross-sectional outputs into the future, the microsimulator enables the tracking of demographic evolution over time, enhances realism in population-based simulations, and supplies credible inputs to agent-based travel demand models.
Demographic modeling↗
Prebiotic Evolution of Nitrogen Compounds
Support from this four year grant has funded our research on two general problems. One involves attempts to model the abiotic formation of simple source compounds for functional biomolecules, their concentration from dilute state in the hydrosphere and, in several cases, surface induced reactions to form precursor monomers for bioactive end products (refs. 1-5). Because of the pervasiveness and antiquity of phosphate based biochemistry and the catalytic activity of RNA we have exploring the hypothesis of an RNA World as an early stage in the emergence of life. This concept is now rather generally considered, but has been questioned due to the earlier lack of an experimentally demonstrated successful scheme for the spontaneous formation of ribose phosphate, the key backbone molecule in RNA. That impediment has now been removed. This has been achieved by demonstrating probable sources of activated (condensed) highly soluble and strongly sorbed phosphates in nature (Refs. 1,2) and effective condensation of aldehyde phosphates to form ribose phosphate in high yield (ref.6), thereby placing the RNA World concept on a somewhat safer experimental footing. Like all work in this field these experiments are oversimplifications that largely ignore competing side reactions with other compounds expected to be present. None the less our choice of experimental conditions aim at selective processes that eliminate interfering reactions. We have also sought to narrow the credibility gap by simulating geophysically and geochemically plausible conditions surrounding the putative prebiotic reactions.
Arrhenius, G.↗
Effect of Under-Resolved Grids on High Order Methods
There has been much discussion on verification and validation processes for establishing the credibility of CFD simulations. Since the early 1990s, many of the aeronautical and mechanical engineering related reference journals mandated that any accepted articles in numerical simulations (without known solutions to compared with) need to perform a minimum of one level of grid refinement and time step reduction. Due to the difficulty in analysis, the effect of under-resolved grids and the nonlinear behavior of available spatial discretizations, are scarcely discussed in the literature. Here, an under-resolved numerical simulation is one where the grid spacing being used is too coarse to resolve the smallest physically relevant scales of the chosen continuum governing equations that are of interest to the numerical modeler. With the advent of new developments in fourth-order or higher spatial schemes, it has become common to regard high order schemes as more accurate, reliable and require less grid points. The danger comes when one tries to perform computations with the coarsest grid possible while still hoping to maintain numerical results sufficiently accurate for complex flows, and especially, data-limited problems. On one hand, high order methods when applies to highly coupled multidimensional complex nonlinear problems might have different stability, convergence and reliability behavior than their well studied low order counterparts, especially for nonlinear schemes such as TVD, MUSCL with limiters, ENO, WENO and discrete Galerkin. On the other hand, high order methods involve more operation counts and systematic grid convergence study can be time consuming and prohibitively expansive. At the same time it is difficult to fully understand or categorize the different nonlinear behavior of finite discretizations, especially at the limits of under-resolution when different types of bifurcation phenomena might occur, depending on the combination of grid spacings, time steps, initial conditions and numerical treatments of boundary conditions.
Yee, H. C.↗
JPL Thermal Design Modeling Philosophy and NASA-STD-7009 Standard for Models and Simulations - A Case Study
The Standard JPL thermal engineering practice prescribes worst-case methodologies for design. In this process, environmental and key uncertain thermal parameters (e.g., thermal blanket performance, interface conductance, optical properties) are stacked in a worst case fashion to yield the most hot- or cold-biased temperature. Thus, these simulations would represent the upper and lower bounds. This, effectively, represents JPL thermal design margin philosophy. Uncertainty in the margins and the absolute temperatures is usually estimated by sensitivity analyses and/or by comparing the worst-case results with "expected" results. Applicability of the analytical model for specific design purposes along with any temperature requirement violations are documented in peer and project design review material. In 2008, NASA released NASA-STD-7009, Standard for Models and Simulations. The scope of this standard covers the development and maintenance of models, the operation of simulations, the analysis of the results, training, recommended practices, the assessment of the Modeling and Simulation (M&S) credibility, and the reporting of the M&S results. The Mars Exploration Rover (MER) project thermal control system M&S activity was chosen as a case study determining whether JPL practice is in line with the standard and to identify areas of non-compliance. This paper summarizes the results and makes recommendations regarding the application of this standard to JPL thermal M&S practices.
thermal blanket performance↗
Structural uncertainty assessment for fire-engulfed objects in crosswind: Establishing credibility for a multiphysics wall-modeled large-eddy simulation paradigm
A structural uncertainty validation study for a large-scale, fire-engulfed, elevated object subjected to crosswind is presented to establish the credibility of a high-fidelity, low-Mach, turbulent reacting flow wall-modeled large-eddy simulation (WMLES) approach that includes multiphysics coupling to participating media radiation and conjugate heat transfer. To establish that WMLES can accurately predict surface quantities including drag and pressure coefficient in the low-Mach crosswind regime, a foundational elevated isothermal cylinder validation case is presented at a similar gap-to-diameter ratio of 0.25, spanning the subcritical to supercritical drag regime (Re 𝐷 = 1.1 × 10 5 and 4.3 × 10 5 , respectively). Here, this study exercised both static and dynamic coefficient LES (Smagorinsky and 𝑘 sgs ) with both local and exchange-based velocity sampling. Results showcase that the drag crisis (or the sudden drop in drag coefficient at increased Re 𝐷 ) is well captured when using an exchange-based dynamic coefficient WMLES methodology, while noting lack of mesh convergence and overall drag and pressure coefficient predictively when using a static coefficient, local velocity sampling WMLES. For the 𝒪(10) m JP-8 liquid pool fire crosswind validation study presented, two experimental crosswind configurations (2 m/s and 9.5 m/s) are showcased for a fire-engulfed mock fuselage roughly 4 m in diameter. Using the best model-form practices identified in the isothermal study, dynamic coefficient 𝑘 sgs exchange-based WMLES fire validation findings demonstrate accurate peak irradiation and skin temperature predictions as a function of crosswind magnitude. Excessive yaw in the low-crosswind fuselage configuration, consistent with experimental findings, captured a significant predicted asymmetry in flame attachment and heat flux toward the downwind cylindrical cap—indicative of axial vortex structures transporting the flame along the upper and lower fuselage leeward surface. All fire mesh resolution simulations captured the experimental finding that as crosswind increased, predicted flame shape and peak irradiation magnitude onto the fuselage transitioned from a windward to a leeward cylinder location due to the migration of the upper- to lower-shear fuel/air mixing layer thereby demonstrating the novelty, significance, and credibility of this high-fidelity WMLES reacting flow framework.
Domino, Stefan Paul [Sandia National Laboratories ↗
Development of NASA's Models and Simulations Standard
From the Space Shuttle Columbia Accident Investigation, there were several NASA-wide actions that were initiated. One of these actions was to develop a standard for development, documentation, and operation of Models and Simulations. Over the course of two-and-a-half years, a team of NASA engineers, representing nine of the ten NASA Centers developed a Models and Simulation Standard to address this action. The standard consists of two parts. The first is the traditional requirements section addressing programmatics, development, documentation, verification, validation, and the reporting of results from both the M&S analysis and the examination of compliance with this standard. The second part is a scale for evaluating the credibility of model and simulation results using levels of merit associated with 8 key factors. This paper provides an historical account of the challenges faced by and the processes used in this committee-based development effort. This account provides insights into how other agencies might approach similar developments. Furthermore, we discuss some specific applications of models and simulations used to assess the impact of this standard on future model and simulation activities.
simulation↗
Stratospheric Impact of Varying Sea Surface Temperatures
The Finite-Volume General Circulation Model (FVGCM) has been run in 50 year simulations with the: 1) 1949-1999 Hadley Centre sea surface temperatures (SST), and 2) a fixed annual cycle of SSTs. In this presentation we first show that the 1949-1999 FVGCM simulation produces a very credible stratosphere in comparison to an NCEP/NCAR reanalysis climatology. In particular, the northern hemisphere has numerous major and minor stratospheric warming, while the southern hemisphere has only a few over the 50-year simulation. During the northern hemisphere winter, temperatures are both warmer in the lower stratosphere and the polar vortex is weaker than is found in the mid-winter southern hemisphere. Mean temperature differences in the lower stratosphere are shown to be small (less than 2 K), and planetary wave forcing is found to be very consistent with the climatology. We then will show the differences between our varying SST simulation and the fixed SST simulation in both the dynamics and in two parameterized trace gases (ozone and methane). In general, differences are found to be small, with subtle changes in planetary wave forcing that lead to reduced temperatures in the SH and increased temperatures in the NH.
Newman, Paul A.↗
A method motion simulator design based on modeling characteristics of the human operator
A design criteria is obtained to compare two simulators and evaluate their equivalence or credibility. In the subsequent analysis the comparison of two simulators can be considered as the same problem as the comparison of a real world situation and a simulation's representation of this real world situation. The design criteria developed involves modeling of the human operator and defining simple parameters to describe his behavior in the simulator and in the real world situation. In the process of obtaining human operator parameters to define characteristics to evaluate simulators, measures are also obtained on these human operator characteristics which can be used to describe the human as an information processor and controller. First, a study is conducted on the simulator design problem in such a manner that this modeling approach can be used to develop a criteria for the comparison of two simulators.
Repperger, D. W.↗
Towards a Comprehensive Dynamic-chemistry Assimilation for Eos-Chem: Plans and Status in NASA's Data Assimilation Office
In order to support the EOS-Chem project, a comprehensive assimilation package for the coupled chemical-dynamical system is being developed by the Data Assimilation Office at NASA GSFC. This involves development of a coupled chemistry/meteorology model and of data assimilation techniques for trace species and meteorology. The model is being developed using the flux-form semi-Lagrangian dynamical core of Lin and Rood, the physical parameterizations from the NCAR Community Climate Model, and atmospheric chemistry modules from the Atmospheric Chemistry and Dynamics branch at NASA GSFC. To date the following results have been obtained: (i) multi-annual simulations with the dynamics-radiation model show the credibility of the package for atmospheric simulations; (ii) initial simulations including a limited number of middle atmospheric trace gases reveal the realistic nature of transport mechanisms, although there is still a need for some improvements. Samples of these results will be shown. A meteorological assimilation system is currently being constructed using the model; this will form the basis for the proposed meteorological/chemical assimilation package. The latter part of the presentation will focus on areas targeted for development in the near and far terms, with the objective of Providing a comprehensive assimilation package for the EOS-Chem science experiment. The first stage will target ozone assimilation. The plans also encompass a reanalysis (ReSTS) for the 1991-1995 period, which includes the Mt. Pinatubo eruption and the time when a large number of UARS observations were available. One of the most challenging aspects of future developments will be to couple theoretical advances in tracer assimilation with the practical considerations of a real environment and eventually a near-real-time assimilation system.
Pawson, Steven↗
Simulation of the Ozone Monitoring Instrument Aerosol Index Using the NASA Goddard Earth Observing System Aerosol Reanalysis Products
We provide an analysis of the commonly used Ozone Monitoring Instrument (OMI) aerosol index (AI) product for qualitative detection of the presence and loading of absorbing aerosols. In our analysis, simulated top-of-atmosphere (TOA) radiances are produced at the OMI footprints from a model atmosphere and aerosol profile provided by the NASA Goddard Earth Observing System (GEOS-5) Modern-Era Retrospective Analysis for Research and Applications aerosol reanalysis (MERRAero). Having established the credibility of the MERRAero simulation of the OMI AI in a previous paper we describe updates in the approach and aerosol optical property assumptions. The OMI TOA radiances are computed in cloud-free conditions from the MERRAero atmospheric state, and the AI is calculated. The simulated TOA radiances are fed to the OMI aerosol retrieval algorithms, and its retrieved AI (OMAERUV AI) is compared to the MERRAero calculated AI. Two main sources of discrepancy are discussed: one pertaining the OMI algorithm assumptions of the surface pressure, which are generally different from what the actual surface pressure of an observation is, and the other related to simplifying assumptions in the molecular atmosphere radiative transfer used in the OMI algorithms. Surface pressure assumptions lead to systematic biases in the OMAERUV AI, particularly over the oceans. Simplifications in the molecular radiative transfer lead to biases particularly in regions of topography intermediate to surface pressures of 600hPa and 1013.25hPa. Generally, the errors in the OMI AI due to these considerations are less than 0.2 in magnitude, though larger errors are possible, particularly over land. We recommend that future versions of the OMI algorithms use surface pressures from readily available atmospheric analyses combined with high-spatial resolution topographic maps and include more surface pressure nodal points in their radiative transfer lookup tables.
remote sensing↗
Quantifying the Sensitivity of Condition Incidence Parameters in the Evidence Library
One approach to quantifying spaceflight risk at NASA makes use event driven probabilistic techniques. The Medical Extensible Dynamic Probabilistic Risk Assessment Tool (MEDPRAT) is such a tool that estimates medical risk metrics via simulation and enables optimization of medical resources subject to mission constraints [1]. Previous analyses have informed medical set composition, exercise countermeasures, and water intake, where each analysis quantifies the risk associated with proposed variations in system design. As future mission profiles extend beyond Low-Earth Orbit (LEO) and lengthen in duration, understanding these risks and contributing factors is critical. MEDPRAT employs Monte Carlo sampling techniques to simulate missions and track the occurrence of medical events. These events follow fault-tree-like progressions through levels of severity and mitigation via medical treatment to many possible outcomes and these are reported throughout the mission. Making this possible, are the medical databases that contain evidence gathered by the Human Research Program (HRP). Quantifying the impact of uncertainty or variability in the input data is an important step in evaluating the credibility of modeling and simulation results. In this work, we investigate the sensitivity of medical risk metrics with respect to the condition incidence parameters within the Evidence Library (EL) [2] as the medical database input for MEDPRAT. The medical conditions, contained in the EL, are equipped with incidence rates that describe the likelihood that the condition will occur. These incidence rates reflect historical spaceflight data or when appropriate, terrestrial data. In this presentation, we will explore how uncertainty in these rates propagate to the medical risk described by MEDPRAT. These results identify the conditions and parameters with the largest contribution to medical risks.
Ian Lim↗