Results for “Simulation Credibility”

NASA NTRS2016

Simulation Credibility: Advances in Verification, Validation, and Uncertainty Quantification

Decision makers and other users of simulations need to know quantified simulation credibility to make simulation-based critical decisions and effectively use simulations, respectively. The credibility of a simulation is quantified by its accuracy in terms of uncertainty, and the responsibility of establishing credibility lies with the creator of the simulation. In this volume, we present some state-of-the-art philosophies, principles, and frameworks. The contributing authors involved in this publication have been dedicated to advancing simulation credibility. They detail and provide examples of key advances over the last 10 years in the processes used to quantify simulation credibility: verification, validation, and uncertainty quantification. The philosophies and assessment methods presented here are anticipated to be useful to other technical communities conducting continuum physics-based simulations; for example, issues related to the establishment of simulation credibility in the discipline of propulsion are discussed. We envision that simulation creators will find this volume very useful to guide and assist them in quantitatively conveying the credibility of their simulations.

Simulation Credibility↗

NASA NTRS2008

Notional Scoring for Technical Review Weighting As Applied to Simulation Credibility Assessment

NASA's Modeling and Simulation Standard requires a credibility assessment for critical engineering data produced by models and simulations. Credibility assessment is thus a "qualifyingfactor" in reporting results from simulation-based analysis. The degree to which assessors should be independent of the simulation developers, users and decision makers is a recurring question. This paper provides alternative "weighting algorithms" for calculating the value-added for independence of the levels of technical review defined for the NASA Modeling and Simulation Standard.

Hale, Joseph Peter↗

NASA NTRS2025

Modeling and Simulation Credibility Assessments of Musculoskeletal Computational Models for Simulating Astronaut Injuries Due to a Poor Spacesuit Fit

The musculoskeletal (MS) system of astronauts is subject to physiological changes, potentially leading to injuries due to the exposure to different gravitational environments experienced during spaceflight. These injuries can occur while an astronaut is performing an Extravehicular Activity (EVA) in space, on lunar or planetary surfaces or while wearing a spacesuit during terrestrial training for an EVA. The OpenSim MS modeling software can assess EVA induced MS injury mechanisms such as muscle strains, ligament injuries and joint injuries. One area of concern, since there are only a few different spacesuit sizes with limited adjustability, is the possibility of a poorly fitting spacesuit. This can cause unnatural joint motions and torques resulting in various MS injuries. A credibility assessment of the OpenSim modeling and simulation procedures is performed per NASA-STD-7009A to provide information on the credibility of the model’s use in simulating EVA related injury mechanisms. The credibility assessment evaluated various OpenSim models against the following eight credibility factors: data pedigree, input pedigree, code verification, solution verification, conceptual validation, referent validation, results uncertainty and results robustness (sensitivity). The models evaluated for EVA injuries will require additional credibility factor analysis and upgrades to the model features, such as adding ligaments to a whole-body model, to reliably predict and analyze the EVA injuries expected to occur due to a poor spacesuit fit. The degree of elevation strategy required to increase the credibility assessment scores will depend on the model complexity and the injury mechanism.

Christopher A. Gallo↗

NASA NTRS2024

Modeling and Simulation Credibility Assessments of Musculoskeletal Computational Models for Simulating Astronaut Injuries Due to a Poor Spacesuit Fit

The musculoskeletal (MS) system of astronauts is subject to physiological changes, potentially leading to injuries due to the exposure to different gravitational environments experienced during spaceflight. These injuries can occur while an astronaut is performing an Extravehicular Activity (EVA) in space, on lunar or planetary surfaces or while wearing a spacesuit during terrestrial training for an EVA. The OpenSim MS modeling software can assess EVA induced MS injury mechanisms such as muscle strains, ligament injuries and joint injuries. One area of concern, since there are only a few different spacesuit sizes with limited adjustability, is the possibility of a poorly fitting spacesuit. This can cause unnatural joint motions and torques resulting in various MS injuries. A credibility assessment of the OpenSim modeling and simulation procedures is performed per NASA-STD-7009A to provide information on the credibility of the model’s use in simulating EVA related injury mechanisms. The credibility assessment evaluated various OpenSim models against the following eight credibility factors: data pedigree, input pedigree, code verification, solution verification, conceptual validation, referent validation, results uncertainty and results robustness (sensitivity). The models evaluated for EVA injuries will require additional credibility factor analysis and upgrades to the model features, such as adding ligaments to a whole-body model, to reliably predict and analyze the EVA injuries expected to occur due to a poor spacesuit fit. The degree of elevation strategy required to increase the credibility assessment scores will depend on the model complexity and the injury mechanism.

Christopher A Gallo↗

NASA NTRS2015

Can Simulation Credibility Be Improved Using Sensitivity Analysis to Understand Input Data Effects on Model Outcome?

Model and simulation (MS) credibility is defined as, the quality to elicit belief or trust in MS results. NASA-STD-7009 [1] delineates eight components (Verification, Validation, Input Pedigree, Results Uncertainty, Results Robustness, Use History, MS Management, People Qualifications) that address quantifying model credibility, and provides guidance to the model developers, analysts, and end users for assessing the MS credibility. Of the eight characteristics, input pedigree, or the quality of the data used to develop model input parameters, governing functions, or initial conditions, can vary significantly. These data quality differences have varying consequences across the range of MS application. NASA-STD-7009 requires that the lowest input data quality be used to represent the entire set of input data when scoring the input pedigree credibility of the model. This requirement provides a conservative assessment of model inputs, and maximizes the communication of the potential level of risk of using model outputs. Unfortunately, in practice, this may result in overly pessimistic communication of the MS output, undermining the credibility of simulation predictions to decision makers. This presentation proposes an alternative assessment mechanism, utilizing results parameter robustness, also known as model input sensitivity, to improve the credibility scoring process for specific simulations.

Credibility↗

NASA NTRSDate not supplied

Modeling and Simulation Credibility Assessment of Finite Element Models for Fall From Heights Injury Scenarios

Finite element (FE) computational models of the whole human body are being used to investigate the injury modalities and mechanisms arising from suited astronauts’ ExtraVehicular Activities (EVAs). These FE whole body models – simplified Elemance and THUMS pedestrian versions – have undergone a series of verification and validation (V&V) assessments for applications in automotive, domestic, and contact sports arenas. However, the FE models’ Modeling and Simulation (M&S) credibility assessments for simulating suited astronaut injuries due to fall from heights in altered gravity environments are limited. The current study evaluates the M&S credibility of simplified Elemance and THUMS pedestrian FE models based on eight credibility factors – data pedigree, input pedigree, code verification, solution verification, conceptual validation, referent validation, results uncertainty, and results robustness – as per NASA-STD-7009A [1]. These credibility factors were assigned ordinal scores ranging between 0 to 4, with a score of 0 in the case of insufficient evidence and 4 when all data for the model is known, used appropriately, and traceable to the Real-World System (RWS) with acceptable accuracy, precision, and uncertainty. The credibility factor sufficiency thresholds were assigned upon guidance from NASA EVA subject matter experts. The factor scores were ascertained through the evaluation of M&S credibility assessment procedures described in the published literature and FE model reports. The M&S credibility scores for the Elemance and THUMS FE models vary between 1 and 3 for data pedigree, input pedigree, and referent validation credibility factors, and between 0 and 1 for code verification, solution verification, conceptual validation, results uncertainty and results robustness. The results show that these FE models are credible for simulating fall from heights injury scenarios in specific kinematic and kinetic ranges that correlate with highway and contact sports-related external conditions. Multiple credibility elevation strategies are prescribed to improve the Elemance and THUMS models’ credibility for use in the EVA fall from height injury scenarios.

R A Perkins↗

NASA NTRS2023

Modeling and Simulation Credibility Assessments of Whole-Body Finite Element Computational Models for Use in NASA Extravehicular Activity Applications

Computational finite element (FE) models are used in suited astronaut injury risk assessments; however, these models’ verification, validation, and credibility (VV&C) procedures for simulating injuries in altered gravity environments are limited. Our study conducts VV&C assessments of THUMS and Elemance whole-body FE models for predicting suited astronaut injury biomechanics using eight credibility factors, as per NASA-STD-7009A. Credibility factor ordinal scores are assigned by reviewing existing documentation describing VV&C practices, and credibility sufficiency thresholds are assigned based on input from subject matter experts. Our results show the FE models are credible for suited astronaut injury investigation in specific ranges of kinematic and kinetic conditions correlating to highway and contact sports events. Nevertheless, these models are deficient when applied outside these ranges. Several credibility elevation strategies are prescribed to improve models’ credibility for the NASA-centric application domain.

Finite Element↗

NASA NTRS2023

Modeling and simulation credibility assessments of whole-body finite element computational models for use in NASA extravehicular activity applications

Computational finite element (FE) models are used in suited astronaut injury risk assessments; however, these models’ verification, validation, and credibility (VV&C) procedures for simulating injuries in altered gravity environments are limited. Our study conducts VV&C assessments of THUMS and Elemance whole-body FE models for predicting suited astronaut injury biomechanics using eight credibility factors, as per NASA-STD-7009A. Credibility factor ordinal scores are assigned by reviewing existing documentation describing VV&C practices, and credibility sufficiency thresholds are assigned based on input from subject matter experts. Our results show the FE models are credible for suited astronaut injury investigation in specific ranges of kinematic and kinetic conditions correlating to highway and contact sports events. Nevertheless, these models are deficient when applied outside these ranges. Several credibility elevation strategies are prescribed to improve models’ credibility for the NASA-centric application domain.

Finite Element↗

DOE OSTI2024

Interlaced Characterization and Calibration (ICC) for Improved Computational Simulation Credibility

Accurate material characterization and model calibration are pivotal for simulations used for high-consequence engineering decisions. Current characterization and calibration methods (1) use simplified test specimen geometries and global data, (2) cannot guarantee that sufficient characterization data is collected for a specific model of interest, (3) provide only mean parameter values with no uncertainty quantification, and (4) are sequential, inflexible, and time-consuming. This work developed a new paradigm—coined Interlaced Characterization and Calibration (ICC)—which drives forward the state-of-the-art in model calibration by bringing together recent advancements into one improved workflow. The ICC paradigm (1) employs tools to efficiently use full-field data to calibrate high-fidelity material models, (2) aligns the data needed with the data collected by adopting an optimal experimental design protocol, (3) provides uncertainty metrics on the calibrated model parameters, and (4) incorporates these advances into a quasi real-time feedback loop. The ICC framework was validated synthetically with both low-fidelity and high-fidelity simulations paired with several different elastoplastic material models, and was also demonstrated experimentally with an aluminum 6061 cruciform exemplar specimen. Results showed that the ICC framework—in which Bayesian optimal experimental design actively guided the experiment— resulted in calibrations with similar or better accuracy than predetermined experiments based on subject matter expertise. Moreover, the ICC framework produced a complete model calibration— with quantified uncertainties on model parameters—in 1 week, a 5 - 10× increase in efficiency over traditional approaches. Thus, the ICC paradigm improves both the calibration process and quality, by (1) improving efficiency, which increases agility of solid mechanics modeling and enables utilization of computational simulation (CompSim) at earlier stages of the design cycle and (2) providing quantified, and in some cases reduced, parameter uncertainties, which increases confidence in model predictions and supports credible decision making.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2009

NASA Standard for Models and Simulations: Credibility Assessment Scale

As one of its many responses to the 2003 Space Shuttle Columbia accident, NASA decided to develop a formal standard for models and simulations (M&S). Work commenced in May 2005. An interim version was issued in late 2006. This interim version underwent considerable revision following an extensive Agency-wide review in 2007 along with some additional revisions as a result of the review by the NASA Engineering Management Board (EMB) in the first half of 2008. Issuance of the revised, permanent version, hereafter referred to as the M&S Standard or just the Standard, occurred in July 2008. Bertch, Zang and Steeleiv provided a summary review of the development process of this standard up through the start of the review by the EMB. A thorough recount of the entire development process, major issues, key decisions, and all review processes are available in Ref. v. This is the second of a pair of papers providing a summary of the final version of the Standard. Its focus is the Credibility Assessment Scale, a key feature of the Standard, including an example of its application to a real-world M&S problem for the James Webb Space Telescope. The companion paper summarizes the overall philosophy of the Standard and an overview of the requirements. Verbatim quotes from the Standard are integrated into the text of this paper, and are indicated by quotation marks.

Babula, Maria↗

NASA NTRS2008

NASA Standard for Models and Simulations: Credibility Assessment Scale

As one of its many responses to the 2003 Space Shuttle Columbia accident, NASA decided to develop a formal standard for models and simulations (M and S)ii. Work commenced in May 2005. An interim version was issued in late 2006. This interim version underwent considerable revision following an extensive Agency-wide review in 2007 along with some additional revisions as a result of the review by the NASA Engineering Management Board (EMB) in the first half of 2008. Issuance of the revised, permanent version,hereafter referred to as the M and S Standard or just the Standard, occurred in July 2008.

verification↗

NASA NTRS2011

Simulation Credibility Scale and Credibility Assessment Scale

No abstract available

Mehta, Unmeel B.↗

NASA NTRSDate not supplied

Credibility Assessment and Evaluation of the Astronaut Musculoskeletal Boot/Ankle Injury Scenario Using Open SIM(TM) Computational Modeling

The musculoskeletal system of astronauts is subject to physiological changes, potentially leading to injuries due to the exposure to different gravitational environments experienced during spaceflight. These injuries can occur while an astronaut is performing an extravehicular activity (EVA) in space, on lunar or planetary surfaces or while wearing a spacesuit during terrestrial training for an EVA. One area of concern are injuries occurring at the feet and ankles due to a poor fit of the spacesuit boot. These injuries can occur if the boot does not support the ankle sufficiently and the astronaut steps awkwardly, twisting the ankle or foot, during an EVA. The OpenSimTM software can assess EVA induced musculoskeletal injuries for muscle strains, ligament injuries and joint injuries. A modeling and simulation credibility assessment of the OpenSim musculoskeletal modeling and simulation procedures, applied to the boot/ankle injury scenario injury mechanisms that may occur while carrying out tasks in these environments, is performed. These modeling and simulation credibility assessments give information on the credibility of the model’s use in simulating foot/ankle related injury mechanisms.

C A Gallo↗

NASA NTRS1998

Credible Software and Simulation Uncertainty

The utility of software primarily depends on its reliability and performance; whereas, its significance depends solely on its credibility for intended use. The credibility of simulations confirms the credibility of software. The level of veracity and the level of validity of simulations determine the degree of credibility of simulations. The process of assessing this credibility in fields such as computational mechanics (CM) differs from that followed by the Defense Modeling and Simulation Office in operations research. Verification and validation (V&V) of CM simulations is not the same as V&V of CM software. Uncertainty is the measure of simulation credibility. Designers who use software are concerned with management of simulation uncertainty. Terminology and concepts are presented with a few examples from computational fluid dynamics.

Mehta, Unmeel B.↗

DOE OSTI2024

Method for Real-Time Salt Aerosol Concentration and Size Measurements for Molten Salt Reactor Safety Assessments

Licensing a new nuclear reactor involves evaluating credible accident scenarios by developing models that simulate accident progression to predict possible outcomes and their impact on safety. The formation of radionuclide-bearing aerosols in the respirable size range can significantly affect dose consequence and threaten human health and is therefore a focus of nuclear reactor safety assessments. Recent reviews of the literature identified a lack of existing experimental data describing the mechanisms of formation and properties (size, concentration, and composition) of radionuclide-bearing aerosols that may be produced from molten salt reactor (MSR) facilities during postulated accident scenarios. Experiments on aerosol formation from molten salt systems are a high priority need that will support MSR licensing by indicating the radiological consequences of aerosol formation and providing the data required for model development and validation. The evolving conditions that occur during a MSR accident, such as a spill of molten fuel salt, may affect aerosol formation mechanisms and aerosol properties over time. There is a need to conduct experiments that simulate credible MSR accidents in a laboratory to generate aerosols with realistic characteristics and behaviors and a need to measure these aerosols in real time under accident-relevant conditions. This report describes the development of a method that can be used to quantify the size and concentration of salt aerosol particles that form from molten salt systems in real time. This method will be employed in future integral effects tests that are conducted at an engineering scale to simulate realistic MSR accidents and in future separate effects tests that will provide mechanistic insight into aerosol formation and properties to support process model development.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

NASA NTRS2024

Expansion of Check-Cases for 6DOF Simulation

This effort expands upon a previous NASA activity that developed flight simulation benchmark check-cases to include new check-cases for the Cislunar domain, comparing multiple NASA simulation tools. The results of this effort describe the benefits of standardizing inputs, simulation comparisons and describe an interactive website that enables comparison of externally provided simulation data. Participating simulations improved their software and identified implementation errors. This activity elevated simulation credibility and provided a measure of validation for the simulations actively in use for NASA’s Human Landing Systems (HLS).

Modeling↗

NASA NTRS2024

Expansion of Check-Cases for 6DOF Simulation: Appendix A

This is the Appendix containing figures of simulation output data plots for comparison from the assessment, “Expansion of Check-Cases for 6DOF Simulation”. This effort expands upon a previous NASA activity that developed flight simulation benchmark check-cases to include new check-cases for the Cislunar domain, comparing multiple NASA simulation tools. The results of this effort describe the benefits of standardizing inputs, simulation comparisons and describe an interactive website that enables comparison of externally provided simulation data. Participating simulations improved their software and identified implementation errors. This activity elevated simulation credibility and provided a measure of validation for the simulations actively in use for NASA’s Human Landing Systems (HLS).

Modeling↗

NASA NTRS2016

Development and Use of Engineering Standards for Computational Fluid Dynamics for Complex Aerospace Systems

Computational fluid dynamics (CFD) and other advanced modeling and simulation (M&S) methods are increasingly relied on for predictive performance, reliability and safety of engineering systems. Analysts, designers, decision makers, and project managers, who must depend on simulation, need practical techniques and methods for assessing simulation credibility. The AIAA Guide for Verification and Validation of Computational Fluid Dynamics Simulations (AIAA G-077-1998 (2002)), originally published in 1998, was the first engineering standards document available to the engineering community for verification and validation (V&V) of simulations. Much progress has been made in these areas since 1998. The AIAA Committee on Standards for CFD is currently updating this Guide to incorporate in it the important developments that have taken place in V&V concepts, methods, and practices, particularly with regard to the broader context of predictive capability and uncertainty quantification (UQ) methods and approaches. This paper will provide an overview of the changes and extensions currently underway to update the AIAA Guide. Specifically, a framework for predictive capability will be described for incorporating a wide range of error and uncertainty sources identified during the modeling, verification, and validation processes, with the goal of estimating the total prediction uncertainty of the simulation. The Guide's goal is to provide a foundation for understanding and addressing major issues and concepts in predictive CFD. However, this Guide will not recommend specific approaches in these areas as the field is rapidly evolving. It is hoped that the guidelines provided in this paper, and explained in more detail in the Guide, will aid in the research, development, and use of CFD in engineering decision-making.

Lee, Hyung B.↗