Search NASA⌕ Search

SEARCH · Search NASA

Results for “Credibility”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Differential credibility assessment for statistical downscaling

Climate science is increasingly using (i) ensembles of climate projections from multiple models derived using different assumptions and/or scenarios and (ii) process-oriented diagnostics of model fidelity. Efforts to assign differential credibility to projections and/or models are also rapidly advancing. A framework to quantify and depict the credibility of statistically downscaled model output is presented and demonstrated. Here, the approach employs transfer functions in the form of robust and resilient generalized linear models applied to downscale daily minimum and maximum temperature anomalies at 10 locations using predictors drawn from ERA-Interim reanalysis and two global climate models (GCM; GFDL-ESM2M and MPI-ESM-LR). The downscaled time series are used to derive several impact relevant CLIMDEX temperature indices that are assigned credibility based on (1) the reproduction of relevant large-scale predictors by the GCMs (i.e. fraction of regression beta-weights derived from predictors that are well-reproduced) and (2) the degree of variance in the observations reproduced in the downscaled series following application of a new variance inflation technique. Credibility of the downscaled predictands varies across locations, between the two GCM and is generally higher for minimum temperature than maximum temperature. The differential credibility assessment framework demonstrated here is easy to use and flexible. It can be applied as is to inform decision makers regarding projection confidence, and/or extended to include other components of the transfer functions, and/or used to weight members of a statistically downscaled ensemble.

54 ENVIRONMENTAL SCIENCES↗

Credible Practice of Models and Simulations for Healthcare

As computational models penetrate clinical care and drive healthcare policy decisions, there is increasing need to communicate that model creation and testing follows reasonable credible practice. Although verification and validation have been generally accepted as significant components of a model’s credibility, they cannot be assumed to equate to a holistic credible practice, which includes activities that can impact comprehension and in-depth examination inherent in the development and reuse of the models. This presentation introduces the Ten Rules for credible practice of modeling and simulation in healthcare developed from a comparative multidisciplinary analysis and practitioner survey. These rules establish a unified conceptual framework for modeling and simulation design, implementation, evaluation, dissemination and usage across the modeling and simulation life-cycle. Although some of these are common sense guidelines, many are often missed or misconstrued, even by seasoned practitioners.

credibility↗

Uncertainty quantification's role in modeling and simulation planning, and credibility assessment through the predictive capability maturity model

The importance of credible, trustworthy numerical simulations is obvious especially when using the results for making high-consequence decisions. Determining the credibility of such numerical predictions is much more difficult and requires a systematic approach to assessing predictive capability, associated uncertainties and overall confidence in the computational simulation process for the intended use of the model. This process begins with an evaluation of the computational modeling of the identified, important physics of the simulation for its intended use. This is commonly done through a Phenomena Identification Ranking Table (PIRT). Then an assessment of the evidence basis supporting the ability to computationally simulate these physics can be performed using various frameworks such as the Predictive Capability Maturity Model (PCMM). There were several critical activities that follow in the areas of code and solution verification, validation and uncertainty quantification, which will be described in detail in the following sections. Here, we introduce the subject matter for general applications but specifics are given for the failure prediction project. In addition, the first task that must be completed in the verification & validation procedure is to perform a credibility assessment to fully understand the requirements and limitations of the current computational simulation capability for the specific application intended use. The PIRT and PCMM are tools used at Sandia National Laboratories (SNL) to provide a consistent manner to perform such an assessment. Ideally, all stakeholders should be represented and contribute to perform an accurate credibility assessment. PIRTs and PCMMs are both described in brief detail below and the resulting assessments for an example project are given.

97 MATHEMATICS AND COMPUTING↗

The aerospace plane design challenge - Credible computational fluid dynamics results

In order to establish the credibility of CFD results utilized in aerospace plane design, the following topics are discussed: CFD validation in relation to 'measureable' fluid dynamics (MFD) validation, credibility requirements, responsibility for credibility, and a guide for establishing credibility. What is of paramount concern for fluid dynamic design is not CFD code validation but qualification of CFD unknowns so that their magnitude is greatly reduced and that these uncertainties are employed for designing with margin. The designers must be trained to properly use CFD if they are to produce good designs. In approximately 70 percent of the flight envelopes of SSTO aerospace planes with supersonic combustion, CFD will be necessary to determine dynamics performance and specifications.

Mehta, Unmeel B.↗

How to Develop and Interpret a Credibility Assessment of Numerical Models for Human Research: NASA-STD-7009 Demystified

In the wake of the Columbia accident, the NASA-STD-7009 [1] credibility assessment was developed as a unifying platform to describe model credibility and the uncertainties in its modeling predictions. This standard is now being adapted by NASAs Human Research Program to cover a wide range of numerical models for human research. When used properly, the standard can improve the process of code development by encouraging the use of best practices. It can also give management more insight in making informed decisions through a better understanding of the models capabilities and limitations.To a newcomer, the abstractions presented in NASA-STD-7009 and the sheer volume of information that must be absorbed can be overwhelming. This talk is aimed at describing the credibility assessment, which is the heart of the standard, in plain terms. It will outline how to develop a credibility assessment under the standard. It will also show how to quickly interpret the graphs and tables that result from the assessment and how to drill down from the top-level view to the foundation of the assessment. Finally, it will highlight some of the resources that are available for further study.

numerical analysis↗

Toward the credibility of Northeast United States summer precipitation projections in CMIP5 and NARCCAP simulations

Abstract Precipitation projections for the northeast United States and nearby Canada (Northeast) are examined for 15 Fifth Phase of the Coupled Model Intercomparison Project (CMIP5) models. A process‐based evaluation of atmospheric circulation features associated with wet Northeast summers is performed to examine whether credibility can be differentiated within the multimodel ensemble. Based on these evaluations, and an analysis of the interannual statistical properties of area‐averaged precipitation, model subsets were formed. Multimodel precipitation projections from each subset were compared to the multimodel projection from all of the models. Higher‐resolution North American Regional Climate Change Assessment Program (NARCCAP) regional climate models (RCMs) were subjected to a similar evaluation, grouping into subsets, and examination of future projections. CMIP5 models adequately simulate most large‐scale circulation features associated with wet Northeast summers, though all have errors in simulating observed sea level pressure and moisture divergence anomalies in the western tropical Atlantic/Gulf of Mexico. Relevant large‐scale processes simulated by the RCMs resemble those of their driving global climate models (GCMs), which are not always realistic. Future RCM studies could benefit from a process analysis of potential driving GCMs prior to dynamical downscaling. No CMIP5 or NARCCAP models were identified as clearly more credible, but six GCMs and four RCMs performed consistently better. Among the “Better” models, there is no consistency in the direction of future summer precipitation change. CMIP5 projections suggest that the Northeast precipitation response depends on the dynamics of the North Atlantic anticyclone and associated circulation and moisture convergence patterns, which vary among “Better” models. Even when model credibility cannot be clearly differentiated, examination of simulated processes provides important insights into their evolution under greenhouse warming.

Thibeault, Jeanne M.↗

Credibility-enhancing displays promote the provision of non-normative public goods

Promoting the adoption of public goods that are not yet widely accepted is particularly challenging. This is because most tools for increasing cooperation—such as reputation concerns and information about social norms—are typically effective only for behaviours that are commonly practiced, or at least generally agreed upon as being desirable. Here we examine how advocates can successfully promote non-normative (that is, rare or unpopular) public goods. We do so by applying the cultural evolutionary theory of credibility-enhancing displays3, which argues that beliefs are spread more effectively by actions than by words alone—because actions provide information about the actor’s true beliefs. Based on this logic, people who themselves engage in a given behaviour will be more effective advocates for that behaviour than people who merely extol its virtues—specifically because engaging in a behaviour credibly signals a belief in its value. As predicted, a field study of a programme that promotes residential solar panel installation in 58 towns in the United States—comprising 1.4 million residents in total—found that community organizers who themselves installed through the programme recruited 62.8% more residents to install solar panels than community organizers who did not. This effect was replicated in three pre-registered randomized survey experiments (total n = 1,805). These experiments also support the theoretical prediction that this effect is specifically driven by subjects’ beliefs about what the community organizer believes about solar panels (that is, second-order beliefs), and demonstrate generalizability to four other highly non-normative behaviours. Our findings shed light on how to spread non-normative prosocial behaviours, offer an empirical demonstration of credibility-enhancing displays and have substantial implications for practitioners and policy-makers.

14 SOLAR ENERGY↗

Toward shell model interactions with credible uncertainties

Background: The nuclear shell model is a powerful framework for predicting nuclear structure observables, but relies on interaction matrix elements fit to experimental data as its inputs. Extending the shell model's applicability, particularly toward dripline nuclei, requires efficient fitting methods and credible uncertainty quantification. Traditional approaches face computational challenges and may underestimate uncertainties. Purpose: We develop and test a framework combining eigenvector continuation and Markov chain Monte Carlo to efficiently fit shell model interaction matrix elements and quantify their uncertainties. Methods: Eigenvector continuation is used to emulate shell model calculations, reducing computational costs. The emulator enables Markov chain Monte Carlo sampling to optimize interaction matrix elements and rigorously assess parametric uncertainties. Here, the framework is benchmarked using the USDB interaction in the 𝑠⁢𝑑 shell. Results: The emulator reproduces the USDB interaction with negligible error, validating its use in shell model fitting applications. However, we find that to obtain credible predictive intervals, the model defect of the shell model itself, rather than experimental or emulator error, must be taken into account in order to obtain credible uncertainties. Conclusions: The proposed framework provides an efficient and rigorous approach for fitting shell model interactions and quantifying uncertainties. Further, the normality assumption used in the past appears sufficient to describe the distribution of interaction matrix elements. However, it is crucial to account for model correlations to avoid underestimating uncertainties.

Nuclear forces↗

What Questions Would a Systems Engineer Ask to Assess Systems Engineering Models as Credible

Digital Systems Engineering strategies typically call for digital Systems Engineering models to be retained in repositories and certified as an authoritative source of truth (enabling model reuse, qualification, and collaboration). In order for digital Systems Engineering models to be certified as authoritative (credible), they need to be assessed - verified and validated - and with the amount of uncertainty in the model quantified (consider reusing someone else's model without knowing the author). Due to this increasing model complexity, the authors assert that traditional human-based methods for validating, verifying, and uncertainty quantification - such as human-based peer-review sessions - cannot sufficiently establish that a digital Systems Engineering model of a complex system is credible. Digital Systems Engineering models of complex systems can contain millions of nodes and edges. The authors assert that this level of detail is beyond the ability of any group of humans - even working for weeks at a time - to discern and catch every minor model infraction. In contrast, computers are highly effective at discerning infractions with massive amounts of information. The authors suggest that a better approach might be to focus the humans at what model patterns should be assessed and enable the computer to assess the massive details in accordance with those patterns - by running through perhaps 100,000 test loops. In anticipation of future projects to implement and automate the assessment of models at Sandia National Laboratories, a study was initiated to elicit input from a group of 25 Systems Engineering experts. The authors positioning query began with - 'What questions would a Systems Engineer ask to assess Systems Engineering models for credibility?" This report documents the results of that survey.

42 ENGINEERING↗

HOLISTIC CREDIBILITY ASSESSMENT OF A MACHINE LEARNING-BASED SURROGATE MODEL IN AN AERODYNAMICS APPLICATION

Elements of ASME V&V 20 credibility assessment methodology are used in tandem with credibility assessment tools from the machine learning community to assess the credibility of a deep neural network based surrogate model. This surrogate model is trained, tested, validated, and employed in the context of aerodynamic coefficient prediction for a NACA 0012 airfoil in subsonic and transonic flow. The parameter space is defined by angle of attack, Reynolds number, and Mach number.

Kirsch, Jared Roelof [Sandia National Laboratories↗

An analysis of possible applications of fuzzy set theory to the actuarial credibility theory

In this work, we review the basic concepts of actuarial credibility theory from the point of view of introducing applications of the fuzzy set-theoretic method. We show how the concept of actuarial credibility can be modeled through the fuzzy set membership functions and how fuzzy set methods, especially fuzzy pattern recognition, can provide an alternative tool for estimating credibility.

Ostaszewski, Krzysztof↗

Credible Computations: Standard and Uncertainty

The discipline of computational fluid dynamics (CFD) is at a crossroad. Most of the significant advances related to computational methods have taken place. The emphasis is now shifting from methods to results. Significant efforts are made in applying CFD to solve design problems. The value of CFD results in design depends on the credibility of computed results for the intended use. The process of establishing credibility requires a standard so that there is a consistency and uniformity in this process and in the interpretation of its outcome. The key element for establishing the credibility is the quantification of uncertainty. This paper presents salient features of a proposed standard and a procedure for determining the uncertainty. A customer of CFD products - computer codes and computed results - expects the following: A computer code in terms of its logic, numerics, and fluid dynamics and the results generated by this code are in compliance with specified requirements. This expectation is fulfilling by verification and validation of these requirements. The verification process assesses whether the problem is solved correctly and the validation process determines whether the right problem is solved. Standards for these processes are recommended. There is always some uncertainty, even if one uses validated models and verified computed results. The value of this uncertainty is important in the design process. This value is obtained by conducting a sensitivity-uncertainty analysis. Sensitivity analysis is generally defined as the procedure for determining the sensitivities of output parameters to input parameters. This analysis is a necessary step in the uncertainty analysis, and the results of this analysis highlight which computed quantities and integrated quantities in computations need to be determined accurately and which quantities do not require such attention. Uncertainty analysis is generally defined as the analysis of the effect of the uncertainties involved in all stages of a process on the final responses. There are two approaches for conducting the uncertainty analysis: experimental and computational. These analyses and approaches are briefly described.

Mehta, Unmeel B.↗

Executive Summary: Special Section on Credible Computational Fluid Dynamics Simulations

This summary presents the motivation for the Special Section on the credibility of computational fluid dynamics (CFD) simulations, its objective, its background and context, its content, and its major conclusions. Verification and validation (V&V) are the processes for establishing the credibility of CFD simulations. Validation assesses whether correct things are performed and verification assesses whether they are performed correctly. Various aspects of V&V are discussed. Progress is made in verification of simulation models. Considerable effort is still needed for developing a systematic validation method that can assess the credibility of simulated reality.

Mehta, Unmeel B.↗

Towards a Credibility Assessment of Models and Simulations

A scale is presented to evaluate the rigor of modeling and simulation (M&S) practices for the purpose of supporting a credibility assessment of the M&S results. The scale distinguishes required and achieved levels of rigor for a set of M&S elements that contribute to credibility including both technical and process measures. The work has its origins in an interest within NASA to include a Credibility Assessment Scale in development of a NASA standard for models and simulations.

Blattnig, Steve R.↗

Credibility Assessment of Deterministic Computational Models and Simulations for Space Biomedical Research and Operations

Human missions beyond low earth orbit to destinations, such as to Mars and asteroids will expose astronauts to novel operational conditions that may pose health risks that are currently not well understood and perhaps unanticipated. In addition, there are limited clinical and research data to inform development and implementation of health risk countermeasures for these missions. Consequently, NASA's Digital Astronaut Project (DAP) is working to develop and implement computational models and simulations (M&S) to help predict and assess spaceflight health and performance risks, and enhance countermeasure development. In order to effectively accomplish these goals, the DAP evaluates its models and simulations via a rigorous verification, validation and credibility assessment process to ensure that the computational tools are sufficiently reliable to both inform research intended to mitigate potential risk as well as guide countermeasure development. In doing so, DAP works closely with end-users, such as space life science researchers, to establish appropriate M&S credibility thresholds. We will present and demonstrate the process the DAP uses to vet computational M&S for space biomedical analysis using real M&S examples. We will also provide recommendations on how the larger space biomedical community can employ these concepts to enhance the credibility of their M&S codes.

Mulugeta, Lealem↗

Integrated Medical Model Verification, Validation, and Credibility

The Integrated Medical Model (IMM) was designed to forecast relative changes for a specified set of crew health and mission success risk metrics by using a probabilistic (stochastic process) model based on historical data, cohort data, and subject matter expert opinion. A probabilistic approach is taken since exact (deterministic) results would not appropriately reflect the uncertainty in the IMM inputs. Once the IMM was conceptualized, a plan was needed to rigorously assess input information, framework and code, and output results of the IMM, and ensure that end user requests and requirements were considered during all stages of model development and implementation. METHODS: In 2008, the IMM team developed a comprehensive verification and validation (VV) plan, which specified internal and external review criteria encompassing 1) verification of data and IMM structure to ensure proper implementation of the IMM, 2) several validation techniques to confirm that the simulation capability of the IMM appropriately represents occurrences and consequences of medical conditions during space missions, and 3) credibility processes to develop user confidence in the information derived from the IMM. When the NASA-STD-7009 (7009) was published, the IMM team updated their verification, validation, and credibility (VVC) project plan to meet 7009 requirements and include 7009 tools in reporting VVC status of the IMM. RESULTS: IMM VVC updates are compiled recurrently and include 7009 Compliance and Credibility matrices, IMM VV Plan status, and a synopsis of any changes or updates to the IMM during the reporting period. Reporting tools have evolved over the lifetime of the IMM project to better communicate VVC status. This has included refining original 7009 methodology with augmentation from the NASA-STD-7009 Guidance Document. End user requests and requirements are being satisfied as evidenced by ISS Program acceptance of IMM risk forecasts, transition to an operational model and simulation tool, and completion of service requests from a broad end user consortium including Operations, Science and Technology Planning, and Exploration Planning. CONCLUSIONS: The VVC approach established by the IMM project of combining the IMM VV Plan with 7009 requirements is comprehensive and includes the involvement of end users at every stage in IMM evolution. Methods and techniques used to quantify the VVC status of the IMM have not only received approval from the local NASA community but have also garnered recognition by other federal agencies seeking to develop similar guidelines in the medical modeling community.

Validation↗

Integrated Medical Model (IMM) Project Verification, Validation, and Credibility (VVandC)

The Integrated Medical Model (IMM) Project supports end user requests by employing the Integrated Medical Evidence Database (iMED) and IMM tools as well as subject matter expertise within the Project. The iMED houses data used by the IMM. The IMM is designed to forecast relative changes for a specified set of crew health and mission success risk metrics by using a probabilistic model based on historical data, cohort data, and subject matter expert opinion. A stochastic approach is taken because deterministic results would not appropriately reflect the uncertainty in the IMM inputs. Once the IMM was conceptualized, a plan was needed to rigorously assess input information, framework and code, and output results of the IMM, and ensure that end user requests and requirements were considered during all stages of model development and implementation, as well as lay the foundation for external review and application. METHODS: In 2008, the Project team developed a comprehensive verification and validation (VV) plan, which specified internal and external review criteria encompassing 1) verification of data and IMM structure to ensure proper implementation of the IMM, 2) several validation techniques to confirm that the simulation capability of the IMM appropriately represents occurrences and consequences of medical conditions during space missions, and 3) credibility processes to develop user confidence in the information derived from the IMM. When the NASA-STD-7009 (7009) [1] was published, the Project team updated their verification, validation, and credibility (VVC) project plan to meet 7009 requirements and include 7009 tools in reporting VVC status of the IMM. Construction of these tools included meeting documentation and evidence requirements sufficient to meet external review success criteria. RESULTS: IMM Project VVC updates are compiled recurrently and include updates to the 7009 Compliance and Credibility matrices. Reporting tools have evolved over the lifetime of the IMM Project to better communicate VVC status. This has included refining original 7009 methodology with augmentation from the HRP NASA-STD-7009 Guidance Document working group and the NASA-HDBK-7009 [2]. End user requests and requirements are being satisfied as evidenced by ISS Program acceptance of IMM risk forecasts, transition to an operational model and simulation tool, and completion of service requests from a broad end user consortium including operations, science and technology planning, and exploration planning. IMM v4.0 is slated for operational release in the FY015 and current VVC assessments illustrate the expected VVC status prior to the completion of customer lead external review efforts. CONCLUSIONS: The VVC approach established by the IMM Project of incorporating Project-specific recommended practices and guidelines for implementing the 7009 requirements is comprehensive and includes the involvement of end users at every stage in IMM evolution. Methods and techniques used to quantify the VVC status of the IMM Project represented a critical communication tool in providing clear and concise suitability assessments to IMM customers. These processes have not only received approval from the local NASA community but have also garnered recognition by other federal agencies seeking to develop similar guidelines in the medical modeling community.

Walton, M.↗

Credibility Assessment and Evaluation of the Astronaut Musculoskeletal Boot/Ankle Injury Scenario Using Open SIM(TM) Computational Modeling

The musculoskeletal system of astronauts is subject to physiological changes, potentially leading to injuries due to the exposure to different gravitational environments experienced during spaceflight. These injuries can occur while an astronaut is performing an extravehicular activity (EVA) in space, on lunar or planetary surfaces or while wearing a spacesuit during terrestrial training for an EVA. One area of concern are injuries occurring at the feet and ankles due to a poor fit of the spacesuit boot. These injuries can occur if the boot does not support the ankle sufficiently and the astronaut steps awkwardly, twisting the ankle or foot, during an EVA. The OpenSimTM software can assess EVA induced musculoskeletal injuries for muscle strains, ligament injuries and joint injuries. A modeling and simulation credibility assessment of the OpenSim musculoskeletal modeling and simulation procedures, applied to the boot/ankle injury scenario injury mechanisms that may occur while carrying out tasks in these environments, is performed. These modeling and simulation credibility assessments give information on the credibility of the model’s use in simulating foot/ankle related injury mechanisms.

C A Gallo↗