Search NASA⌕ Search

SEARCH · Search NASA

Results for “model credibility”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

The atmospheric effects of stratospheric aircraft. Report of the 1992 Models and Measurements Workshop. Volume 3: Special diagnostic studies

This Workshop on Stratospheric Models and Measurements (M&M) marks a significant expansion in the history of model intercomparisons. It provides a foundation for establishing the credibility of stratospheric models used in environmental assessments of chlorofluorocarbons, aircraft emissions, and climate-chemistry interactions. The core of the M&M comparisons involves the selection of observations of the current stratosphere (i.e., within the last 15 years): these data are believed to be accurate and representative of certain aspects of stratospheric chemistry and dynamics that the models should be able to simulate.

Prather, Michael J.↗

HOLISTIC CREDIBILITY ASSESSMENT OF A MACHINE LEARNING-BASED SURROGATE MODEL IN AN AERODYNAMICS APPLICATION

Elements of ASME V&V 20 credibility assessment methodology are used in tandem with credibility assessment tools from the machine learning community to assess the credibility of a deep neural network based surrogate model. This surrogate model is trained, tested, validated, and employed in the context of aerodynamic coefficient prediction for a NACA 0012 airfoil in subsonic and transonic flow. The parameter space is defined by angle of attack, Reynolds number, and Mach number.

Kirsch, Jared Roelof [Sandia National Laboratories↗

Future precipitation increase constrained by climatological pattern of cloud effect

The fractional increase in global mean precipitation (?P ¯/P ¯) is a first-order measure of the hydrological cycle intensification under anthropogenic warming. However, ?P ¯/P ¯ varies by a factor of more than three among model projections, hindering credible assessments of the associated climate impacts. The uncertainty in ?P ¯/P ¯ stems from uncertainty in both hydrological sensitivity (global mean precipitation increase per unit warming) and climate sensitivity (global mean temperature increase per forcing). Here, by investigating hydrological and climate sensitivities in a unified surface-energy-balance perspective, we find that both sensitivities are significantly correlated with surface shortwave cloud feedback, which is further linked to the climatological pattern of cloud shortwave effect. The observed pattern of cloud effect thus constrains both sensitivities and consequently constrains ?P ¯/P ¯. The 5%-95% uncertainty range of ?P ¯/P ¯ from 1979-2005 to 2080-2100 under the high-emission (moderate-emission) scenario is constrained from 6.34±3.53% (4.19±2.28%) in the raw ensemble-model projection to 7.03±2.59% (4.63±1.71%). The constraint thus suggests a higher most-likely ?P ¯/P ¯ and reduces the uncertainty by ~25%, providing valuable information for impact assessments.

Zhou, Wenyu↗

Physical Review Journal Club: Structural uncertainty assessment for fire-engulfed objects in crosswind: Establishing credibility for a multiphysics wall-modeled large-eddy simulation paradigm

This is an invited presentation for APS's monthly Journal Club where important papers that have recently been accepted for publication are highlighted in a seminar-like setting. The presentation will be "live" (May 1, 2025) and streamed across the world (including an archived recording).

Domino, Stefan Paul [Sandia National Laboratories ↗

Simplified ISCCP Cloud Regimes for Evaluating Cloudiness in CMIP5 Models

We take advantage of the ISCCP simulator implemented in many CMIP5 models, in order to introduce a credible method to compare model cloud output with corresponding ISCCP observations based on the cloud regime (CR) concept. A novel simplified CR derivation method is introduced that exploits the co-variations of three variables, cloud optical thickness, cloud top pressure and cloud fraction (, pc, CF). Following evaluation criteria established in a previous paper of ours, we assess model cloud simulation performance based on how well the simplified CRs are simulated in terms of similarity of centroids, global values and map correlations of relative frequency-of-occurrence (RFO), and long term total clouds amounts (as opposed to average instantaneous cloud fractions). Mirroring prior results, modeled clouds tend to be too thick optically and not as extensive as in observations. Averaging across models does not help in this or other evaluation aspects, except for cloud geographical locations. A good model for one criterion is usually found to be also a good performer for other criteria despite plenty of exceptions for the criterion of centroid similarity. On the other hand, CRs that include convectively generated high-altitude clouds are not as well simulated here compared to the previous study, but other regimes containing low clouds with high CF improve. Models that have previously performed well distinguish themselves again here, but improvements from other models are also found. The simplified cloud evaluation method thus proves to be more forgiving while still good enough to reveal model weaknesses.

global climate models↗

Evaluation of GEOS-5 Sulfur Dioxide Simulations During the Frostburg, MD 2010 Field Campaign.

Sulfur dioxide (SO2) is a major atmospheric pollutant with a strong anthropogenic component mostly produced by the combustion of fossil fuel and other industrial activities. As a precursor of sulfate aerosols that affect climate, air quality, and human health, this gas needs to be monitored on a global scale. Global climate and chemistry models including aerosol processes along with their radiative effects are important tools for climate and air quality research. Validation of these models against in-situ and satellite measurements is essential to ascertain the credibility of these models and to guide model improvements. In this study the Goddard Chemistry, Aerosol, Radiation, and Transport (GOCART) module running on-line inside the Goddard Earth Observing System version 5 (GEOS-5) model is used to simulate aerosol and SO2 concentrations. Data taken in November 2010 over Frostburg, Maryland during an SO2 field campaign involving ground instrumentation and aircraft are used to evaluate GEOS-5 simulated SO2 concentrations. Preliminary data analysis indicated the model overestimated surface SO2 concentration, which motivated the examination of mixing processes in the model and the specification of SO2 anthropogenic emission rates. As a result of this analysis, a revision of anthropogenic emission inventories in GEOS-5 was implemented, and the vertical placement of SO2 sources was updated. Results show that these revisions improve the model agreement with observations locally and in regions outside the area of this field campaign. In particular, we use the ground-based measurements collected by the United States Environmental Protection Agency (US EPA) for the year 2010 to evaluate the revised model simulations over North America.

Air Quality Campaign↗

Adopting Code Verification Methodology Based on Model Form

Code verification is an essential part of credibility analysis for computational models. It assesses whether the mathematical model is implemented correctly into the code and whether the numerical methods behave consistently, and is done before solution verification and validation. Robust guidance for code verification exists in the literature. However, there is no known, concise guide for selecting the approach based on the model form that also presents an overview of the common elements. This document was written to address this gap as an accessible reference for beginning a code-verification effort.

97 MATHEMATICS AND COMPUTING↗

Adapting Code Verification Methodology to Model Form

Code verification is an essential part of credibility analysis for computational models. It assesses whether the mathematical model is implemented correctly into the code and whether the numerical methods behave consistently, and is done before solution verification and validation. Robust guidance for code verification exists in the literature. However, there is no known, concise guide for selecting the approach based on the model form that also presents an overview of the common elements. This document was written to address this gap as an accessible reference for beginning a code-verification effort.

97 MATHEMATICS AND COMPUTING↗

White paper on Verification and Validation for Cyber Emulation Models

All disciplines that use models to predict the behavior of real-world systems need to determine the accuracy of the models’ results. Techniques for verification, validation, and uncertainty quantification (VVUQ) focus on improving the credibility of computational models and assessing their predictive capability. VVUQ emphasizes rigorous evaluation of models and how they are applied to improve understanding of model limitations and quantify the accuracy of model predictions.

97 MATHEMATICS AND COMPUTING↗

Photometric redshift uncertainties in weak gravitational lensing shear analysis: models and marginalization

ABSTRACT Recovering credible cosmological parameter constraints in a weak lensing shear analysis requires an accurate model that can be used to marginalize over nuisance parameters describing potential sources of systematic uncertainty, such as the uncertainties on the sample redshift distribution n(z). Due to the challenge of running Markov chain Monte Carlo (MCMC) in the high-dimensional parameter spaces in which the n(z) uncertainties may be parametrized, it is common practice to simplify the n(z) parametrization or combine MCMC chains that each have a fixed n(z) resampled from the n(z) uncertainties. In this work, we propose a statistically principled Bayesian resampling approach for marginalizing over the n(z) uncertainty using multiple MCMC chains. We self-consistently compare the new method to existing ones from the literature in the context of a forecasted cosmic shear analysis for the HSC three-year shape catalogue, and find that these methods recover statistically consistent error bars for the cosmological parameter constraints for predicted HSC three-year analysis, implying that using the most computationally efficient of the approaches is appropriate. However, we find that for data sets with the constraining power of the full HSC survey data set (and, by implication, those upcoming surveys with even tighter constraints), the choice of method for marginalizing over n(z) uncertainty among the several methods from the literature may modify the 1σ uncertainties on Ωm–S8 constraints by ∼4 per cent, and a careful model selection is needed to ensure credible parameter intervals.

Zhang, Tianqing (ORCID:000000025596198X)↗

Building a Database for a Quantitative Model

A database can greatly benefit a quantitative analysis. The defining characteristic of a quantitative risk, or reliability, model is the use of failure estimate data. Models can easily contain a thousand Basic Events, relying on hundreds of individual data sources. Obviously, entering so much data by hand will eventually lead to errors. Not so obviously entering data this way does not aid linking the Basic Events to the data sources. The best way to organize large amounts of data on a computer is with a database. But a model does not require a large, enterprise-level database with dedicated developers and administrators. A database built in Excel can be quite sufficient. A simple spreadsheet database can link every Basic Event to the individual data source selected for them. This database can also contain the manipulations appropriate for how the data is used in the model. These manipulations include stressing factors based on use and maintenance cycles, dormancy, unique failure modes, the modeling of multiple items as a single "Super component" Basic Event, and Bayesian Updating based on flight and testing experience. A simple, unique metadata field in both the model and database provides a link from any Basic Event in the model to its data source and all relevant calculations. The credibility for the entire model often rests on the credibility and traceability of the data.

Kahn, C. Joseph↗

HRP's Healthcare Spin-Offs Through Computational Modeling and Simulation Practice Methodologies

Spaceflight missions expose astronauts to novel operational and environmental conditions that pose health risks that are currently not well understood, and perhaps unanticipated. Furthermore, given the limited number of humans that have flown in long duration missions and beyond low Earth-orbit, the amount of research and clinical data necessary to predict and mitigate these health and performance risks are limited. Consequently, NASA's Human Research Program (HRP) conducts research and develops advanced methods and tools to predict, assess, and mitigate potential hazards to the health of astronauts. In this light, NASA has explored the possibility of leveraging computational modeling since the 1970s as a means to elucidate the physiologic risks of spaceflight and develop countermeasures. Since that time, substantial progress has been realized in this arena through a number of HRP funded activates such as the Digital Astronaut Project (DAP) and the Integrated Medical Model (IMM). Much of this success can be attributed to HRP's endeavor to establish rigorous verification, validation, and credibility (VV&C) processes that ensure computational models and simulations (M&S) are sufficiently credible to address issues within their intended scope. This presentation summarizes HRP's activities in credibility of modeling and simulation, in particular through its outreach to the community of modeling and simulation practitioners. METHODS: The HRP requires all M&S that can have moderate to high impact on crew health or mission success must be vetted in accordance to NASA Standard for Models and Simulations, NASA-STD-7009 (7009) [5]. As this standard mostly focuses on engineering systems, the IMM and DAP have invested substantial efforts to adapt the processes established in this standard for their application to biological M&S, which is more prevalent in human health and performance (HHP) and space biomedical research and operations [6,7]. These methods have also generated substantial interest by the broader medical community though institutions like the National Institutes of Health (NIH) and the Food and Drug Administration (FDA) to develop similar standards and guidelines applicable to the larger medical operations and research community. DISCUSSION: Similar to NASA, many leading government agencies, health institutions and medical product developers around the world are recognizing the potential of computational M&S to support clinical research and decision making. In this light, substantial investments are being made in computational medicine and notable discoveries are being realized [8]. However, there is a lack of broadly applicable practice guidance for the development and implementation of M&S in clinical care and research in a manner that instills confidence among medical practitioners and biological researchers [9,10]. In this presentation, we will give an overview on how HRP is working with the NIH's Interagency Modeling and Analysis Group (IMAG), the FDA and the American Society of Mechanical Engineers (ASME) to leverage NASA's biomedical VV&C processes to establish a new regulatory standard for Verification and Validation in Computational Modeling of Medical Devices, and Guidelines for Credible Practice of Computational Modeling and Simulation in Healthcare.

Mulugeta, Lealem↗

Multi-fidelity thermal modeling of laser powder bed additive manufacturing

Laser powder bed fusion (LPBF) Additive manufacturing (AM) has attracted interest as an agile method of building production metal parts to reduce design-build-test cycle times for systems. However, predicting part performance is difficult due to inherent process variabilities. This makes qualification challenging. Computational process models have attempted to address some of these challenges, including mesoscale, full physics models and reduced fidelity conduction models. The goal of this work is credible multi-fidelity modeling of the LPBF process by investigating methods for estimating the error between models of two different fidelities. Two methods of error estimation are investigated, adjoint-based error estimation and Bayesian calibration. Adjoint-based error estimation is found to effectively bounding the error between the two models, but with very conservative bounds, making predictions highly uncertain. Bayesian parameter calibration applied to conduction model heat source parameters is found to effectively bound the observed error between the models for melt pool morphology quantities of interest. However, the calibrations do not effectively bound the error in heat distribution.

36 MATERIALS SCIENCE↗

Manufactured solutions for an electromagnetic slot model

The accurate modeling of electromagnetic penetration is an important topic in computational electromagnetics. Electromagnetic penetration occurs through intentional or inadvertent openings in an otherwise closed electromagnetic scatterer, which prevent the contents from being fully shielded from external fields. To efficiently model electromagnetic penetration, aperture or slot models can be used with surface integral equations to solve Maxwell's equations. A necessary step towards establishing the credibility of these models is to assess the correctness of the implementation of the underlying numerical methods through code verification. Surface integral equations and slot models yield multiple interacting sources of numerical error and other challenges, which render traditional code-verification approaches ineffective. In this paper, we provide approaches to separately measure the numerical errors arising from these different error sources for the method-of-moments implementation of the electric-field integral equation with a slot model. Finally, we demonstrate the effectiveness of these approaches for a variety of cases.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

An Overview of NASA's IM&S Verification and Validation Process Plan and Specification for Space Exploration

NASA's Exploration Systems Mission Directorate (ESMD) is implementing a management approach for modeling and simulation (M&S) that will provide decision-makers information on the model's fidelity, credibility, and quality. This information will allow the decision-maker to understand the risks involved in using a model's results in the decision-making process. This presentation will discuss NASA's approach for verification and validation (V&V) of its models or simulations supporting space exploration. This presentation will describe NASA's V&V process and the associated M&S verification and validation (V&V) activities required to support the decision-making process. The M&S V&V Plan and V&V Report templates for ESMD will also be illustrated.

Gravitz, Robert M.↗

NASA's M and S Accreditation Process Plan and Specification for Space Exploration

NASA's Exploration Systems Mission Directorate (ESMD) is implementing a management approach for modeling and simulation (M&S) that will provide decision-makers information on the model's fidelity, credibility, and quality. This information will allow the decision-maker to understand the risks involved in using a model's results in the decision-making process. This presentation will discuss NASA's overall approach to achieving formal accreditation of its models or simulations supporting space exploration. The development of a formal Accreditation Plan is a key component in the preliminary activities for modeling and simulation (M&S) assessment. This presentation will describe NASA's process for identifying risks associated with M&S use and the associated M&S assessments that will dictate the level of data certification and M&S verification and validation (V&V) activities required to support the decision-making process. The M&S Accreditation Plan and Report templates for ESMD will also be illustrated.

O'Neil, David↗

NASA Handbook for Models and Simulations: An Implementation Guide for NASA-STD-7009

The purpose of this Handbook is to provide technical information, clarification, examples, processes, and techniques to help institute good modeling and simulation practices in the National Aeronautics and Space Administration (NASA). As a companion guide to NASA-STD- 7009, Standard for Models and Simulations, this Handbook provides a broader scope of information than may be included in a Standard and promotes good practices in the production, use, and consumption of NASA modeling and simulation products. NASA-STD-7009 specifies what a modeling and simulation activity shall or should do (in the requirements) but does not prescribe how the requirements are to be met, which varies with the specific engineering discipline, or who is responsible for complying with the requirements, which depends on the size and type of project. A guidance document, which is not constrained by the requirements of a Standard, is better suited to address these additional aspects and provide necessary clarification. This Handbook stems from the Space Shuttle Columbia Accident Investigation (2003), which called for Agency-wide improvements in the "development, documentation, and operation of models and simulations"' that subsequently elicited additional guidance from the NASA Office of the Chief Engineer to include "a standard method to assess the credibility of the models and simulations."2 General methods applicable across the broad spectrum of model and simulation (M&S) disciplines were sought to help guide the modeling and simulation processes within NASA and to provide for consistent reporting ofM&S activities and analysis results. From this, the standardized process for the M&S activity was developed. The major contents of this Handbook are the implementation details of the general M&S requirements ofNASA-STD-7009, including explanations, examples, and suggestions for improving the credibility assessment of an M&S-based analysis.

Steele, Martin J.↗