Search NASA⌕ Search

SEARCH · Search NASA

Results for “model credibility”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

A ``Tail" of Two Cities: Fatality-based Modeling of COVID-19 Evolution in New York City and Cook County, IL

I describe $SIR$ modeling of the COVID-19 pandemic in two U.S. urban environments, New York City (NYC) and Cook County, IL, from onset through the month of June, 2020. Since testing was not widespread early in the pandemic in the U.S., I do not use data on confirmed cases and rely solely on public fatality data to estimate model parameters. Fits to the first 20 days of data determine a degenerate combination of the basic reproduction number, $R_0$, and the mean time to removal from the infectious population, $\gamma^{-1}$, with $\gamma (R_0-1) = 0.25 (0.21)$ inverse days for NYC (Cook County). Equivalently, the initial doubling time was $t_d = 2.8 (3.4)$ days for NYC (Cook). The early fatality data suggest that both locations had infections in early February. I model the mitigation measures implemented in mid-March in both locations (distancing, quarantine, isolation, etc) via a time-dependent reproduction number $R_t$ that declines monotonically from $R_0$ to a sma ller asymptotic value, with a parameterized functional form. The timing (mid-March) and duration (several days) of the transitions in $R_t$ appear well determined by the data. However, the fatality data determine only a degenerate combination of the parameters $R_0$, the percentage reduction in social contact due to mitigation measures, $X$, and the infection fatality rate (IFR), $f$. With flat priors, based on simulations the NYC model parameters have 95.45\% credible intervals of $R_0 = 3.0-5.4$, $X= 80-99.9\%$ and $f=2-6\%$, with $5-13\%$ of the population asymptotically infected. A strong external prior indicating a lower value of $f$ or of $\gamma^{-1}$ would imply lower values of $R_0$ and $X$ and higher percentage infection of the population. For Cook County, the evolution was qualitatively different: after mitigation measures were implemented, the daily fatality counts reached a plateau for about a month before tailing off. This is consistent with an $SIR$ model that exhibits ``critical slowing-down", in which $R_t$ plateaus ! at a value just above unity. For Cook County, the 95.45\% credible intervals for the model parameters are much broader and shifted downward, $R_0=1.4-4.7$, $X=26-54\%$, and $f=0.1-0.6\%$ with $15-88\%$ of the population asymptotically infected. Despite the apparently lower efficacy of its social contact reduction measures, Cook County has had significantly fewer fatalities per population than NYC, $D_\infty/N=100$ vs. 270 per 100,000. In the model, this is attributed to the lower inferred IFR for Cook; an external prior pointing to similar values of the IFR for the two locations would instead chalk up the difference in $D/N$ to differences in the relative growth rate of the disease. I derive a model-dependent threshold, $X^{crit}_e$, for `safe' re-opening, that is, for easing of contact reduction that would not trigger a second wave; for NYC, the models predict that increasing social contact by more than 20\% from post-mitigation levels will lead to renewed spread, while for Cook County the threshold value is very uncertain, given the parameter degeneracies. The timing of 2nd-wave growth will depend on the amplitude of contact increase relative to $X^{crit}_e$ and on the asymptotic growth rate, and the impact in terms of fatalities will depend on the parameter $f$.

Frieman, Joshua↗
DOE OSTI2026

Polywell Revisited

The Polywell fusion concept, originally proposed by Robert W. Bussard in 1985, has been investigated for over four decades as a potential solution for achieving net fusion energy in a compact and economically viable reactor. It combines two distinct approaches: high-beta magnetic cusp confinement of electrons using polyhedral coil configurations and electrostatic ion confinement via a potential well formed by injected electron beams. While the hybrid nature of the Polywell system offers advantages in plasma stability and engineering simplicity, previous efforts have been limited by persistent challenges in achieving sufficient plasma confinement required to generate a net energy gain. Here, in this study, we examine previous works and identify limitations of several Polywell embodiments that have historically impeded progress. We present an updated Polywell physics model incorporating experimental findings and recent first-principles particle-in-cell simulations. This updated model outlines a credible path toward overcoming confinement losses and achieving net energy gain using deuterium-tritium (D-T) fuels. Our findings provide a renewed scientific basis for the continued development of the Polywell fusion concept as a practical and scalable approach to fusion energy.

Electrostatic potential well↗
DOE OSTI2025

Robust Solution Verification Experiments on Nonuniform Meshes

The activities of verification, validation, and uncertainty quantification (VVUQ) provide a comprehensive means to assess the credibility of computational models. Within VVUQ, solution verification assesses numerical errors and evaluates whether the simulation is sufficiently accurate for its intended applications. As computational modeling gains traction in the development of complex, high-consequence systems, the need for robust solution verification intensifies, particularly because experimental data for these systems are often limited. This work examines improvements in the robustness of Richardson extrapolation (RE), a method commonly used in solution verification to study the discretization error of computational models using a power law. Nonuniform mesh refinement is discussed alongside other pollutants that affect the robustness of the power law model. Maximum likelihood estimation (MLE) is proposed as a robust strategy to address the uncertainty generated by nonuniform mesh refinement. An exploratory computational fluid dynamics (CFD) study of a 2D planar Poiseuille flow is conducted to determine if nonuniform mesh noise can be modeled with this MLE approach for more robust RE.

Weinmeister, Justin [ORNL] (ORCID:0000000160090237↗
NASA NTRS1986

F-106 data summary and model results relative to threat criteria and protection design analysis

The NASA F-106 has acquired considerable data on the rates-of-change of electromagnetic parameters on the aircraft surface during 690 direct lightning strikes while penetrating thunderstorms at altitudes ranging from 15,000 to 40,000 feet. These in-situ measurements have provided the basis for the first statistical quantification of the lightning electromagnetic threat to aircrat appropriate for determining lightning indirect effects on aircraft. The data are presently being used in updating previous lightning criteria and standards developed over the years from ground-based measurements. The new lightning standards will, therefore, be the first which reflect actual aircraft responses measured at flight altitudes. The modeling technique developed to interpret and understand the direct strike electromagnetic data acquired on the F-106 provides a means to model the interaction of the lightning channel with the F-106. The reasonable results obtained with the model, compared to measured responses, yield confidence that the model may be credibly applied to other aircraft types and uses in the prediction of internal coupling effects in the design of lightning protection for new aircraft.

Pitts, F. L.↗
NASA NTRS1987

New methods and results for quantification of lightning-aircraft electrodynamics

The NASA F-106 collected data on the rates of change of electromagnetic parameters on the aircraft surface during over 700 direct lightning strikes while penetrating thunderstorms at altitudes from 15,000 t0 40,000 ft (4,570 to 12,190 m). These in situ measurements provided the basis for the first statistical quantification of the lightning electromagnetic threat to aircraft appropriate for determining indirect lightning effects on aircraft. These data are used to update previous lightning criteria and standards developed over the years from ground-based measurements. The proposed standards will be the first which reflect actual aircraft responses measured at flight altitudes. Nonparametric maximum likelihood estimates of the distribution of the peak electromagnetic rates of change for consideration in the new standards are obtained based on peak recorder data for multiple-strike flights. The linear and nonlinear modeling techniques developed provide means to interpret and understand the direct-strike electromagnetic data acquired on the F-106. The reasonable results obtained with the models, compared with measured responses, provide increased confidence that the models may be credibly applied to other aircraft.

Pitts, Felix L.↗
NASA NTRS2008

A Capable and Temporary Test Facility on a Shoestring Budget: The MSL Touchdown Test Facility

The Mars Science Laboratory mission (MSL) has undertaken a developmental Touchdown Test Program that utilizes a full-scale rover vehicle and an overhead winch system to replicate the skycrane landing event. Landing surfaces consisting of flat and sloped granular media, planar, rigid surfaces, and various combinations of rocks and slopes were studied. Information gathered from these tests was vital for validating the rover analytical model, validating certain design or system behavior assumptions, and for exploring events and phenomenon that are either very difficult or too costly to model in a credible way. This paper describes this test program, with a focus on the creation of test facility, daily test operations, and some of the challenges faced and lessons learned along the way.

rover↗
NASA NTRS2014

NASA-STD-7009 Guidance Document for Human Health and Performance Models and Simulations

Rigorous verification, validation, and credibility (VVC) processes are imperative to ensure that models and simulations (MS) are sufficiently reliable to address issues within their intended scope. The NASA standard for MS, NASA-STD-7009 (7009) [1] was a resultant outcome of the Columbia Accident Investigation Board (CAIB) to ensure MS are developed, applied, and interpreted appropriately for making decisions that may impact crew or mission safety. Because the 7009 focus is engineering systems, a NASA-STD-7009 Guidance Document is being developed to augment the 7009 and provide information, tools, and techniques applicable to the probabilistic and deterministic biological MS more prevalent in human health and performance (HHP) and space biomedical research and operations.

numerical analysis↗
NASA NTRS2017

The Cloud Feedback Model Intercomparison Project (CFMIP) contribution to CMIP6.

The primary objective of CFMIP is to inform future assessments of cloud feedbacks through improved understanding of cloud-climate feedback mechanisms and better evaluation of cloud processes and cloud feedbacks in climate models. However, the CFMIP approach is also increasingly being used to understand other aspects of climate change, and so a second objective has now been introduced, to improve understanding of circulation, regional-scale precipitation, and non-linear changes. CFMIP is supporting ongoing model inter-comparison activities by coordinating a hierarchy of targeted experiments for CMIP6, along with a set of cloud-related output diagnostics. CFMIP contributes primarily to addressing the CMIP6 questions 'How does the Earth system respond to forcing?' and 'What are the origins and consequences of systematic model biases?' and supports the activities of the WCRP Grand Challenge on Clouds, Circulation and Climate Sensitivity. A compact set of Tier 1 experiments is proposed for CMIP6 to address this question: (1) what are the physical mechanisms underlying the range of cloud feedbacks and cloud adjustments predicted by climate models, and which models have the most credible cloud feedbacks? Additional Tier 2 experiments are proposed to address the following questions. (2) Are cloud feedbacks consistent for climate cooling and warming, and if not, why? (3) How do cloud-radiative effects impact the structure, the strength and the variability of the general atmospheric circulation in present and future climates? (4) How do responses in the climate system due to changes in solar forcing differ from changes due to CO2, and is the response sensitive to the sign of the forcing? (5) To what extent is regional climate change per CO2 doubling state-dependent (non-linear), and why? (6) Are climate feedbacks during the 20th century different to those acting on long-term climate change and climate sensitivity? (7) How do regional climate responses (e.g. in precipitation) and their uncertainties in coupled models arise from the combination of different aspects of CO2 forcing and sea surface warming? CFMIP also proposes a number of additional model outputs in the CMIP DECK, CMIP6 Historical and CMIP6 CFMIP experiments, including COSP simulator outputs and process diagnostics to address the following questions. 1. How well do clouds and other relevant variables simulated by models agree with observations? 2. What physical processes and mechanisms are important for a credible simulation of clouds, cloud feedbacks and cloud adjustments in climate models? 3. Which models have the most credible representations of processes relevant to the simulation of clouds? 4. How do clouds and their changes interact with other elements of the climate system?

carbon dioxide↗
DOE OSTI2023

Test and Evaluation of Systems with Embedded Machine Learning Components

As Machine Learning (ML) continues to advance, it is being integrated into more systems. Often, the ML component represents a significant portion of the system that reduces the burden on the end user or significantly improves task performance. However, the ML component represents an unknown complex phenomenon that is learned from collected data without the need to be explicitly programmed. Despite the improvement in task performance, the models are often black boxes. Evaluating the credibility and the vulnerabilities of ML models poses a gap in current test and evaluation practice. For high consequence applications, the lack of testing and evaluation procedures represents a significant source of uncertainty and risk. To help reduce that risk, here we present considerations to evaluate systems embedded with an ML component within a red-teaming inspired methodology. We focus on (1) cyber vulnerabilities to an ML model, (2) evaluating performance gaps, and (3) adversarial ML vulnerabilities.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1978

Numerical derivation of linear perturbation model from nonlinear real-time vehicle simulations - A means of validation and increased credibility

A numerical technique has been developed which generates linear perturbation models from nonlinear real-time aircraft vehicle simulations at Langley Research Center. The technique is very general and can be applied to simulations of any system described by nonlinear differential equations. Linear models about various trim conditions have been obtained from simulations of the Boeing 737, Northrop F5-E, Cessna 180, Sikorsky CH-54, and NASA/Army Rotor Systems Research Aircraft (RSRA). One use of the derived linear models is as a validation and verification tool for the nonlinear simulation.

Dieudonne, J. E.↗
DOE OSTI2020

FACETS Dynamical Downscaling Simulations over North America by the CAM-MPAS Variable-Resolution Model

Comprehensive assessment of climate datasets created by statistical or dynamical models is important for effectively communicating model projection and associated uncertainty to stakeholders and decision-makers. The Department of Energy FACETS project aims to foster such communication through development of metrics and their demonstration on a hierarchy of downscaled climate datasets to quantify aspects of climate change projections that are credible, particularly for supporting decisions related to the energy-water-land nexus. As a part of this effort, we have produced a regional climate dataset using the Model for Prediction Across Scales coupled to the Community Atmosphere Model (CAM-MPAS). This global modeling framework is configured with variable-resolution meshes featuring higher resolutions over North America, as well as quasi-uniform resolution meshes across the globe. The variable-resolution configurations allow fine-scale features to be better resolved inside the refinement and interact with global-scale circulations. The dataset includes multiple uniform- (240km and 120km) and variable-resolution (200-50km, 100-25km, and 46-12km) simulations that are designed to be compatible with other regional climate simulations that contribute to the hierarchy of downscaled climate datasets of the project. Furthermore, the dataset consists of simulations for both the present-day (1989-2010) and future (2079-2100) climate and post-processing of the model output has been coordinated across the project for consistency to facilitate common analysis across the hierarchy of datasets. Altogether, this CAM-MPAS model dataset provides a unique opportunity to assess the influence of resolutions and modeling framework on model credibility and climate change projection.

Sakaguchi, Koichi↗
DOE OSTI2022

Machine learning-based observation-constrained projections reveal elevated global socioeconomic risks from wildfire

Reliable projections of wildfire and associated socioeconomic risks are crucial for the development of efficient and effective adaptation and mitigation strategies. The lack of or limited observational constraints for modeling outputs impairs the credibility of wildfire projections. Here, we present a machine learning framework to constrain the future fire carbon emissions simulated by 13 Earth system models from the Coupled Model Intercomparison Project phase 6 (CMIP6), using historical, observed joint states of fire-relevant variables. During the twenty-first century, the observation-constrained ensemble indicates a weaker increase in global fire carbon emissions but higher increase in global wildfire exposure in population, gross domestic production, and agricultural area, compared with the default ensemble. Such elevated socioeconomic risks are primarily caused by the compound regional enhancement of future wildfire activity and socioeconomic development in the western and central African countries, necessitating an emergent strategic preparedness to wildfires in these countries.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2010

An Extreme-Value Approach to Anomaly Vulnerability Identification

The objective of this paper is to present a method for importance analysis in parametric probabilistic modeling where the result of interest is the identification of potential engineering vulnerabilities associated with postulated anomalies in system behavior. In the context of Accident Precursor Analysis (APA), under which this method has been developed, these vulnerabilities, designated as anomaly vulnerabilities, are conditions that produce high risk in the presence of anomalous system behavior. The method defines a parameter-specific Parameter Vulnerability Importance measure (PVI), which identifies anomaly risk-model parameter values that indicate the potential presence of anomaly vulnerabilities, and allows them to be prioritized for further investigation. This entails analyzing each uncertain risk-model parameter over its credible range of values to determine where it produces the maximum risk. A parameter that produces high system risk for a particular range of values suggests that the system is vulnerable to the modeled anomalous conditions, if indeed the true parameter value lies in that range. Thus, PVI analysis provides a means of identifying and prioritizing anomaly-related engineering issues that at the very least warrant improved understanding to reduce uncertainty, such that true vulnerabilities may be identified and proper corrective actions taken.

Everett, Chris↗
NASA NTRS1995

Verification and Validation of Simulation Models for High-Speed Flow Fields

This paper is largely based on the paper titled "Guide to Credible Computational Fluid Dynamics Simulations (Invited)" (AIAA Paper 95-2225). The significance of computational fluid dynamics (CFD) simulations depends solely on their credibility. A customer of CFD products -- simulations and software -- expects that these products are credible for the intended use. Simulation model verification and validation are critical in establishing the credibility of simulations and in certifying simulation software. The only thing that matters in establishing the credibility is uncertainty, not veracity or validity. The sensitivity-uncertainty analysis is the key to the establishment of this credibility. Assessing the credibility of complex simulation results poses a significant challenge. Terminology, concepts, framework, and guidelines are presented for addressing this challenge. Verification assesses whether the problem is solved correctly and estimates the level of computational accuracy of simulations; validation determines whether the right problem is solved and assesses the level of validity of the simulation model by estimating the degree to which simulations accurately represent reality. These concepts and the related guidelines are discussed with examples from high-speed flow fields.

Mehta, Unmeel B.↗
NASA NTRSDate not supplied

Exploitation of a Validation Hierarchy for Modeling and Simulation

Across engineering there is an evolving need to increase reliance on physics-based simulation to develop, design and optimize engineering systems. This increased reliance on modeling and simulation has highlighted a growing need to transform the confidence that modeling and simulation analysts have in their results into credibility for systems engineers to design and field systems more quickly and with less physical testing. For isolated components of a complex system, where a single discipline may drive product design, this is less of a concern as the relationship is often straightforward and easy to explain. However, when these isolated components are integrated, and are expected to operate in a multi-disciplinary context in which safety critical systems are involved, new concepts and model assurance standards are required. In this paper we address this challenge by showing how a model validation hierarchy can be exploited to identify those model validation experiments that will contribute most to increasing confidence and credibility of modeling and simulation predictions. The approach that is adopted contains four main steps. The first step is the construction of a model validation hierarchy that links subsystems, assemblies, and components to a hierarchy of physical experiments that can be used support model validation. This hierarchy connects the concerns of systems engineers to those of the modeling and simulation analyst in a clear and logical way. The structure and content of this hierarchy is then used in a second step to establish which physical phenomena have the greatest impact on overall system performance metrics. A gap analysis technique, based upon modeling and simulation concerns, is then used to prioritize the important physical phenomenon. Unfortunately, a common outcome of such gap analyses is the identification of many important gaps and so, in the final step of our process, we advocate the use of a global sensitivity analysis as a means to complete the prioritization.

Verification and Validation↗
NASA NTRSDate not supplied

Exploitation of a Validation Hierarchy for Modeling and Simulation

Across engineering there is an evolving need to increase reliance on physics-based simulation to develop, design and optimize engineering systems. This increased reliance on modeling and simulation has highlighted a growing need to transform the confidence that modeling and simulation analysts have in their results into credibility for systems engineers to design and field systems more quickly and with less physical testing. For isolated components of a complex system, where a single discipline may drive product design, this is less of a concern as the relationship is often straightforward and easy to explain. However, when these isolated components are integrated, and are expected to operate in a multi-disciplinary context in which safety critical systems are involved, new concepts and model assurance standards are required. In this paper we address this challenge by showing how a model validation hierarchy can be exploited to identify those model validation experiments that will contribute most to increasing confidence and credibility of modeling and simulation predictions. The approach that is adopted contains four main steps. The first step is the construction of a model validation hierarchy that links subsystems, assemblies, and components to a hierarchy of physical experiments that can be used support model validation. This hierarchy connects the concerns of systems engineers to those of the modeling and simulation analyst in a clear and logical way. The structure and content of this hierarchy is then used in a second step to establish which physical phenomena have the greatest impact on overall system performance metrics. A gap analysis technique, based upon modeling and simulation concerns, is then used to prioritize the important physical phenomenon. Unfortunately, a common outcome of such gap analyses is the identification of many important gaps and so, in the final step of our process, we advocate the use of a global sensitivity analysis as a means to complete the prioritization.

Verification and Validation↗
NASA NTRS2022

NASA Orbital Debris Engineering Model (ORDEM) 3.1: Model Verification and Validation

The NASA Orbital Debris Engineering Model (ORDEM) 3.1 Model Verification and Validation (V&V) document accompanies the delivery of the latest ORDEM 3.1 model (Vavrin & Manis, 2019) and provides a detailed description of the V&V activities used to verify that the model was built correctly and validate the model against independent, real world sources of data obtained from sampling the orbital debris environment. This ORDEM 3.1 Model V&V document, along with the related ORDEM 3.1 Model Process document – which covers details of the mathematical, statistical, and physical basis of the model – are intended to inform credibility assessments, risk analyses, uncertainty characterizations, and other applications derived from use of the model by the ORDEM 3.1 user community.

Timothy Kennedy↗