SEARCH · Search NASA
Results for “model credibility”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Establishing model credibility for process-microstructure-property relationships in additive manufacturing using exascale computing
Additive Manufacturing (AM) of alloys holds significant promise as a disruptive technology in various industries, yet its adoption is often hindered by challenges in achieving consistent part quality. These issues are primarily due to the complex process-microstructure-property (PSP) relationships inherent to AM. Computational models can greatly aid in understanding these relationships, but their widespread impact and adoption has been limited by a lack of validated, open-source, and computationally efficient PSP modeling frameworks and hardware limitations. Here, this study leverages the ExaAM software suite and data from the AMBench-2018 series of laser powder bed fusion (LPBF) benchmark experiments to perform a comprehensive model assessment, including verification, validation, sensitivity analysis, and uncertainty quantification. The RADICAL-EnTK workflow manager was used to perform an ensemble of heat transport, solidification, and mechanical response simulations on the exascale computer Frontier, considering uncertainties in critical model inputs such as laser spot size and nucleation parameters, and consisting of 125 explicit grain structure simulations and 7875 crystal plasticity simulations. For a selected location within the Inconel 625 AMBench-2018 test artifact, sensitivity analysis and uncertainty quantification were performed using the predicted distributions of grain structure and mechanical properties. Qualitative agreement was found between the predicted grain size and texture and the observed AMBench-2018 microstructure, the mean predicted yield stress was within 5% of the experimental measurement mean, and the mean predicted engineering stress at 5% strain was within 10% of the experimental measurement mean. The insights gained from development and validation of the ExaAM PSP modeling framework will help guide future directions for enhancing the credibility and reliability of PSP models in AM, thereby accelerating the adoption of AM technologies in various industries.
An Extension to the Predictive Capability Maturity Model to Assess Model Credibility Using Dempster-Shafer Theory
Explore the source record for details and available documents.
A Demonstration of the Quantitative Approach for Model Credibility on an Electromagnetic Application
Explore the source record for details and available documents.
Model credibility and risk quantification
Explore the source record for details and available documents.
Assessing the Accuracy of ML Explanations for Model Credibility.
Abstract not provided.
Robust Assessment of Surrogate Model Credibility
Explore the source record for details and available documents.
What Questions Would a Systems Engineer Ask to Assess Systems Engineering Models as Credible
Digital Systems Engineering strategies typically call for digital Systems Engineering models to be retained in repositories and certified as an authoritative source of truth (enabling model reuse, qualification, and collaboration). In order for digital Systems Engineering models to be certified as authoritative (credible), they need to be assessed - verified and validated - and with the amount of uncertainty in the model quantified (consider reusing someone else's model without knowing the author). Due to this increasing model complexity, the authors assert that traditional human-based methods for validating, verifying, and uncertainty quantification - such as human-based peer-review sessions - cannot sufficiently establish that a digital Systems Engineering model of a complex system is credible. Digital Systems Engineering models of complex systems can contain millions of nodes and edges. The authors assert that this level of detail is beyond the ability of any group of humans - even working for weeks at a time - to discern and catch every minor model infraction. In contrast, computers are highly effective at discerning infractions with massive amounts of information. The authors suggest that a better approach might be to focus the humans at what model patterns should be assessed and enable the computer to assess the massive details in accordance with those patterns - by running through perhaps 100,000 test loops. In anticipation of future projects to implement and automate the assessment of models at Sandia National Laboratories, a study was initiated to elicit input from a group of 25 Systems Engineering experts. The authors positioning query began with - 'What questions would a Systems Engineer ask to assess Systems Engineering models for credibility?" This report documents the results of that survey.
Material Models and Credibility for System Level Abnormal Mechanical ModSim Applications
The purpose of this document is to provide evidence for assessing the adequacy of parameterized material models for a collection of materials used in a finite element analyses setting. “Adequacy” is relative to the intended use of the material in particular analyses. The intended application of the material models covered within this document is for system level abnormal mechanical solid mechanics analyses. Generally, material model parameterizations should be valid from temperatures of approximately -50 to 70° C, across a range of strain rates, and (depending on details of the parts involved) large deformations. Each material covered in this document is presented in its own chapter with a common format across materials. Model assumptions, limitations, existing validation results, readiness for use with uncertainty quantification, general usage guidance, and failure considerations are all provided along with specific parameterization inputs suitable for the finite element analysis code Sierra/Solid Mechanics.
What Questions Would a Systems Engineer Ask to Assess MBSE Models as Credible?.
Abstract not provided.
Simplifying and Visualizing the Ontology of Systems Engineering Models
The credibility of an engineering model is of critical importance in large-scale projects. How concerned should an engineer be when reusing someone else's model when they may not know the author or be familiar with the tools that were used to create it? In this report, the authors advance engineers' capabilities for assessing models through examination of the underlying semantic structure of a model--the ontology. This ontology defines the objects in a model, types of objects, and relationships between them. In this study, two advances in ontology simplification and visualization are discussed and are demonstrated on two systems engineering models. These advances are critical steps toward enabling engineering models to interoperate, as well as assessing models for credibility. For example, results of this research show an 80% reduction in file size and representation size, dramatically improving the throughput of graph algorithms applied to the analysis of these models. Finally, four future problems are outlined in ontology research toward establishing credible models--ontology discovery, ontology matching, ontology alignment, and model assessment.
Toward shell model interactions with credible uncertainties
Background: The nuclear shell model is a powerful framework for predicting nuclear structure observables, but relies on interaction matrix elements fit to experimental data as its inputs. Extending the shell model's applicability, particularly toward dripline nuclei, requires efficient fitting methods and credible uncertainty quantification. Traditional approaches face computational challenges and may underestimate uncertainties. Purpose: We develop and test a framework combining eigenvector continuation and Markov chain Monte Carlo to efficiently fit shell model interaction matrix elements and quantify their uncertainties. Methods: Eigenvector continuation is used to emulate shell model calculations, reducing computational costs. The emulator enables Markov chain Monte Carlo sampling to optimize interaction matrix elements and rigorously assess parametric uncertainties. Here, the framework is benchmarked using the USDB interaction in the 𝑠𝑑 shell. Results: The emulator reproduces the USDB interaction with negligible error, validating its use in shell model fitting applications. However, we find that to obtain credible predictive intervals, the model defect of the shell model itself, rather than experimental or emulator error, must be taken into account in order to obtain credible uncertainties. Conclusions: The proposed framework provides an efficient and rigorous approach for fitting shell model interactions and quantifying uncertainties. Further, the normality assumption used in the past appears sufficient to describe the distribution of interaction matrix elements. However, it is crucial to account for model correlations to avoid underestimating uncertainties.
Connecting Large‐Scale Meteorological Patterns to Extratropical Cyclones in CMIP6 Climate Models Using Self‐Organizing Maps
Abstract Extratropical cyclones (ETCs) are responsible for the majority of cool‐season extreme events in the northeastern United States (NEUS), often leading to high‐impact weather conditions that can have wide‐ranging socioeconomic impacts. Evaluating the ability of climate models to adequately simulate ETC dynamics is essential for improving model performance and increasing confidence in future projections used by stakeholders and policymakers. ETCs are traditionally studied using techniques such as case studies and synoptic typing, however, these approaches can be time‐consuming, require subjective analysis, and do not necessarily identify the coincident large‐scale meteorological patterns (LSMPs). Here, we apply self‐organizing maps (SOMs) as an automated machine‐learning approach to characterize the LSMPs and associated frequency and intensity of discrete ETC events over NEUS. The dominant patterns of geopotential height variability are identified through SOM analysis of five reanalysis products during the last four decades. ETC events are tracked using TempestExtremes and are integrated with SOMs to classify the accumulated cyclone activity (ACA) associated with each pattern. We then evaluate the skill of CMIP6 historical experiments in simulating the LSMPs and ETC events identified in the SOM. Our results identify a robust bias toward more zonal patterns, with models struggling to reproduce the more amplified patterns typically associated with the highest cyclone activity. While model resolution has some impact on simulation credibility, model configuration appears to be more important in LSMP representation. The vast majority of CMIP6 models produce too few ETCs, although model errors are distributed around historical reanalyses when ACA is normalized by storm frequency.
Identifying synoptic controls on boundary layer thermodynamic and cloud properties in a regional forecast model
Although most of our understanding of boundary layer cloudiness is based on idealized, subtropical, barotropic marine environments, boundary layer clouds exist across a range of conditions. In this study, we use the Naval Research Laboratory's Coupled Ocean/Atmosphere Mesoscale Prediction System (COAMPS) and an automated cold-front-relative analysis framework to explore the boundary layer structure associated with low clouds across a transect through the cold front of a midlatitude synoptic cyclone. The model credibly captures boundary layer structure in line with conceptual models of midlatitude cyclones and ground-based observations at Graciosa Island in the Azores. The warm sector is conditionally unstable, with clouds that are too shallow and with too little liquid water, compared to cloud property retrievals from satellite and surface-based instruments. The cold-frontal region exhibits convection associated with weak stability and ascent. Northwest of the cold front, the boundary layer is well mixed, deeper, and capped by a strong inversion maintained by large-scale subsidence. Simulated clouds in frontal and post-frontal regions are mostly too thick, with too much liquid water and too little cloud base drizzle. The post-frontal clouds are associated with grid-scale updrafts, which appear to be the model's attempt to represent mesoscale organization of cellular convection typically observed in the cold sector of midlatitude cyclones. The deep, well-mixed post-frontal boundary layers and cloudiness are maintained by strong surface fluxes, as in cold air outbreaks. Our analysis framework serves as a unique approach to model verification, and our results offer insights into differences in boundary layer cloud behavior between subtropical and synoptic cold-sector regimes.
Preliminary Results for Using Uncertainty and Out-of-distribution Detection to Identify Unreliable Predictions.
As machine learning (ML) models are deployed into an ever-diversifying set of application spaces, ranging from self-driving cars to cybersecurity to climate modeling, the need to carefully evaluate model credibility becomes increasingly important. Uncertainty quantification (UQ) provides important information about the ability of a learned model to make sound predictions, often with respect to individual test cases. However, most UQ methods for ML are themselves data-driven and therefore susceptible to the same knowledge gaps as the models themselves. Specifically, UQ helps to identify points near decision boundaries where the models fit the data poorly, yet predictions can score as certain for points that are under-represented by the training data and thus out-of-distribution (OOD). One method for evaluating the quality of both ML models and their associated uncertainty estimates is out-of-distribution detection (OODD). We combine OODD with UQ to provide insights into the reliability of the individual predictions made by an ML model.
A Distinct Atmospheric Mode for California Precipitation
The hydrologic cycle in California is strongly influenced by wet-season (November–April) precipitation. Here, we demonstrate the existence of an influential mode of North Pacific atmospheric pressure variability that regulates wet-season precipitation variability over both northern and southern California. This mode, named as the “California precipitation mode” (CPM), is statistically distinct from other well-known modes of pressure variability such as the Pacific-North American pattern. In addition to controlling wet-season mean precipitation, positive days of the CPM coincide with up to 90% of the extreme (>99th percentile) precipitation days and 76% of detected atmospheric rivers (ARs) days, while the negative days correspond with 60% of the dry days. CMIP6 models capture the CPM remarkably well, including its statistical separation from the other well-known modes of pressure variability. The models also reproduce the CPM's strong association with California wet-season precipitation, giving confidence in the models’ dynamics relating to regional hydrologic extremes. However, the models also exhibit biases in regional hydrologic extremes. The CPM is a useful way to understand the origins of those biases and select the more credible models for further analysis: Models with unrealistically strong gradients in the CPM pressure pattern generally oversimulate larger wet extremes and produce excessively long dry intervals in the historical period. Thus the hydrologic biases can be traced to the particular aspects of North Pacific atmospheric dynamics.