The Need for Credibility Guidance for Analyses Quantifying Margin and Uncertainty.
Abstract not provided.
Schroeder, Benjamin↗
SEARCH · Search NASA
Results for “Credibility”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Off-Axis Input Characterization of Random Vibration Laboratory Data for Model Credibility.
Abstract not provided.
Blecke, Jill↗
Experimental Credibility and its Role in Model Validation and Decision Making.
Abstract not provided.
Kieweg, Sarah↗
Credibility Framework ? an End-to-End Credibility Workflow Platform.
Abstract not provided.
Orient, George Edgar↗
Differential credibility assessment for statistical downscaling
Climate science is increasingly using (i) ensembles of climate projections from multiple models derived using different assumptions and/or scenarios and (ii) process-oriented diagnostics of model fidelity. Efforts to assign differential credibility to projections and/or models are also rapidly advancing. A framework to quantify and depict the credibility of statistically downscaled model output is presented and demonstrated. Here, the approach employs transfer functions in the form of robust and resilient generalized linear models applied to downscale daily minimum and maximum temperature anomalies at 10 locations using predictors drawn from ERA-Interim reanalysis and two global climate models (GCM; GFDL-ESM2M and MPI-ESM-LR). The downscaled time series are used to derive several impact relevant CLIMDEX temperature indices that are assigned credibility based on (1) the reproduction of relevant large-scale predictors by the GCMs (i.e. fraction of regression beta-weights derived from predictors that are well-reproduced) and (2) the degree of variance in the observations reproduced in the downscaled series following application of a new variance inflation technique. Credibility of the downscaled predictands varies across locations, between the two GCM and is generally higher for minimum temperature than maximum temperature. The differential credibility assessment framework demonstrated here is easy to use and flexible. It can be applied as is to inform decision makers regarding projection confidence, and/or extended to include other components of the transfer functions, and/or used to weight members of a statistically downscaled ensemble.
54 ENVIRONMENTAL SCIENCES↗
Uncertainty quantification's role in modeling and simulation planning, and credibility assessment through the predictive capability maturity model
The importance of credible, trustworthy numerical simulations is obvious especially when using the results for making high-consequence decisions. Determining the credibility of such numerical predictions is much more difficult and requires a systematic approach to assessing predictive capability, associated uncertainties and overall confidence in the computational simulation process for the intended use of the model. This process begins with an evaluation of the computational modeling of the identified, important physics of the simulation for its intended use. This is commonly done through a Phenomena Identification Ranking Table (PIRT). Then an assessment of the evidence basis supporting the ability to computationally simulate these physics can be performed using various frameworks such as the Predictive Capability Maturity Model (PCMM). There were several critical activities that follow in the areas of code and solution verification, validation and uncertainty quantification, which will be described in detail in the following sections. Here, we introduce the subject matter for general applications but specifics are given for the failure prediction project. In addition, the first task that must be completed in the verification & validation procedure is to perform a credibility assessment to fully understand the requirements and limitations of the current computational simulation capability for the specific application intended use. The PIRT and PCMM are tools used at Sandia National Laboratories (SNL) to provide a consistent manner to perform such an assessment. Ideally, all stakeholders should be represented and contribute to perform an accurate credibility assessment. PIRTs and PCMMs are both described in brief detail below and the resulting assessments for an example project are given.
97 MATHEMATICS AND COMPUTING↗
Credibility-enhancing displays promote the provision of non-normative public goods
Promoting the adoption of public goods that are not yet widely accepted is particularly challenging. This is because most tools for increasing cooperation—such as reputation concerns and information about social norms—are typically effective only for behaviours that are commonly practiced, or at least generally agreed upon as being desirable. Here we examine how advocates can successfully promote non-normative (that is, rare or unpopular) public goods. We do so by applying the cultural evolutionary theory of credibility-enhancing displays3, which argues that beliefs are spread more effectively by actions than by words alone—because actions provide information about the actor’s true beliefs. Based on this logic, people who themselves engage in a given behaviour will be more effective advocates for that behaviour than people who merely extol its virtues—specifically because engaging in a behaviour credibly signals a belief in its value. As predicted, a field study of a programme that promotes residential solar panel installation in 58 towns in the United States—comprising 1.4 million residents in total—found that community organizers who themselves installed through the programme recruited 62.8% more residents to install solar panels than community organizers who did not. This effect was replicated in three pre-registered randomized survey experiments (total n = 1,805). These experiments also support the theoretical prediction that this effect is specifically driven by subjects’ beliefs about what the community organizer believes about solar panels (that is, second-order beliefs), and demonstrate generalizability to four other highly non-normative behaviours. Our findings shed light on how to spread non-normative prosocial behaviours, offer an empirical demonstration of credibility-enhancing displays and have substantial implications for practitioners and policy-makers.
14 SOLAR ENERGY↗
Toward shell model interactions with credible uncertainties
Background: The nuclear shell model is a powerful framework for predicting nuclear structure observables, but relies on interaction matrix elements fit to experimental data as its inputs. Extending the shell model's applicability, particularly toward dripline nuclei, requires efficient fitting methods and credible uncertainty quantification. Traditional approaches face computational challenges and may underestimate uncertainties. Purpose: We develop and test a framework combining eigenvector continuation and Markov chain Monte Carlo to efficiently fit shell model interaction matrix elements and quantify their uncertainties. Methods: Eigenvector continuation is used to emulate shell model calculations, reducing computational costs. The emulator enables Markov chain Monte Carlo sampling to optimize interaction matrix elements and rigorously assess parametric uncertainties. Here, the framework is benchmarked using the USDB interaction in the 𝑠𝑑 shell. Results: The emulator reproduces the USDB interaction with negligible error, validating its use in shell model fitting applications. However, we find that to obtain credible predictive intervals, the model defect of the shell model itself, rather than experimental or emulator error, must be taken into account in order to obtain credible uncertainties. Conclusions: The proposed framework provides an efficient and rigorous approach for fitting shell model interactions and quantifying uncertainties. Further, the normality assumption used in the past appears sufficient to describe the distribution of interaction matrix elements. However, it is crucial to account for model correlations to avoid underestimating uncertainties.
Nuclear forces↗
What Questions Would a Systems Engineer Ask to Assess Systems Engineering Models as Credible
Digital Systems Engineering strategies typically call for digital Systems Engineering models to be retained in repositories and certified as an authoritative source of truth (enabling model reuse, qualification, and collaboration). In order for digital Systems Engineering models to be certified as authoritative (credible), they need to be assessed - verified and validated - and with the amount of uncertainty in the model quantified (consider reusing someone else's model without knowing the author). Due to this increasing model complexity, the authors assert that traditional human-based methods for validating, verifying, and uncertainty quantification - such as human-based peer-review sessions - cannot sufficiently establish that a digital Systems Engineering model of a complex system is credible. Digital Systems Engineering models of complex systems can contain millions of nodes and edges. The authors assert that this level of detail is beyond the ability of any group of humans - even working for weeks at a time - to discern and catch every minor model infraction. In contrast, computers are highly effective at discerning infractions with massive amounts of information. The authors suggest that a better approach might be to focus the humans at what model patterns should be assessed and enable the computer to assess the massive details in accordance with those patterns - by running through perhaps 100,000 test loops. In anticipation of future projects to implement and automate the assessment of models at Sandia National Laboratories, a study was initiated to elicit input from a group of 25 Systems Engineering experts. The authors positioning query began with - 'What questions would a Systems Engineer ask to assess Systems Engineering models for credibility?" This report documents the results of that survey.
42 ENGINEERING↗
HOLISTIC CREDIBILITY ASSESSMENT OF A MACHINE LEARNING-BASED SURROGATE MODEL IN AN AERODYNAMICS APPLICATION
Elements of ASME V&V 20 credibility assessment methodology are used in tandem with credibility assessment tools from the machine learning community to assess the credibility of a deep neural network based surrogate model. This surrogate model is trained, tested, validated, and employed in the context of aerodynamic coefficient prediction for a NACA 0012 airfoil in subsonic and transonic flow. The parameter space is defined by angle of attack, Reynolds number, and Mach number.
Kirsch, Jared Roelof [Sandia National Laboratories↗
Maximum credibly yield for deuteriuim-filled double shell imaging targets meeting requirements for yield bin Category A
We are anticipating our first NIF double shell shot using an aluminum ablator and a glass inner shell filled with deuterium shown in figure 1. The expected yield is between a few 10 10 to a few 10 11 dd neutrons. The maximum credible yield is 5e+13. This memo describes why, and what would be expected with variations on the target. This memo evaluates the maximum credible yield for deuterium filled double shell capsule targets with an aluminum ablator shell and a glass inner shell in yield Category A (< 10 14 neutrons). It also pertains to fills of gas diluted with hydrogen, helium ( 3 He or 4 He), or any other fuel except tritium. This memo does not apply to lower z ablator dopants, such as beryllium, as this would increase the ablation efficiency. This evaluation is for 5.75 scale hohlraum targets of either gold or uranium with helium gas fills with density between 0 and 1.6 mg/cc. It could be extended to other hohlraum sizes and shapes with slight modifications. At present only laser pulse energies up to 1.5 MJ were considered with a single step laser pulse of arbitrary shape. Since yield decreases with laser energy for this target, the memo could be extended to higher laser energies if desired. These maximum laser parameters of pulses addressed here are near the edge of NIF’s capability, and constitute the operating envelope for experiments covered by this memo. We have not considered multiple step pulses, would probably create no advantages in performance, and are not planned for double shell capsules. The main target variables are summarized in Table 1 and explained in detail in the memo. Predicted neutron yields are based on 1D and 2D clean simulations.
38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗
A framework for strategic discovery of credible neural network surrogate models under uncertainty
The widespread integration of deep neural networks in developing data-driven surrogate models for high-fidelity simulations of complex physical systems highlights the critical necessity for robust uncertainty quantification techniques and credibility assessment methodologies, ensuring the reliable deployment of surrogate models in consequential decision-making. Here, this study presents the Occam Plausibility Algorithm for surrogate models (OPAL-surrogate), providing a systematic framework to uncover predictive neural network-based surrogate models within the large space of potential models, including various neural network classes and choices of architecture and hyperparameters. The framework is grounded in hierarchical Bayesian inferences and employs model validation tests to evaluate the credibility and prediction reliability of the surrogate models under uncertainty. Leveraging these principles, OPAL-surrogate introduces a systematic and efficient strategy for balancing the trade-off between model complexity, accuracy, and prediction uncertainty. The effectiveness of OPAL-surrogate is demonstrated through two modeling problems, including the deformation of porous materials for building insulation and turbulent combustion flow for ablation of solid fuels within hybrid rocket motors.
42 ENGINEERING↗
Structural uncertainty assessment for fire-engulfed objects in crosswind: Establishing credibility for a multiphysics wall-modeled large-eddy simulation paradigm
A structural uncertainty validation study for a large-scale, fire-engulfed, elevated object subjected to crosswind is presented to establish the credibility of a high-fidelity, low-Mach, turbulent reacting flow wall-modeled large-eddy simulation (WMLES) approach that includes multiphysics coupling to participating media radiation and conjugate heat transfer. To establish that WMLES can accurately predict surface quantities including drag and pressure coefficient in the low-Mach crosswind regime, a foundational elevated isothermal cylinder validation case is presented at a similar gap-to-diameter ratio of 0.25, spanning the subcritical to supercritical drag regime (Re 𝐷 = 1.1 × 10 5 and 4.3 × 10 5 , respectively). Here, this study exercised both static and dynamic coefficient LES (Smagorinsky and 𝑘 sgs ) with both local and exchange-based velocity sampling. Results showcase that the drag crisis (or the sudden drop in drag coefficient at increased Re 𝐷 ) is well captured when using an exchange-based dynamic coefficient WMLES methodology, while noting lack of mesh convergence and overall drag and pressure coefficient predictively when using a static coefficient, local velocity sampling WMLES. For the 𝒪(10) m JP-8 liquid pool fire crosswind validation study presented, two experimental crosswind configurations (2 m/s and 9.5 m/s) are showcased for a fire-engulfed mock fuselage roughly 4 m in diameter. Using the best model-form practices identified in the isothermal study, dynamic coefficient 𝑘 sgs exchange-based WMLES fire validation findings demonstrate accurate peak irradiation and skin temperature predictions as a function of crosswind magnitude. Excessive yaw in the low-crosswind fuselage configuration, consistent with experimental findings, captured a significant predicted asymmetry in flame attachment and heat flux toward the downwind cylindrical cap—indicative of axial vortex structures transporting the flame along the upper and lower fuselage leeward surface. All fire mesh resolution simulations captured the experimental finding that as crosswind increased, predicted flame shape and peak irradiation magnitude onto the fuselage transitioned from a windward to a leeward cylinder location due to the migration of the upper- to lower-shear fuel/air mixing layer thereby demonstrating the novelty, significance, and credibility of this high-fidelity WMLES reacting flow framework.
Domino, Stefan Paul [Sandia National Laboratories ↗
Credibility Framework (CF) v.1.0
Credibility Framework provides tools such as PIRT (Phenomena Identification and Ranking Table) and PCMM (Predictive Capability Maturity Model) configurable by non-programmers through simple Excel spreadsheets for tailoring the credibility process to meet the needs of the analysis provider organizations, customers and regulatory agencies. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND2020-10230 M
Clay, RobertL.↗
We Know Where We Don't Know: 3D Bayesian CNNs for Credible Geometric Uncertainty
Deep learning has been successfully applied to the segmentation of 3D Computed Tomography (CT) scans. Establishing the credibility of these segmentations requires uncertainty quantification (UQ) to identify untrustworthy predictions. Recent UQ architectures include Monte Carlo dropout networks (MCDNs), which approximate deep Gaussian processes, and Bayesian neural networks (BNNs), which learn the distribution of the weight space. BNNs are advantageous over MCDNs for UQ but are thought to be computationally infeasible in high dimension, and neither architecture has produced interpretable geometric uncertainty maps. We propose a novel 3D Bayesian convolutional neural network (BCNN), the first deep learning method which generates statistically credible geometric uncertainty maps and scales for application to 3D data. We present experimental results on CT scans of graphite electrodes and laser-welded metals and show that our BCNN outperforms an MCDN in recent uncertainty metrics. The geometric uncertainty maps generated by our BCNN capture distributions of sigmoid values that are interpretable as confidence intervals, critical for applications that rely on deep learning for high-consequence decisions.
97 MATHEMATICS AND COMPUTING↗
The Difficulty of Defining the Terms Credible and Unlikely for Nuclear Criticality Safety Purposes
During the last revision of the ANSI/ANS-8.1-2014, extensive efforts were made by the working group (ANS-8.1) to ensure the statements in the standard clearly reflected their intent This effort involved the review of each requirement and recommendation in the standard. In some cases, definitions were added to the standard glossary to minimize ambiguity in the standard (e.g., “process conditions” or “parameter”). The working group also considered adding definitions for the words “credible” and “unlikely” because definitions had been provided to the nuclear criticality safety (NCS) community from the US Department of Energy that caused confusion at some sites. A Nuclear Regulatory Commission document intends for licensees to provide appropriate definitions for these words in their site license. The purpose of this paper is to discuss why the working group ultimately decided not to include definitions for credible and unlikely in the 2014 revision of ANS-8.1
96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
The Difficulty of Defining the Terms 'Credible' and 'Unlikely' for Nuclear Criticality Safety Purposes
During the last revision of the ANSI/ANS-8.1-2014, extensive efforts were made by the working group (ANS- 8.1) to ensure the statements in the standard clearly reflected their intent. This effort involved the review of each requirement and recommendation in the standard. In some cases, definitions were added to the standard glossary to minimize ambiguity in the standard (e.g., 'process conditions' or 'parameter'). The working group also considered adding definitions for the words 'credible' and 'unlikely' because definitions had been provided to the nuclear criticality safety (NCS) community from the US Department of Energy that caused confusion at some sites. A Nuclear Regulatory Commission document intends for licensees to provide appropriate definitions for these words in their site license. The purpose of this paper is to discuss why the working group ultimately decided not to include definitions for credible and unlikely in the 2014 revision of ANS-8.1.
22 GENERAL STUDIES OF NUCLEAR REACTORS↗
Overview of the fusion nuclear science facility, a credible break-in step on the path to fusion energy
Here the Fusion Nuclear Science Facility (FNSF) is examined as part of a two step program from ITER to commercial power plants. This first step is considered mandatory to establish the materials and component database in the real fusion in-service environment before proceeding to larger electricity producing facilities. The FNSF can be shown to make tremendous advances beyond ITER, toward a power plant, particularly in plasma duration and fusion nuclear environment. A moderate FNSF is studied in detail, which does not generate net electricity, but does reach the power plant blanket operating temperatures. The full poloidal Dual Coolant Lead Lithium (DCLL) blanket is chosen, with alternates being the Helium Cooled Lead Lithium (HCLL) and Helium Cooled Ceramic Breeder/Pebble Bed (HCCB/PB). Several power plant relevant choices are made in order to follow the philosophy of targeted technologies. Any fusion core component must be qualified by fusion relevant neutron testing and highly integrated non-nuclear testing before it can be installed on the FNSF in order to avoid the high probability of constant failures in a plasma-vacuum system. A range of missions for the FNSF, or any fusion nuclear facility on the path toward fusion power plants, are established and characterized by several metrics. A conservative physics strategy is pursued to accommodate the transition to ultra-long plasma pulses, and parameters are chosen to represent the power plant regime to the extent possible. An operating space is identified, and from this, one point is chosen for further detailed analysis, with R = 4.8 m, a = 1.2 m, I P = 7.9 MA, B T = 7.5 T, β N < 2.7, n/n Gr = 0.9, f BS = 0.52, q95 = 6.0, H 98 ~1.0, and Q = 4.0. The operating space is shown to be robust to parameter variations. A program is established for the FNSF to show how the missions for the facility are met, with a He/H, a DD and 5 DT phases. The facility requires ~25 years to complete its DT operation, including 7.8 years of neutron production, and the remaining spent on inspections and maintenance. The DD phase is critical to establish the ultra-long plasma pulse lengths. The blanket testing strategy is examined, and shows that many sectors have penetrations for heating and current drive (H/CD), diagnostics, or Test Blanket Modules (TBMs). The hot cell is a critical facility element in order for the FNSF to perform its function of developing the in-service material and component database. The pre-FNSF R&D is laid out in terms of priority topics, with the FNSF phases driving the time-lines for R&D completion. A series of detailed technical assessments of the FNSF operating point are reported in this issue, showing the credibility of such a step, and more detailed emphasis on R&D items to pursue. These include nuclear analysis, thermo-mechanics and thermal-hydraulics, liquid metal thermal hydraulics, transient thermo-mechanics, tritium analysis, maintenance assessment, magnet specification and analysis, materials assessments, core and scrape-off layer (SOL)/divertor plasma examinations.
70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗