Search NASA⌕ Search

SEARCH · Search NASA

Results for “Bayesian Testing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Monte Carlo Bayesian Inference on a Statistical Model of Sub-gridcolumn Moisture Variability Using High-resolution Cloud Observations : Sensitivity Tests and Results - Part II

Part I presented a Monte Carlo Bayesian method for constraining a complex statistical model of GCM sub-gridcolumn moisture variability using high-resolution MODIS cloud data, thereby permitting large-scale model parameter estimation and cloud data assimilation. This part performs some basic testing of this new approach, verifying that it does indeed significantly reduce mean and standard deviation biases with respect to the assimilated MODIS cloud optical depth, brightness temperature and cloud top pressure, and that it also improves the simulated rotational-Ramman scattering cloud optical centroid pressure (OCP) against independent (non-assimilated) retrievals from the OMI instrument. Of particular interest, the Monte Carlo method does show skill in the especially difficult case where the background state is clear but cloudy observations exist. In traditional linearized data assimilation methods, a subsaturated background cannot produce clouds via any infinitesimal equilibrium perturbation, but the Monte Carlo approach allows finite jumps into regions of non-zero cloud probability. In the example provided, the method is able to restore marine stratocumulus near the Californian coast where the background state has a clear swath. This paper also examines a number of algorithmic and physical sensitivities of the new method and provides guidance for its cost-effective implementation. One obvious difficulty for the method, and other cloud data assimilation methods as well, is the lack of information content in the cloud observables on cloud vertical structure, beyond cloud top pressure and optical thickness, thus necessitating strong dependence on the background vertical moisture structure. It is found that a simple flow-dependent correlation modification due to Riishojgaard (1998) provides some help in this respect, by better honoring inversion structures in the background state.

Monte Carlo Bayesian inference↗

Statistical Symbolic Execution with Informed Sampling

Symbolic execution techniques have been proposed recently for the probabilistic analysis of programs. These techniques seek to quantify the likelihood of reaching program events of interest, e.g., assert violations. They have many promising applications but have scalability issues due to high computational demand. To address this challenge, we propose a statistical symbolic execution technique that performs Monte Carlo sampling of the symbolic program paths and uses the obtained information for Bayesian estimation and hypothesis testing with respect to the probability of reaching the target events. To speed up the convergence of the statistical analysis, we propose Informed Sampling, an iterative symbolic execution that first explores the paths that have high statistical significance, prunes them from the state space and guides the execution towards less likely paths. The technique combines Bayesian estimation with a partial exact analysis for the pruned paths leading to provably improved convergence of the statistical analysis. We have implemented statistical symbolic execution with in- formed sampling in the Symbolic PathFinder tool. We show experimentally that the informed sampling obtains more precise results and converges faster than a purely statistical analysis and may also be more efficient than an exact symbolic analysis. When the latter does not terminate symbolic execution with informed sampling can give meaningful results under the same time and memory limits.

Reliability↗

Quantifying parameters for Bayesian prior assumptions when estimating the probability of failure of software

Software reliability has become increasingly important, especially in life-critical situations. The ability to measure the results of testing and to quantify software reliability is needed. If this is accomplished, a certain minimum amount of reliability for a piece of software can be specified, and testing and/or other analysis may be done until that minimum number has been attained. There are many models for estimating software reliability. The accuracy of these models has been challenged and many revisions for the models and recalibration techniques have been devised. Of particular interest is the method of estimating the probability of failure of software when no failures have yet occurred in its current version as described by Miller. This model uses black box testing with formulae based on Bayesian estimation. The focus is on three interrelated issues: estimating the probability of failure when testing has revealed no errors; modifying this estimation when the input use distribution does not match the test distribution; and combining the results from random testing with other relevant information to obtain a possibly more accurate estimate of the probability of failure. Obtaining relevant information about the software and combining the results for a better estimate for the Miller model are discussed.

Long, Jacquelyn E.↗

The effect of modeling dose uncertainty on low-boom community noise dose-response curves

In logistic dose-response modeling, failing to account for uncertainty in estimated doses can cause an artificial flattening or attenuation of the slope of the summary curve. In Lee et al. [J. Acoust. Soc. Am. 147(4), pp. 2222-2234 (2020)], data from two NASA low-amplitude sonic boom community noise survey tests were modeled using a Bayesian multilevel logistic regression (MLR) statistical model that assumed there was no uncertainty in the noise dose estimates. However, in these community tests, the noise dose uncertainty was estimated by Page et al. [NASA/CR-2014-218180 and NASA/CR-2020-220589/Volume I] using a leave-one-out method. In the current work, a term was added to extend the Bayesian MLR model to account for the estimated noise dose uncertainty quantified in the Page et al. analyses. This uncertainty term was included in two ways, either as classical or as Berkson uncertainty, and yield similar results. When the uncertainty is accounted for in the Bayesian MLR model, the dose-response curves become 5-10% steeper, but the difference in the noise dose that elicits a 5% highly annoyed response is small (less than 1 dB). This result is encouraging for future X-59 community tests whose survey area will be sparsely populated with noise monitors.

X-59↗

A note on minimizing time assurance tests for repairable systems

We consider assurance testing for repairable systems when supplementary information is available in addition to the data collected in the assurance test. Here, the supplementary information is incorporated using a Bayesian inferential framework. Here we consider assurance testing for a homogeneous Poisson process. In this note we consider an alternative criterion that minimizes the test time while ensuring that the requirements on the producer's and consumer's risks are met. We illustrate the use of this alternative criterion with an example.

42 ENGINEERING↗

Testing non-minimally coupled inflation with CMB data: a Bayesian analysis

We use the most recent cosmic microwave background (CMB) data to perform a Bayesian statistical analysis and discuss the observational viability of inflationary models with a non-minimal coupling, ξ, between the inflaton field and the Ricci scalar. We particularize our analysis to two examples of small and large field inflationary models, namely, the Coleman-Weinberg and the chaotic quartic potentials. We find that ( i ) the ξ parameter is closely correlated with the primordial amplitude ; ( ii ) although improving the agreement with the CMB data in the r − n {sub s} plane, where r is the tensor-to-scalar ratio and n {sub s} the primordial spectral index, a non-null coupling is strongly disfavoured with respect to the minimally coupled standard ΛCDM model, since the upper bounds of the Bayes factor (odds) for ξ parameter are greater than 150:1.

79 ASTRONOMY AND ASTROPHYSICS↗

Sensitivity of modelled passive margin stratigraphy to variations in sea level, sediment supply and subsidence

Abstract We produced a 10 Myr synthetic stratigraphic section using a forward stratigraphic model that generates marine deltaic stratigraphy over geological timescales. We recursively fit the model using a Bayesian inversion algorithm to test: (1) if it could be accurately reconstructed; (2) if the parameters used to create it could be recovered; and (3) the sensitivity of the model output to given model parameters and the attendant physical processes. The original synthetic stratigraphic section was produced with cyclical sea‐level variations of 40 and 30 m with 2.4 and 10 Myr periods respectively. Sediment was also supplied cyclically, in 2.4 and 10 Myr cycles with amplitudes of 30 and 80 tons/100 kyr, respectively, varying from a mean of 232 tons/100 kyr. Parameter values were sampled to fit the model using a Markov chain Monte Carlo algorithm, resulting in a ±5 m (1σ) variation between the experimental output and the original. Sea level varied by ±7 m (1σ) within the posterior distribution of parameters. As a result, both the 10 Myr and 2.4 Myr sea‐level cycles could be extracted from the original output. The variation in sediment supply was approximately ±38 tons/100 kyr (1σ) and, as a result, only the larger long‐term supply variations could be accurately recovered in refitting the model. The variation in thermal, flexural and total subsidence across those parameter sets is less than ±10 m (1σ). The original section experienced 150 m of total subsidence at the depocentre. Our results demonstrate the distinct and interpretable imprint of sea level and subsidence on continental margin stratigraphy can be quantified. Moreover, we conclude that sea‐level change produces a defined effect on the geometries of stratigraphic architecture, and that techniques applied for the purpose of delineating sea‐level variation from continental margin strata have a well‐founded conceptual basis.

Schmelz, William J.↗

Asynchronous demographic responses to Pleistocene climate change in Eastern Nearctic vertebrates

Abstract Pleistocene climatic cycles altered species distributions in the Eastern Nearctic of North America, yet the degree of congruent demographic response to the Pleistocene among codistributed taxa remains unknown. We use a hierarchical approximate Bayesian computational approach to test if population sizes across lineages of snakes, lizards, turtles, mammals, birds, salamanders and frogs in this region expanded synchronously to Late Pleistocene climate changes. Expansion occurred in 75% of 74 lineages, and of these, population size trajectories across the community were partially synchronous, with coexpansion found in at least 50% of lineages in each taxonomic group. For those taxa expanding outside of these synchronous pulses, factors related to when they entered the community, ecological thresholds or biotic interactions likely condition their timing of response to Pleistocene climate change. Unified timing of population size change across communities in response to Pleistocene climate cycles is likely rare in North America.

Burbrink, Frank T.↗

XRD-PUAT (X-Ray Diffraction - Parameter Uncertainty Analysis Toolkit)

This XRD uncertainty toolkit was built to investigate uncertainty and local least squares topology over a specified parameter space on a single histogram in a gpx GSAS-II file. The parameter uncertainty can be investigated using either a frequentist F-test approach or a Bayesian Inference statistical inversion method on a weighted least squares or peak fit refinements.SAND Number: SAND2020-12227 M Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Moore, Alexander↗

QMU as an Approach to Strengthening the Predictive Capabilities of Complex Models

Complex systems are made up of multiple interdependent parts, and the behavior of the entire system cannot always be directly inferred from the behavior of the individual parts. They are nonlinear and system responses are not necessarily additive. Examples of complex systems include energy, cyber and telecommunication infrastructures, human and animal social structures, and biological structures such as cells. To meet the goals of infrastructure development, maintenance, and protection for cyber-related complex systems, novel modeling and simulation technology is needed. Sandia has shown success using M&S in the nuclear weapons (NW) program. However, complex systems represent a significant challenge and relative departure from the classical M&S exercises, and many of the scientific and mathematical M&S processes must be re-envisioned. Specifically, in the NW program, requirements and acceptable margins for performance, resilience, and security are well-defined and given quantitatively from the start. The Quantification of Margins and Uncertainties (QMU) process helps to assess whether or not these safety, reliability and performance requirements have been met after a system has been developed. In this sense, QMU is used as a sort of check that requirements have been met once the development process is completed. In contrast, performance requirements and margins may not have been defined a priori for many complex systems, (i.e. the Internet, electrical distribution grids, etc.), particularly not in quantitative terms. This project addresses this fundamental difference by investigating the use of QMU at the start of the design process for complex systems. Three major tasks were completed. First, the characteristics of the cyber infrastructure problem were collected and considered in the context of QMU-based tools. Second, UQ methodologies for the quantification of model discrepancies were considered in the context of statistical models of cyber activity. Third, Bayesian methods for optimal testing in the QMU framework were developed. This completion of this project represent an increased understanding of how to apply and use the QMU process as a means for improving model predictions of the behavior of complex systems.

Gray, Genetha Anne↗

Interpreting High-resolution Spectroscopy of Exoplanets using Cross-correlations and Supervised Machine Learning

We present a new method for performing atmospheric retrieval on ground-based, high-resolution data of exoplanets. Our method combines cross-correlation functions with a random forest, a supervised machine-learning technique, to overcome challenges associated with high-resolution data. A series of cross-correlation functions are concatenated to give a “CCF-sequence” for each model atmosphere, which reduces the dimensionality by a factor of ∼100. The random forest, trained on our grid of ∼65,000 models, provides a likelihood-free method of retrieval. The precomputed grid spans 31 values of both temperature and metallicity, and incorporates a realistic noise model. We apply our method to HARPS-N observations of the ultra-hot Jupiter KELT-9b and obtain a metallicity consistent with solar (logM = − 0.2 ± 0.2). Our retrieved transit chord temperature (T=6000{sub −200}{sup +0}K) is unreliable as strong ion lines lie outside of the extent of the training set, which we interpret as being indicative of missing physics in our atmospheric model. We compare our method to traditional nested sampling, as well as other machine-learning techniques, such as Bayesian neural networks. We demonstrate that the likelihood-free aspect of the random forest makes it more robust than nested sampling to different error distributions, and that the Bayesian neural network we tested is unable to reproduce complex posteriors. We also address the claim in Cobb et al. 2019 that our random forest retrieval technique can be overconfident but incorrect. We show that this is an artifact of the training set, rather than of the machine-learning method, and that the posteriors agree with those obtained using nested sampling.

79 ASTRONOMY AND ASTROPHYSICS↗

Composite Overwrapped Pressure Vessels (COPV): Flight Rationale for the Space Shuttle Program

Each Orbiter Vehicle (Space Shuttle Program) contains up to 24 Kevlar49/Epoxy Composite Overwrapped Pressure Vessels (COPV) for storage of pressurized gases. In the wake of the Columbia accident and the ensuing Return To Flight (RTF) activities, Orbiter engineers reexamined COPV flight certification. The original COPV design calculations were updated to include recently declassified Kevlar COPV test data from Lawrence Livermore National Laboratory (LLNL) and to incorporate changes in how the Space Shuttle was operated as opposed to orinigially envisioned. 2005 estimates for the probability of a catastrophic failure over the life of the program (from STS-1 through STS-107) were one-in-five. To address this unacceptable risk, the Orbiter Project Office (OPO) initiated a comprehensive investigation to understand and mitigate this risk. First, the team considered and eventually deemed unfeasible procuring and replacing all existing flight COPVs. OPO replaced the two vessels with the highest risk with existing flight spare units. Second, OPO instituted operational improvements in ground procedures to signficiantly reduce risk, without adversely affecting Shuttle capability. Third, OPO developed a comprehensive model to quantify the likelihood of occurrance. A fully-instrumented burst test (recording a lower burst pressure than expected) on a flight-certified vessel provided critical understanding of the behavior of Orbiter COPVs. A more accurate model was based on a newly-compiled comprehensive database of Kevlar data from LLNL and elsewhere. Considering hardware changes, operational improvements and reliability model refinements, the mean reliability was determined to be 0.998 for the remainder of the Shuttle Program (from 2007, for STS- 118 thru STS-135). Since limited hardware resources precluded full model validation through multiple tests, additional model confidence was sought through the first-ever Accelerated Stress Rupture Test (ASRT) of a flown flight article. A Bayesian statistical approach was developed to interpret possible test results. Since the lifetime observed in the ASRT exceeded initial estimates by one to two orders of magnitude, the Space Shuttle Program deemed there was significant conservatism in the model and accepted continued operation with existing flight hardware. Given the variability in tank-to-tank original prooftest response, a non-destructive evaluation (NDE) technique utilizing Raman Spectroscopy was developed to directly measure COPV residual stress state. Preliminary results showed that patterns of low fiber elastic strains over the outside vessel surface, together with measured permanent volume growth during proof, could be directly correlated to increased fiber stress ratios on the inside fibers adjacent to the liner, and thus reduced reliability.

Kezirian, Michael T.↗

BAYESIAN CALIBRATION OF SAFETY CODES USING DATA FROM SEPARATE-AND INTEGRAL EFFECTS TESTS

Large-scale system codes for simulation of safety performance of nuclear plants may contain parameters whose values are not known very accurately. In order to be able to use the results of these simulation codes with confidence, it is important to learn how the uncertainty on the values of these parameters affects the output of the codes. New information from tests or operating experience is incorporated into safety codes by a process known as calibration, which reduces uncertainty in the output of the safety code, and thereby improves its support for decision-making. Modern analysis capabilities afford very significant improvements on classical ways of doing calibration, and the work reported here implements some of those improvements. The key innovation has come from development of safety code surrogate model (code emulator) construction and prediction algorithms. A surrogate is needed for calibration of plant-scale simulation codes because the multivariate nature of the problem (i.e., the need to adjust multiple uncertain parameters at once to fit multiple pieces of new information) calls for multiple evaluations of performance, which, for a computation-intensive model, makes calibration very computation-intensive. Use of a fast surrogate makes the calibration processes used here with Markov Chain Monte Carlo (MCMC) sampling feasible. Moreover, most traditional surrogates do not provide uncertainty information along with their predictions, but the Gaussian Process (GP) based code surrogates used here do. This improves the soundness of the code calibration process. Results are demonstrated on a simplified scenario with data from Separate and Integral Effect Tests.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Monte Carlo goodness-of-fit tests for degree corrected and related stochastic blockmodels

Abstract We construct Bayesian and frequentist finite-sample goodness-of-fit tests for three different variants of the stochastic blockmodel for network data. Since all of the stochastic blockmodel variants are log-linear in form when block assignments are known, the tests for the latent block model versions combine a block membership estimator with the algebraic statistics machinery for testing goodness-of-fit in log-linear models. We describe Markov bases and marginal polytopes of the variants of the stochastic blockmodel and discuss how both facilitate the development of goodness-of-fit tests and understanding of model behaviour. The general testing methodology developed here extends to any finite mixture of log-linear models on discrete data, and as such is the first application of the algebraic statistics machinery for latent-variable models.

Karwa, Vishesh↗

Proxy-Based Bayesian Inversion Of Poroelastic Simulations To Interpret Strain Tensor Data Measured During Well Testing

The long runtimes of 3D poroelastic numerical simulators makes it impractical to interpret deformation datasets using many inversion schemes. Recent advances in instrumentation have made it possible to measure the strain tensor during well testing, but the lack of robust inversion methods is limiting the ability to interpret these data. We have developed an inversion workflow that reduces the number of computations required to complete a Bayesian inversion using DREAMzs. The workflow trains a KNN model using output from the poroelastic simulator, and then uses the KNN model as a proxy for the simulator during inversion. The workflow also includes a strategy for ensuring the results from the proxy model converge to the results from the simulator, ensuring the accuracy of the final results. An idealized example configured to represent a well test in a deep aquifer is used to verify that the workflow correctly identifies parameters and characterizes noise. Field data measured using strainmeters during an injection test at an oil reservoir in Oklahoma are used to evaluate performance with a real dataset. The workflow identified 265 history matching solutions out of 1240 total simulation runs (21% acceptance ratio), and the results are used to characterize posterior parameter distribution and evaluate the prediction uncertainty. This approach makes it feasible to invert strain data measured during well testing and this has the potential to improve the characterization of aquifers and reservoirs.

Roudini, Soheil↗

Determining the Bayesian optimal sampling strategy in a hierarchical system.

Consider a classic hierarchy tree as a basic model of a 'system-of-systems' network, where each node represents a component system (which may itself consist of a set of sub-systems). For this general composite system, we present a technique for computing the optimal testing strategy, which is based on Bayesian decision analysis. In previous work, we developed a Bayesian approach for computing the distribution of the reliability of a system-of-systems structure that uses test data and prior information. This allows for the determination of both an estimate of the reliability and a quantification of confidence in the estimate. Improving the accuracy of the reliability estimate and increasing the corresponding confidence require the collection of additional data. However, testing all possible sub-systems may not be cost-effective, feasible, or even necessary to achieve an improvement in the reliability estimate. To address this sampling issue, we formulate a Bayesian methodology that systematically determines the optimal sampling strategy under specified constraints and costs that will maximally improve the reliability estimate of the composite system, e.g., by reducing the variance of the reliability distribution. This methodology involves calculating the 'Bayes risk of a decision rule' for each available sampling strategy, where risk quantifies the relative effect that each sampling strategy could have on the reliability estimate. A general numerical algorithm is developed and tested using an example multicomponent system. The results show that the procedure scales linearly with the number of components available for testing.

99 GENERAL AND MISCELLANEOUS↗

Bayesian estimation of life parameters in the Weibull distribution.

Development of a Bayesian analysis of the scale and shape parameters in the Weibull distribution and the corresponding reliability function with respect to the usual life-testing procedures. For the scale parameter theta, Bayesian estimates of theta and reliability are obtained for the uniform, exponential, and inverted gamma prior probability densities. Bhattacharya's results (1967) for the one-parameter exponential life-testing distribution are reduced to a special case of these results. A fully Bayesian analysis of both the scale and shape parameters is developed by assuming independent prior distributions; since in the latter case, analytical tractability is not possible, Bayesian estimates are obtained through a conjunction of Monte Carlo simulation and numerical-integration techniques. In both cases, a computer simulation is carried out, and a comparison is made between the Bayesian and the corresponding minimum-variance unbiased, or maximum likelihood, estimates. As expected, the Bayesian estimates are superior.

Canavos, G. C.↗