Search NASA⌕ Search

SEARCH · Search NASA

Results for “Model misspecification”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events

The Weather Risk Attribution Forecast (WRAF) is a forecasting tool that uses output from global climate models to make simultaneous attribution statements about whether and how greenhouse gas emissions have contributed to extreme weather across the globe. However, in conducting a large number of simultaneous hypothesis tests, the WRAF is prone to identifying false “discoveries.” A common technique for addressing this multiple testing problem is to adjust the procedure in a way that controls the proportion of true null hypotheses that are incorrectly rejected, or the false discovery rate (FDR). Unfortunately, generic FDR procedures suffer from low power when the hypotheses are dependent, and techniques designed to account for dependence are sensitive to misspecification of the underlying statistical model. In this paper, we develop a Bayesian decision-theoretical approach for dependent multiple testing and a nonparametric hierarchical statistical model that flexibly controls false discovery and is robust to model misspecification. We illustrate the robustness of our procedure to model error with a simulation study, using a framework that accounts for generic spatial dependence and allows the practitioner to flexibly specify the decision criteria. Finally, we apply our procedure to several seasonal forecasts and discuss implementation for the WRAF workflow. Lastly, supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

54 ENVIRONMENTAL SCIENCES↗

Deep Learning and Likelihood Approaches for Viral Phylogeography Converge on the Same Answers Whether the Inference Model Is Right or Wrong

Abstract Analysis of phylogenetic trees has become an essential tool in epidemiology. Likelihood-based methods fit models to phylogenies to draw inferences about the phylodynamics and history of viral transmission. However, these methods are often computationally expensive, which limits the complexity and realism of phylodynamic models and makes them ill-suited for informing policy decisions in real-time during rapidly developing outbreaks. Likelihood-free methods using deep learning are pushing the boundaries of inference beyond these constraints. In this paper, we extend, compare, and contrast a recently developed deep learning method for likelihood-free inference from trees. We trained multiple deep neural networks using phylogenies from simulated outbreaks that spread among 5 locations and found they achieve close to the same levels of accuracy as Bayesian inference under the true simulation model. We compared robustness to model misspecification of a trained neural network to that of a Bayesian method. We found that both models had comparable performance, converging on similar biases. We also implemented a method of uncertainty quantification called conformalized quantile regression that we demonstrate has similar patterns of sensitivity to model misspecification as Bayesian highest posterior density (HPD) and greatly overlap with HPDs, but have lower precision (more conservative). Finally, we trained and tested a neural network against phylogeographic data from a recent study of the SARS-Cov-2 pandemic in Europe and obtained similar estimates of region-specific epidemiological parameters and the location of the common ancestor in Europe. Along with being as accurate and robust as likelihood-based methods, our trained neural networks are on average over 3 orders of magnitude faster after training. Our results support the notion that neural networks can be trained with simulated data to accurately mimic the good and bad statistical properties of the likelihood functions of generative phylogenetic models.

Evolutionary Biology↗

Quantifying Model Form Uncertainty of Epidemic Forecasting Models from Incidence Data

Objective We present a mathematical framework for non-parametric estimation of the force of infection, together with statistical upper and lower confidence bands. The resulting estimates allow to assess how well simpler models, such as SEIR, fit the observed time series of incidence data.IntroductionUncertainty Quantification (UQ), the ability to quantify the impact of sample-to-sample variations and model misspecification on predictions and forecasts, is a critical aspect of disease surveillance. While quantifying the impact of stochastic uncertainty in the data is well understood, quantifying the impact of model misspecification is significantly harder. For the latter, one needs a "universal model" to which more restrictive parametric models are compared too.MethodsThis talk presents a useful modeling framework for time series of incidence data from contagious diseases that enables one to identify and quantify the impact of model form uncertainty. Specifically, we propose to focus on estimating the timedependent force of infection. The latter is a universal parameters for all contagious disease model. Using a machine learning technique for estimating monotone functions, i.e., isotonic regression and its variants, one can estimate the force ofinfection without addtional assumptions. We note that most contagious disease model do satisfy this monotonicity assumption, due to a combination of factors: depletion of susceptibles, implementation of mitigation strategies, behavior change, etc. Comparing the resulting "non-parametric" estimate with parametric estimates, obtained by fitting an SEIR for example, can reveal model deficiencies and help quantify model form uncertainties.Finally, we discuss how ideas from "strict bound theory" can be used to develop upper and lower uncertainty bands for force of infection that acknowledge the intrinsic stochasticity in the data.ResultsWe demonstrate the application of the methodology to weekly Influenza Like Illness (ILI) incidence data from France andcompare the results to fitted SIR and SEIR models. This comparison can be seen as a nonparametric goodness of fit test, providing one with tools to do simple model selection.ConclusionsWe present a novel and flexible model to statistically describe the force of infection as a function of time. Comparing the fit to incidence data of that model with the fit of simpler parametric models enables the quantification of model form uncertainty and associated model selection.

Hengartner, Nicholas↗

Robust Consumption-Investment Problem on Infinite Horizon

In our paper we consider an infinite horizon consumption-investment problem under a model misspecification in a general stochastic factor model. We formulate the problem as a stochastic game and finally characterize the saddle point and the value function of that game using an ODE of semilinear type, for which we provide a proof of an existence and uniqueness theorem for its solution. Such equation is interested on its own right, since it generalizes many other equations arising in various infinite horizon optimization problems.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Modeling neutrino-induced scale-dependent galaxy clustering for photometric galaxy surveys

Abstract The increasing statistical precision of photometric redshift surveys requires improved accuracy of theoretical predictions for large-scale structure observables to obtain unbiased cosmological constraints. In ΛCDM cosmologies, massive neutrinos stream freely at small cosmological scales, suppressing the small-scale power spectrum. In massive neutrino cosmologies, galaxy bias modeling needs to accurately relate the scale-dependent growth of the underlying matter field to observed galaxy clustering statistics. In this work, we implement a computationally efficient approximation of the neutrino-induced scale-dependent bias (NISDB). Through simulated likelihood analyses of Dark Energy Survey Year 3 (DESY3) and Legacy Survey of Space and Time Year 1 (LSSTY1) synthetic data that contain an appreciable NISDB, we examine the impact of linear galaxy bias and neutrino mass modeling choices on cosmological parameter inference. We find model misspecification of the NISDB approximation and neutrino mass models to decrease the constraining power of photometric galaxy surveys and cause parameter biases in the cosmological interpretation of future surveys. We quantify these biases and devise mitigation strategies.

Astronomy & Astrophysics↗

Statistically Rigorous Uncertainty Quantification for Physical Parameter Model Calibration with Functional Output

In experiments conducted on the Z-machine at Sandia National Laboratories, dynamic material properties cannot be analyzed using traditional analytic methods, necessitating solving an inverse problem. Bayesian model calibration is a statistical framework for solving an inverse problem to estimate parameters input into a computational model in the presence of multiple uncertainties. Disentangling input parameter uncertainty and model misspecification is often poorly identified problem. When using computational models for physical parameter estimation, the issue of parameter identifiability must be carefully considered to obtain accurate and precise estimates of physical parameters. Additionally, in dynamic material properties applications, the experimental output is a function, velocity over time. While we can sample an arbitrarily large number of points from the measured velocity, these curves only contain a finite amount of information about the calibration parameters. In this report, we propose modifications to the Bayesian model calibration framework to simplify and improve the estimation of physical parameters with functional outputs. Specifically, we propose scaling the likelihood function by an effective sample size rather than modeling the discrepancy function; and modularizing input nuisance parameters with weakly identified parameters. We evaluate the performance of these proposed methods using a statistical simulation study and then apply these methods to estimate parameters of the tantalum equation of state. We conclude that these proposed methods can provide simple, fast, and statistically valid alternatives to the full Bayesian model calibration procedure; and that these methods can be used to estimate parameters of the equation of state for tantalum.

97 MATHEMATICS AND COMPUTING↗

Photometric redshifts and intrinsic alignments: Degeneracies and biases in the 3 × 2 pt analysis

We present a systematic study of cosmological parameter bias in weak lensing and large-scale structure analyses for upcoming imaging surveys induced by the interplay of intrinsic alignments (IA) and photometric redshift (photo-z) model misspecification error. We first examine the degeneracies between the parameters of the tidal alignment–tidal torquing (TATT) model for IA and of a photo-z model including a mean shift ( Δ z ¯ ) and variance ( σ z ) for each tomographic bin of lenses and sources, under a variety of underlying true IA behaviors. We identify strong degeneracies between: (1) the redshift scaling of the tidal alignment amplitude and the mean shift and variances of source bins, (2) the redshift scaling of the tidal torquing amplitude and the variance of the lowest- z source bin, and (3) the IA source density weighting and the mean shift and variance of several source bins. We then use this information to guide our exploration of the level of cosmological parameter bias which can be induced given incorrect modeling of IA, photo-z, or both. We find that marginalizing over all the parameters of TATT is generally sufficient to preclude cosmological parameter bias in the scenarios we consider. However, this does not necessarily mean that IA and photo-z parameters are themselves unbiased, nor does it mean that the best-fit model is a good fit to the data. We also find scenarios where the inferred parameters produce χ d . o . f . 2 values indicative of a good fit but cosmological parameter bias is significant, particularly when the IA source density weighting parameter is not marginalized over. Published by the American Physical Society 2024

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

The Optical and Infrared Are Connected

Galaxies are often modeled as composites of separable components with distinct spectral signatures, implying that different wavelength ranges are only weakly correlated. They are not. We present a data-driven model that exploits subtle correlations between physical processes to accurately predict infrared (IR) Wide-field Infrared Survey Explorer (WISE) photometry from a neural summary of optical Sloan Digital Sky Survey spectra. The model achieves accuracies of $χ^{2}_{N} ≈ 1$ for all photometric bands in WISE, as well as good colors. We are able to tightly constrain typically IR-derived properties, e.g., the bolometric luminosities of active galactic nuclei (AGN) and dust parameters such as q PAH . We also test whether current spectral energy distribution (SED) fitting methods reproduce such panchromatic relations, but find their predictions biased and overconfident, likely due to model misspecification, with correlated biases in star-formation rates (SFRs) and AGN luminosities being most evident. To help improve SED models, we determine which features of the optical spectrum are responsible for our improved predictions, and identify several lines (Ca II , Sr II , Fe I , [O II ], and Hα), which point to the complex chronology of star formation and chemical enrichment being incorrectly modeled.

Jespersen, Christian Kragh [Princeton Univ., NJ (U↗

Sequential Monte Carlo for Cut-Bayesian Posterior Computation

We propose a sequential Monte Carlo (SMC) method to efficiently and accurately compute cut-Bayesian posterior quantities of interest, variations of standard Bayesian approaches constructed primarily to account for model misspecification. We prove finite sample concentration bounds for estimators derived from the proposed method along with a linear tempering extension and apply these results to a realistic setting where a computer model is misspecified. We then illustrate the SMC method for inference in a modular chemical reactor example that includes submodels for reaction kinetics, turbulence, mass transfer, and diffusion. The samples obtained are commensurate with a direct-sampling approach that consists of running multiple Markov chains, with computational efficiency gains using the SMC method. Overall, the SMC method presented yields a novel, rigorous approach to computing with cut-Bayesian posterior distributions.

97 MATHEMATICS AND COMPUTING↗

Dark Energy Survey: Modeling strategy for multiprobe cluster cosmology and validation for the Full Six-year Dataset

We introduce an updated To&Krause2021 model for joint analyses of cluster abundances and large-scale two-point correlations of weak lensing and galaxy and cluster clustering (termed CL+3x2pt analysis) and validate that this model meets the systematic accuracy requirements of analyses with the statistical precision of the final Dark Energy Survey (DES) Year 6 (Y6) dataset. The validation program consists of two distinct approaches, (1) identification of modeling and parameterization choices and impact studies using simulated analyses with each possible model misspecification (2) end-to-end validation using mock catalogs from customized Cardinal simulations that incorporate realistic galaxy populations and DES-Y6-specific galaxy and cluster selection and photometric redshift modeling, which are the key observational systematics. In combination, these validation tests indicate that the model presented here meets the accuracy requirements of DES-Y6 for CL+3x2pt based on a large list of tests for known systematics. In addition, we also validate that the model is sufficient for several other data combinations: the CL+GC subset of this data vector (excluding galaxy--galaxy lensing and cosmic shear two-point statistics) and the CL+3x2pt+BAO+SN (combination of CL+3x2pt with the previously published Y6 DES baryonic acoustic oscillation and Y5 supernovae data).

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Dark energy survey: Modeling strategy for multiprobe cluster cosmology and validation for the full six-year dataset

Here, we introduce an updated To&Krause2021 model for joint analyses of cluster abundances and large-scale two-point correlations of weak lensing and galaxy and cluster clustering (termed CL+3×2 pt analysis) and validate that this model meets the systematic accuracy requirements of analyses with the statistical precision of the final Dark Energy Survey (DES) Year 6 (Y6) dataset. The validation program consists of two distinct approaches, (i) identification of modeling and parametrization choices and impact studies using simulated analyses with each possible model misspecification and (ii) end-to-end validation using mock catalogs from customized Cardinal simulations that incorporate realistic galaxy populations and DES-Y6-specific galaxy and cluster selection and photometric redshift modeling, which are the key observational systematics. In combination, these validation tests indicate that the model presented here meets the accuracy requirements of DES-Y6 for CL+3×2 pt based on a large list of tests for known systematics. In addition, we also validate that the model is sufficient for several other data combinations: the CL+GC subset of this data vector (excluding galaxy–galaxy lensing and cosmic shear two-point statistics) and the CL+3×2 pt+BAO+SN (combination of CL+3×2 pt with the previously published Y6 DES baryonic acoustic oscillation and Y5 supernovae data).

79 ASTRONOMY AND ASTROPHYSICS↗

Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons

Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator mappings between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. Further, to demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed. Further, to help facilitate the deployment of UQ in Scientific Machine Learning research and practice, we present and develop in [1] an open-source Python library (github.com/Crunch-UQ4MI/neuraluq), termed NeuralUQ, that is accompanied by an educational tutorial and additional computational experiments.

11 physics-informed neural networks↗

Accurate field-level weak lensing inference for precision cosmology

We present miko, a catalog-to-cosmology pipeline for general flat-sky field-level inference, which provides access to cosmological information beyond the two-point statistics. In the context of weak lensing, we identify several new field-level analysis systematics (such as aliasing, Fourier mode-coupling, and density-induced shape noise), quantify their impact on cosmological constraints, and correct the biases to a percent level. Next, we find that model misspecification can lead to both absolute bias and incorrect uncertainty quantification for the inferred cosmological parameters in realistic simulations. The Gaussian map prior infers unbiased cosmological parameters, regardless of the true data distribution, but it yields overconfident uncertainties. The log-normal map prior quantifies the uncertainties accurately, although it requires careful calibration of the shift parameters for unbiased cosmological parameters. Here, we demonstrate systematics control down to the 2% level for both models, making them suitable for ongoing weak lensing surveys.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗