Search NASA⌕ Search

SEARCH · Search NASA

Results for “Model selection”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Contextual Active Online Model Selection with Expert Advice

How can we collect the most useful labels to learn a model selection policy, when presented with arbitrary heterogeneous data streams? In this paper, we formulate this task as a contextual active model selection problem, where at each round the learner receives an unlabeled data point along with a context. The goal is to output the best model for any given context without obtaining an excessive amount of labels. In particular, we focus on the task of selecting pre-trained classifiers, and propose a contextual active model selection algorithm (CAMS), which relies on a novel uncertainty sampling query criterion defined on a given policy class for adaptive model selection. In comparison to prior art, our algorithm does not assume a globally optimal model. We provide rigorous theoretical analysis for the regret and query complexity under both adversarial and stochastic settings. Our experiments on several benchmark classification datasets demonstrate the algorithm’s effectiveness in terms of both regret and query complexity. Notably, to achieve the same accuracy, CAMS incurs less than 10% of the label cost when compared to the best online model selection baselines on CIFAR10.

Liu, Xuefeng↗

Inflation model selection meets dark radiation

We investigate how inflation model selection is affected by the presence of additional free-streaming relativistic degrees of freedom, i.e. dark radiation. We perform a full Bayesian analysis of both inflation parameters and cosmological parameters taking reheating into account self-consistently. We compute the Bayesian evidence for a few representative inflation scenarios in both the standard ΛCDM model and an extension including dark radiation parametrised by its effective number of relativistic species N {sub eff}. Using a minimal dataset (Planck low-ℓ polarisation, temperature power spectrum and lensing reconstruction), we find that the observational status of most inflationary models is unchanged. The exceptions are potentials such as power-law inflation that predict large values for the scalar spectral index that can only be realised when N {sub eff} is allowed to vary. Adding baryon acoustic oscillations data and the B-mode data from BICEP2/Keck makes power-law inflation disfavoured, while adding local measurements of the Hubble constant H {sub 0} makes power-law inflation slightly favoured compared to the best single-field plateau potentials. This illustrates how the dark radiation solution to the H {sub 0} tension would have deep consequences for inflation model selection.

79 ASTRONOMY AND ASTROPHYSICS↗

Hybrid nested sampling algorithm for Bayesian model selection applied to inverse subsurface flow problems

A Hybrid Nested Sampling (HNS) algorithm is proposed for efficient Bayesian model calibration and prior model selection. The proposed algorithm combines, Nested Sampling (NS) algorithm, Hybrid Monte Carlo (HMC) sampling and gradient estimation using Stochastic Ensemble Method (SEM). NS is an efficient sampling algorithm that can be used for Bayesian calibration and estimating the Bayesian evidence for prior model selection. Nested sampling has the advantage of computational feasibility. Within the nested sampling algorithm, a constrained sampling step is performed. For this step, we utilize HMC to reduce the correlation between successive sampled states. HMC relies on the gradient of the logarithm of the posterior distribution, which we estimate using a stochastic ensemble method based on an ensemble of directional derivatives. SEM only requires forward model runs and the simulator is then used as a black box and no adjoint code is needed. The developed HNS algorithm is successfully applied for Bayesian calibration and prior model selection of several nonlinear subsurface flow problems.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Quantitative Parameter Estimation, Model Selection, and Variable Selection in Battery Science

Numerical physics-based models are useful in understanding battery performance and devel- oping optimal battery design architectures. Data science developments have enabled software algorithms to perform data analysis and decision making that traditionally could only be per- formed by technical experts. Traditional workflows of model development - manual parameter estimation through visual comparison of simulations to experimental observations, and model discrimination through expert intuition - are time-consuming, and difficult to justify. This paper compares the conclusions derived from computationally scalable algorithms to conclu- sions developed by expert researchers. This paper illustrates how data science techniques, such as cross-validation and lasso regression, can be used to augment physics-based simulations to perform data analysis such as parameter estimation, model selection, variable selection, and model-guided design of experiment. The validation of these algorithms is that they produce results similar to those of the expert modeler. The algorithms outlined are well-established and the approaches are general, so can be applied to a variety of battery chemistries and architectures. The conclusions reached using these approaches are in agreement with expert analysis (literature results), were reached with minimal human intervention, and provide quantitative justification. By minimizing the amount of expert time, and providing rigorous quantitative justifications, these methods may accelerate battery development.

25 ENERGY STORAGE↗

Agent-Based vs. Equation-based Epidemiological Models:A Model Selection Case Study

This paper is motivated by the need to design model validation strategies for epidemiological disease-spread models. We consider both agent-based and equation-based models of pandemic disease spread and study the nuances and complexities one has to consider from the perspective of model validation. For this purpose, we instantiate an equation based model and an agent based model of the 1918 Spanish flu and we leverage data published in the literature for our case- study. We present our observations from the perspective of each implementation and discuss the application of model-selection criteria to compare the risk in choosing one modeling paradigm to another. We conclude with a discussion of our experience and document future ideas for a model validation framework.

agent-based models↗

An empirical approach to model selection: weak lensing and intrinsic alignments

ABSTRACT In cosmology, we routinely choose between models to describe our data, and can incur biases due to insufficient models or lose constraining power with overly complex models. In this paper, we propose an empirical approach to model selection that explicitly balances parameter bias against model complexity. Our method uses synthetic data to calibrate the relation between bias and the χ2 difference between models. This allows us to interpret χ2 values obtained from real data (even if catalogues are blinded) and choose a model accordingly. We apply our method to the problem of intrinsic alignments – one of the most significant weak lensing systematics, and a major contributor to the error budget in modern lensing surveys. Specifically, we consider the example of the Dark Energy Survey Year 3 (DES Y3), and compare the commonly used non-linear alignment (NLA) and tidal alignment and tidal torque (TATT) models. The models are calibrated against bias in the Ωm–S8 plane. Once noise is accounted for, we find that it is possible to set a threshold Δχ2 that guarantees an analysis using NLA is unbiased at some specified level Nσ and confidence level. By contrast, we find that theoretically defined thresholds (based on, e.g. p-values for χ2) tend to be overly optimistic, and do not reliably rule out cosmological biases up to ∼1–2σ. Considering the real DES Y3 cosmic shear results, based on the reported difference in χ2 from NLA and TATT analyses, we find a roughly $30{{\ \rm per\ cent}}$ chance that were NLA to be the fiducial model, the results would be biased (in the Ωm–S8 plane) by more than 0.3σ. More broadly, the method we propose here is simple and general, and requires a relatively low level of resources. We foresee applications to future analyses as a model selection tool in many contexts.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

On Optimal Input Design and Model Selection for Communication Channels

In this paper, the optimal model (structure) selection and input design which minimize the worst case identification error for communication systems are provided. The problem is formulated using metric complexity theory in a Hilbert space setting. It is pointed out that model selection and input design can be handled independently. Kolmogorov n-width is used to characterize the representation error introduced by model selection, while Gel fand and Time n-widths are used to represent the inherent error introduced by input design. After the model is selected, an optimal input which minimizes the worst case identification error is shown to exist. In particular, it is proven that the optimal model for reducing the representation error is a Finite Impulse Response (FIR) model, and the optimal input is an impulse at the start of the observation interval. FIR models are widely popular in communication systems, such as, in Orthogonal Frequency Division Multiplexing (OFDM) systems.

fast identification↗

Progress Towards Prognostic Health Management of Passive Components in Advanced Reactors – Model Selection and Evaluation

This paper presents recent progress towards developing a prognostic health management framework for passive components of advanced reactors (AR). The focus of this paper is on lifecycle prognostics for passive components using a Bayesian prognostic algorithm that provides a natural framework for incorporating different sources of variability, and uncertainties inherent in the operations of AR. High-temperature creep damage, a prototypic failure mechanism in AR materials, is used as the context for this research. A Bayesian model selection approach is implemented to select the appropriate creep degradation model at any given time, using relevant sensor measurements reflecting the material degradation state. The model selection approach, based on reversible jump Markov chain Monte Carlo methods, is integrated with Bayesian particle filter-based prognostic framework. The proposed approach is evaluated using strain measurements obtained from accelerated creep testing of stainless steel specimens. Results indicate feasibility of the proposed approach in accurately identifying the creep degradation stage from the available measurements at a given time. Effect of uncertainties in material degradation model and measurement noise on the performance of the prognostic algorithm is also investigated.

advanced reactor prognostics↗

Simulating the Dynamics of the CX-100 Wind Turbine Blade: Model Selection Using a Robustness Criterion

Several plausible modeling strategies are available to develop finite element (FE) models of ever-increasingly complex phenomena. Expert judgment is typically used to choose which strategy to employ, while the “best” modeling approach remains unknown. This paper proposes a decision analysis framework that offers a systematic and rigorous methodology for comparing plausible modeling strategies. The proposed framework departs from the conventional approach that considers only test-analysis correlation to select the model that provides the highest degree of fidelity-to-data. The novelty of the framework lies in an exploration of the trade-offs between robustness to uncertainty and fidelity-to-data. Exploring robustness to model imprecision and inexactness, in addition to fidelity-to-data, lends credibility to the simulation by guaranteeing that its predictions can be trusted even if some of the modeling assumptions and input parameters are incorrect. To demonstrate this approach, an experimental configuration is analyzed in which large masses are used to load the CX-100 wind turbine blade in bending during vibration testing. Two plausible simulations are developed with differing strategies to implement these large masses using (i) a combination of point-mass and spring elements or (ii) solid elements. In this paper, the authors study the ability of the two FE models to predict the experimentally obtained natural frequencies, and the robustness of these competing models to uncertainties in the input parameters. Considering robustness for model selection provides the extent to which prediction accuracy deteriorates as the lack-of-knowledge increases. Therefore, the preferable modeling strategy is the one that offers the best compromise between fidelity-to-data and robustness to uncertainty. To predict the bending vibration of the CX-100 wind turbine blade, it is observed that the modeling strategy with solid elements is far superior to the other one in its ability to provide a compromise between fidelity-to-data and robustness to the modeling assumptions. (Approved for unlimited, public release on October xx, 2012, LA-UR-12-xxxx, Unclassified.)

17 WIND ENERGY↗

'Chain pooling' model selection as developed for the statistical analysis of a rotor burst protection experiment

A statistical decision procedure called chain pooling had been developed for model selection in fitting the results of a two-level fixed-effects full or fractional factorial experiment not having replication. The basic strategy included the use of one nominal level of significance for a preliminary test and a second nominal level of significance for the final test. The subject has been reexamined from the point of view of using as many as three successive statistical model deletion procedures in fitting the results of a single experiment. The investigation consisted of random number studies intended to simulate the results of a proposed aircraft turbine-engine rotor-burst-protection experiment. As a conservative approach, population model coefficients were chosen to represent a saturated 2 to the 4th power experiment with a distribution of parameter values unfavorable to the decision procedures. Three model selection strategies were developed.

Holms, A. G.↗

A meta-learning based distribution system load forecasting model selection framework

This paper presents a meta-learning based, automatic distribution system load forecasting model selection framework. Furthermore, the framework includes the following processes: feature extraction, candidate model preparation and labeling, offline training, and online model recommendation. Using load forecasting needs and data characteristics as input features, multiple metalearners are used to rank the candidate load forecast models based on their forecasting accuracy. Then, a scoring-voting mechanism is proposed to weights recommendations from each meta-leaner and make the final recommendations. Heterogeneous load forecasting tasks with different temporal and technical requirements at different load aggregation levels are set up to train, validate, and test the performance of the proposed framework. Simulation results demonstrate that the performance of the meta-learning based approach is satisfactory in both seen and unseen forecasting tasks.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Multiverse: Bayesian model selection for neural networks

Multiverse is a code repository for a set of tools for Bayesian model selection for neural networks. The goal of the tools is to provide capabilities for selecting among prior and model specifications for Bayesian neural networks. Multiverse will include tools for creating Bayesian neural networks, evaluating the Bayesian model evidence, and performing inference in Bayesian neural networks. These components are written in Python, a high-level programming language that takes advantage of the Python ecosystem of high-quality open-source packages for machine learning and signal processing.

Klein, Natalie↗

Deconvolving distribution of relaxation times, resistances and inductance from electrochemical impedance spectroscopy via statistical model selection: Exploiting structural-sparsity regularization and data-driven parameter tuning

The distribution of relaxation times (DRT) has drawn increasing attention for interpreting electrochemical impedance spectroscopy (EIS). Deconvolution of DRT from EIS is a challenging ill-posed problem that requires regularization methods. In this work, we formulate DRT reconstruction task as a statistical model selection problem with structural-sparsity penalties. We utilize the Elastic net regularization that simultaneously benefits from Ridge and Lasso regularizations with optimal tuning parameter automatically determined by the information criteria. We benchmark our approach on four synthetic experiments (a ZARC element, ZARC mixtures, a RC circuit and a Fractal element) and two real EIS datasets of a Lithium ion battery and an organic-inorganic halide class of perovskites in oxygen environment at different gas pressures. We demonstrate the superiority of proposed model selection procedure, that is capable of eliminating pseudo peaks and representing asymmetries in DRT as well as precisely estimating resistances. Here, we highlight our approach is robust to reducing and subsampling EIS frequency range, making it a promising tool for timing-resolved, localized and large scale EIS data analysis. For the Lithium ion battery data analysis, we extend the classical DRT model to incorporate the inductive effect and illustrate DRT as a guidance for equivalent circuit modeling to refine impedance reconstruction at low risks of overfitting. Furthermore, the structural-sparsity regularization could be extended for multidimensional and Bayesian EIS data analysis.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Robust Decision-making Applied to Model Selection

The scientific and engineering communities are relying more and more on numerical models to simulate ever-increasingly complex phenomena. Selecting a model, from among a family of models that meets the simulation requirements, presents a challenge to modern-day analysts. To address this concern, a framework is adopted anchored in info-gap decision theory. The framework proposes to select models by examining the trade-offs between prediction accuracy and sensitivity to epistemic uncertainty. The framework is demonstrated on two structural engineering applications by asking the following question: Which model, of several numerical models, approximates the behavior of a structure when parameters that define each of those models are unknown? One observation is that models that are nominally more accurate are not necessarily more robust, and their accuracy can deteriorate greatly depending upon the assumptions made. It is posited that, as reliance on numerical models increases, establishing robustness will become as important as demonstrating accuracy.

17 WIND ENERGY↗

Characterizing Quantum Devices Using Model Selection

Quantum information processors are a new kind of computing device with the potential to solve certain problems very quickly. The quantum bits, or "ʺqubits,"ʺ that enable such processors are being developed at Sandia and around the world. In support of this effort, Sandia’s Center for Computing Research is designing methods and algorithms to characterize and debug the behavior of experimental qubits, such as trapped ions and electron spins in silicon. This is a challenge, because qubits are hard to characterize for exactly the same reason that they are powerful. Their behavior (and misbehavior) is far richer and more complicated than that of classical bits! The most reliable models of quantum errors have infinitely many parameters, and we never have enough data to fit all of them. Until now, researchers simply threw out most of these parameters resulting in a crude approximation that makes the problem tractable, but isn’t reliable or accurate enough for Sandia’s experimental work. Our new techniques use statistical model selection to identify “significant” error parameters on the fly, applying a data-­‐‑driven information criterion to choose a Hilbert space that describes the data well without overfitting. To find a reliable criterion, we needed to evaluate many candidates on an enormous corpus of test cases–for which we relied on Red Sky’s ability to run parallel Monte Carlo simulations. Our computations demonstrate how classic model selection methods have to be modified for quantum devices. These results will enable new techniques that identify the effective dimension of as-­‐‑built qubits on the fly, and then use this information to reliably pin down error behaviors and produce successively better and more useful qubits. This project highlights the close and productive relationship between Sandia and the University of New Mexico, where Travis Scholten is working towards a Ph.D. in physics. Both the intellectual environment of UNM and the high-­‐‑powered supercomputing available at Sandia were critical ingredients in achieving our research goals.

97 MATHEMATICS AND COMPUTING↗