Results for “Over-parameterization”

DOE OSTI2023

On the influence of over-parameterization in manifold based surrogates and deep neural operators

Constructing accurate and generalizable approximators (surrogate models) for complex physico-chemical processes exhibiting highly non-smooth dynamics is challenging. The main question is what type of surrogate models we should construct and should these models be under-parameterized or over-parameterized. In this work, we propose new developments and perform comparisons for two promising approaches: manifold-based polynomial chaos expansion (m-PCE) and the deep neural operator (DeepONet), and we examine the effect of over-parameterization on generalization. While m-PCE enables the construction of a mapping by first identifying low-dimensional embeddings of the input functions, parameters, and quantities of interest (QoIs), a neural operator learns the nonlinear mapping via the use of deep neural networks. Here, we demonstrate the performance of these methods in terms of generalization accuracy by solving the 2D time-dependent Brusselator reaction-diffusion system with uncertainty sources, modeling an autocatalytic chemical reaction between two species. We first propose an extension of the m-PCE by constructing a mapping between latent spaces formed by two separate embeddings of the input functions and the output QoIs. To further enhance the accuracy of the DeepONet, we introduce weight self-adaptivity in the loss function. We demonstrate that the performance of m-PCE and DeepONet is comparable for cases of relatively smooth input-output mappings. However, when highly non-smooth dynamics is considered, DeepONet shows higher approximation accuracy. We also find that for m-PCE, modest over-parameterization leads to better generalization, both within and outside of distribution, whereas aggressive over-parameterization leads to over-fitting. In contrast, an even highly over-parameterized DeepONet leads to better generalization for both smooth and non-smooth dynamics. Furthermore, we compare the performance of the above models with another recently proposed operator learning model, the Fourier Neural Operator, and show that its over-parameterization also leads to better generalization. Taken together, our studies show that m-PCE can provide very good accuracy at very low training cost, whereas a highly over-parameterized DeepONet can provide better accuracy and robustness to noise but at higher training cost. In both methods, the inference cost is negligible.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

A deep learning-based Bayesian framework for high-resolution calibration of building energy models

Calibrating building energy models (BEMs), i.e., closing discrepancy between modeling and field measurements, is of significance to support its applications in building sustainability and resilience analysis. However, as being widely used in practice, current Bayesian calibration is mostly performed in low-resolution (annual or monthly), instead of high-resolution (hourly or sub-hourly), which is crucial to support emerging BEM applications, such as building-renewable energy integration (demand response) and smart control. This is attributable to the gaps in current Bayesian calibration process, including (1) difficulty in supporting reliable high-resolution calibration with over-parameterization and multi-solution issues, (2) inadequacy of meta-model to capture temporal building dynamics in high-resolution, and (3) excessive computational burdens of covariance matrix calculation in Bayesian inference. Therefore, to close these gaps, this research proposes a novel deep learning-based Bayesian calibration framework, involving pre-calibration mechanism, Long Short-Term Memory as surrogate models, and simplified covariance matrix calculation, to calibrate BEMs in high temporal resolution (i.e., hourly) with enhanced accuracy and computational efficiency. Finally, the case study demonstrates its effectiveness to match modeling outcomes with measurements and realize CV-RMSE of < 30 % and NMBE of < 6 % in hourly resolution, as well as a significant reduction of calibration time (by > 99 %, from > 600 h to ~ 1.5 h).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2023

Comparative verification of hydro-mechanical fracture behavior: Task G of international research project DECOVALEX–2023

Numerical simulations become a necessity when experimental approaches cannot cover the required physical and time scale of interest. One of such area is a simulation of long-term host rock behaviors for nuclear waste disposal and simulation tools involved in the assessment must go through rigorous validation tests. The DECOVALEX project (Development of COupled models and their VAlidation against EXperiments) is dedicated to this purpose by international participants. This work is part of the ongoing phase DECOVALEX–2023 (D–2023, Task G) particularly aiming to simulate fracture behaviors under various conditions. Here, in this study, we cross-verified a variety of numerical methods including continuous and discontinuous approaches against four benchmark exercises with emphasis on numerical accuracy and parameterization of the various numerical approaches. The systematic inter-comparisons of test cases highlight advantages and disadvantages of the different numerical models. Numerical details on discretization effects (e.g. mesh density and orientation) and domain size were investigated in detail for practical applications. It became evident that meticulous attention to mesh resolution and domain size is imperative for achieving accurate numerical simulations, even for static cracks. Moreover, when comparing numerical methods to closed-form solutions for static cracks, all models successfully reproduced the maximum crack opening but encountered challenges near the crack tips. Finally, the paper discusses how to convert between and therefore compare parameters of various numerical approaches. Our benchmark studies reveal that each model necessitates a distinct number of parameters, even in simple scenarios like static crack aperture benchmarks. It is generally more practical to employ fewer parameters to mitigate model over-parameterization and enhance experimental feasibility.

58 GEOSCIENCES↗

DOE OSTI2020

Nonlinear sparse Bayesian learning for physics-based models

This paper addresses the issue of overfitting while calibrating unknown parameters of over-parameterized physics-based models with noisy and incomplete observations. Here, a semi-analytical Bayesian framework of nonlinear sparse Bayesian learning (NSBL) is proposed to identify sparsity among model parameters during Bayesian inversion. NSBL offers significant advantages over machine learning algorithm of sparse Bayesian learning (SBL) for physics-based models, such as 1) the likelihood function or the posterior parameter distribution is not required to be Gaussian, and 2) prior parameter knowledge is incorporated into sparse learning (i.e. not all parameters are treated as questionable). NSBL employs the concept of automatic relevance determination (ARD) to facilitate sparsity among questionable parameters through parameterized prior distributions. The analytical tractability of NSBL is enabled by employing Gaussian ARD priors and by building a Gaussian mixture-model approximation of the posterior parameter distribution that excludes the contribution of ARD priors. Subsequently, type-II maximum likelihood is executed using Newton's method whereby the evidence and its gradient and Hessian information are computed in a semi-analytical fashion. We show numerically and analytically that SBL is a special case of NSBL for linear regression models. Subsequently, a linear regression example involving multimodality in both parameter posterior pdf and model evidence is considered to demonstrate the performance of NSBL in cases where SBL is inapplicable. Next, NSBL is applied to identify sparsity among the damping coefficients of a mass-spring-damper model of a shear building frame. These numerical studies demonstrate the robustness and efficiency of NSBL in alleviating overfitting during Bayesian inversion of nonlinear physics-based models.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2021

Bayesian sparse learning with preconditioned stochastic gradient MCMC and its applications

Deep neural networks have been successfully employed in an extensive variety of research areas, including solving partial differential equations. Despite its significant success, there are some challenges in effectively training DNN, such as avoiding overfitting in over-parameterized DNNs and accelerating the optimization in DNNs with pathological curvature. Here, we propose a Bayesian type sparse deep learning algorithm. The algorithm utilizes a set of spike-and-slab priors for the parameters in the deep neural network. The hierarchical Bayesian mixture will be trained using an adaptive empirical method. That is, one will alternatively sample from the posterior using preconditioned stochastic gradient Langevin Dynamics (PSGLD), and optimize the latent variables via stochastic approximation. The sparsity of the network is achieved while optimizing the hyperparameters with adaptive searching and penalizing. A popular SG-MCMC approach is Stochastic gradient Langevin dynamics (SGLD). However, considering the complex geometry in the model parameter space in nonconvex learning, updating parameters using a universal step size in each component as in SGLD may cause slow mixing. To address this issue, we apply a computationally manageable preconditioner in the updating rule, which provides a step-size parameter to adapt to local geometric properties. Moreover, by smoothly optimizing the hyperparameter in the preconditioning matrix, our proposed algorithm ensures a decreasing bias, which is introduced by ignoring the correction term in the preconditioned SGLD. According to the existing theoretical framework, we show that the proposed algorithm can asymptotically converge to the correct distribution with a controllable bias under mild conditions. Numerical tests are performed on both synthetic regression problems and learning solutions of elliptic PDE, which demonstrate the accuracy and efficiency of the present work.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2023

Encoding nonlinear and unsteady aerodynamics of limit cycle oscillations using nonlinear sparse Bayesian learning

This article investigates the applicability of a recently proposed, nonlinear sparse Bayesian learning (NSBL) algorithm to identify and estimate the complex aerodynamics of limit cycle oscillations. NSBL provides a semi-analytical framework for determining the data-optimal sparse model nested within a (potentially) over-parameterized model. This is particularly relevant to nonlinear dynamical systems where modelling approaches involve the use of physics-based and data-driven components. In such cases, the data-driven components, where analytical descriptions of the physical processes are not readily available, are often prone to overfitting, meaning that the empirical aspects of these models will often involve the calibration of an unnecessarily large number of parameters. While an overparameterized model may fit the observed data well, such models may be inadequate for making predictions in regimes that are different from those wherein the data were recorded. In view of this, it is desirable to not only calibrate the model parameters, but also identify the optimal compromise between data fit and model complexity. In this article, we exhibit the optimal model discovery for an aeroelastic system wherein the structural dynamics are well-known and described by a differential equation model, coupled with a semi-empirical aerodynamic model for laminar separation flutter, resulting in low-amplitude limit cycle oscillations (LCO). To illustrate the performance of the algorithm, in this article, we use synthetic data and demonstrate the ability of the algorithm to correctly rediscover the optimal model and model parameters, given a known data-generating model. The synthetic data are generated from a forward simulation of a known differential equation model with parameters selected so as to mimic the dynamics observed in wind-tunnel experiments. Subsequently, we demonstrate the performance of the algorithm for model selection using noisy LCO data from wind tunnel experiments. As there is no ground truth available for the experimental data case, we provide a comparison between NSBL and Bayesian model selection to validate the results, and demonstrate the use of NSBL as an efficient alternative to traditional methods.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Bayesian tensorized neural networks with automatic rank selection

Tensor decomposition is an effective approach to compress over-parameterized neural networks and to enable their deployment on resource-constrained hardware platforms. However, directly applying tensor compression in the training process is a challenging task due to the difficulty of choosing a proper tensor rank. In order to address this challenge, this paper proposes a low-rank Bayesian tensorized neural network. Our Bayesian method performs automatic model compression via an adaptive tensor rank determination. We also present approaches for posterior density calculation and maximum a posteriori (MAP) estimation for the end-to-end training of our tensorized neural network. Here, we provide experimental validation on a two-layer fully connected neural network, a 6-layer CNN and a 110-layer residual neural network where our work produces 7.4x to 137x more compact neural networks directly from the training while achieving high prediction accuracy.

Low-rank tensor↗

DOE OSTI2019

A Novel Pruning Method for Convolutional Neural Networks Based off Identifying Critical Filters

Convolutional Neural Networks (CNNs) are one of the most extensively used tools in machine learning, but they are still not well understood and in many cases they are over-parameterized, leading to slow inference and impeding their deployment on low-power devices. In the last few years, many methods for decreasing the number of parameters in a network by pruning its output channels have been suggested, but a very recent work has argued that random pruning of channels performs on-par with state-of-the-art pruning methods. While random and other pruning methods might be effectively used for lowering the number of parameters in a CNN, none of these methods can be used to gain any further understanding of the model that the CNN has built. In this work, we propose a novel method for pruning a network, that at the same time can lead to a better understanding of what the individual filters of the network learn about the data. The method proposed aims to keep only the filters that are "important" for a class. We define a filter as important for a class if its removal has the highest negative impact on the accuracy for that class. We demonstrate that our method is better than random pruning on two networks used on the EMNIST and CIFAR10 datasets. By analyzing the important filters, we find that the important filters in the pruned networks learn features which are more general across classes. We demonstrate the importance and applicability of that observation in two transfer-learning tasks.

Dimovska, Mihaela↗

DOE OSTI2024

Reinforcement Learning as a Parsimonious Alternative to Prediction Cascades: A Case Study on Image Segmentation

Deep learning architectures have achieved state-of-the-art (SOTA) performance on computer vision tasks such as object detection and image segmentation. This may be attributed to the use of over-parameterized, monolithic deep learning architectures executed on large datasets. Although such large architectures lead to increased accuracy, this is usually accompanied by a larger increase in computation and memory requirements during inference. While this is a non-issue in traditional machine learning (ML) pipelines, the recent confluence of machine learning and fields like the Internet of Things (IoT) has rendered such large architectures infeasible for execution in low-resource settings. For some datasets, large monolithic pipelines may be overkill for simpler inputs. To address this problem, previous efforts have proposed decision cascades where inputs are passed through models of increasing complexity until the desired performance is achieved. However, we argue that cascaded prediction leads to sub-optimal throughput and increased computational cost due to wasteful intermediate computations. To address this, we propose PaSeR (Parsimonious Segmentation with Reinforcement Learning) a non-cascading, cost-aware learning pipeline as an efficient alternative to cascaded decision architectures. Through experimental evaluation on both real-world and standard datasets, we demonstrate that PaSeR achieves better accuracy while minimizing computational cost relative to cascaded models. Further, we introduce a new metric IoU/GigaFlop to evaluate the balance between cost and performance. On the real-world task of battery material phase segmentation, PaSeR yields 179% improvement over SOTA MatPhase model and a 196% improvement over IDK Cascades under the IoU/GigaFlop metric. We also demonstrate PaSeR’s adaptability to complementary models trained on a noisy MNIST dataset, where it outperforms all baselines on IoU/GigaFlop by an average of 44%.

Srikshan, Bharat↗

DOE OSTI2026

Determining Optimal Magnetometer Configuration on MAGIS-100

Long-baseline atom interferometers such as the Matter-wave Atomic Gradiometer Interferometric Sensor (MAGIS-100) require stringent control and continuous characterization of background magnetic fields and spatial gradients to prevent systemic phase shifts that mimic ultralight dark matter or gravitational wave signatures. Because direct sensor placement within the ultra-high vacuum beam pipe is infeasible, in-situ magnetic field monitoring relies on external sensor arrays situated in the surrounding annular region. This work demonstrates a field reconstruction framework for a 5.3-meter MAGIS-100 modular section using finite-element Opera simulations. Transverse magnetic fields are expanded using a cylindrical multipole framework as informed by Fermilab’s Muon g-2 experiment, with magnetometer array configurations optimized via Fisher information matrix D-optimality. Inverting external sensor readings through a Gauss-Newton scheme recovers interior tube fields across distinct axial positions. In the discontinuity-averse uniform region (slice pair P4), the model achieves sub-noise-floor performance with a cross-validated root-mean-square error (RMSE) of $6.7227 \times 10^{-4}\text{ A/m}$ ($0.845\times$ sensor noise floor) and an interior field coefficient of variation of $1.71\%$. An elbow criterion in the Fisher bounds establishes $n_{\text{max}} = 2$ as the optimal multipole truncation order to prevent noise amplification from over-parameterization, with $n_{\text{max}} = 3$ (sextupole) order chosen for analysis to demonstrate further complexity and cross-pair comparison. Furthermore, analytical differentiation of the fitted multipole coefficients yields dense spatial maps of the transverse Jacobian gradient matrix $\nabla \mathbf{H}$ along with propagated $1\sigma$ uncertainty bounds across the beam region ($r \le 2.75\text{ in}$). This operational framework confirms that external magnetometer arrays can reliably monitor magnetic field uniformity and spatial gradients along the 100-meter flight path given appropriate sampling for any complexity order.

Appleby, Darwin [William Rainey Harper Coll.] (ORC↗

DOE OSTI2026

Determining Optimal Magnetometer Configuration on MAGIS-100

Long-baseline atom interferometers such as the Matter-wave Atomic Gradiometer Interferometric Sensor (MAGIS-100) require stringent control and continuous characterization of background magnetic fields and spatial gradients to prevent systemic phase shifts that mimic ultralight dark matter or gravitational wave signatures. Because direct sensor placement within the ultra-high vacuum beam pipe is infeasible, in-situ magnetic field monitoring relies on external sensor arrays situated in the surrounding annular region. This work demonstrates a field reconstruction framework for a 5.3-meter MAGIS-100 modular section using finite-element Opera simulations. Transverse magnetic fields are expanded using a cylindrical multipole framework as informed by Fermilab’s Muon g-2 experiment, with magnetometer array configurations optimized via Fisher information matrix D-optimality. Inverting external sensor readings through a Gauss-Newton scheme recovers interior tube fields across distinct axial positions. In the discontinuity-averse uniform region (slice pair P4), the model achieves sub-noise-floor performance with a cross-validated root-mean-square error (RMSE) of $6.7227 \times 10^{-4}\text{ A/m}$ ($0.845\times$ sensor noise floor) and an interior field coefficient of variation of $1.71\%$. An elbow criterion in the Fisher bounds establishes $n_{\text{max}} = 2$ as the optimal multipole truncation order to prevent noise amplification from over-parameterization, with $n_{\text{max}} = 3$ (sextupole) order chosen for analysis to demonstrate further complexity and cross-pair comparison. Furthermore, analytical differentiation of the fitted multipole coefficients yields dense spatial maps of the transverse Jacobian gradient matrix $\nabla \mathbf{H}$ along with propagated $1\sigma$ uncertainty bounds across the beam region ($r \le 2.75\text{ in}$). This operational framework confirms that external magnetometer arrays can reliably monitor magnetic field uniformity and spatial gradients along the 100-meter flight path given appropriate sampling for any complexity order.

Appleby, Darwin [William Rainey Harper Coll.] (ORC↗

DOE OSTI2024

Yet Another Discriminant Analysis (YADA): A Probabilistic Model for Machine Learning Applications

This paper presents a probabilistic model for various machine learning (ML) applications. While deep learning (DL) has produced state-of-the-art results in many domains, DL models are complex and over-parameterized, which leads to high uncertainty about what the model has learned, as well as its decision process. Further, DL models are not probabilistic, making reasoning about their output challenging. In contrast, the proposed model, referred to as Yet Another Discriminate Analysis(YADA), is less complex than other methods, is based on a mathematically rigorous foundation, and can be utilized for a wide variety of ML tasks including classification, explainability, and uncertainty quantification. YADA is thus competitive in most cases with many state-of-the-art DL models. Ideally, a probabilistic model would represent the full joint probability distribution of its features, but doing so is often computationally expensive and intractable. Hence, many probabilistic models assume that the features are either normally distributed, mutually independent, or both, which can severely limit their performance. YADA is an intermediate model that (1) captures the marginal distributions of each variable and the pairwise correlations between variables and (2) explicitly maps features to the space of multivariate Gaussian variables. Numerous mathematical properties of the YADA model can be derived, thereby improving the theoretic underpinnings of ML. Validation of the model can be statistically verified on new or held-out data using native properties of YADA. However, there are some engineering and practical challenges that we enumerate to make YADA more useful.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2004

Closed Loop System Identification with Genetic Algorithms

High performance control design for a flexible space structure is challenging since high fidelity plant models are di.cult to obtain a priori. Uncertainty in the control design models typically require a very robust, low performance control design which must be tuned on-orbit to achieve the required performance. Closed loop system identi.cation is often required to obtain a multivariable open loop plant model based on closed-loop response data. In order to provide an accurate initial plant model to guarantee convergence for standard local optimization methods, this paper presents a global parameter optimization method using genetic algorithms. A minimal representation of the state space dynamics is employed to mitigate the non-uniqueness and over-parameterization of general state space realizations. This control-relevant system identi.cation procedure stresses the joint nature of the system identi.cation and control design problem by seeking to obtain a model that minimizes the di.erence between the predicted and actual closed-loop performance.

Whorton, Mark S.↗

DOE OSTI2021

Identification of Life Models for Li-Ion Batteries Using Penalized Regression and Bilevel Optimization

Reduced-order physics-based life models are extremely useful for rapidly predicting battery state-of-health and for simulating battery lifetime in arbitrary aging conditions. However, identification of well-parameterized models is difficult. This is because, for maximum usefulness in predicting lifetime under a variety of conditions, aging test data exhibits many degradation mechanisms, which all need to be accurately modeled. However, because aging tests are time-consuming and expensive, especially for large-format batteries, a minimum of tests are conducted while probing many stress factors. Building a well-parameterized model is then very challenging: an under-parameterized model will neglect critical degradation modes, and an over-parameterized model will extrapolate poorly to new testing conditions. To complicate this matter, the functional form of the model for any individual degradation rate can be very difficult to identify. In this work, the statistical tools of penalized regression and bilevel optimization are used to help identify both the functional forms of and optimize the parameters of reduced-order life models, accelerating identification of robust models. Model robustness is demonstrated through traditional statistics methods of cross-validation and sensitivity analysis, uncertainty quantification through bootstrap resampling and Monte-Carlo simulation, and simulation of real-world use cases.

47 OTHER INSTRUMENTATION↗

DOE OSTI2026

Benchmarking quantum trial wavefunctions for phaseless auxiliary-field quantum Monte Carlo

The phaseless auxiliary-field quantum Monte Carlo (ph-AFQMC) method is a stochastic imaginary-time projection technique for computing ground-state properties of strongly correlated quantum systems, with accuracy that depends critically on the choice of trial wavefunction. Here, we investigate ph-AFQMC with trial states prepared using parameterized quantum circuits. In this work, we present a comprehensive benchmarking study of quantum trial wavefunctions spanning unitary coupled-cluster, Hamiltonian-informed, Jastrow-inspired, and adaptively constructed ansatze. The benchmarking evaluates accuracy, expressibility, and scalability of these ansatze within the QC-AFQMC framework. We test these ansatze on linear hydrogen chains under bond stretching and find that several ansatz families produce chemically accurate ph-AFQMC energies across the dissociation curve. We have performed simulations using the CUDA-Q quantum development platform on the GPU partition of the Perlmutter supercomputer. When comparing ansatze at similar numbers of variational parameters, we find that different ansatz families yield comparable ph-AFQMC results despite exhibiting substantially different variational energies, optimization costs, and circuit depths. Our results indicate that the variational energy of an ansatz is not always a reliable indicator of its quality for ph-AFQMC and reveal instances of over-parameterization. In the strongly correlated regime, trial wavefunctions obtained from adaptive ansatze, exemplified here by ADAPT-VQE with the UCCSD operator pool, can outperform their fixed-ansatz counterparts (UCCSD) in terms of projected energies while using substantially more compact circuits, providing a flexible route to optimize quantum resources within the ph-AFQMC framework.

Rofougaran, Rod [LBNL, Berkeley; Columbia U.; PNL,↗

DOE OSTI2026

In-Situ Magnetic Field Reconstruction in the MAGIS-100 Experiment

Long-baseline atom interferometers such as the Matter-wave Atomic Gradiometer Interferometric Sensor (MAGIS-100) require stringent control and continuous characterization of background magnetic fields and spatial gradients to prevent systemic phase shifts that mimic ultralight dark matter or gravitational wave signatures. Because direct sensor placement within the ultra-high vacuum beam pipe is infeasible, in-situ magnetic field monitoring relies on external sensor arrays situated in the surrounding annular region. This work demonstrates a field reconstruction framework for a 5.3-meter MAGIS-100 modular section using finite-element Opera simulations. Transverse magnetic fields are expanded using a cylindrical multipole framework as informed by Fermilab’s Muon g-2 experiment, with magnetometer array configurations optimized via Fisher information matrix D-optimality. Inverting external sensor readings through a Gauss-Newton scheme recovers interior tube fields across distinct axial positions. In the discontinuity-averse uniform region (slice pair P4), the model achieves sub-noise-floor performance with a cross-validated root-mean-square error (RMSE) of $6.7227 \times 10^{-4}\text{ A/m}$ ($0.845\times$ sensor noise floor) and an interior field coefficient of variation of $1.71\%$. An elbow criterion in the Fisher bounds establishes $n_{\text{max}} = 2$ as the optimal multipole truncation order to prevent noise amplification from over-parameterization, with $n_{\text{max}} = 3$ (sextupole) order chosen for analysis to demonstrate further complexity and cross-pair comparison. Furthermore, analytical differentiation of the fitted multipole coefficients yields dense spatial maps of the transverse Jacobian gradient matrix $\nabla \mathbf{H}$ along with propagated $1\sigma$ uncertainty bounds across the beam region ($r \le 2.75\text{ in}$). This operational framework confirms that external magnetometer arrays can reliably monitor magnetic field uniformity and spatial gradients along the 100-meter flight path given appropriate sampling for any complexity order.

Appleby, Darwin [William Rainey Harper Coll.; Ferm↗