Search NASA⌕ Search

SEARCH · Search NASA

Results for “Generalization error”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness

We report the accuracy of deep learning, i.e., deep neural networks, can be characterized by dividing the total error into three main types: approximation error, optimization error, and generalization error. Whereas there are some satisfactory answers to the problems of approximation and optimization, much less is known about the theory of generalization. Most existing theoretical works for generalization fail to explain the performance of neural networks in practice. To derive a meaningful bound, we study the generalization error of neural networks for classification problems in terms of data distribution and neural network smoothness. We introduce the cover complexity (CC) to measure the difficulty of learning a data set and the inverse of the modulus of continuity to quantify neural network smoothness. A quantitative bound for expected accuracy/error is derived by considering both the CC and neural network smoothness. Although most of the analysis is general and not specific to neural networks, we validate our theoretical assumptions and results numerically for neural networks by several data sets of images. The numerical results confirm that the expected error of trained networks scaled with the square root of the number of classes has a linear relationship with respect to the CC. We also observe a clear consistency between test loss and neural network smoothness during the training process. In addition, we demonstrate empirically that the neural network smoothness decreases when the network size increases whereas the smoothness is insensitive to training dataset size.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Sensor Error Estimation for Reactor Coolant System with Generalization Error Equations

We consider a simplified version of the sensor error estimation problem in pressure sensors in the primary coolant system of a nuclear power plant under the condition that a majority of sensors are error-free. We present a sensor fusion method to estimate sensor errors in the primary coolant system of a power reactor by using measurements collected by multiple sensors. The underlying relationship between the measured and estimated parameters is inferred using measurements collected under controlled no-error conditions of the plant. We propose a machine learning approach to train fusers that provide an estimate of the sensor measurement by using measurements from other sensors, and the difference between the actual sensor measurement and its estimate provides the sensor estimate. We present generalization error equations for these methods that characterize the performance of their sensor error estimates on future measurements using the distribution-free machine learning formulation.

Rao, Nageswara S.↗
DOE OSTI2023

Deep nonparametric estimation of intrinsic data structures by chart autoencoders: Generalization error and robustness

Autoencoders have demonstrated remarkable success in learning low-dimensional latent features of high-dimensional data across various applications. Assuming that data are sampled near a low-dimensional manifold, we employ chart autoencoders, which encode data into low-dimensional latent features on a collection of charts, preserving the topology and geometry of the data manifold. Our paper establishes statistical guarantees on the generalization error of chart autoencoders, and we demonstrate their denoising capabilities by considering n noisy training samples, along with their noise-free counterparts, on a d-dimensional manifold. By training autoencoders, we show that chart autoencoders can effectively denoise the input data with normal noise. We prove that, under proper network architectures, chart autoencoders achieve a squared generalization error in the order of n–$\frac{2}{d+2}$log 4 n, which depends on the intrinsic dimension of the manifold and only weakly depends on the ambient dimension and noise level. We further extend our theory on data with noise containing both normal and tangential components, where chart autoencoders still exhibit a denoising effect for the normal component. As a special case, our theory also applies to classical autoencoders, as long as the data manifold has a global parametrization. Furthermore, our results provide a solid theoretical foundation for the effectiveness of autoencoders, which is further validated through several numerical experiments.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Generalization error guaranteed auto-encoder-based nonlinear model reduction for operator learning

Many physical processes in science and engineering are naturally represented by operators between infinite-dimensional function spaces. The problem of operator learning, in this context, seeks to extract these physical processes from empirical data, which is challenging due to the infinite or high dimensionality of data. An integral component in addressing this challenge is model reduction, which reduces both the data dimensionality and problem size. In this paper, we utilize low-dimensional nonlinear structures in model reduction by investigating Auto-Encoder-based Neural Network (AENet). AENet first learns the latent variables of the input data and then learns the transformation from these latent variables to corresponding output data. Our numerical experiments validate the ability of AENet to accurately learn the solution operator of nonlinear partial differential equations. Furthermore, we establish a mathematical and statistical estimation theory that analyzes the generalization error of AENet. Finally, our theoretical framework shows that the sample complexity of training AENet is intricately tied to the intrinsic dimension of the modeled process, while also demonstrating the robustness of AENet to noise.

Auto-encoder↗
NASA NTRS2004

AveBoost2: Boosting for Noisy Data

AdaBoost is a well-known ensemble learning algorithm that constructs its constituent or base models in sequence. A key step in AdaBoost is constructing a distribution over the training examples to create each base model. This distribution, represented as a vector, is constructed to be orthogonal to the vector of mistakes made by the pre- vious base model in the sequence. The idea is to make the next base model's errors uncorrelated with those of the previous model. In previous work, we developed an algorithm, AveBoost, that constructed distributions orthogonal to the mistake vectors of all the previous models, and then averaged them to create the next base model s distribution. Our experiments demonstrated the superior accuracy of our approach. In this paper, we slightly revise our algorithm to allow us to obtain non-trivial theoretical results: bounds on the training error and generalization error (difference between training and test error). Our averaging process has a regularizing effect which, as expected, leads us to a worse training error bound for our algorithm than for AdaBoost but a superior generalization error bound. For this paper, we experimented with the data that we used in both as originally supplied and with added label noise-a small fraction of the data has its original label changed. Noisy data are notoriously difficult for AdaBoost to learn. Our algorithm's performance improvement over AdaBoost is even greater on the noisy data than the original data.

Oza, Nikunj C.↗
NASA NTRS1990

Performance analysis of a generalized concurrent error detection procedure

A general procedure for error detection in complex systems, called the data block capture and analysis monitoring process, is described and analyzed. It is assumed that, in addition to being exposed to potential external fault sources, a complex system will in general always contain embedded hardware and software fault mechanisms which can cause the system to perform incorrect computations and/or produce incorrect output. Thus, in operation, the system continuously moves back and forth between error and no-error states. These external fault sources or internal fault mechanisms are extremely difficult to detect. The data block capture and analysis monitoring process is concerned with detecting deviations from the normal performance of the system, known as errors, which are symptomatic of fault conditions. The process consists of repeatedly recording a fixed amount of data from a set of predetermined observation lines of the system being monitored (i.e., capturing a block of data) and then analyzing the captured block in an attempt to determine whether the system is functioning correctly. The performances of linear, quadratic, and logarithmic data analysis algorithms are rigorously characterized in terms of the probability of correctly detecting an error, the expectation and variance of the number of false alarms per error, and the expectation and variance of the latency in detection of errors. Insight into the nature of the general problem of error detection is obtained.

Blough, Douglas M.↗
NASA NTRS1988

An extremum principle for computation of the zone of tooth contact and generalized transmission error of spiral bevel gears

For a given set of forces transmitted by the gears, each of the three components of the generalized transmission error of spiral bevel gears is shown to be stationary with respect to small independent variations in the positions of the endpoints of the lines of tooth contact about their true values. The tangential generalized transmission error component is shown to take on a minimum value at the true endpoint positions. A computational procedure based on the method of steepest descent is described for computing the true line of contact endpoint positions and the three components of the generalized transmission error. A method for computing the Fourier series coefficients of the tooth meshing harmonics of the three generalized transmission error components also is provided.

Mark, W. D.↗
NASA NTRS1987

Use of the generalized transmission error in the equations of motion of gear systems

The vibratory excitation arising from a gear pair is widely recognized to be a consequence of the nonuniform transmission of motion by the gear pair. In the preceding paper in this issue, it is shown that a three-component transmission error is required to describe the nonuniform transmission of motion by bevel gears for vibration excitation characterization purposes. The expression for the three-component transmission error derived in that paper is combined in the present paper with an analysis of the mesh forces and mesh elasticity to yield an equation of constraint involving the six degree-of-freedom unknown vibratory displacements of the gear shaft centerliners, the three unknown components of the generalized force transmitted by the mesh, and the geometric deviations of the tooth running surfaces from perfect involute surfaces which are assumed known. This matrix equation can be combined with the equations of motion of a gear system to predict the vibratory response of the system to the generalized transmission error excitation arising from meshing gear pairs within the system.

Mark, W. D.↗
DOE OSTI2025

Parameter uncertainties for imperfect surrogate models in the low-noise regime

Abstract Bayesian regression determines model parameters by minimizing the expected loss, an upper bound to the true generalization error. However, this loss ignores model form error, or misspecification, meaning parameter uncertainties are significantly underestimated and vanish in the large data limit. As misspecification is the main source of uncertainty for surrogate models of low-noise calculations, such as those arising in atomistic simulation, predictive uncertainties are systematically underestimated. We analyze the true generalization error of misspecified, near-deterministic surrogate models, a regime of broad relevance in science and engineering. We show that posterior parameter distributions must cover every training point to avoid a divergence in the generalization error and design a compatible ansatz which incurs minimal overhead for linear models. The approach is demonstrated on model problems before application to thousand-dimensional datasets in atomistic machine learning. Our efficient misspecification-aware scheme gives accurate prediction and bounding of test errors in terms of parameter uncertainties, allowing this important source of uncertainty to be incorporated in multi-scale computational workflows.

Swinburne, Thomas D. (ORCID:0000000232554257)↗
NASA NTRS1987

Jet stream velocity errors in general circulation models

The longitude and time dependence of excessive wind speed errors above subtropical jets in current GCM forecasts is studied for 14 five-day winter forecasts using the NASA Goddard Laboratory for Atmospheres fourth-order GCM. Several distinct phenomena, which may be divided into four categories, are found to contribute to the excess winds. These categories are: (1) error growth above the jet near the Himalayas; (2) error growth above the jet initiated elsewhere followed by advection; (3) tropical moisture bursts appearing in equatorial regions and migrating northeastward to merge with and distort the jet; and (4) undulatory growth of waves in the meridional component of the jet stream velocity. The results of additional sensitivity studies of the Himalayan region error are also reported.

Tenenbaum, J.↗
DOE OSTI2026

Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study

Neural scaling laws play a pivotal role in the performance of deep neural networks and have been observed in a wide range of tasks. However, a complete theoretical framework for understanding these scaling laws remains underdeveloped. In this paper, we explore the neural scaling laws for deep operator networks, which involve learning mappings between function spaces, with a focus on the Chen and Chen style architecture. These approaches, which include the popular Deep Operator Network (DeepONet), approximate the output functions using a linear combination of learnable basis functions and coefficients that depend on the input functions. We establish a theoretical framework to quantify the neural scaling laws by analyzing its approximation and generalization errors. We articulate the relationship between the approximation and generalization errors of deep operator networks and key factors such as network model size and training data size. Moreover, we address cases where input functions exhibit low-dimensional structures, allowing us to derive tighter error bounds. These results also hold for deep ReLU networks and other similar structures. Our results offer a partial explanation of the neural scaling laws in operator learning and provide a theoretical foundation for their applications.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1990

Attitude determination error analysis - General model and specific application

This paper presents a comprehensive approach to filter and dynamics modeling for attitude determination error analysis. The discussion includes models for both batch least-squares and sequential estimators, a specific dynamic model for attitude determination error analysis of a three-axis stabilized spacecraft equipped with strapdown gyros, and the incorporation of general attitude sensor observations. An analyst using this approach to perform an error analysis chooses a subset of the spacecraft parameters to be 'solve-for' parameters, which are to be estimated, and another subset to be 'consider' parameters, which are assumed to have errors but not to be estimated. The result of the error analysis is an indication of overall uncertainties in the 'solve-for' parameters, as well as the contributions of the various error sources to these uncertainties, including those of errors in the a priori 'solve-for' estimates, of measurement noise, of dynamic noise (also known as process noise or plant noise), and of 'consider' parameter uncertainties. The analysis of attitude, star tracker alignment, and gyro bias uncertainties for the Gamma Ray Observatory spacecraft provide a specific example of the use of a general-purpose software package incorporating these models.

Markley, F. Landis↗
NASA NTRS1987

The generalized transmission error of spiral bevel gears

The traditional definition of the transmission error of parallel-axis gear pairs is reviewed and shown to be unsuitable for characterizing the deviation from conjugate action of bevel gear pairs for vibration excitation characterization purposes. This situation is rectified by generalizing the concept of the transmission error of parallel-axis gears to a three-component transmission error for spiral bevel gears of nominal spherical involute design. A general relationship is derived which expresses the contributions to the three-component transmission error from each gear of a meshing spiral bevel pair as a linear transformation of the six coordinates that describe the deviation of the shaft centerline position of each gear of the pair from the position of its rigid perfect involute counterpart.

Mark, W. D.↗
DOE OSTI2024

Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces

Learning operators between infinitely dimensional spaces is an important learning task arising in machine learning, imaging science, mathematical modeling and simulations, etc. This paper studies the nonparametric estimation of Lipschitz operators using deep neural networks. Non-asymptotic upper bounds are derived for the generalization error of the empirical risk minimizer over a properly chosen network class. Under the assumption that the target operator exhibits a low dimensional structure, our error bounds decay as the training sample size increases, with an attractive fast rate depending on the intrinsic dimension in our estimation. Our assumptions cover most scenarios in real applications and our results give rise to fast rates by exploiting low dimensional structures of data in operator estimation. We also investigate the influence of network structures (e.g., network width, depth, and sparsity) on the generalization error of the neural network estimator and propose a general suggestion on the choice of network structures to maximize the learning efficiency quantitatively.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Deep nonparametric estimation of operators between infinite dimensional spaces

Learning operators between infinitely dimensional spaces is an important learning task arising in machine learning, imaging science, mathematical modeling and simulations, etc. This paper studies the nonparametric estimation of Lipschitz operators using deep neural networks. Non-asymptotic upper bounds are derived for the generalization error of the empirical risk minimizer over a properly chosen network class. Under the assumption that the target operator exhibits a low dimensional structure, our error bounds decay as the training sample size increases, with an attractive fast rate depending on the intrinsic dimension in our estimation. Our assumptions cover most scenarios in real applications and our results give rise to fast rates by exploiting low dimensional structures of data in operator estimation. We also investigate the influence of network structures (e.g., network width, depth, and sparsity) on the generalization error of the neural network estimator and propose a general suggestion on the choice of network structures to maximize the learning efficiency quantitatively.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Generalization in quantum machine learning from few training data

Modern quantum machine learning (QML) methods involve variationally optimizing a parameterized quantum circuit on a training data set, and subsequently making predictions on a testing data set (i.e., generalizing). In this work, we provide a comprehensive study of generalization performance in QML after training on a limited number N of training data points. We show that the generalization error of a quantum machine learning model with T trainable gates scales at worst as $\sqrt{T/N}$. When only K$\ll$T gates have undergone substantial change in the optimization process, we prove that the generalization error improves to $\sqrt{K/N}$. Our results imply that the compiling of unitaries into a polynomial number of native gates, a crucial application for the quantum computing industry that typically uses exponential-size training data, can be sped up significantly. We also show that classification of quantum states across a phase transition with a quantum convolutional neural network requires only a very small training data set. Other potential applications include learning quantum error correcting codes or quantum dynamical simulation. Our work injects new hope into the field of QML, as good generalization is guaranteed from few training data.

97 MATHEMATICS AND COMPUTING↗