Estimating model-form errors using random field models
Explore the source record for details and available documents.
Ray, Jaideep↗
SEARCH · Search NASA
Results for “error models”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Effects of machine learning errors on human decision-making: manipulations of model accuracy, error types, and error importance
Abstract This study addressed the cognitive impacts of providing correct and incorrect machine learning (ML) outputs in support of an object detection task. The study consisted of five experiments that manipulated the accuracy and importance of mock ML outputs. In each of the experiments, participants were given the T and L task with T-shaped targets and L-shaped distractors. They were tasked with categorizing each image as target present or target absent. In Experiment 1, they performed this task without the aid of ML outputs. In Experiments 2–5, they were shown images with bounding boxes, representing the output of an ML model. The outputs could be correct (hits and correct rejections), or they could be erroneous (false alarms and misses). Experiment 2 manipulated the overall accuracy of these mock ML outputs. Experiment 3 manipulated the proportion of different types of errors. Experiments 4 and 5 manipulated the importance of specific types of stimuli or model errors, as well as the framing of the task in terms of human or model performance. These experiments showed that model misses were consistently harder for participants to detect than model false alarms. In general, as the model’s performance increased, human performance increased as well, but in many cases the participants were more likely to overlook model errors when the model had high accuracy overall. Warning participants to be on the lookout for specific types of model errors had very little impact on their performance. Overall, our results emphasize the importance of considering human cognition when determining what level of model performance and types of model errors are acceptable for a given task.
97 MATHEMATICS AND COMPUTING↗
Accuracy Criterion for Structural Calculations
Simple method of establishing an accuracy criterion based on comparing allowable errors and modeling errors. On basis of comparison, amounts of change required to improve modeling error used in convergence criterion.
Chen, J. C.↗
Testing contamination source identification methods for water distribution networks
In the event of contamination in a water distribution network (WDN), source identification (SI) methods that analyze sensor data can be used to identify the source location(s). Knowledge of the source location and characteristics are important to inform contamination control and cleanup operations. Various SI strategies that have been developed by researchers differ in their underlying assumptions and solution techniques. The following manuscript presents a systematic procedure for testing and evaluating SI methods. The performance of these SI methods is affected by various factors including the size of WDN model, measurement error, modeling error, time and number of contaminant injections, and time and number of measurements. This paper includes test cases that vary these factors and evaluates three SI methods on the basis of accuracy and specificity. The tests are used to review and compare these different SI methods, highlighting their strengths in handling various identification scenarios. These SI methods and a testing framework that includes the test cases and analysis tools presented in this paper have been integrated into EPA’s Water Security Toolkit (WST), a suite of software tools to help researchers and others in the water industry evaluate and plan various response strategies in case of a contamination incident. Lastly, a set of recommendations are made for users to consider when working with different categories of SI methods.
54 ENVIRONMENTAL SCIENCES↗
Information transmission with continuous variable quantum erasure channels
Quantum capacity, as the key figure of merit for a given quantum channel, upper bounds the channel's ability in transmitting quantum information. Identifying different types of channels, evaluating the corresponding quantum capacity, and finding the capacity-approaching coding scheme are the major tasks in quantum communication theory. Quantum channel in discrete variables has been discussed enormously based on various error models, while error model in the continuous variable channel has been less studied due to the infinite dimensional problem. In this paper, we investigate a general continuous variable quantum erasure channel. By defining an effective subspace of the continuous variable system, we find a continuous variable random coding model. We then derive the quantum capacity of the continuous variable erasure channel in the framework of decoupling theory. The discussion in this paper fills the gap of a quantum erasure channel in continuous variable setting and sheds light on the understanding of other types of continuous variable quantum channels.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Obtaining consistent models of helicopter flight-data measurement errors using kinematic-compatibility and state-reconstruction methods
A new method was developed for application of Kalman Filter/Smoothers to post-flight processing of helicopter flight test dynamic measurements. This processing includes checking for kinematic compatibility among the measurements, identification of a measurement error model, and reconstruction of both measured and unmeasured time histories. Emphasis is placed on identification of a parametric measurement error model which is valid for a set of flight test data. This is facilitated through a new method of concatenating several maneuver time histories. The method also includes a model structure determination step which ensures that a physically realistic parameterization has been achieved. Application of the method to a set of BO-105 flight test data is illustrated. The resulting minimally parameterized error model is shown to characterize the measurement errors of the entire data set with very little variation in the parameter values. Reconstructed time histories are shown to have increased bandwidths and signal to noise ratios.
Fletcher, Jay W.↗
Error control in the GCF: An information-theoretic model for error analysis and coding
The structure of data-transmission errors within the Ground Communications Facility is analyzed in order to provide error control (both forward error correction and feedback retransmission) for improved communication. Emphasis is placed on constructing a theoretical model of errors and obtaining from it all the relevant statistics for error control. No specific coding strategy is analyzed, but references to the significance of certain error pattern distributions, as predicted by the model, to error correction are made.
Adeyemi, O.↗
Bayesian characterization and forward propagation of the uncertainty in thermodynamic models for redox active materials
Demonstrating the thermodynamic efficiency of hydrogen conversion processes using various materials is a critical step in developing new technologies for storing concentrated solar energy, and is largely accomplished by using a thermodynamic model derived from experimental data. A main goal of this project is to calculate the uncertainty of the thermodynamic efficiency by calculating the uncertainty of the components that feed into the efficiency. Many different models and data sets were used to test the workflow. First, the models were fit to the data using a Bayesian Inference and a method called Markov Chain Monte Carlo (MCMC), which found the maximum a priori parameters, and a posterior probability distribution of the parameters. Next, the different models were compared to each other using model evidence values. It was found that for cleaner data sets, overfitting had not yet been reached, and the most complicated model was ideal, but on the noisier data sets, the less complex models were favored because the more complicated models resulted in overfitting. Next, forward propagation was used to calculate the enthalpy change and its associated uncertainty. A few variations on the models were tried, such as fitting in a different variable, producing negligible or negative effects on the fits of the models. Thus, the original models were used. A sensitivity analysis was performed, and used to calculate the model error. On the cleaner data sets, there was very minimal experimental noise, and thus, all resulting error was from the model. With consideration of the model error, the models fit the data very well, and the simpler model had a high model error, as expected. All these components will then be used to calculate the thermodynamic efficiency of the different materials.
14 SOLAR ENERGY↗
Model Form Error Correction for a Black-Box Thermal Battery Heat Transfer Simulation
Thermal batteries are crucial for supplying power to high-consequence engineering applications such as rockets. Computational simulations have been developed to predict thermal battery behavior, but these simulations often suffer from modeling errors, including model form uncertainty. Addressing this uncertainty can be achieved by quantifying either the model discrepancy in the output or the model form error (MFE) in the governing equation. MFE is particularly valuable as it can be better extrapolated beyond observed outputs, which is essential for predictions involving changes in external system loading, system configuration and geometry, or output quantities. This paper employs a state estimation approach to estimate MFE using experimental data and then utilizes machine learning (ML) to model its relationship with state variables. A nonintrusive technique is used to estimate MFE in a black-box thermal battery heat transfer simulation. The trained machine learning model for MFE is then applied to correct simulation predictions under extrapolated initial conditions and battery configurations. In conclusion, the methodology's performance is evaluated using additional experimental data, demonstrating its effectiveness in improving prediction accuracy.
Batteries↗
Dominant modes of mechanical systems
To obtain a reduced model of a mechanical system, one can identify the dominant modes and retain those as the reduced model. Two widely used schemes for selection of dominant modes are modal cost analysis (MCA) and balancing, but neither of these methods guarantees that the model error is small, where model error measures the difference between the output of the full model and the output of the reduced model. The authors introduce a set of coordinates called pseudo uncorrelated modal coordinates (PUnc) that can be used in conjunction with component cost analysis to select dominant modes yielding a reduced model with minimum model error. Furthermore, unlike MCA and balancing, the accuracy of the reduced model (the model error) in PUnc coordinates can be evaluated before reduction.
Breida, M.↗
Precision X-band radio Doppler and ranging navigation: Mars Observer interplanetary cruise scenario
This article describes an error covariance analysis based on a Mars Observer mission scenario; the study was performed to establish the navigation performance that can potentially be achieved in a demonstration of precision two-way X-band (8.4-GHz) Doppler and ranging with the Mars Observer spacecraft planned for next year, and to evaluate the sensitivity of the predicted performance to variations in ground system error modeling assumptions. Orbit determination error statistics computed for a 182-day Doppler and ranging data arc predicted Mars approach orbit determination accuracies of about 0.45 micro-rad in an angular sense, using a conservative ground system error model as a baseline. When less-conservative error model assumptions were employed, it was found that orbit determination accuracies of 0.19 to 0.30 micro-rad could be obtained; the level of accuracy of the assumed Mars ephemeris is about 0.11 micro-rad. In comparison, Doppler-only performance with the baseline error model was predicted to be about 1.30 to 1.51 micro-rad, although it was found that when improved station location accuracies and Global Positioning System-based tropospheric calibration accuracies were assumed, accuracies of 0.44 to 0.52 micro-rad were predicted. In the Doppler plus ranging cases, the results were relatively insensitive to variations in ranging system and station delay calibration uncertainties of a few meters and tropospheric zenith delay calibration uncertainties of a few centimeters.
Estefan, J. A.↗
Modeling mechanical behavior of an additively manufactured metal structure with local texture variations: a study on model form error
Here, the repetitive rapid solidification that occurs in metal additive manufacturing (AM) processes creates microstructures distinctly different from wrought materials. Local variability in AM microstructures (either by design or unintentional) raises questions as to how AM structures should be modeled at the part-scale to minimize modeling error. The key goal of this work is to demonstrate a posteriori error estimation applied to an AM part. It is assumed that the actual microstructure is unknown and an approximate, spatially uniform material model is used. Error bounds are calculated for many reference models based on AM microstructures with elongated grain morphology and localized or global fiber textures during a post-processing step. The current findings promote confidence that a posteriori model form error estimation could be used effectively in mechanical performance simulations of AM parts to quickly obtain quantitative error metrics between an approximate model result and many microstructure-based reference models. The a posteriori error estimation introduces significant time savings compared to computing the full reference model solutions. Tight bounds on model form error are obtained when texture variations in the reference models occur on large length scales. For materials with property variation at small length scales, multi-scale error estimation techniques are needed to properly account for the many interfaces present between areas with different properties.
36 MATERIALS SCIENCE↗
Data for A Hybrid Biophysical-Machine Learning Framework for Diurnal Surface Energy Flux Estimation Using Proximal Sensing
Thermal infrared-based remote sensing of surface energy fluxes has traditionally relied on high spatial resolution satellite data with revisit frequencies on the order of weeks. In this study, we evaluate a biophysics-based analytical surface energy balance model for predicting latent energy (LE) and sensible heat (H) fluxes using proximal sensing observations. The Surface Temperature Initiated Closure (STIC1.2) model has been extensively validated across a wide range of spatial and temporal scales using various satellite-derived thermal infrared data sets. Here we extend this validation by applying STIC at sub-hourly temporal resolution over multiple growing seasons for four distinct agricultural systems. We further develop and evaluate novel STIC variants that incorporate machine learning (ML) techniques to eliminate the need for surface energy balance observations, specifically net radiation and soil heat flux, thereby enhancing model applicability in data-sparse settings. The integration of a ML component to estimate surface available energy is shown to have strong predictive performance for both LE (R2 = 0.81–0.94) and H (R2 = 0.46–0.72) across all agricultural systems examined here, demonstrating the potential of hybrid biophysical-machine learning approaches for surface energy balance modeling with minimal data requirements. This study concludes with a novel application of explainable machine learning (exML) to diagnose sources of model error. This exML framework attributes residual prediction errors to both model input variables and environmental drivers not explicitly included in the simulation experiments. This approach provides a new pathway for improving model design and integrating previously overlooked yet influential variables into future model iterations.
AI/ML↗
Error Propagation Analysis in the SAE Architecture Analysis and Design Language (AADL) and the EDICT Tool Framework
This report documents the capabilities of the EDICT tools for error modeling and error propagation analysis when operating with models defined in the Architecture Analysis & Design Language (AADL). We discuss our experience using the EDICT error analysis capabilities on a model of the Scalable Processor-Independent Design for Enhanced Reliability (SPIDER) architecture that uses the Reliable Optical Bus (ROBUS). Based on these experiences we draw some initial conclusions about model based design techniques for error modeling and analysis of highly reliable computing architectures.
LaValley, Brian W.↗
Learn-as-you-go acceleration of cosmological parameter estimates
Cosmological analyses can be accelerated by approximating slow calculations using a training set, which is either precomputed or generated dynamically. However, this approach is only safe if the approximations are well understood and controlled. This paper surveys issues associated with the use of machine-learning based emulation strategies for accelerating cosmological parameter estimation. We describe a learn-as-you-go algorithm that is implemented in the Cosmo++ code and (1) trains the emulator while simultaneously estimating posterior probabilities; (2) identifies unreliable estimates, computing the exact numerical likelihoods if necessary; and (3) progressively learns and updates the error model as the calculation progresses. We explicitly describe and model the emulation error and show how this can be propagated into the posterior probabilities. We apply these techniques to the Planck likelihood and the calculation of ΛCDM posterior probabilities. The computation is significantly accelerated without a pre-defined training set and uncertainties in the posterior probabilities are subdominant to statistical fluctuations. We have obtained a speedup factor of 6.5 for Metropolis-Hastings and 3.5 for nested sampling. Finally, we discuss the general requirements for a credible error model and show how to update them on-the-fly.
79 ASTRONOMY AND ASTROPHYSICS↗
Is There an Optimal CO2 Partial Column for Flux Inversions?
The fidelity of flux estimates from an atmospheric inversion depends on the ability of atmospheric transport models to simulate the measured quantity. For species such as CO2, with surface fluxes and a large network of surface measurements, this means correctly simulating the dynamics of the planetary boundary layer (PBL), which is one of the most uncertain aspects of atmospheric transport modeling. In contrast, the simulated total column average mole fraction of CO2 (XCO2) is largely insensitive to simulated PBL dynamics. Therefore, measurements of XCO2 provided by current and future short wave infrared (SWIR) greenhouse gas (GHG) satellites such as GOSAT and the OCO family would seem to be more appropriate to flux inversions, as far as minimizing transport model errors (the "noise") is concerned. Unfortunately, the flux-induced variation of CO2 (the "signal") is the largest within the PBL and smallest in XCO2. Therefore, assimilating XCO2 as opposed to PBL CO2 need not give us the strongest "signal to noise" in flux inversions. Recent work on GOSAT and OCO2 retrievals suggest that SWIR satellite spectra may be used to estimate a lower partial column CO2, which could be assimilated in a flux inversion, instead of XCO2.Here we report on a study to assess whether there is an optimal partial column average CO2, intermediate between PBL CO2 and XCO2, whose assimilation might yield the best signal to noise in flux inversions, where (as before) "signal" is the flux-induced variation and "noise" is the error in transport modeling. We simulate atmospheric CO2 with five different global transport models and a common surface CO2 flux over ten years. We consider the spread across the five models to be a proxy for transport model error (the "noise"), and the common variation of CO2 in all five models to be a proxy for the "signal". We compare these signals and noises at different spatiotemporal scales for different partial column specifications to investigate whether there exists an optimal partial column that has large surface flux-driven variations and yet is relatively insensitive to errors in transport models. Finally, we comment on the feasibility of estimating such a partial column from current and future SWIR GHG satellites in the light of recent work on vertically resolved CO2 from current SWIR GHG satellites.
Basu, Sourish↗
A Method for the Quantification of Model Form Error Associated with Physical Systems
In the process of model validation, models are often declared valid when the differences between model predictions and experimental data sets are satisfactorily small. However, little consideration is given to the effectiveness of a model using parameters that deviate slightly from those that were fitted to data, such as a higher load level. Furthermore, few means exist to compare and choose between two or more models that reproduce data equally well. These issues can be addressed by analyzing model form error, which is the error associated with the differences between the physical phenomena captured by models and that of the real system. This report presents a new quantitative method for model form error analysis and applies it to data taken from experiments on tape joint bending vibrations. Two models for the tape joint system are compared, and suggestions for future improvements to the method are given. As the available data set is too small to draw any statistical conclusions, the focus of this paper is the development of a methodology that can be applied to general problems.
97 MATHEMATICS AND COMPUTING↗
An error bound for a discrete reduced order model of a linear multivariable system
The design of feasible controllers for high dimension multivariable systems can be greatly aided by a method of model reduction. In order for the design based on the order reduction to include a guarantee of stability, it is sufficient to have a bound on the model error. Previous work has provided such a bound for continuous-time systems for algorithms based on balancing. In this note an L-infinity bound is derived for model error for a method of order reduction of discrete linear multivariable systems based on balancing.
Al-Saggaf, Ubaid M.↗