Control of modeling error in calibration and validation processes for predictive stochastic models
The idea of adaptive control of modeling error is expanded to include ideas of statistical calibration, validation, and uncertainty quantification.
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
The idea of adaptive control of modeling error is expanded to include ideas of statistical calibration, validation, and uncertainty quantification.
The approach used to obtain reliable countywide forest resource data, including statistics and a forest cover distribution map from LANDSAT data is described. Comparison of results of the LANDSAT anlysis and U.S. Forestry Survey data is presented in a table. Pine, oak/pine, and oak/hickory accounted for 22.3%, 32.6% and 45.2% respectively of the total forest area in the LANDSAT analysis, compared to 22.8%, 30.1%. and 47.1% in the Forest Survey. The corresponding figures for each forest type differ at most 2.5% for the mixed forest. The total county area in forest land measured using LANDSAT was underestimated by 1,409 acres, a 2.2% error based on the Forest Survey statistics. However, the estimate of 61,242 acres fell well within the known 4% uncertainty of the estimate for the Forest Survey figures (62,651 acres + or - 2,544 acres).
A small error in the mathematics described in the previous paper (C.M. Parish and M.K. Miller: Multivariate statistical analysis of atom probe tomography data, Ultramicroscopy 110(11) (2010) 1362-1373) has been found. Although the results and conclusions are completely unchanged, it is important to correct the error.
We introduce a new methodology for inferring the accuracy of computational simulations through the practice of solution verification. We demonstrate this methodology on examples from computational heat transfer, fluid dynamics and radiation transport. Our methodology is suited to both well- and ill-behaved sequences of simulations. Our approach to the analysis of these sequences of simulations incorporates expert judgment into the process directly via a flexible optimization framework, and the application of robust statistics. The expert judgment is systematically applied as constraints to the analysis, and together with the robust statistics guards against over-emphasis on anomalous analysis results. We have named our methodology Robust Verification. Our methodology is based on utilizing multiple constrained optimization problems to solve the verification model in a manner that varies the analysis' underlying assumptions. Constraints applied in the analysis can include expert judgment regarding convergence rates (bounds and expectations) as well as bounding values for physical quantities (e.g., positivity of energy or density). This approach then produces a number of error models, which are then analyzed through robust statistical techniques (median instead of mean statistics). This provides self-contained, data and expert informed error estimation including uncertainties for both the solution itself and order of convergence. Our method produces high quality results for the well-behaved cases relatively consistent with existing practice. The methodology can also produce reliable results for ill-behaved circumstances predicated on appropriate expert judgment. We demonstrate the method and compare the results with standard approaches used for both code and solution verification on well-behaved and ill-behaved simulations.
The emergence of small-scale deformation mapping by a combination of scanning electron microscopy and digital image correlation (SEM-DIC) has enabled full-field investigations into the complex roles of microstructure on microscale deformation mechanisms. However, these investigations are hindered by errors in alignment between the microstructure data, generally acquired by electron backscatter diffraction (EBSD), and the microscale strain data obtained by SEM-DIC. Distortions, stitching artifacts, and spatial resolution differences between microstructure and strain fields can lead to misalignments on the order of μms. This alignment uncertainty introduces error into microstructure-strain localization analyses and precludes the examination of deformation mechanisms near grain boundaries. To improve alignment between EBSD-obtained grain boundaries and SEM-DIC strain data, an approach was created wherein a mantle was first established around the EBSD-acquired grain boundaries. Strain data was then clustered within this mantle using a k-means algorithm to identify grain boundary strain localization, and these boundary points were fit to obtain a continuous curve for each individual boundary. Clustered point outliers, such as those due to grain boundary-local dislocation slip, were statistically identified by using an absolute error threshold and removed from the grain boundary fit. The ending identification of grain boundaries in the microscale strain data is significantly improved from EBSD-identified boundaries.
Normality is one of the basic assumptions in applying statistical procedures. For example in linear regression most of the inferential procedures are based on the assumption of normality, i.e. the disturbance vector is assumed to be normally distributed. Failure to assess non-normality of the error terms may lead to incorrect results of usual statistical inference techniques such as t-test or F-test. Thus, error terms should be normally distributed in order to allow us to make exact inferences. As a consequence, normally distributed stochastic errors are necessary in order to make a not misleading inferences which explains a necessity and importance of robust tests of normality. Therefore, the aim of this contribution is to discuss normality testing of error terms in regression models. In this contribution, we introduce the general RT class of robust tests for normality, and present and discuss the trade-off between power and robustness of selected classical and robust normality tests of error terms in regression models.
Abstract. This study investigated the computational benefits of using multi-fidelity statistical estimation (MFSE) algorithms to quantify uncertainty in the mass change of Humboldt Glacier, Greenland, between 2007 and 2100 using a single climate change scenario. The goal of this study was to determine whether MFSE can use multiple models of varying cost and accuracy to reduce the computational cost of estimating the mean and variance of the projected mass change of a glacier. The problem size and complexity were chosen to reflect the challenges posed by future continental-scale studies while still facilitating a computationally feasible investigation of MFSE methods. When quantifying uncertainty introduced by a high-dimensional parameterization of the basal friction field, MFSE was able to reduce the mean-squared error in the estimates of the statistics by well over an order of magnitude when compared to a single-fidelity approach that only used the highest-fidelity model. This significant reduction in computational cost was achieved despite the low-fidelity models used being incapable of capturing the local features of the ice-flow fields predicted by the high-fidelity model. The MFSE algorithms were able to effectively leverage the high correlation between each model's predictions of mass change, which all responded similarly to perturbations in the model inputs. Consequently, our results suggest that MFSE could be highly useful for reducing the cost of computing continental-scale probabilistic projections of sea-level rise due to ice-sheet mass change.
Purpose: The Monte Carlo (MC) method is a gold standard for dose calculation in radiotherapy. However, it is not a priori clear how many particles need to be simulated to achieve a given dose accuracy. Prior error estimate and stopping criterion are not well established for MC. This work aims to fill this gap. Methods: Due to the statistical nature of MC, our approach is based on one-sample t-test. We design the prior error estimate method based on the t-test, and then use this t-test based error estimate for developing a simulation stopping criterion. The three major components are as follows.First, the source particles are randomized in energy, space and angle, so that the dose deposition from a particle to the voxel is independent and identically distributed (i.i.d.).Second, a sample under consideration in the t-test is the mean value of dose deposition to the voxel by sufficiently large number of source particles. Then according to central limit theorem, the sample as the mean value of i.i.d. variables is normally distributed with the expectation equal to the true deposited dose.Third, the t-test is performed with the null hypothesis that the difference between sample expectation (the same as true deposited dose) and on-the-fly calculated mean sample dose from MC is larger than a given error threshold, in addition to which users have the freedom to specify confidence probability and region of interest in the t-test based stopping criterion. Results: The method is validated for proton dose calculation. The difference between the MC Result based on the t-test prior error estimate and the statistical Result by repeating numerous MC simulations is within 1%. Conclusion: The t-test based prior error estimate and stopping criterion are developed for MC and validated for proton dose calculation. Xiang Hong and Hao Gao were partially supported by the NSFC (#11405105), the 973 Program (#2015CB856000) and the Shanghai Pujiang Talent Program (#14PJ1404500)
The Fixed-Energy Response-Function Analysis with Multiple Efficiency (FRAM) code was developed at Los Alamos National Laboratory to measure the gamma-ray spectrometry of the isotopic composition of plutonium, uranium, and other actinides. For FRAM version 4 and earlier, the reported uncertainties of the results come from the propagation of the statistics in the peak areas only. No systematic error components are included in the reported uncertainties. We did several studies with FRAM v.4 and found that FRAM’s statistical precision can be reasonably represented by its reported uncertainties. FRAM’s biases or systematic uncertainties can come from a variety of sources and can be difficult to determine. For FRAM version 5, we carefully examined the FRAM analytical results of both the archival plutonium data and the data specifically acquired for the isotopic uncertainty analysis project and found the relationship between the bias and other parameters. We then worked out the equations representing the biases of the measured isotopes from each measurement using internal spectral parameters, such as peak resolution and shape, region of analysis, and burnup (for plutonium) or enrichment (for uranium). The resulting biases were included in the reported uncertainties of FRAM v.5. For the upcoming FRAM version 6.1, we are doing the same study that we did for FRAM v.5. The resulting biases will be included in its reported uncertainties.
We use ASCA observations of the Extended Medium Sensitivity Survey sample of clusters of galaxies to construct the first z = 0.5 - 0.8 cluster temperature function. This distant cluster temperature function, when compared to local z approximately 0 and to a similar moderate redshift (z = 0.3 - 0.4) temperature function strongly constrains the matter density of the universe. Best fits to the distributions of temperatures and redshifts of these cluster samples results in Omega(sub M) = 0.45 +/- 0.1 if Lambda = 0 and Omega = 0.27 +/- 0.1 if Lambda + Omega(sub M) = 1. The uncertainties are 1sigma statistical. We examine the systematics of our approach and find that systematics, stemming mainly from model assumptions and not measurement errors, are about the same size as the statistical uncertainty +/- 0.1. In this poster proceedings, we clarify the issue of a8 as reported in our paper Donahue & Voit (1999), since this was a matter of discussion at the meeting.
The utility of quantum computers for simulating lattice gauge theories is currently limited by the noisiness of the physical hardware. Various quantum error mitigation strategies exist to reduce the statistical and systematic uncertainties in quantum simulations via improved algorithms and analysis strategies. We perform quantum simulations of Z 2 gauge theory with matter to study the efficacy and interplay of different error mitigation methods: readout error mitigation, randomized compiling, rescaling, and dynamical decoupling. We compute Minkowski correlation functions in this confining gauge theory and extract the mass of the lightest spin-1 state from fits to their time dependence. Quantum error mitigation extends the range of times over which our correlation function calculations are accurate by a factor of 6 and is therefore essential for obtaining reliable masses.
A high-fidelity lidar turbulence measurement technique relies on accurate estimates of radial velocity variance that are subject to both systematic and random errors determined by the autocorrelation function of radial velocity, the sampling rate, and the sampling duration. Our paper quantifies the effect of the volumetric averaging in lidar radial velocity measurements on the autocorrelation function and the dependence of the systematic and random errors on the sampling duration, using both statistically simulated and observed data. For current-generation scanning lidars and sampling durations of about 30 min and longer, during which the stationarity assumption is valid for atmospheric flows, the systematic error is negligible but the random error exceeds about 10%.
Purpose: A novel computer software system, namely APDV (Automatic Pre-Delivery Verification), has been developed for verifying patient treatment plan parameters right prior to treatment deliveries in order to automatically detect and prevent catastrophic errors. Methods: APDV is designed to continuously monitor new DICOM plan files on the TMS computer at the treatment console. When new plans to be delivered are detected, APDV checks the consistencies of plan parameters and high-level plan statistics using underlying rules and statistical properties based on given treatment site, technique and modality. These rules were quantitatively derived by retrospectively analyzing all the EBRT treatment plans of the past 8 years at authors’ institution. Therapists and physicists will be notified with a warning message displayed on the TMS computer if any critical errors are detected, and check results, confirmation, together with dismissal actions will be saved into database for further review. Results: APDV was implemented as a stand-alone program using C# to ensure required real time performance. Mean values and standard deviations were quantitatively derived for various plan parameters including MLC usage, MU/cGy radio, beam SSD, beam weighting, and the beam gantry angles (only for lateral targets) per treatment site, technique and modality. 2D-based rules of combined MU/cGy ratio and averaged SSD values were also derived using joint probabilities of confidence error ellipses. The statistics of these major treatment plan parameters quantitatively evaluate the consistency of any treatment plans which facilitates automatic APDV checking procedures. Conclusion: APDV could be useful in detecting and preventing catastrophic errors immediately before treatment deliveries. Future plan including automatic patient identify and patient setup checks after patient daily images are acquired by the machine and become available on the TMS computer. This project is supported by the Agency for Healthcare Research and Quality (AHRQ) under award 1R01HS0222888. The senior author received research grants from ViewRay Inc. and Varian Medical System.
Physicists have long wondered whether the gravitational interactions between matter and antimatter might be different from those between matter and itself. Although there are many indirect indications that no such differences exist and that the weak equivalence principle holds, there have been no direct, free-fall style, experimental tests of gravity on antimatter. Here we describe a novel direct test methodology; we search for a propensity for antihydrogen atoms to fall downward when released from the ALPHA antihydrogen trap. In the absence of systematic errors, we can reject ratios of the gravitational to inertial mass of antihydrogen 475 at a statistical significance level of 5%; worst-case systematic errors increase the minimum rejection ratio to 110. A similar search places somewhat tighter bounds on a negative gravitational mass, that is, on antigravity. This methodology, coupled with ongoing experimental improvements, should allow us to bound the ratio within the more interesting near equivalence regime.
Physicists are often expected to have a solid grounding in experimental design and statistical analysis, sometimes filling in when biostatisticians or other experts are not available for consultation. Unfortunately, graduate education on these topics is seldom emphasized and few opportunities for continuing education exist. Clinical physicists incorporate new technology and methods into their practice based on published literature. A poor understanding of experimental design and analysis could Result in inappropriate use of new techniques. Clinical physicists also improve current practice through quality initiatives that require sound experimental design and analysis. Academic physicists with a poor understanding of design and analysis may produce ambiguous (or misleading) results. This can Result in unnecessary rewrites, publication rejection, and experimental redesign (wasting time, money, and effort). This symposium will provide a practical review of error and uncertainty, common study designs, and statistical tests. Instruction will primarily focus on practical implementation through examples and answer questions such as: where would you typically apply the test/design and where is the test/design typically misapplied (i.e., common pitfalls)? An analysis of error and uncertainty will also be explored using biological studies and associated modeling as a specific use case. Learning Objectives: Understand common experimental testing and clinical trial designs, what questions they can answer, and how to interpret the results Determine where specific statistical tests are appropriate and identify common pitfalls Understand the how uncertainty and error are addressed in biological testing and associated biological modeling.
Zero-noise extrapolation (ZNE) is an increasingly popular technique for mitigating errors in noisy quantum computations without using additional quantum resources. We review the fundamentals of ZNE and propose several improvements to noise scaling and extrapolation, the two key components in the technique. We introduce unitary folding and parameterized noise scaling. Furthermore, these are digital noise scaling frameworks, i.e. one can apply them using only gate-level access common to most quantum instruction sets. We also study different extrapolation methods, including a new adaptive protocol that uses a statistical inference framework. Benchmarks of our techniques show error reductions of 18X to 24X over non-mitigated circuits and demonstrate ZNE effectiveness at larger qubit numbers than have been tested previously. In addition to presenting new results, this work is a self-contained introduction to the practical use of ZNE by quantum programmers.
Abstract This study proposes and assesses a methodology to obtain high-quality probabilistic predictions and uncertainty information of near-landfall tropical cyclone–driven (TC-driven) storm tide and inundation with limited time and resources. Forecasts of TC track, intensity, and size are perturbed according to quasi-random Korobov sequences of historical forecast errors with assumed Gaussian and uniform statistical distributions. These perturbations are run in an ensemble of hydrodynamic storm tide model simulations. The resulting set of maximum water surface elevations are dimensionality reduced using Karhunen–Loève expansions and then used as a training set to develop a polynomial chaos (PC) surrogate model from which global sensitivities and probabilistic predictions can be extracted. The maximum water surface elevation is extrapolated over dry points incorporating energy head loss with distance to properly train the surrogate for predicting inundation. We find that the surrogate constructed with third-order PCs using elastic net penalized regression with leave-one-out cross validation provides the most robust fit across training and test sets. Probabilistic predictions of maximum water surface elevation and inundation area by the surrogate model at 48-h lead time for three past U.S. landfalling hurricanes (Irma in 2017, Florence in 2018, and Laura in 2020) are found to be reliable when compared to best track hindcast simulation results, even when trained with as few as 19 samples. The maximum water surface elevation is most sensitive to perpendicular track-offset errors for all three storms. Laura is also highly sensitive to storm size and has the least reliable prediction. Significance Statement The purpose of this study is to develop and evaluate a methodology that can be used to provide high-quality probabilistic predictions of hurricane-induced storm tide and inundation with limited time and resources. This is important for emergency management purposes during or after the landfall of hurricanes. Our results show that sampling forecast errors using quasi-random sequences combined with machine learning techniques that fit polynomial functions to the data are well suited to this task. The polynomial functions also have the benefit of producing exact sensitivity indices of storm tide and inundation to the forecasted hurricane properties such as path, intensity, and size, which can be used for uncertainty estimation. The code implementing the presented methodology is publicly available on GitHub.
Here in this investigation, a data-driven turbulence closure framework is introduced and deployed for the subgrid modelling of Kraichnan turbulence. The novelty of the proposed method lies in the fact that snapshots from high-fidelity numerical data are used to inform artificial neural networks for predicting the turbulence source term through localized grid-resolved information. In particular, our proposed methodology successfully establishes a map between inputs given by stencils of the vorticity and the streamfunction along with information from two well-known eddy-viscosity kernels. Through this we predict the subgrid vorticity forcing in a temporally and spatially dynamic fashion. Our study is both a priori and a posteriori in nature. In the former, we present an extensive hyper-parameter optimization analysis in addition to learning quantification through probability-density-function-based validation of subgrid predictions. In the latter, we analyse the performance of our framework for flow evolution in a classical decaying two-dimensional turbulence test case in the presence of errors related to temporal and spatial discretization. Statistical assessments in the form of angle-averaged kinetic energy spectra demonstrate the promise of the proposed methodology for subgrid quantity inference. In addition, it is also observed that some measure of a posteriori error must be considered during optimal model selection for greater accuracy. The results in this article thus represent a promising development in the formalization of a framework for generation of heuristic-free turbulence closures from data.