Search NASA⌕ Search

SEARCH · Search NASA

Results for “Error Metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Using a surrogate-assisted Bayesian framework to calibrate the runoff-generation scheme in the Energy Exascale Earth System Model (E3SM) v1

Abstract. Runoff is a critical component of the terrestrial water cycle, and Earth system models (ESMs) are essential tools to study its spatiotemporal variability. Runoff schemes in ESMs typically include many parameters so that model calibration is necessary to improve the accuracy of simulated runoff. However, runoff calibration at a global scale is challenging because of the high computational cost and the lack of reliable observational datasets. In this study, we calibrated 11 runoff relevant parameters in the Energy Exascale Earth System Model (E3SM) Land Model (ELM) using a surrogate-assisted Bayesian framework. First, the polynomial chaos expansion machinery with Bayesian compressed sensing is used to construct computationally inexpensive surrogate models for ELM-simulated runoff at 0.5∘ × 0.5∘ for 1991–2010. The error metric between the ELM simulations and the benchmark data is selected to construct the surrogates, which facilitates efficient calibration and avoids the more conventional, but challenging, construction of high-dimensional surrogates for the ELM simulated runoff. Second, the Sobol' index sensitivity analysis is performed using the surrogate models to identify the most sensitive parameters, and our results show that, in most regions, ELM-simulated runoff is strongly sensitive to 3 of the 11 uncertain parameters. Third, a Bayesian method is used to infer the optimal values of the most sensitive parameters using an observation-based global runoff dataset as the benchmark. Our results show that model performance is significantly improved with the inferred parameter values. Although the parametric uncertainty of simulated runoff is reduced after the parameter inference, it remains comparable to the multimodel ensemble uncertainty represented by the global hydrological models in ISMIP2a. Additionally, the annual global runoff trend during the simulation period is not well constrained by the inferred parameter values, suggesting the importance of including parametric uncertainty in future runoff projections.

58 GEOSCIENCES↗

Automatic documentation system extension to multi-manufacturers' computers and to measure, improve, and predict software reliability

The DOMONIC system has been modified to run on the Univac 1108 and the CDC 6600 as well as the IBM 370 computer system. The DOMONIC monitor system has been implemented to gather data which can be used to optimize the DOMONIC system and to predict the reliability of software developed using DOMONIC. The areas of quality metrics, error characterization, program complexity, program testing, validation and verification are analyzed. A software reliability model for estimating program completion levels and one on which to base system acceptance have been developed. The DAVE system which performs flow analysis and error detection has been converted from the University of Colorado CDC 6400/6600 computer to the IBM 360/370 computer system for use with the DOMONIC system.

Simmons, D. B.↗

Robust Method For Robotic Mapping

Robot constructs map from experience. Topological model consists of nodes and arcs corresponding to distinctive places and local travel edges linking nearby distinctive places. Model created by linking places and edges. Enables accumulation of metrical information with reduced vulnerability to metrical errors. Applications include robotic sentires, robotic delivery trucks, robotic floor cleaners, and robotic lawnmowers.

Kuipers, Benjamin J.↗

Phase-retrieval algorithms for a complicated optical system

Phase-retrieval algorithms have been developed that handle a complicated optical system that requires multiple Fresnellike transforms to propagate from one end of the system to the other including the absorption by apertures in more than one plane and allowance for bad detector pixels. Gradient-search algorithms and generalizations of the iterative-transform phase-retrieval algorithms are derived. Analytic expressions for the gradient of an error metric, with respect to polynomial coefficients and with respect to point-by-point phase descriptions, are given. The entire gradient can be computed with the number of transforms required to propagate a wave front from one end of the optical system to the other and back again, independent of the number of coefficients or phase points. This greatly speeds the computation. The reconstruction of pupil amplitude is also given. A convergence proof of the generalized iterative transform algorithm is given. These improved algorithms permit a more accurate characterization of complicated optical systems from their point spread functions.

Fienup, J. R.↗

Nonlinear aerodynamic modeling using multivariate orthogonal functions

A technique was developed for global modeling of nonlinear aerodynamic coefficients using multivariate orthogonal functions based on the data. Each orthogonal function retained in the model was decomposed into an expansion of ordinary polynomials in the independent variables, so that the final model could be interpreted as selectively retained terms from a multivariable power series expansion. A predicted squared-error metric was used to determine the orthogonal functions to be retained in the model; analytical derivatives were easily computed. The approach was demonstrated on the Z-body axis aerodynamic force coefficient (Cz) wind tunnel data for an F-18 research vehicle which came from a tabular wind tunnel and covered the entire subsonic flight envelope. For a realistic case, the analytical model predicted experimental values of Cz very well. The modeling technique is shown to be capable of generating a compact, global analytical representation of nonlinear aerodynamics. The polynomial model has good predictive capability, global validity, and analytical differentiability.

Morelli, Eugene A.↗

Comparison of Image Restoration Methods for Lunar Epithermal Neutron Emission Mapping

Orbital measurements of neutrons by the Lunar Exploring Neutron Detector (LEND) onboard the Lunar Reconnaissance Orbiter are being used to quantify the spatial distribution of near surface hydrogen (H). Inferred H concentration maps have low signal-to-noise (SN) and image restoration (IR) techniques are being studied to enhance results. A single-blind. two-phase study is described in which four teams of researchers independently developed image restoration techniques optimized for LEND data. Synthetic lunar epithermal neutron emission maps were derived from LEND simulations. These data were used as ground truth to determine the relative quantitative performance of the IR methods vs. a default denoising (smoothing) technique. We review and used factors influencing orbital remote sensing of neutrons emitted from the lunar surface to develop a database of synthetic "true" maps for performance evaluation. A prior independent training phase was implemented for each technique to assure methods were optimized before the blind trial. Method performance was determined using several regional root-mean-square error metrics specific to epithermal signals of interest. Results indicate unbiased IR methods realize only small signal gains in most of the tested metrics. This suggests other physically based modeling assumptions are required to produce appreciable signal gains in similar low SN IR applications.

McClanahan, T. P.↗

An OSSE on Mesoscale Model Assimilation of Simulated HIRAD-Observed Hurricane Surface Winds

The hazards of landfalling hurricanes are well known, but progress on improving the intensity forecasts of these deadly storms at landfall has been slow. Many cite a lack of high-resolution data sets taken inside the core of a hurricane, and the lack of reliable measurements in extreme conditions near the surface of hurricanes, as possible reasons why even the most state-of-the-art forecasting models cannot seem to forecast intensity changes better. The Hurricane Imaging Radiometer (HIRAD) is a new airborne microwave remote sensor for observing hurricanes, and is operated and researched by NASA Marshall Space Flight Center in partnership with the NOAA Atlantic Oceanographic and Meteorological Laboratory/Hurricane Research Division, the University of Central Florida, the University of Michigan, and the University of Alabama in Huntsville. This instrument?s purpose is to study the wind field of a hurricane, specifically observing surface wind speeds and rain rates, in what has traditionally been the most difficult areas for other instruments to study; the high wind and heavy rain regions. Dr. T. N. Krishnamurti has studied various data assimilation techniques for hurricane and monsoon rain rates, and this study builds off of results obtained from utilizing his style of physical initializations of rainfall observations, but obtaining reliable observations in heavy rain regions has always presented trouble to our research of high-resolution rainfall forecasting. Reliable data from these regions at such a high resolution and wide swath as HIRAD provides is potentially very valuable to mesoscale forecasting of hurricane intensity. This study shows how the data assimilation technique of Ensemble Kalman Filtering (EnKF) in the Weather Research and Forecasting (WRF) model can be used to incorporate wind, and later rain rate, data into a mesoscale model forecast of hurricane intensity. The study makes use of an Observing System Simulation Experiment (OSSE) with a simulated HIRAD dataset sampled during a hurricane and uses EnKF to forecast the track and intensity prediction of the hurricane. Comparisons to truth and error metrics are used to assess the model?s forecast performance.

Albers, Cerese↗

Using Dispersed Modes During Model Correlation

The model correlation process for the modal characteristics of a launch vehicle is well established. After a test, parameters within the nominal model are adjusted to reflect structural dynamics revealed during testing. However, a full model correlation process for a complex structure can take months of man-hours and many computational resources. If the analyst only has weeks, or even days, of time in which to correlate the nominal model to the experimental results, then the traditional correlation process is not suitable. This paper describes using model dispersions to assist the model correlation process and decrease the overall cost of the process. The process creates thousands of model dispersions from the nominal model prior to the test and then compares each of them to the test data. Using mode shape and frequency error metrics, one dispersion is selected as the best match to the test data. This dispersion is further improved by using a commercial model correlation software. In the three examples shown in this paper, this dispersion based model correlation process performs well when compared to models correlated using traditional techniques and saves time in the post-test analysis.

Stewart, Eric C.↗

An AeroCom–AeroSat study: intercomparison of satellite AOD datasets for aerosol model evaluation

To better understand and characterize current uncertainties in the important observational constraint of climate models of aerosol optical depth (AOD), we evaluate and intercompare 14 satellite products, representing nine different retrieval algorithm families using observations from five different sensors on six different platforms. The satellite products (super-observations consisting of 1°×1° daily aggregated retrievals drawn from the years 2006, 2008 and 2010) are evaluated with AErosol RObotic NETwork (AERONET) and Maritime Aerosol Network (MAN) data. Results show that different products exhibit different regionally varying biases (both under- and overestimates) that may reach ±50 %, although a typical bias would be 15 %–25 % (depending on the product). In addition to these biases, the products exhibit random errors that can be 1.6 to 3 times as large. Most products show similar performance, although there are a few exceptions with either larger biases or larger random errors. The intercomparison of satellite products extends this analysis and provides spatial context to it. In particular, we show that aggregated satellite AOD agrees much better than the spatial coverage (often driven by cloud masks) within the 1°×1° grid cells. Up to ∼50 % of the difference between satellite AOD is attributed to cloud contamination. The diversity in AOD products shows clear spatial patterns and varies from 10 % (parts of the ocean) to 100 % (central Asia and Australia). More importantly, we show that the diversity may be used as an indication of AOD uncertainty, at least for the better performing products. This provides modellers with a global map of expected AOD uncertainty in satellite products, allows assessment of products away from AERONET sites, can provide guidance for future AERONET locations and offers suggestions for product improvements. We account for statistical and sampling noise in our analyses. Sampling noise, variations due to the evaluation of different subsets of the data, causes important changes in error metrics. The consequences of this noise term for product evaluation are discussed.

Aerosol↗

Autogenous Pressurization of a Cryogenic Tank Using Computational Fluid Dynamics

Well-validated simulation tools can be used to predict long-term storage and transfer of cryogenic propellants, which are essential to NASA’s mission plan to return to the Moon and continue to Mars. Autogenous pressurization of propellant tanks is used to supply rocket engine turbopumps with pressurized liquid fuel and oxidizer, preventing cavitation. Additionally, autogenous pressurization can be used to pressurize propellant tanks to support on-orbit propellant transfer. In lieu of expensive tests conducted on-orbit, accurate predictive computational models of these processes can be used to reduce system and propellant mass as well as mission risk. Using simulation tools also reduces the cost of analyzing and developing this technology. This study presents a multiphase computational fluid dynamics model capable of simulating autogenous pressurization of a large cryogenic tank using the commercial code STAR-CCM+. Experimental data from a full-scale tank pressurization test under terrestrial gravity is compared to simulation results using transient error metrics. Comparisons show good agreement and give confidence in using these and other validated simulation tools to develop cryogenic pressurization systems.

Cryogenics↗

Autogenous Pressurization of a Cryogenic Tank using Computational Fluid Dynamics

Well-validated simulation tools can be used to predict long-term storage and transfer of cryogenic propellants, which are essential to NASA’s mission plan to return to the Moon and continue to Mars. Autogenous pressurization of propellant tanks is used to supply rocket engine turbopumps with pressurized liquid fuel and oxidizer, preventing cavitation. Additionally, autogenous pressurization can be used to pressurize propellant tanks to support on-orbit propellant transfer. In lieu of expensive tests conducted on-orbit, accurate predictive computational models of these processes can be used to reduce system and propellant mass as well as mission risk. Using simulation tools also reduces the cost of analyzing and developing this technology. This study presents a multiphase computational fluid dynamics model capable of simulating autogenous pressurization of a large cryogenic tank using the commercial code STAR-CCM+. Experimental data from a full-scale tank pressurization test under terrestrial gravity is compared to simulation results using transient error metrics. Comparisons show good agreement and give confidence in using these and other validated simulation tools to develop cryogenic pressurization systems. Note: There is no audio file included or available for this presentation.

cryogenics↗

An L1 Adaptive Control Augmentation for a Lift Plus-Cruise Vehicle

This paper presents anL1adaptive control augmentation for a Lift-Plus-Cruise (L+C)vehicle. This class of vehicles operates in three flight modes with different dynamic behavior: vertical, transition, and forward flight. A robust uniform controller is used as a baseline to stabilize the system throughout these flight modes. The uniform controller is a linear control law designed around trim conditions of the aircraft and includes control allocation to achieve the desired forces and moments on the vehicle. TheL1control augmentation is designed for each of these trim conditions to compensate for the nonlinear time- and state-dependent uncertainties in the vehicle dynamics. The augmented control output is then added to the desired force and moment commands on the vehicle. Simulation results demonstrate the effectiveness of control augmentation for reducing the effects of unmodeled dynamics, reduced actuator effectiveness, and time-dependent disturbances. Effectiveness is demonstrated through tracking error metrics.

Andrew Patterson↗

Taming Wild Horses: The Need for Virtual Time-based Scheduling of VMs in Network Simulations

The next generation of scalable network simulators employ virtual machines (VMs) to act as high-fidelity models of traffic producer/consumer nodes in simulated networks. However, network simulations could be inaccurate if VMs are not scheduled according to virtual time, especially when many VMs are hosted per simulator core in a multi-core simulator environment. Since VMs are by default free-running, on the outset, it is not clear if, and to what extent, their untamed execution affects the results in simulated scenarios. Here, we provide the first quantitative basis for establishing the need for generalized virtual time scheduling of VMs in network simulators, based on an actual prototyped implementations. To exercise breadth, our system is tested with multiple disparate applications: (a) a set of message passing parallel programs, (b) a computer worm propagation phenomenon, and (c) a mobile ad-hoc wireless network simulation. We define and use error metrics and benchmarks in scaled tests to empirically report the poor match of traditional, fairness-based VM scheduling to VM-based network simulation, and also clearly show the better performance of our simulation-specific scheduler, with up to 64 VMs hosted on a 12-core simulator node.

99 GENERAL AND MISCELLANEOUS↗

Surrogate Models for TREAT Transient Calculations

The Transient Reactor Test Facility (TREAT) at Idaho National Laboratory (INL) is currently preparing for resumption of operation after more than 20 years in a long-term standby state. In parallel with preparation for operations and a new transient testing program, the DOE-NE Advanced Modeling and Simulation program (NEAMS) has been supporting the development of three-dimensional transient multi-physics modeling capabilities within the reactor physics application MAMMOTH [1]. Recently, the MAMMOTH reactor physics team has begun development of uncertainty and sensitivity analysis for the TREAT core. This research will be used to support the analysis of reactor operation and experiment planning. Traditional approaches, i.e., Monte Carlo simulations, for such tasks require a large number of model evaluations under both steady state and transient conditions. However, exhaustive sets of sampling runs are simply not practical here since evaluations of each model can take a substantial amount of time. A solution to this problem is to build a surrogate model employing a smaller number of model evaluations. The surrogate model is designed to provide an accurate and fast approximation to the detailed TREAT model. In this manner, the surrogate model can be used in place of the TREAT transient model for uncertainty and sensitivity analysis at a fraction of the computational cost. The high dimensional model representation (HDMR) [2] method is employed here for approximating the responses of interest in terms of functions of lower dimensions. This method has been implemented in RAVEN [3] and all TREAT transient calculations are performed using the multiphysics code MAMMOTH. In addition, a code interface has also been developed in order to couple RAVEN with MAMMOTH. The surrogate model is then tested and compared for the MAMMOTH model of the TREAT startup testing program Transient Test #15 core [4], and its accuracy is measured by its ability to estimate the mean, coefficient of variance, skewness, excess kurtosis and probability density function of the outputs of Transient Test #15 model, as well as root mean square error metric.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

The Effect of Measurement Error on Surveillance Metrics

The purpose of this manuscript is to describe different simulation studies that CCS-6 has performed for the purpose of understanding the effects of measurement error on the surveillance metrics. We assume that the measured items come from a larger population of items. We denote the random variable associate with an item's value of an attribute of interest as X and that $X \text{~} N(\mu, \sigma^2$). This distribution represents the variability in the population of interest and we wish to make inference on the parameters $\mu$ and $\sigma$ or on some function of these parameters. When an item X is selected from the larger population, a measurement is made on some attribute of it. This measurement is made with error and the true value of X is not observed. The rest of this section presents simulation results for different measurement cases encountered.

42 ENGINEERING↗

MO-FG-202-07: Real-Time EPID-Based Detection Metric For VMAT Delivery Errors

Purpose: To create and test an accurate EPID-frame-based VMAT QA metric to detect gross dose errors in real-time and to provide information about the source of error. Methods: A Swiss cheese model was created for an EPID-based real-time QA process. The system compares a treatmentplan- based reference set of EPID images with images acquired over each 2° gantry angle interval. The metric utilizes a sequence of independent consecutively executed error detection Methods: a masking technique that verifies infield radiation delivery and ensures no out-of-field radiation; output normalization checks at two different stages; global image alignment to quantify rotation, scaling and translation; standard gamma evaluation (3%, 3 mm) and pixel intensity deviation checks including and excluding high dose gradient regions. Tolerances for each test were determined. For algorithm testing, twelve different types of errors were selected to modify the original plan. Corresponding predictions for each test case were generated, which included measurement-based noise. Each test case was run multiple times (with different noise per run) to assess the ability to detect introduced errors. Results: Averaged over five test runs, 99.1% of all plan variations that resulted in patient dose errors were detected within 2° and 100% within 4° (∼1% of patient dose delivery). Including cases that led to slightly modified but clinically equivalent plans, 91.5% were detected by the system within 2°. Based on the type of method that detected the error, determination of error sources was achieved. Conclusion: An EPID-based during-treatment error detection system for VMAT deliveries was successfully designed and tested. The system utilizes a sequence of methods to identify and prevent gross treatment delivery errors. The system was inspected for robustness with realistic noise variations, demonstrating that it has the potential to detect a large majority of errors in real-time and indicate the error source. J. V. Siebers receives funding support from Varian Medical Systems.

60 APPLIED LIFE SCIENCES↗

Alternatives to accuracy and bias metrics based on percentage errors for radiation belt modeling applications

This report reviews existing literature describing forecast accuracy metrics, concentrating on those based on relative errors and percentage errors. We then review how the most common of these metrics, the mean absolute percentage error (MAPE), has been applied in recent radiation belt modeling literature. Finally, we describe metrics based on the ratios of predicted to observed values (the accuracy ratio) that address the drawbacks inherent in using MAPE. Specifically, we define and recommend the median log accuracy ratio as a measure of bias and the median symmetric accuracy as a measure of accuracy.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗