Search NASA⌕ Search

SEARCH · Search NASA

Results for “Error Metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Accelerating Time-Varying Hardware Volume Rendering Using TSP Trees and Color-Based Error Metrics

This paper describes a new hardware volume rendering algorithm for time-varying data. The algorithm uses the Time-Space Partitioning (TSP) tree data structure to identify regions within the data that have spatial or temporal coherence. By using this coherence, the rendering algorithm can improve performance when the volume data is larger than the texture memory capacity by decreasing the amount of textures required. This coherence can also allow improved speed by appropriately rendering flat-shaded polygons instead of textured polygons, and by not rendering transparent regions. To reduce the polygonization overhead caused by the use of the hierarchical data structure, we introduce an optimization method using polygon templates. The paper also introduces new color-based error metrics, which more accurately identify coherent regions compared to the earlier scalar-based metrics. By showing experimental results from runs using different data sets and error metrics, we demonstrate that the new methods give substantial improvements in volume rendering performance.

Ellsworth, David↗

Effectiveness of various error metrics in SCEPTRE.

The objective of this project is to investigate accuracy of error metrics in SCEPTRE and produce useful benchmarks, identify metrics that do not work well, identify metrics that do work well, and produce easy to reference results.

99 GENERAL AND MISCELLANEOUS↗

Performance Metrics, Error Modeling, and Uncertainty Quantification

A common set of statistical metrics has been used to summarize the performance of models or measurements-­ the most widely used ones being bias, mean square error, and linear correlation coefficient. They assume linear, additive, Gaussian errors, and they are interdependent, incomplete, and incapable of directly quantifying un­certainty. The authors demonstrate that these metrics can be directly derived from the parameters of the simple linear error model. Since a correct error model captures the full error information, it is argued that the specification of a parametric error model should be an alternative to the metrics-based approach. The error-modeling meth­odology is applicable to both linear and nonlinear errors, while the metrics are only meaningful for linear errors. In addition, the error model expresses the error structure more naturally, and directly quantifies uncertainty. This argument is further explained by highlighting the intrinsic connections between the performance metrics, the error model, and the joint distribution between the data and the reference.

Quantification↗

WE-H-BRC-05: Catastrophic Error Metrics for Radiation Therapy

Purpose: Intuitive evaluation of complex radiotherapy treatments is impractical, while data transfer anomalies create the potential for catastrophic treatment delivery errors. Contrary to prevailing wisdom, logical scrutiny can be applied to patient-specific machine settings. Such tests can be automated, applied at the point of treatment delivery and can be dissociated from prior states of the treatment plan, potentially revealing errors introduced early in the process. Methods: Analytical metrics were formulated for conventional and intensity modulated RT (IMRT) treatments. These were designed to assess consistency between monitor unit settings, wedge values, prescription dose and leaf positioning (IMRT). Institutional metric averages for 218 clinical plans were stratified over multiple anatomical sites. Treatment delivery errors were simulated using a commercial treatment planning system and metric behavior assessed via receiver-operator-characteristic (ROC) analysis. A positive result was returned if the erred plan metric value exceeded a given number of standard deviations, e.g. 2. The finding was declared true positive if the dosimetric impact exceeded 25%. ROC curves were generated over a range of metric standard deviations. Results: Data for the conventional treatment metric indicated standard deviations of 3%, 12%, 11%, 8%, and 5 % for brain, pelvis, abdomen, lung and breast sites, respectively. Optimum error declaration thresholds yielded true positive rates (TPR) between 0.7 and 1, and false positive rates (FPR) between 0 and 0.2. Two proposed IMRT metrics possessed standard deviations of 23% and 37%. The superior metric returned TPR and FPR of 0.7 and 0.2, respectively, when both leaf position and MUs were modelled. Isolation to only leaf position errors yielded TPR and FPR values of 0.9 and 0.1. Conclusion: Logical tests can reveal treatment delivery errors and prevent large, catastrophic errors. Analytical metrics are able to identify errors in monitor units, wedging and leaf positions with favorable sensitivity and specificity. In part by Varian.

60 APPLIED LIFE SCIENCES↗

Error metrics for partially coherent wave fields

Lensless imaging methods that account for partial coherence have become very common in the past decade. However, there are no metrics in use for comparing partially coherent light fields, despite the widespread use of such metrics to compare fully coherent objects and wave fields. Here, we show how reformulating the mean squared error and Fourier ring correlation in terms of quantum state fidelity naturally generalizes them to partially coherent wave fields. We report these results fill an important gap in the lensless imaging literature and will enable quantitative assessments of the reliability and resolution of reconstructed partially coherent wave fields.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Radio metric errors due to mismatch and offset between a DSN antenna beam and the beam of a troposphere calibration instrument

Two components of the error of a troposphere calibration measurement were quantified by theoretical calculations. The first component is a beam mismatch error, which occurs when the calibration instrument senses a conical volume different from the cylindrical volume sampled by a Deep Space Network (DSN) antenna. The second component is a beam offset error, which occurs if the calibration instrument is not mounted on the axis of the DSN antenna. These two error sources were calculated for both delay (e.g., VLBI) and delay rate (e.g., Doppler) measurements. The beam mismatch error for both delay and delay rate drops rapidly as the beamwidth of the troposphere calibration instrument (e.g., a water vapor radiometer or an infrared Fourier transform spectrometer) is reduced. At a 10-deg elevation angle, the instantaneous beam mismatch error is 1.0 mm for a 6-deg beamwidth and 0.09 mm for a 0.5-deg beam (these are the full angular widths of a circular beam with uniform gain out to a sharp cutoff). Time averaging for 60-100 sec will reduce these errors by factors of 1.2-2.2. At a 20-deg elevation angle, the lower limit for current Doppler observations, the beam-mismatch delay rate error is an Allan standard deviation over 100 sec of 1.1 x 10(exp -14) with a 4-deg beam and 1.3 x 10(exp -l5) for a 0.5-deg beam. A 50-m beam offset would result in a fairly modest (compared to other expected error sources) delay error (less than or equal to 0.3 mm for 60-sec integrations at any elevation angle is greater than or equal to 6 deg). However, the same offset would cause a large error in delay rate measurements (e.g., an Allan standard deviation of 1.2 x 10(exp -14) over 100 sec at a 20-deg elevation angle), which would dominate over other known error sources if the beamwidth is 2 deg or smaller. An on-axis location is essential for accurate troposphere calibration of delay rate measurements. A half-power beamwidth (for a beam with a tapered gain profile) of 1.2 deg or smaller is desired for calibration of all types of radio metrics. A water-vapor radiometer calibration beam of this size with very low sidelobes would require a clear aperture antenna with a diameter of at least 1.5 m if the primary water vapor sensing channel were in the 20-22 GHz range.

R P Linfield↗

Accounting for Training Data Error in Machine Learning Applied to Earth Observations

Remote sensing, or Earth Observation (EO), is increasingly used to understand Earth system dynamics and create continuous and categorical maps of biophysical properties and land cover, especially based on recent advances in machine learning (ML). ML models typically require large, spatially explicit training datasets to make accurate predictions. Training data (TD) are typically generated by digitizing polygons on high spatial-resolution imagery, by collecting in situ data, or by using pre-existing datasets. TD are often assumed to accurately represent the truth, but in practice almost always have error, stemming from (1) sample design, and (2) sample collection errors. The latter is particularly relevant for image-interpreted TD, an increasingly commonly used method due to its practicality and the increasing training sample size requirements of modern ML algorithms. TD errors can cause substantial errors in the maps created using ML algorithms, which may impact map use and interpretation. Despite these potential errors and their real-world consequences for map-based decisions, TD error is often not accounted for or reported in EO research. Here we review the current practices for collecting and handling TD. We identify the sources of TD error, and illustrate their impacts using several case studies representing different EO applications (infrastructure mapping, global surface flux estimates, and agricultural monitoring), and provide guidelines for minimizing and accounting for TD errors. To harmonize terminology, we distinguish TD from three other classes of data that should be used to create and assess ML models: training reference data, used to assess the quality of TD during data generation; validation data, used to iteratively improve models; and map reference data, used only for final accuracy assessment. We focus primarily on TD, but our advice is generally applicable to all four classes, and we ground our review in established best practices for map accuracy assessment literature. EO researchers should start by determining the tolerable levels of map error and appropriate error metrics. Next, TD error should be minimized during sample design by choosing a representative spatio-temporal collection strategy, by using spatially and temporally relevant imagery and ancillary data sources during TD creation, and by selecting a set of legend definitions supported by the data. Furthermore, TD error can be minimized during the collection of individual samples by using consensus-based collection strategies, by directly comparing interpreted training observations against expert-generated training reference data to derive TD error metrics, and by providing image interpreters with thorough application-specific training. We strongly advise that TD error is incorporated in model outputs, either directly in bias and variance estimates or, at a minimum, by documenting the sources and implications of error. TD should be fully documented and made available via an open TD repository, allowing others to replicate and assess its use. To guide researchers in this process, we propose three tiers of TD error accounting standards. Finally, we advise researchers to clearly communicate the magnitude and impacts of TD error on map outputs, with specific consideration given to the likely map audience.

58 GEOSCIENCES↗

Response Surface Modeling Using Multivariate Orthogonal Functions

A nonlinear modeling technique was used to characterize response surfaces for non-dimensional longitudinal aerodynamic force and moment coefficients, based on wind tunnel data from a commercial jet transport model. Data were collected using two experimental procedures - one based on modem design of experiments (MDOE), and one using a classical one factor at a time (OFAT) approach. The nonlinear modeling technique used multivariate orthogonal functions generated from the independent variable data as modeling functions in a least squares context to characterize the response surfaces. Model terms were selected automatically using a prediction error metric. Prediction error bounds computed from the modeling data alone were found to be- a good measure of actual prediction error for prediction points within the inference space. Root-mean-square model fit error and prediction error were less than 4 percent of the mean response value in all cases. Efficacy and prediction performance of the response surface models identified from both MDOE and OFAT experiments were investigated.

Morelli, Eugene A.↗

Evaluating the WFIP2 updates to the HRRR model using scanning Doppler lidar measurements in the complex terrain of the Columbia River Basin

The wind-energy (WE) industry relies on numerical weather prediction (NWP) forecast models as foundational or base models for many purposes, including wind-resource assessment and wind-power forecasting. During the Second Wind Forecast Improvement Project (WFIP2) in the Columbia River Basin of Oregon and Washington, a significant effort was made to improve NWP forecasts through focused model development, to include experimental refinements to the High Resolution Rapid Refresh (HRRR) model physics and horizontal grid spacing. In this study, the performance of an experimental version of HRRR that includes these refinements is tested against a control version, which corresponds to that of the operational HRRR run by National Oceanic and Atmospheric Administration/National Centers for Environmental Protection at the outset of WFIP2. Furthermore, the effects of horizontal grid resolution were also tested by comparing wind forecasts from the HRRR (with 3-km grid spacing) with those from a finer-resolution HRRR nest with 750-m grid spacing. Model forecasts are validated against accurate wind-profile measurements by three scanning, pulsed Doppler lidars at sites separated by a total distance of 71 km. Model skill and improvements in model skill, attributable to physics refinements and improved horizontal grid resolution, varied by season, by site, and during periods of atmospheric phenomena relevant to WE. In general, model errors were the largest below 150 m above ground level (AGL). Experimental HRRR refinements tended to reduce the mean absolute error (MAE) and other error metrics for many conditions, but degradation in skill (increased MAE) was noted below 150 m AGL at the two lowest-elevation sites at night. Finer resolution was found to produce the most significant reductions in the error metrics.

17 WIND ENERGY↗

Short-term electricity load forecasting: Application-driven evaluation of machine learning models across spatial and temporal scales

As we transition towards a decarbonized economy, the integration of variable renewable energy resources and new demands (e.g., electric vehicles, heat pumps) into the electricity grid places unprecedented pressure on grid operators to effectively anticipate and manage peak load. In this context, machine learning algorithms are proving to be indispensable for accurate short-term load forecasting, a crucial task to address these challenges. This study benchmarks 6 machine learning algorithms, including three neural networks and three tree-based algorithms, across various levels of spatial aggregation and time horizons (1, 4, 8, 24, and 48 h). The central contribution of this work is the comparison and analysis of load forecasting models not only based on statistical metrics, but also based on a novel error metric, which evaluates the cost implications of forecast errors for power system stakeholders. Results show that tree-based models outperform neural networks, based on statistical metrics, and yield less skewed error distributions for most spatial scales. However, through the lens of the novel error metric, neural networks are the more competitive choice, especially for forecast horizons that exceed 8 h. The study concludes with actionable recommendations to grid operators and highlights the need for the development of error metrics that link forecasting accuracy to operational costs. To promote transparency and open science, the datasets and Python code are open-sourced via a supplementary repository.

Houben, Nikolaus↗

Comparing field data using Alpert multi-wavelets

In this paper we introduce a method to compare sets of full-field data using Alpert tree-wavelet transforms. The Alpert tree-wavelet methods transform the data into a spectral space allowing the comparison of all points in the fields by comparing spectral amplitudes. The methods are insensitive to translation, scale and discretization and can be applied to arbitrary geometries. This makes them especially well suited for comparison of field data sets coming from two different sources such as when comparing simulation field data to experimental field data. We have developed both global and local error metrics to quantify the error between two fields. Additionally, we verify the methods on two-dimensional and three-dimensional discretizations of analytical functions. Furthermore, we then deploy the methods to compare full-field strain data from a simulation of elastomeric syntactic foam.

42 ENGINEERING↗

Physics-informed transformation toward improving the machine-learned NLTE models of ICF simulations

The integration of machine-learning techniques into inertial confinement fusion (ICF) simulations has emerged as a powerful approach for enhancing computational efficiency. By replacing the costly nonlocal thermodynamic equilibrium (NLTE) model with machine-learning models, significant reductions in calculation time have been achieved. However, determining how to optimize machine-learning-based NLTE models in order to match ICF simulation dynamics remains challenging, underscoring the need for physically relevant error metrics and strategies to enhance model accuracy with respect to these metrics. Thus, we propose novel physics-informed transformations designed to emphasize energy transport, use these transformations to establish new error metrics, and demonstrate that they yield smaller errors within reduced principal-component spaces compared to conventional transformations. Published by the American Physical Society 2025

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Commercial Supersonic Fan Noise Modeling

This presentation describes the development of an empirical noise model for a two-stage fan designed for a supersonic commercial aircraft. The existing fan noise models are reviewed and compared with noise data from a wind tunnel experiment where a model scale two-stage fan was tested. An error metric was developed to quantify how well the measurements and empirical model match. This error metric was then used to determine new coefficients and parameters for a version of the fan model that is a good match for the experimental data.

supersonics, acoustics, turbofan↗

NESC GN&C TDT Workshop on 2D Image Motion Optical Transfer Functions, Pointing Performance Analysis, and Requirements

What You Will Learn: The focus is on payload imaging performance due to pointing motion. Some historical background on pointing performance analysis is given. The Optical Transfer Function (OTF) and Modulation Transfer Function (MTF) are defined. The imaging performance due to pointing motion is measured by image motion optical transfer functions (IM OTF). IM OTFs are defined for displacement, smear, and jitter motions, which are all rigorously defined. Deterministic and Statistical IM OTFs are briefly derived and graphically illustrated and compared. The IM OTFs are parameterized by pointing error metrics(PEM), which are means and covariances of displacement, smear, and jitter. Emphasis is on procedures and algorithms to evaluate the image motion optical transfer functions and pointing error metrics. Three procedures are covered, which depend on whether the pointing error data is from time-domain simulation, frequency-domain analysis, or stochastic modeling. A method to evaluate the relative contribution of disturbance sources and to identify the most significant contributors is presented. The presentation includes pertinent discussion of flexible structures and control-structure interaction. No single book can adequately cover this subject, so a book is not required for the course. A list of selected articles, reports, documents, and books is provided for reference and further study. Mathis kept to the minimum necessary to convey principles; lengthy derivations are left to the reference material. Graphics are used to illustrate concepts. As with any such learning endeavor, the knowledge gained will be retained and strengthened through actual practice.c©2019–2021 Mark E. Pittelkau— 5

NASA Engineering and Safety Center (NESC)↗

Intercomparison of wind and wave data from the ECMWF Reanalysis Interim and the NCEP Climate Forecast System Reanalysis

The recent release of the ECMWF Reanalysis Interim (ERA-I) and NCEP Climate Forecast System Reanalysis (CFSR) allows for studies of global climate and its cycles with unprecedented detail. While the developers have performed verification and validation, there is little information on their relative performance in particular related to their use in ocean modeling. This study focuses on the intercomparison of wind and wave reanalysis datasets from ERA-I and CFSR utilizing the same set of altimetry and buoy observations and error metrics to assess their consistency in time and space. Both products have good spatial homogeneity with consistent levels of errors in the Northern and Southern Hemispheres. ERA-I proves to be homogenous through time, while CFSR exhibits an abrupt decrease in the level of errors in the Southern Ocean beginning 1994. ERA-I generally underestimates the wind speed and wave height with lower standard deviations in comparison to observations, but maintains slightly better error metrics. Despite having a slightly positive bias, CFSR provides a better description of the variability of the observations and improved performance in the upper percentiles associated with extreme events. Altogether ERA-I has better homogeneity through time deeming it more reliable for modeling of long-term processes; however caution must be applied with analysis of the upper percentiles.

16 TIDAL AND WAVE POWER↗

MS25: Materials Science-Focused Benchmark Data Set for Machine Learning Interatomic Potentials

Here, we present MS25, a benchmark data set for evaluating machine learning interatomic potentials (MLIPs) across diverse materials-relevant systems including MgO surfaces, liquid water, zeolites, a catalytic Pt surface reaction, high-entropy alloys (HEAs), and disordered Zr-oxides. Five MLIP architectures (MACE, NequIP, Allegro, MTP, and Torch-ANI) are trained and tested, focusing not only on traditional metrics (energies, forces, and stresses) but also explicitly validating derived physical observables such as lattice constants, volumes, and reaction barriers. We find that most models reach comparable accuracy on standard error metrics across the simple systems, although equivariant MLIPs offer 1.5–2× improvements over nonequivariant MLIPs in energy and force error for structurally complex or compositionally disordered environments such as HEAs and Zr–O systems. Our analysis highlights that low errors in energy and force predictions do not guarantee reliable observables, emphasizing the necessity of explicit validation. We demonstrate limitations in cross-framework transferability, as models trained on one zeolite framework (CHA) fail to reliably generalize to predictions of structurally distinct frameworks (e.g., MFI). Size-extensive tests show some dependence on system size for MgO, resulting from forced periodicity. The HEA and Zr–O data sets are identified as challenging tests for future benchmarks and MLIP model architecture developments as they show significant differentiation in error between MLIP architectures and are still relatively difficult at 1000 training images. Moving forward, we recommend that benchmarking efforts shift their focus from marginal accuracy improvements in energy and force errors toward identifying and understanding model failure modes, rigorously assessing transferability, and evaluating how their errors affect observable predictions. For researchers looking to choose an MLIP architecture, we suggest selecting equivariant MLIP architectures if the complexity of the system is a challenge. For simple materials problems, auxiliary features such as integration with molecular dynamics engines, trade-offs between computational data set generation cost vs MLIP inference speed, and framework integration may play a more important decision factor than small differences in error metrics that are unlikely to matter for production-level research.

chemical structure↗