Search NASA⌕ Search

SEARCH · Search NASA

Results for “Error Metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Deployment of Traditional and Hybrid Machine Learning for Critical Heat Flux Prediction in the CTF Thermal-Hydraulics Code

Critical heat flux (CHF) marks the transition from nucleate to film boiling, where heat transfer to the working fluid can rapidly deteriorate. Accurate CHF prediction is essential for efficiency, safety, and preventing equipment damage, particularly in nuclear reactors. Although widely used, empirical correlations frequently exhibit discrepancies when compared to experimental data, limiting their reliability in diverse operational conditions. Traditional machine learning (ML) approaches have demonstrated potential for CHF prediction but often suffer from limited interpretability, data scarcity, and insufficient knowledge of physical principles. Hybrid model approaches, which combine data-driven ML with base models, mitigate these concerns by incorporating prior knowledge of the domain. This study integrates an externally trained purely data-driven ML model and two hybrid models (using the Biasi and Bowring CHF correlations) within the CTF subchannel code via a custom Fortran framework. Performance was evaluated using two validation cases: a subset of the Nuclear Regulatory Commission (NRC) CHF database and the Bennett dryout experiments. In both cases, the hybrid models demonstrated significantly lower error metrics compared to conventional empirical correlations, with the best models often reducing relative error by about 5 percentage points. The pure ML model achieved comparable accuracy, outperforming the hybrid Biasi model in the NRC test case (3.3% versus 5.5% relative error) but exhibiting slightly higher error against the hybrid Bowring model in the Bennett test case (7.7% versus 6.1%). Trend analysis of error parity indicated that ML-based models reduced the tendency for CHF overprediction, improving overall accuracy. These results demonstrate that ML-based CHF models can be effectively integrated into subchannel codes and could potentially increase performance compared to conventional methods.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Assessment of Computational Fluid Dynamics (CFD) Models for Shock Boundary-Layer Interaction

A workshop on the computational fluid dynamics (CFD) prediction of shock boundary-layer interactions (SBLIs) was held at the 48th AIAA Aerospace Sciences Meeting. As part of the workshop numerous CFD analysts submitted solutions to four experimentally measured SBLIs. This paper describes the assessment of the CFD predictions. The assessment includes an uncertainty analysis of the experimental data, the definition of an error metric and the application of that metric to the CFD solutions. The CFD solutions provided very similar levels of error and in general it was difficult to discern clear trends in the data. For the Reynolds Averaged Navier-Stokes methods the choice of turbulence model appeared to be the largest factor in solution accuracy. Large-eddy simulation methods produced error levels similar to RANS methods but provided superior predictions of normal stresses.

DeBonis, James R.↗

Modeling mechanical behavior of an additively manufactured metal structure with local texture variations: a study on model form error

Here, the repetitive rapid solidification that occurs in metal additive manufacturing (AM) processes creates microstructures distinctly different from wrought materials. Local variability in AM microstructures (either by design or unintentional) raises questions as to how AM structures should be modeled at the part-scale to minimize modeling error. The key goal of this work is to demonstrate a posteriori error estimation applied to an AM part. It is assumed that the actual microstructure is unknown and an approximate, spatially uniform material model is used. Error bounds are calculated for many reference models based on AM microstructures with elongated grain morphology and localized or global fiber textures during a post-processing step. The current findings promote confidence that a posteriori model form error estimation could be used effectively in mechanical performance simulations of AM parts to quickly obtain quantitative error metrics between an approximate model result and many microstructure-based reference models. The a posteriori error estimation introduces significant time savings compared to computing the full reference model solutions. Tight bounds on model form error are obtained when texture variations in the reference models occur on large length scales. For materials with property variation at small length scales, multi-scale error estimation techniques are needed to properly account for the many interfaces present between areas with different properties.

36 MATERIALS SCIENCE↗

Gleipnir: toward practical error analysis for Quantum programs

Practical error analysis is essential for the design, optimization, and evaluation of Noisy Intermediate-Scale Quantum(NISQ) computing. However, bounding errors in quantum programs is a grand challenge, because the effects of quantum errors depend on exponentially large quantum states. In this work, we present Gleipnir, a novel methodology toward practically computing verified error bounds in quantum programs. Gleipnir introduces the (ρ,δ)-diamond norm, an error metric constrained by a quantum predicate consisting of the approximate state ρ and its distance δ to the ideal state ρ. This predicate (ρ,δ) can be computed adaptively using tensor networks based on the Matrix Product States. Gleipnir features a lightweight logic for reasoning about error bounds in noisy quantum programs, based on the (ρ,δ)-diamond norm metric. Furthermore, our experimental results show that Gleipnir is able to efficiently generate tight error bounds for real-world quantum programs with 10 to 100 qubits, and can be used to evaluate the error mitigation performance of quantum compiler transformations.

Tao, Runzhou↗

An Objective and Efficient Method for Assessing the Impact of Reduced-Precision Calculations On Solution Correctness

Recent studies have shown that reducing the precision of floating-point calculations in an atmospheric model can improve the models' computational performance without affecting model fidelity, but code changes are needed to accommodate lower precision, and a small number of calculations might need to retain higher precision to prevent undue round-off error. For complex and computationally expensive systems like the Energy Exascale Earth System Model (E3SM), a method is needed to objectively assess the quality of the lower-precision simulations and to quickly identify problematic code pieces This paper demonstrates that solution correctness can be evaluated through a simple and quantitative error metric based on time step convergence. The proposed test method can unambiguously detect the accumulation of precision error and objectively assess its impact on solution accuracy. Since fast physical processes are known to affect key features of the multi-year mean climate in an atmospheric model, we show that the convergence test applied to short simulations can provide useful information about the impact of reduced precision on a model’s long-term behavior. In contrast, the traditional way of climate model evaluation requires multi-year simulations and involves inspecting many physical quantities for statistical significance in the presence of natural variability. The simplicity, computational efficiency, and objectivity of the proposed method makes it very attractive for high-resolution model development. The method is also expected to be applicable to other models that numerically solve time evolution equations.

97 MATHEMATICS AND COMPUTING↗

Assessing Clouds Using Satellite Observations Through Three Generations of Global Atmosphere Models

Abstract Clouds are parameterized in climate models using quantities on the model grid‐scale to approximate the cloud cover and impact on radiation. Because of the complexity of processes involved with clouds, these parameterizations are one of the key challenges in climate modeling. Differences in parameterizations of clouds are among the main contributors to the spread in climate sensitivity across models. In this work, the clouds in three generations of an atmosphere model lineage are evaluated against satellite observations. Satellite simulators are used within the model to provide an appropriate comparison with individual satellite products. In some respects, especially the top‐of‐atmosphere cloud radiative effect, the models show generational improvements. The most recent generation, represented by two distinct branches of development, exhibits some regional regressions in the cloud representation; in particular the southern ocean shows a positive bias in cloud cover. The two branches of model development show how choices during model development, both structural and parametric, lead to different cloud climatologies. Several evaluation strategies are used to quantify the spatial errors in terms of the large‐scale circulation and the cloud structure. The Earth mover's distance is proposed as a useful error metric for the passive satellite data products that provide cloud‐top pressure‐optical depth histograms. The cloud errors identified here may contribute to the high climate sensitivity in the Community Earth System Model, version 2 and in the Energy Exascale Earth System Model, version 1.

54 ENVIRONMENTAL SCIENCES↗

A variational framework for residual-based adaptivity in neural PDE solvers and operator learning

Residual-based adaptive strategies are widely used in scientific machine learning yet remain largely heuristic. We introduce a variational framework that formalizes these methods through convex transformations of the residual, where different transformations correspond to distinct objective functionals. For instance, exponential weights target uniform error minimization, while linear weights recover quadratic error minimization. This perspective reveals adaptive weighting as a means of selecting sampling distributions that optimize a primal objective, directly linking discretization choices to error metrics. This principled approach yields three key benefits: it enables systematic design of adaptive schemes, reduces discretization error by lowering estimator variance, and enhances learning dynamics by improving gradient signal-to-noise ratio. Extending the framework to operator learning, we demonstrate substantial performance gains across diverse optimizers and architectures. Our results provide a theoretical perspective for residual-based adaptivity and establish a foundation for principled discretization and training.

97 MATHEMATICS AND COMPUTING↗

Beam-centric algorithm for pretreatment patient position correction in external beam radiation therapy

Purpose: In current image guided pretreatment patient position adjustment methods, image registration is used to determine alignment parameters. Since most positioning hardware lacks the full six degrees of freedom (DOF), accuracy is compromised. The authors show that such compromises are often unnecessary when one models the planned treatment beams as part of the adjustment calculation process. The authors present a flexible algorithm for determining optimal realizable adjustments for both step-and-shoot and arc delivery methods. Methods: The beam shape model is based on the polygonal intersection of each beam segment with the plane in pretreatment image volume that passes through machine isocenter perpendicular to the central axis of the beam. Under a virtual six-DOF correction, ideal positions of these polygon vertices are computed. The proposed method determines the couch, gantry, and collimator adjustments that minimize the total mismatch of all vertices over all segments with respect to their ideal positions. Using this geometric error metric as a function of the number of available DOF, the user may select the most desirable correction regime. Results: For a simulated treatment plan consisting of three equally weighted coplanar fixed beams, the authors achieve a 7% residual geometric error (with respect to the ideal correction, considered 0% error) by applying gantry rotation as well as translation and isocentric rotation of the couch. For a clinical head-and-neck intensity modulated radiotherapy plan with seven beams and five segments per beam, the corresponding error is 6%. Correction involving only couch translation (typical clinical practice) leads to a much larger 18% mismatch. Clinically significant consequences of more accurate adjustment are apparent in the dose volume histograms of target and critical structures. Conclusions: The algorithm achieves improvements in delivery accuracy using standard delivery hardware without significantly increasing total treatment session duration. It encourages parsimonious utilization of all available DOF. Finally, in certain cases, it obviates the need of a robotic couch having six DOF for the correction of patient displacement and rotations.

61 RADIATION PROTECTION AND DOSIMETRY↗

DCT quantization matrices visually optimized for individual images

This presentation describes how a vision model incorporating contrast sensitivity, contrast masking, and light adaptation is used to design visually optimal quantization matrices for Discrete Cosine Transform image compression. The Discrete Cosine Transform (DCT) underlies several image compression standards (JPEG, MPEG, H.261). The DCT is applied to 8x8 pixel blocks, and the resulting coefficients are quantized by division and rounding. The 8x8 'quantization matrix' of divisors determines the visual quality of the reconstructed image; the design of this matrix is left to the user. Since each DCT coefficient corresponds to a particular spatial frequency in a particular image region, each quantization error consists of a local increment or decrement in a particular frequency. After adjustments for contrast sensitivity, local light adaptation, and local contrast masking, this coefficient error can be converted to a just-noticeable-difference (jnd). The jnd's for different frequencies and image blocks can be pooled to yield a global perceptual error metric. With this metric, we can compute for each image the quantization matrix that minimizes the bit-rate for a given perceptual error, or perceptual error for a given bit-rate. Implementation of this system demonstrates its advantages over existing techniques. A unique feature of this scheme is that the quantization matrix is optimized for each individual image. This is compatible with the JPEG standard, which requires transmission of the quantization matrix.

Watson, Andrew B.↗

An Approach for the Assessment of System Upset Resilience

This report describes an approach for the assessment of upset resilience that is applicable to systems in general, including safety-critical, real-time systems. For this work, resilience is defined as the ability to preserve and restore service availability and integrity under stated conditions of configuration, functional inputs and environmental conditions. To enable a quantitative approach, we define novel system service degradation metrics and propose a new mathematical definition of resilience. These behavioral-level metrics are based on the fundamental service classification criteria of correctness, detectability, symmetry and persistence. This approach consists of a Monte-Carlo-based stimulus injection experiment, on a physical implementation or an error-propagation model of a system, to generate a system response set that can be characterized in terms of dimensional error metrics and integrated to form an overall measure of resilience. We expect this approach to be helpful in gaining insight into the error containment and repair capabilities of systems for a wide range of conditions.

Torres-Pomales, Wilfredo↗

A hybrid wavelet de-noising and Rank-Set Pair Analysis approach for forecasting hydro-meteorological time series

Highlights: • A new hybrid approach is proposed to improve forecasts of hydro-meteorological time series. • Rank-Set Pair Analysis combined with wavelet de-noising markedly improves forecasting accuracy. • The performance of the proposed approach proves best among its present competitors even when the extreme value occurs. Accurate, fast forecasting of hydro-meteorological time series is presently a major challenge in drought and flood mitigation. This paper proposes a hybrid approach, wavelet de-noising (WD) and Rank-Set Pair Analysis (RSPA), that takes full advantage of a combination of the two approaches to improve forecasts of hydro-meteorological time series. WD allows decomposition and reconstruction of a time series by the wavelet transform, and hence separation of the noise from the original series. RSPA, a more reliable and efficient version of Set Pair Analysis, is integrated with WD to form the hybrid WD-RSPA approach. Two types of hydro-meteorological data sets with different characteristics and different levels of human influences at some representative stations are used to illustrate the WD-RSPA approach. The approach is also compared to three other generic methods: the conventional Auto Regressive Integrated Moving Average (ARIMA) method, Artificial Neural Networks (ANNs) (BP-error Back Propagation, MLP-Multilayer Perceptron and RBF-Radial Basis Function), and RSPA alone. Nine error metrics are used to evaluate the model performance. Compared to three other generic methods, the results generated by WD-REPA model presented invariably smaller error measures which means the forecasting capability of the WD-REPA model is better than other models. The results show that WD-RSPA is accurate, feasible, and effective. In particular, WD-RSPA is found to be the best among the various generic methods compared in this paper, even when the extreme events are included within a time series.

58 GEOSCIENCES↗

Depletion GPT-free sensitivity analysis for reactor eigenvalue problems

This manuscript introduces a novel approach to solving depletion perturbation theory problems without the need to set up or solve the generalized perturbation theory (GPT) equations. The approach, hereinafter denoted generalized perturbation theory free (GPT-Free), constructs a reduced order model (ROM) using methods based in perturbation theory and computes response sensitivity profiles in a manner that is independent of the number or type of responses, allowing for an efficient computation of sensitivities when many responses are required. Moreover, the reduction error from using the ROM is quantified in the GPT-Free approach by means of a Wilks' order statistics error metric denoted the K-metric. Traditional GPT has been recognized as the most computationally efficient approach for performing sensitivity analyses of models with many input parameters, e.g. when forward sensitivity analyses are computationally intractable. However, most neutronics codes that can solve the fundamental (homogenous) adjoint eigenvalue problem do not have GPT capabilities unless envisioned during code development. The GPT-Free approach addresses this limitation by requiring only the ability to compute the fundamental adjoint. This manuscript demonstrates the GPT-Free approach for depletion reactor calculations performed in SCALE6 using the 7x7 UAM assembly model. A ROM is developed for the assembly over a time horizon of 990 days. The approach both calculates the reduction error over the lifetime of the simulation using the K-metric and benchmarks the obtained sensitivities using sample calculations. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Short-Term Solar Forecasting Performance of Popular Machine Learning Algorithms: Preprint

A framework for assessing the performance of short-term solar forecasting is presented in conjunction with a range of numerical results using global horizontal irradiation (GHI) from the open-source Surface Radiation Budget (SURFRAD) data network. A suite of popular machine learning algorithms is compared according to a set of statistically distinct metrics and benchmarked against the persistence-of-cloudiness forecast and a cloud motion forecast. Results show significant improvement compared to the benchmarks with trade-offs among the machine learning algorithms depending on the desired error metric. Training inputs include time series observations of GHI for a history of years, historical weather and atmospheric measurements, and corresponding date and time stamps such that training sensitivities might be inferred. Prediction outputs are GHI forecasts for 1, 2, 3, and 4 hours ahead of the issue time, and they are made for every month of the year for 7 locations. Photovoltaic power and energy outputs can then be made using the solar forecasts to better understand power system impacts.

14 SOLAR ENERGY↗

Assessing the Effects of Data Compression in Simulations Using Physically Motivated Metrics

This paper examines whether lossy compression can be used effectively in physics simulations as a possible strategy to combat the expected data-movement bottleneck in future high performance computing architectures. We show that, for the codes and simulations we tested, compression levels of 3–5X can be applied without causing significant changes to important physical quantities. Rather than applying signal processing error metrics, we utilize physics-based metrics appropriate for each code to assess the impact of compression. We evaluate three different simulation codes: a Lagrangian shock-hydrodynamics code, an Eulerian higher-order hydrodynamics turbulence modeling code, and an Eulerian coupled laser-plasma interaction code. We compress relevant quantities after each time-step to approximate the effects of tightly coupled compression and study the compression rates to estimate memory and disk-bandwidth reduction. We find that the error characteristics of compression algorithms must be carefully considered in the context of the underlying physics being modeled.

Laney, Daniel↗

Marker-free registration of forest terrestrial laser scanner data pairs with embedded confidence metrics

Terrestrial laser scanning (TLS) has emerged as an effective tool for rapid comprehensive measurement of object structure. Registration of TLS data is an important prerequisite to overcome the limitations of occlusion. However, due to the high dissimilarity of point cloud data collected from disparate viewpoints in the forest environment, adequate marker-free registration approaches have not been developed. The majority of studies instead rely on the utilization of artificial tie points (e.g., reflective tooling balls) placed within a scene to aid in coordinate transformation. We present a technique for generating view-invariant feature descriptors that are intrinsic to the point cloud data and, thus, enable blind marker-free registration in forest environments. To overcome the limitation of initial pose estimation, we employ a voting method to blindly determine the optimal pairwise transformation parameters, without an a priori estimate of the initial sensor pose. To provide embedded error metrics, we developed a set theory framework in which a circular transformation is traversed between disjoint tie point subsets. This provides an upper estimate of the Root Mean Square Error (RMSE) confidence associated with each pairwise transformation. Output RMSE errors are commensurate with the RMSE of input tie points locations. Thus, while the mean output RMSE=16.3cm, improved results could be achieved with a more precise laser scanning system. This study 1) quantifies the RMSE of the proposed marker-free registration approach, 2) assesses the validity of embedded confidence metrics using receiver operator characteristic (ROC) curves, and 3) informs optimal sample spacing considerations for TLS data collection in New England forests. Furthermore, while the implications for rapid, accurate, and precise forest inventory are obvious, the conceptual framework outlined here could potentially be extended to built environments.

47 OTHER INSTRUMENTATION↗

Calibrating Microscopic Car-Following Models for Adaptive Cruise Control Vehicles: Multiobjective Approach

Adaptive cruise control (ACC) vehicles are the first step toward comprehensive vehicle automation. However, the impacts of such vehicles on the underlying traffic flow are not yet clear. Therefore, it is of interest to accurately model vehicle-level dynamics of commercially available ACC vehicles so that they may be used in further modeling efforts to quantify the impact of commercially available ACC vehicles on traffic flow. Importantly, not only model selection but also the calibration approach and error metric used for calibration are critical to accurately model ACC vehicle behavior. In this work, we explore the question of how to calibrate car following models to describe ACC vehicle dynamics. Specifically, we apply a multi-objective calibration approach to understand the tradeoff between calibrating model parameters to minimize speed error vs. spacing error. Three different car-following models are calibrated for data from six vehicles. The results are in line with recent literature and verify that targeting a low spacing error does not compromise the speed accuracy whether the opposite is not true for modeling ACC vehicle dynamics.

33 ADVANCED PROPULSION SYSTEMS↗

Surrogate Models for TREAT Transient Calculations

The Transient Reactor Test Facility (TREAT) at Idaho National Laboratory (INL) is currently preparing for resumption of operation after more than 20 years in a long-term standby state. In parallel with preparation for operations and a new transient testing program, the DOENE Advanced Modeling and Simulation program (NEAMS) has been supporting the development of three-dimensional transient multi-physics modeling capabilities within the reactor physics application MAMMOTH. Recently, the MAMMOTH reactor physics team has begun development of uncertainty and sensitivity analysis for the TREAT core. This research will be used to support the analysis of reactor operation and experiment planning. Traditional approaches, i.e., Monte Carlo simulations, for such tasks require a large number of model evaluations under both steady state and transient conditions. However, exhaustive sets of sampling runs are simply not practical for TREAT since evaluations of each model can take a substantial amount of time. A solution to this problem is to build a surrogate model employing a smaller number of model evaluations. The surrogate model is designed to provide an accurate and fast approximation to the detailed TREAT model. In this manner, the surrogate model can be used in place of the TREAT transient model for uncertainty and sensitivity analysis at a fraction of the computational cost. The high dimensional model representation (HDMR) method is employed here for approximating the responses of interest in terms of functions of lower dimensions. This method has been implemented in RAVEN and all TREAT transient calculations are performed using the multiphysics code MAMMOTH. In addition, a code interface has also been developed in order to couple RAVEN with MAMMOTH. The surrogate model is then tested and compared for TREAT startup testing program Transient Test no. 15 core, and its accuracy is measured by its ability to estimate the mean, coefficient of variance, skewness, excess kurtosis and probability density function of the outputs of Transient Test no. 15 model, as well as root mean square error metric.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Worldwide benchmarking of cost-effective radiometers for direct and diffuse irradiance

Solar energy projects can benefit from direct normal irradiance (DNI) and diffuse horizontal irradiance (DHI) measurements during all project phases. Several commercial measurement systems for DNI and DHI are available. Sun trackers with pyranometers and pyrheliometers can provide highly accurate measurements but are often impractical in solar energy applications. For less expensive and more robust sensors, it is often unclear which accuracy can be expected under a project site's specific atmospheric conditions. We address this challenge through our dedicated experimental comparison of relevant sensor systems (rotating shadowband irradiometer [short RSI], Delta-T SPN1, EKO MS-90, PyranoCam, Sunto CaptPro, Kipp & Zonen CSD3) at up to six sites worldwide. The RSI systems (rRMSD 3 to 8.6%, DNI; 4.8 to 7.6%, DHI) and PyranoCam (rRMSD 2.6 to 5.2%, DNI; 4.4 to 5.8%, DHI) exhibit similar error metrics and are the most accurate systems in the test. Delta-T SPN1 and EKO MS-90 (rRMSD 6.8 to 15%, DNI; 10.6 to 20.1%, DHI) but especially Kipp & Zonen CSD3 and Sunto CaptPro show significant deviations (rRMSD 17.7 to 20%, DNI; 33 to 58%, DHI). We evaluate the influence of relevant atmospheric parameters on the sensors' accuracies by a rather unique measurement setup. MS-90's DNI errors depend on DNI itself, with overestimations for low reference DNI. The deviations of SPN1's DHI and DNI measurements increase sharply in situations with high circumsolar irradiance. Also CaptPro and CSD3's increased measurement errors are related to circumsolar irradiance. For RSI and PyranoCam, only moderate influences on the measurements are identified, indicating a general applicability of these instruments.

14 SOLAR ENERGY↗