Search NASA⌕ Search

SEARCH · Search NASA

Results for “Model error”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26

Assessment of errors in analytic modeling of permanent magnet electron spectrometers for laser-plasma accelerators

Here, we assess the error in three treatments of a critical electron beam diagnostic for laser-plasma acceleration (LPA) experiments: a permanent magnet electron spectrometer. Since LPA electron spectrometers are often difficult to calibrate due to the scarcity of well-characterized, tunable electron beam sources in the appropriate energy range and the mechanical complexity of electron spectrometers, the standard of calibration is Hall probe measurements. We first compare the electron spectrometer performance between SIMION calculations and the Hall probe measurements. We find up to a 60% (<8%) error in determining the absolute energy for electrons below (above) 80 MeV when using the SIMION-modeled magnetic field vs the measured magnetic field due to SIMION overpredicting fringe field strength. The difference in spectrometer energy resolution is within ∼10% between the modeled and measured fields. We then assess a simple block model, which is commonly used. We find that the simple block model presented here sufficiently captures the predicted as-built spectrometer performance for any application provided that the user can tolerate uncertainties on absolute energy determination up to 5% and errors on energy resolution up to 1%.

Electromagnetic optics↗

Multi-phenomenology Yield Characterization

This report serves as the first delivery of a four-year applied science effort to transform and advance the error bounds for the yield estimate of an explosion. Each year’s delivery will be in this form, culminating in the submission of this work for peer review to a scientific journal. Importantly, the yearly progress reports can then also be viewed as expanding drafts working towards a formal journal article submission. For the first tranche of funding, we collaborated with Air Force Technical Applications Center (AFTAC) scientists to identify unclassified real-world data that demonstrate and validate our advanced error propagation methods. Collaboration includes visits to AFTAC and telecons. For this development, we illustrate the fusion of seismic, acoustic, optical, and surface effect signatures from an explosion. The mathematics and code being adapted to this specific application (Williams et al., 2021) involves physics models of multiple sensor signatures. We have also identified related physics models and have integrated them into code. Current methods of underground explosion yield estimation for the Threshold Test Ban Treaty (TTBT) have served the US treaty monitoring mission well for decades. A research objective of the Defense Nuclear Nonproliferation Research and Development (DNN R&D) office of the National Nuclear Security Administration (NNSA) has always been to provide new technical capabilities for monitoring lower thresholds. The general model and error propagation code to be developed in this project is based on significant advances in error modeling and propagation needed to analyze data at lower detection thresholds. The second tranche of funding for this project began on May 1, 2022, and planned work for the second tranche includes: i) completing the integration of physical model code into the general error model framework; this code accommodates a wide range of linear/nonlinear source models, fixed/ random effects, and frequentist/Bayesian analyses (the purpose of which is not to dictate to users how to analyze data, but instead to allow users the maximum flexibility in their work); ii) illustrative application of code to identified data, and; iii) initial planning with AFTAC researchers on delivery of code to the Common Development Environment at AFTAC, and continued writing of the planned final journal article submission (year two of this progress report), with particular emphasis on descriptions of data identified for this effort.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

Data-driven closure modeling for hypersonic turbulent flows

The Reynolds-averaged Navier–Stokes (RANS) equations remain a workhorse technology for simulating compressible fluid flows of practical interest. Due to model-form errors, however, RANS models can yield erroneous predictions that preclude their use on mission-critical problems. This report summarizes work performed from FY22-FY24 focused on improving RANS models for hypersonic flows using data-driven modeling and scientific machine learning. In this work we: 1. Investigate the current capabilities of RANS models in Sandia’s parallel aerodynamics and re-entry code (SPARC) for hypersonic flows with a focus on shock boundary layer interactions (SBLIs), 2. Assess several established corrections that exist in the literature aimed at improving predictions for SBLIs, 3. Develop improved models for the Reynolds stress tensor using tensor-basis neural networks, 4. Develop a neural-network-based variable turbulent Prandtl number model to reduce errors in wall heating in SBLIs. 5. Begin future investigations including employing the LIFE framework to improve wall heating predictions in SBLIs as well as the ensemble Kalman filter. We find that current RANS models in SPARC are deficient for complex SBLI flows. In particular, no current model jointly predicts wall heat flux, wall shear stress, and wall pressure with reasonable accuracy. Existing corrections help, but do not alleviate this issue altogether. The development of improved models for the Reynolds stress tensor via tensor-basis neural networks results in more predictive RANS models across a suite of low-speed and high-speed cases. For hypersonic boundary layers, the inclusion of the wall-normal Reynolds stress via TBNNs has an appreciable impact on the wall-normal momentum balance and wall quantities. However, we find that improvements to the Reynolds stress tensor do not address the over-prediction in wall heat flux in SBLIs. We find that a neural-network-based variable turbulent Prandtl number model systematically and substantially improves wall heating predictions for a range of SBLI cases.

97 MATHEMATICS AND COMPUTING↗

Model validation and error attribution for a drifting qubit

Qubit performance is often reported in terms of a variety of single-value metrics, each providing a facet of the underlying noise mechanism limiting performance. However, the value of these metrics may drift over long timescales, and reporting a single number for qubit performance fails to account for the low-frequency noise processes that give rise to this drift. Here, in this work, we demonstrate how we can use the distribution of these values to validate or invalidate candidate noise models. We focus on the case of randomized benchmarking (RB), where typically a single error rate is reported but this error rate can drift over time when multiple passes of RB are performed. We show that using a statistical test as simple as the Kolmogorov-Smirnov statistic on the distribution of RB error rates can be used to rule out noise models, assuming the experiment is performed over a long enough time interval to capture relevant low frequency noise. With confidence in a noise model, we show how care must be exercised when performing error attribution using the distribution of drifting RB error rate.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

On the Statistical Calibration of Physical Models

ABSTRACT We introduce a novel statistical calibration framework for physical models, relying on probabilistic embedding of model discrepancy error within the model. For clarity of illustration, we take the measurement errors out of consideration, calibrating a chemical model of interest with respect to a more detailed model, considered as “truth” for the present purpose. We employ Bayesian statistical methods for such model‐to‐model calibration and demonstrate their capabilities on simple synthetic models, leading to a well‐defined parameter estimation problem that employs approximate Bayesian computation. The method is then demonstrated on two case studies for calibration of kinetic rate parameters for methane air chemistry, where ignition time information from a detailed elementary‐step kinetic model is used to estimate rate coefficients of a simple chemical mechanism. We show that the calibrated model predictions fit the data and that uncertainty in these predictions is consistent in a mean‐square sense with the discrepancy from the detailed model data.

Sargsyan, K.↗

Improving Lidar Turbulence Estimates for Wind Energy

Remote sensing devices (e.g., lidars) are quickly becoming a cost-effective and reliable alternative to meteorological towers for wind energy applications. Although lidars can measure mean wind speeds accurately, these devices measure different values of turbulence intensity (TI) than an instrument on a tower. In response to these issues, a lidar TI error reduction model was recently developed for commercially available lidars. The TI error model first applies physics-based corrections to the lidar measurements, then uses machine-learning techniques to further reduce errors in lidar TI estimates. The model was tested at two sites in the Southern Plains where vertically profiling lidars were collocated with meteorological towers. This presentation primarily focuses on the physics-based corrections, which include corrections for instrument noise, volume averaging, and variance contamination. As different factors affect TI under different stability conditions, the combination of physical corrections applied in L-TERRA changes depending on the atmospheric stability during each 10-minute time period. This stability-dependent version of L-TERRA performed well at both sites, reducing TI error and bringing lidar TI estimates closer to estimates from instruments on towers. However, there is still scatter evident in the lidar TI estimates, indicating that there are physics that are not being captured in the current version of L-TERRA. Two options are discussed for modeling the remainder of the TI error physics in L-TERRA: machine learning and lidar simulations. Lidar simulations appear to be a better approach, as they can help improve understanding of atmospheric effects on TI error and do not require a large training data set.

17 WIND ENERGY↗

COSP: Satellite simulation software for model assessment

Errors in the simulation of clouds in general circulation models (GCMs) remain a long-standing issue in climate projections, as discussed in the Intergovernmental Panel on Climate Change (IPCC) Fourth Assessment Report. This highlights the need for developing new analysis techniques to improve our knowledge of the physical processes at the root of these errors. The Cloud Feedback Model Intercomparison Project (CFMIP) pursues this objective, and under that framework the CFMIP Observation Simulator Package (COSP) has been developed. COSP is a flexible software tool that enables the simulation of several satellite-borne active and passive sensor observations from model variables. The flexibility of COSP and a common interface for all sensors facilitates its use in any type of numerical model, from high-resolution cloud-resolving models to the coarser-resolution GCMs assessed by the IPCC, and the scales in between used in weather forecast and regional models. The diversity of model parameterization techniques makes the comparison between model and observations difficult, as some parameterized variables (e.g., cloud fraction) do not have the same meaning in all models. The approach followed in COSP permits models to be evaluated against observations and compared against each other in a more consistent manner. This thus permits a more detailed diagnosis of the physical processes that govern the behavior of clouds and precipitation in numerical models. The World Climate Research Programme (WCRP) Working Group on Coupled Modelling has recommended the use of COSP in a subset of climate experiments that will be assessed by the next IPCC report. Here we describe COSP, present some results from its application to numerical models, and discuss future work that will expand its capabilities.

54 ENVIRONMENTAL SCIENCES↗

Detecting and Testing for Structural Error in Computer Models with Application to the Community Atmospheric Model

Bayesian calibration/inversion methods for uncertain input parameters to simulation codes have generated an increasing amount of interest in the climate community in the recent years. This approach uses multiple disparate observational datasets related to the output of the simulator of interest to yield a probabilistic constraint on the uncertain inputs (i.e. probability distribution). A major challenge in applying Bayesian calibration methods to simulation codes is related to the possible presence of a large structural error in the output fields of interest; that is, the presence of discrepancy between the observed and the simulated output that is neither explained by the uncertainty in the inputs nor the observations. We present here methods to test for and visualize structural error in an ensemble of input-perturbed simulations. The methods are based on bootstrap (resmpling) test statistics and well-known metrics to quantify the difference between two probability distributions. The proposed methods are demonstrated using selected precipitation output fields of the Community Atmospheric Model (CAM), which are compared to observations provided by the Global Precipitation Climatology Project (GPCP). The methods are found to be useful in identifying regions and seasons with a large structural error. The proposed structural error detection methodology can be used to screen multiple observational datasets of interest for structural error, making it possible to retain only the datasets with a small structural error for use in Bayesian calibration. In addition, information about the amount of structural error can be valuable to model developers.

58 GEOSCIENCES↗

Balancing aggregation and smoothing errors in inverse models

Inverse models use observations of a system (observation vector) to quantify the variables driving that system (state vector) by statistical optimization. When the observation vector is large, such as with satellite data, selecting a suitable dimension for the state vector is a challenge. A state vector that is too large cannot be effectively constrained by the observations, leading to smoothing error. However, reducing the dimension of the state vector leads to aggregation error as prior relationships between state vector elements are imposed rather than optimized. Here we present a method for quantifying aggregation and smoothing errors as a function of state vector dimension, so that a suitable dimension can be selected by minimizing the combined error. Reducing the state vector within the aggregation error constraints can have the added advantage of enabling analytical solution to the inverse problem with full error characterization. We compare three methods for reducing the dimension of the state vector from its native resolution: (1) merging adjacent elements (grid coarsening), (2) clustering with principal component analysis (PCA), and (3) applying a Gaussian mixture model (GMM) with Gaussian pdfs as state vector elements on which the native-resolution state vector elements are projected using radial basis functions (RBFs). The GMM method leads to somewhat lower aggregation error than the other methods, but more importantly it retains resolution of major local features in the state vector while smoothing weak and broad features.

Turner, A. J.↗

LIKELIHOOD-FREE COSMOLOGICAL INFERENCE WITH TYPE Ia SUPERNOVAE: APPROXIMATE BAYESIAN COMPUTATION FOR A COMPLETE TREATMENT OF UNCERTAINTY

Cosmological inference becomes increasingly difficult when complex data-generating processes cannot be modeled by simple probability distributions. With the ever-increasing size of data sets in cosmology, there is an increasing burden placed on adequate modeling; systematic errors in the model will dominate where previously these were swamped by statistical errors. For example, Gaussian distributions are an insufficient representation for errors in quantities like photometric redshifts. Likewise, it can be difficult to quantify analytically the distribution of errors that are introduced in complex fitting codes. Without a simple form for these distributions, it becomes difficult to accurately construct a likelihood function for the data as a function of parameters of interest. Approximate Bayesian computation (ABC) provides a means of probing the posterior distribution when direct calculation of a sufficiently accurate likelihood is intractable. ABC allows one to bypass direct calculation of the likelihood but instead relies upon the ability to simulate the forward process that generated the data. These simulations can naturally incorporate priors placed on nuisance parameters, and hence these can be marginalized in a natural way. We present and discuss ABC methods in the context of supernova cosmology using data from the SDSS-II Supernova Survey. Assuming a flat cosmology and constant dark energy equation of state, we demonstrate that ABC can recover an accurate posterior distribution. Finally, we show that ABC can still produce an accurate posterior distribution when we contaminate the sample with Type IIP supernovae.

79 ASTRONOMY AND ASTROPHYSICS↗

Machine Learning Models for GPU Error Prediction in a Large Scale HPC System

GPUs are widely deployed on large-scale HPC systems to provide powerful computational capability for scientific applications from various domains. As those applications are normally long-running, investigating the characteristics of GPU errors becomes imperative for reliability. In this paper, we first study the system conditions that trigger GPU errors using six-month trace data collected from a large-scale, operational HPC system. Then, we use machine learning to predict the occurrence of GPU errors, by taking advantage of temporal and spatial dependencies of the trace data. The resulting machine learning prediction framework is robust and accurate under different workloads.

Nie, Bin↗

Lattice Element Error Solver for Modeling of Accelerators, Storage Ring, Transport Systems, and Insertion Devices

The goals of CRADA 350 were achieved in this Phase I SBIR work. Specifically, a software environment was developed which included a nonlinear solver, OASIS. During the Phase I work, we applied the algorithm to a number of nonlinear-solver benchmark problems, including SLAC accelerator problems. By searching over only two optimization-control parameters, this Phase I work has enabled the development of a systematic procedure to facilitate the finding of a “best-guessed” global minimum solution for various types of non-linear problems with different size and complexity. This set of rules was codified and led to a versatile nonlinear program, IMIGO.

43 PARTICLE ACCELERATORS↗

High-Resolution Regional Wave Hindcast for the U.S. Alaska Coast

Regional long-term wave hindcast with state-of-art third-generation models is essential for characterization of regional wave resources. This report summarizes modeling efforts for the simulation of the wave climate along southern Alaska, which used an unstructured, nested-grid modeling approach that incorporates a global-regional nested grid using WAVEWATCH III ® and the high-resolution SWAN (Simulating Waves Nearshore) model. Wave resource and wave bulk parameters were simulated for a 32-year period from 1979 to 2010 and were subsequently validated with wave buoy data in the model domain. Error statistics of model skills were calculated at all buoy stations. Overall, model results, both resource and bulk parameters, match observations well. The model hindcast was able to reproduce the seasonal variation of the sea state with large waves that occur in the winter and early spring months when wind forcing is strong and the calm sea state during the summer when wind forcing is weak. The hindcast also considers the effect of sea ice on wave growth, which affects the southern Bering Sea. The hindcast results, including resource and bulk parameters, were archived at every model grid point for the entire model domain at three-hour intervals. In addition, spectral results were saved at National Data Buoy Center buoy stations and selected virtual buoy locations at hourly intervals. The nested-grid modeling framework employed in this study provides a powerful and efficient modeling approach for accurately simulating wave climate at regional and long-term temporal scales with sufficiently fine resolutions in the nearshore region.

54 ENVIRONMENTAL SCIENCES↗