Search NASA⌕ Search

SEARCH · Search NASA

Results for “Model error”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

A Comparison of Forecast Error Generators for Modeling Wind and Load Uncertainty

This paper presents four algorithms to generate random forecast error time series, including a truncated-normal distribution model, a state-space based Markov model, a seasonal autoregressive moving average (ARMA) model, and a stochastic-optimization based model. The error time series are used to create real-time (RT), hour-ahead (HA), and day-ahead (DA) wind and load forecast time series that statistically match historically observed forecasting data sets, used for variable generation integration studies. A comparison is made using historical DA load forecast and actual load values to generate new sets of DA forecasts with similar stoical forecast error characteristics. This paper discusses and compares the capabilities of each algorithm to preserve the characteristics of the historical forecast data sets.

Lu, Ning↗

Moving beyond the total sea ice extent in gauging model biases

Here, reproducing characteristics of observed sea ice extent remains an important climate modeling challenge. This study describes several approaches to improve how model biases in total sea ice distribution are quantified, and applies them to historically forced simulations contributed to phase 5 of the Coupled Model Intercomparison Project (CMIP5). The quantity of hemispheric total sea ice area, or some measure of its equatorward extent, is often used to evaluate model performance. A new approach is introduced that investigates additional details about the structure of model errors, with an aim to reduce the potential impact of compensating errors when gauging differences between simulated and observed sea ice. Using multiple observational datasets, several new methods are applied to evaluate the climatological spatial distribution and the annual cycle of sea ice cover in 41 CMIP5 models. It is shown that in some models, error compensation can be substantial, for example resulting from too much sea ice in one region and too little in another. Error compensation tends to be larger in models that agree more closely with the observed total sea ice area, which may result from model tuning. The results herein suggest that consideration of only the total hemispheric sea ice area or extent can be misleading when quantitatively comparing how well models agree with observations. Further work is needed to fully develop robust methods to holistically evaluate the ability of models to capture the finescale structure of sea ice characteristics; however, the “sector scale” metric used here aids in reducing the impact of compensating errors in hemispheric integrals.

54 ENVIRONMENTAL SCIENCES↗

Uncertainty quantification for nuclear density functional theory and information content of new measurements

Statistical tools of uncertainty quantification can be used to assess the information content of measured observables with respect to present-day theoretical models, to estimate model errors and thereby improve predictive capability, to extrapolate beyond the regions reached by experiment, and to provide meaningful input to applications and planned measurements. To showcase new opportunities offered by such tools, we make a rigorous analysis of theoretical statistical uncertainties in nuclear density functional theory using Bayesian inference methods. By considering the recent mass measurements from the Canadian Penning Trap at Argonne National Laboratory, we demonstrate how the Bayesian analysis and a direct least-squares optimization, combined with high-performance computing, can be used to assess the information content of the new data with respect to a model based on the Skyrme energy density functional approach. Employing the posterior probability distribution computed with a Gaussian process emulator, we apply the Bayesian framework to propagate theoretical statistical uncertainties in predictions of nuclear masses, two-neutron dripline, and fission barriers. Overall, we find that the new mass measurements do not impose a constraint that is strong enough to lead to significant changes in the model parameters. In addition, the example discussed in this study sets the stage for quantifying and maximizing the impact of new measurements with respect to current modeling and guiding future experimental efforts, thus enhancing the experiment-theory cycle in the scientific method.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Quantification of modeling uncertainty in the Rayleigh damping model

Understanding and accurately characterizing energy dissipation mechanisms in civil structures during earthquakes is an important element of seismic assessment and design. The most commonly used model is attributed to Rayleigh. This paper proposes a systematic approach to quantify the uncertainty associated with Rayleigh's damping model. Bayesian calibration with embedded model error is employed to treat the coefficients of the Rayleigh model as random variables using modal damping ratios. Through a numerical example, we illustrate how this approach works and how the calibrated model can address modeling uncertainty associated with the Rayleigh damping model.

42 ENGINEERING↗

Bayesian Calibration of the Community Land Model using Surrogates

We present results from the Bayesian calibration of hydrological parameters of the Community Land Model (CLM), which is often used in climate simulations and Earth system models. A statistical inverse problem is formulated for three hydrological parameters, conditioned on observations of latent heat surface fluxes over 48 months. Our calibration method uses polynomial and Gaussian process surrogates of the CLM, and solves the parameter estimation problem using a Markov chain Monte Carlo sampler. Posterior probability densities for the parameters are developed for two sites with different soil and vegetation covers. Our method also allows us to examine the structural error in CLM under two error models. We find that accurate surrogate models can be created for CLM in most cases. The posterior distributions lead to better prediction than the default parameter values in CLM. Climatologically averaging the observations does not modify the parameters’ distributions significantly. The structural error model reveals a correlation time-scale which can potentially be used to identify physical processes that could be contributing to it. While the calibrated CLM has a higher predictive skill, the calibration is under-dispersive.

Bayesian calibration↗

AMiBA: cluster Sunyaev-Zel'dovich effect observations with the expanded 13-element array

The Yuan-Tseh Lee Array for Microwave Background Anisotropy (AMiBA) is a co-planar interferometer array operating at a wavelength of 3 mm to measure the Sunyaev–Zel’dovich effect (SZE) of galaxy clusters at arcminute scales. The first phase of operation—with a compact 7-element array with 0.6 m antennas (AMiBA-7)—observed six clusters at angular scales from 5{sup ′} to 23{sup ′}. Here, we describe the expansion of AMiBA to a 13-element array with 1.2 m antennas (AMiBA-13), its subsequent commissioning, and cluster SZE observing program. The most noticeable changes compared to AMiBA-7 are (1) array re-configuration with baselines ranging from 1.4 m to 4.8 m, allowing us to sample structures between 2{sup ′} and 10{sup ′}, (2) 13 new lightweight carbon-fiber-reinforced plastic (CFRP) 1.2 m reflectors, and (3) additional correlators and six new receivers. Since the reflectors are co-mounted on and distributed over the entire six-meter CFRP platform, a refined hexapod pointing error model and phase error correction scheme have been developed for AMiBA-13. These effects—entirely negligible for the earlier central close-packed AMiBA-7 configuration—can lead to additional geometrical delays during observations. Our correction scheme recovers at least 80 ± 5% of the point-source fluxes. We, therefore, apply an upward correcting factor of 1.25 to our visibilities to correct for phase decoherence, and a ±5% systematic uncertainty is added in quadrature with our statistical errors. We demonstrate the absence of further systematics with a noise level consistent with zero in stacked uv-visibilities. From the AMiBA-13 SZE observing program, we present here maps of a subset of 12 clusters with signal-to-noise ratios above five. We demonstrate combining AMiBA-7 with AMiBA-13 observations on Abell 1689, by jointly fitting their data to a generalized Navarro–Frenk–White model. Our cylindrically integrated Compton-y values for five radii are consistent with results from the Berkeley-Illinois-Maryland Array, the Owens Valley Radio Observatory, the Sunyaev–Zel’dovich Array, and the Planck Observatory. We also report the first targeted SZE detection toward the optically selected cluster RCS J1447+0828, and we demonstrate the ability of AMiBA SZE data to serve as a proxy for the total cluster mass. Finally, we show that our AMiBA-SZE derived cluster masses are consistent with recent lensing mass measurements in the literature.

79 ASTRONOMY AND ASTROPHYSICS↗

A novel network-based approach to determining measurement representation error for model evaluation of aerosol microphysical properties

Atmospheric aerosol size and abundance influence radiative effects and climate change. To date, efforts to constrain global climate models’ radiative forcing with in situ aerosol observations have been hamstrung by uncertainty. One source of error, the regional “representation error,” arises when accurate but sparse single-point measurements of atmospheric aerosol distributions are compared with a model value, assuming that the single-point measurement is representative of the model domain. The Portable Optical Particle Spectrometer network in the Southern Great Plains (POPSnet-SGP) campaign has demonstrated that a network of nearly autonomous aerosol instruments operating at ambient temperature and relative humidity (with low measurement error) may be used to quantify measurement representation error and investigate the factors introducing heterogeneity in aerosol distributions across a rural, continental background region. Measurements were made using Portable Optical Particle Spectrometer (POPS) instruments at several sites for five months across the Department of Energy’s Aerosol Radiation Measurement Southern Great Plains (ARM-SGP) User Facility in the central USA. Measurement representation error decreased with longer averaging periods (20-40 % between 1 sec and 1 day), varied between sites by 10 – 20 % for aerosol concentration 140 – 2500 nm in diameter (N_140), and was higher for aerosols > 400 nm in diameter (N_400). Our measurements also show the influence of local meteorology on aerosol surface area (A_140) and size distributions: A_140 is positively correlated with wind speed and relative humidity, negatively correlated with precipitation, and lower given westerly winds. Based on this study, we conclude that the POPSnet approach provides considerably more insight into the spatial variability in the aerosol population that can be used to constrain climate models than would be available from similar networks of PM 2.5 monitors.

54 ENVIRONMENTAL SCIENCES↗

A novel implicit hybrid machine learning model and its application for reinforcement learning

A novel methodology to develop implicit hybrid models is presented. PyTorch is used to integrate physics-based equations with machine learning models. Automatic differentiation of the hybrid model is leveraged to solve the implicit equations. Iterative solving enables gradient based updates to the machine learning model. The novel methodology is compared to an explicit hybrid approach on a continuously stirred tank reactor (CSTR). The novel method results in a lower modelling error. Both hybrid models effectively train with noisy data. To test the implicit hybrid model, it is employed as a reinforcement learning (RL) training model. The RL algorithm trained on the hybrid model outperforms real time optimization of the CSTR and performs nearly as well as RL trained directly on the CSTR and a traditional gradient based approach. Training RL directly on the CSTR requires over 60,000 system interactions compared to 6000 historical data points for hybrid model development.

42 ENGINEERING↗

Regional-scale, sector-specific evaluation of global CO2 inversion models using aircraft data from the ACT-America project

We use vertical profiles of airborne measurements of CO2 from frontal cases during the summer 2016 Atmospheric Carbon and Transport – America (ACT-America) campaign to evaluate the skill of a set of ten global CO2 inversion models participating in the Orbiting Carbon Observatory – 2 (OCO-2) Model Intercomparison Project (MIP). Model errors and biases (model minus observation) were categorized by region (Mid-Atlantic, Midwest, and South), frontal sector (warm or cold), and transport model (predominantly Tracer Model 5 (TM5) and Goddard Earth Observing System – Chemistry (GEOS-Chem)). Overall, the inversions reproduce the general structures of the observed vertical profiles and the enhanced / depleted low-level CO2 in warm / cold sectors, but tend to underestimate the magnitude of the sector difference in each region. In the Midwest and South warm sectors, inversion biases were about 1 ppm above 1500 m AGL, though model spread (quantified by interquartile range) was often even smaller; below 1500 m AGL, model biases were negative and about -2 ppm near the surface. For the Midwest and South cold sectors, models had +2-3 ppm biases below 1500 m AGL, but with comparable model spread. Uniquely, in the Mid-Atlantic there was a consistent difference between TM5 and GEOS-Chem mole fractions for both sectors up to 3000 m AGL (TM5 lower by 2 ppm), on the order of the observation-relative biases. In the MidAtlantic TM5 inversions had negative biases in warm sectors, while GEOS-Chem inversions had positive biases in cold sectors. Possible reasons for the regional variability are discussed.

Gaudet, Brian J.↗

A Hybrid Biophysical‐Machine Learning Framework for Diurnal Surface Energy Flux Estimation Using Proximal Sensing

Thermal infrared-based remote sensing of surface energy fluxes has traditionally relied on high spatial resolution satellite data with revisit frequencies on the order of weeks. In this study, we evaluate a biophysics-based analytical surface energy balance model for predicting latent energy ( LE ) and sensible heat ( H ) fluxes using proximal sensing observations. The Surface Temperature Initiated Closure (STIC1.2) model has been extensively validated across a wide range of spatial and temporal scales using various satellite-derived thermal infrared data sets. Here we extend this validation by applying STIC at sub-hourly temporal resolution over multiple growing seasons for four distinct agricultural systems. We further develop and evaluate novel STIC variants that incorporate machine learning (ML) techniques to eliminate the need for surface energy balance observations, specifically net radiation and soil heat flux, thereby enhancing model applicability in data-sparse settings. The integration of a ML component to estimate surface available energy is shown to have strong predictive performance for both LE (R 2 = 0.81–0.94) and H (R 2 = 0.46–0.72) across all agricultural systems examined here, demonstrating the potential of hybrid biophysical-machine learning approaches for surface energy balance modeling with minimal data requirements. This study concludes with a novel application of explainable machine learning (exML) to diagnose sources of model error. This exML framework attributes residual prediction errors to both model input variables and environmental drivers not explicitly included in the simulation experiments. This approach provides a new pathway for improving model design and integrating previously overlooked yet influential variables into future model iterations.

evapotranspiration↗

Uncertainty quantification in application of the enrichment meter principle for nondestructive assay of special nuclear material

The various methods of nondestructive assay (NDA) of special nuclear material (SNM) have applications in nuclear nonproliferation, including detection and identification of illicit SNM at border crossings, and quantifying SNM at nuclear facilities for safeguards. No assay method is complete without “error bars,” which provide one way of expressing confidence in the assay result. Consequently, NDA specialists typically quantify total uncertainty in terms of “random” and “systematic” components, and then specify error bars for the total mass estimate in multiple items. Uncertainty quantification (UQ) for NDA has always been important, but it is recognized that greater rigor is needed and achievable using modern statistical methods. To this end, we describe the extent to which the guideline for expressing uncertainty in measurements (GUM) can be used for NDA. Also, we propose improvements over GUM for NDA by illustrating UQ challenges that it does not address, including calibration with errors in predictors, model error, and item-specific biases. A case study is presented using low-resolution NaI spectra and applying the enrichment meter principle to estimate the U-235 mass in an item. The case study illustrates how to update the current American Society for Testing and Materials guide for application of the enrichment meter principle using gamma spectra from a NaI detector.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Data for: A hybrid biophysical-machine learning framework for diurnal surface energy flux estimation using proximal sensing

Thermal-based remote sensing of surface energy fluxes has traditionally relied on high spatial resolution satellite data with revisit frequencies on the order of weeks. In this study, we evaluate a biophysics-based analytical surface energy balance model for predicting latent energy (LE) and sensible heat (H) fluxes using proximal sensing observations. The Surface Temperature Initiated Closure (STIC1.2) model has been extensively validated across a wide range of spatial and temporal scales using various satellite-derived thermal datasets. Here we extend this validation by applying STIC at sub-hourly temporal resolution over multiple growing seasons for four distinct agricultural systems. We further develop and evaluate novel STIC variants that incorporate machine learning (ML) techniques to eliminate the need for specific surface energy balance observations, specifically net radiation and soil heat flux, thereby enhancing model applicability in data-sparse settings. The integration of an ML component to estimate surface available energy is shown to have strong predictive performance for both LE (R2 = 0.81-0.94) and H (R2 = 0.46-0.72) across all agricultural systems examined here, demonstrating the potential of hybrid biophysical – machine learning approaches for surface energy balance modeling with minimal data requirements. This study concludes with a novel application of explainable machine learning (exML) to diagnose sources of model error. This exML framework attributes residual prediction errors to both model input variables and environmental drivers not explicitly included in the simulation experiments. This approach provides a new pathway for improving model design and integrating previously overlooked yet influential variables into future model iterations.

Agricultural Sciences↗

Toward Joint Hypothesis-Tests Seismic Event Screening Analysis: Ms|mb and Event Depth

Well established theory can be used to combine single-phenomenology hypothesis tests into a multi-phenomenology event screening hypothesis test (Fisher's and Tippett's tests). Commonly used standard error in Ms:mb event screening hypothesis test is not fully consistent with physical basis. Improved standard error - Better agreement with physical basis, and correctly partitions error to include Model Error as a component of variance, correctly reduces station noise variance through network averaging. For 2009 DPRK test - Commonly used standard error 'rejects' H0 even with better scaling slope ({beta} = 1, Selby et al.), improved standard error 'fails to rejects' H0.

58 GEOSCIENCES↗

Evaluating the WFIP2 updates to the HRRR model using scanning Doppler lidar measurements in the complex terrain of the Columbia River Basin

The wind-energy (WE) industry relies on numerical weather prediction (NWP) forecast models as foundational or base models for many purposes, including wind-resource assessment and wind-power forecasting. During the Second Wind Forecast Improvement Project (WFIP2) in the Columbia River Basin of Oregon and Washington, a significant effort was made to improve NWP forecasts through focused model development, to include experimental refinements to the High Resolution Rapid Refresh (HRRR) model physics and horizontal grid spacing. In this study, the performance of an experimental version of HRRR that includes these refinements is tested against a control version, which corresponds to that of the operational HRRR run by National Oceanic and Atmospheric Administration/National Centers for Environmental Protection at the outset of WFIP2. Furthermore, the effects of horizontal grid resolution were also tested by comparing wind forecasts from the HRRR (with 3-km grid spacing) with those from a finer-resolution HRRR nest with 750-m grid spacing. Model forecasts are validated against accurate wind-profile measurements by three scanning, pulsed Doppler lidars at sites separated by a total distance of 71 km. Model skill and improvements in model skill, attributable to physics refinements and improved horizontal grid resolution, varied by season, by site, and during periods of atmospheric phenomena relevant to WE. In general, model errors were the largest below 150 m above ground level (AGL). Experimental HRRR refinements tended to reduce the mean absolute error (MAE) and other error metrics for many conditions, but degradation in skill (increased MAE) was noted below 150 m AGL at the two lowest-elevation sites at night. Finer resolution was found to produce the most significant reductions in the error metrics.

17 WIND ENERGY↗

Free kick instead of cross-validation in maximum-likelihood refinement of macromolecular crystal structures

The maximum-likelihood free-kick target, which calculates model error estimates from the work set and a randomly displaced model, proved superior in the accuracy and consistency of refinement of crystal structures compared with the maximum-likelihood cross-validation target, which calculates error estimates from the test set and the unperturbed model. The refinement of a molecular model is a computational procedure by which the atomic model is fitted to the diffraction data. The commonly used target in the refinement of macromolecular structures is the maximum-likelihood (ML) function, which relies on the assessment of model errors. The current ML functions rely on cross-validation. They utilize phase-error estimates that are calculated from a small fraction of diffraction data, called the test set, that are not used to fit the model. An approach has been developed that uses the work set to calculate the phase-error estimates in the ML refinement from simulating the model errors via the random displacement of atomic coordinates. It is called ML free-kick refinement as it uses the ML formulation of the target function and is based on the idea of freeing the model from the model bias imposed by the chemical energy restraints used in refinement. This approach for the calculation of error estimates is superior to the cross-validation approach: it reduces the phase error and increases the accuracy of molecular models, is more robust, provides clearer maps and may use a smaller portion of data for the test set for the calculation of R{sub free} or may leave it out completely.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

A New Coupled Biogeochemical Modeling Approach Provides Accurate Predictions of Methane and Carbon Dioxide Fluxes Across Diverse Tidal Wetlands

Abstract Tidal wetlands provide valuable ecosystem services, including storing large amounts of carbon. However, the net exchanges of carbon dioxide (CO 2 ) and methane (CH 4 ) in tidal wetlands are highly uncertain. While several biogeochemical models can operate in tidal wetlands, they have yet to be parameterized and validated against high‐frequency, ecosystem‐scale CO 2 and CH 4 flux measurements across diverse sites. We paired the Cohort Marsh Equilibrium Model (CMEM) with a version of the PEPRMT model called PEPRMT‐Tidal, which considers the effects of water table height, sulfate, and nitrate availability on CO 2 and CH 4 emissions. Using a model‐data fusion approach, we parameterized the model with three sites and validated it with two independent sites, with representation from the three marine coasts of North America. Gross primary productivity (GPP) and ecosystem respiration (R eco ) modules explained, on average, 73% of the variation in CO 2 exchange with low model error (normalized root mean square error (nRMSE) <1). The CH 4 module also explained the majority of variance in CH 4 emissions in validation sites ( R 2 = 0.54; nRMSE = 1.15). The PEPRMT‐Tidal‐CMEM model coupling is a key advance toward constraining estimates of greenhouse gas emissions across diverse North American tidal wetlands. Further analyses of model error and case studies during changing salinity conditions guide future modeling efforts regarding four main processes: (a) the influence of salinity and nitrate on GPP, (b) the influence of laterally transported dissolved inorganic C on R eco , (c) heterogeneous sulfate availability and methylotrophic methanogenesis impacts on surface CH 4 emissions, and (d) CH 4 responses to non‐periodic changes in salinity.

54 ENVIRONMENTAL SCIENCES↗

ARETE: Accurate Error Assessment via Machine Learning-Guided Dynamic-Timing Analysis

Nanometer circuits are increasingly prone to timing errors, escalating the need for fault injection frameworks to accurately evaluate their impact on applications. Here in this paper, we propose ARETE, a novel cross-layer, fault-injection framework that combines dynamic-binary instrumentation with machine learning-guided dynamic-timing analysis. ARETE enables accurate fault-injection into any application by estimating the location of the injecting errors via dynamic-timing analysis. To accelerate fault-injection, we develop a novel, data-aware, machine learning-based mechanism that dynamically pre-selects the error-prone instructions and limits the application of the costly dynamic-timing analysis only to them. To evaluate ARETE's accuracy, our fully automated toolflow is configured to support fault-injection based on detailed post-layout gate-level simulations as well as via existing workload-agnostic error models. Our results for various workloads, including an autonomous-driving library, show that the location and time of injected errors performed by ARETE, is 89.9% consistent with fault-injection based on full gate-level simulation. On average, ARETE executes 84.6x faster than gate-level simulation and at a cost of 3.4% loss in the program output quality estimation. When compared to the existing statistical fault-injection tools that are based on workload-agnostic error models, ARETE improves the accuracy of fault-injection rate and output quality estimation by 143.9% and 40.4% on average, respectively.

97 MATHEMATICS AND COMPUTING↗