Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

Aerosols at the poles: an AeroCom Phase II multi-model evaluation

Abstract. Atmospheric aerosols from anthropogenic and natural sources reach the polar regions through long-range transport and affect the local radiation balance. Such transport is, however, poorly constrained in present-day global climate models, and few multi-model evaluations of polar anthropogenic aerosol radiative forcing exist. Here we compare the aerosol optical depth (AOD) at 550 nm from simulations with 16 global aerosol models from the AeroCom Phase II model intercomparison project with available observations at both poles. We show that the annual mean multi-model median is representative of the observations in Arctic, but that the intermodel spread is large. We also document the geographical distribution and seasonal cycle of the AOD for the individual aerosol species: black carbon (BC) from fossil fuel and biomass burning, sulfate, organic aerosols (OAs), dust, and sea-salt. For a subset of models that represent nitrate and secondary organic aerosols (SOAs), we document the role of these aerosols at high latitudes.The seasonal dependence of natural and anthropogenic aerosols differs with natural aerosols peaking in winter (sea-salt) and spring (dust), whereas AOD from anthropogenic aerosols peaks in late spring and summer. The models produce a median annual mean AOD of 0.07 in the Arctic (defined here as north of 60° N). The models also predict a noteworthy aerosol transport to the Antarctic (south of 70° S) with a resulting AOD varying between 0.01 and 0.02. The models have estimated the shortwave anthropogenic radiative forcing contributions to the direct aerosol effect (DAE) associated with BC and OA from fossil fuel and biofuel (FF), sulfate, SOAs, nitrate, and biomass burning from BC and OA emissions combined. The Arctic modelled annual mean DAE is slightly negative (−0.12 W m−2), dominated by a positive BC FF DAE in spring and a negative sulfate DAE in summer. The Antarctic DAE is governed by BC FF. We perform sensitivity experiments with one of the AeroCom models (GISS modelE) to investigate how regional emissions of BC and sulfate and the lifetime of BC influence the Arctic and Antarctic AOD. A doubling of emissions in eastern Asia results in a 33 % increase in Arctic AOD of BC. A doubling of the BC lifetime results in a 39 % increase in Arctic AOD of BC. However, these radical changes still fall within the AeroCom model range.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Taking climate model evaluation to the next level

We report that earth system models are complex and represent a large number of processes, resulting in a persistent spread across climate projections for a given future scenario. Owing to different model performances against observations and the lack of independence among models, there is now evidence that giving equal weight to each available model projection is suboptimal. This Perspective discusses newly developed tools that facilitate a more rapid and comprehensive evaluation of model simulations with observations, process-based emergent constraints that are a promising way to focus evaluation on the observations most relevant to climate projections, and advanced methods for model weighting. Lastly, these approaches are needed to distill the most credible information on regional climate changes, impacts, and risks for stakeholders and policy-makers.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

System integration of wind and solar power in integrated assessment models: A cross-model evaluation of new approaches

Mitigation-Process Integrated Assessment Models (MP-IAMs) are used to analyze long-term transformation pathways of the energy system required to achieve stringent climate change mitigation targets. Due to their substantial temporal and spatial aggregation, IAMs cannot explicitly represent all detailed challenges of integrating the variable renewable energies (VRE) wind and solar in power systems, but rather rely on parameterized modeling approaches. In the ADVANCE project, six international modeling teams have developed new approaches to improve the representation of power sector dynamics and VRE integration in IAMs. In this study, we qualitatively and quantitatively evaluate the last years' modeling progress and study the impact of VRE integration modeling on VRE deployment in IAM scenarios. For a comprehensive and transparent qualitative evaluation, we first develop a framework of 18 features of power sector dynamics and VRE integration. We then apply this framework to the newly-developed modeling approaches to derive a detailed map of strengths and limitations of the different approaches. For the quantitative evaluation, we compare the IAMs to the detailed hourly-resolution power sector model REMIX. We find that the new modeling approaches manage to represent a large number of features of the power sector, and the numerical results are in reasonable agreement with those derived from the detailed power sector model. Updating the power sector representation and the cost and resources of wind and solar substantially increased wind and solar shares across models: Under a carbon price of 30$/tCO2 in 2020 (increasing by 5% per year), the model-average cost-minimizing VRE share over the period 2050-2100 is 62% of electricity generation, 24%-points higher than with the old model version.

(IAM) Variable Renewable Energy (VRE)↗
DOE OSTI2016

Ground heat flux: An analytical review of 6 models evaluated at 88 sites and globally

Uncertainty in ground heat flux (G) means that evaluation of the other terms in the surface energy balance (e.g., latent and sensible heat fluxes (LE and H)) remains problematic. Algorithms that calculate LE and H require available energy, the difference between net radiation, R NET , and G. There are a wide range of approaches to model G for large-scale applications, with a subsequent wide range of estimates and accuracies. Here, we provide the largest review of these methods to date (N = 6), evaluating modeled G against measured G from 88 FLUXNET sites. The instantaneous midday variability in G is best captured by models forced with net radiation, while models forced by temperature show the least error at both instantaneous and daily time scales. We produce global decadal data sets of G to illustrate regional and seasonal sensitivities, as well as uncertainty. Global model mean midmorning instantaneous G is highest during September, October, and November at 63.42 (±16.84) Wm -2 , while over December, January, and February G is lowest at 53.86 (±18.09) Wm -2 but shows greater intermodel uncertainty. Results from this work have the potential to improve evapotranspiration estimates and guide appropriate G model selection and development for various land uses.

58 GEOSCIENCES↗
DOE OSTI2016

An inverse problem strategy based on forward model evaluations: Gradient-based optimization without adjoint solves

This study presents a new nonlinear programming formulation for the solution of inverse problems. First, a general inverse problem formulation based on the compliance error functional is presented. The proposed error functional enables the computation of the Lagrange multipliers, and thus the first order derivative information, at the expense of just one model evaluation. Therefore, the calculation of the Lagrange multipliers does not require the solution of the computationally intensive adjoint problem. This leads to significant speedups for large-scale, gradient-based inverse problems.

42 ENGINEERING↗
NASA NTRS1994

Evaluating modeling tools for the EDOS

The Earth Observing System (EOS) Data and Operations System (EDOS) Project is developing a functional, system performance model to support the system implementation phase of the EDOS which is being designed and built by the Goddard Space Flight Center (GSFC). The EDOS Project will use modeling to meet two key objectives: (1) manage system design impacts introduced by unplanned changed in mission requirements; and (2) evaluate evolutionary technology insertions throughout the development of the EDOS. To select a suitable modeling tool, the EDOS modeling team developed an approach for evaluating modeling tools and languages by deriving evaluation criteria from both the EDOS modeling requirements and the development plan. Essential and optional features for an appropriate modeling tool were identified and compared with known capabilities of several modeling tools. Vendors were also provided the opportunity to model a representative EDOS processing function to demonstrate the applicability of their modeling tool to the EDOS modeling requirements. This paper emphasizes the importance of using a well defined approach for evaluating tools to model complex systems like the EDOS. The results of this evaluation study do not in any way signify the superiority of any one modeling tool since the results will vary with the specific modeling requirements of each project.

Knoble, Gordon↗
DOE OSTI2023

BISON-FIPD integration enhanced low-burnup SFR metallic fuel swelling model evaluation framework

Experiments indicated that metallic fuel in sodium-cooled fast reactors (SFRs) rapidly swells radially and axially at low burnup. Despite that, prior studies have been focused on describing high burnup axial fuel elongation. With recent conventional and non-conventional metallic fuel concepts being considered for license applications, understanding multidimensional fuel swelling at a wide range of burnup levels is important to fuel analysis and qualification activities. Here, we report the development and demonstration efforts of a low-burnup SFR metallic fuel swelling model evaluation framework using the BISON advanced fuel performance code. The framework leverages the Integral Fast Reactor (IFR) program X423 experiment data through the ongoing integration project to enable standardized and automated use of legacy metallic fuel irradiation data maintained in the SFR fuel irradiation and physics database (FIPD) for BISON metallic fuel model verification and validation. In conclusion, the performance of the framework was demonstrated using the two representative metallic fuel swelling model sets implemented in BISON, with a series of insights about future advanced swelling model development.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2022

The DOE E3SM Model Version 2: Overview of the Physical Model and Initial Model Evaluation

Abstract This work documents version two of the Department of Energy's Energy Exascale Earth System Model (E3SM). E3SMv2 is a significant evolution from its predecessor E3SMv1, resulting in a model that is nearly twice as fast and with a simulated climate that is improved in many metrics. We describe the physical climate model in its lower horizontal resolution configuration consisting of 110 km atmosphere, 165 km land, 0.5° river routing model, and an ocean and sea ice with mesh spacing varying between 60 km in the mid‐latitudes and 30 km at the equator and poles. The model performance is evaluated with Coupled Model Intercomparison Project Phase 6 Diagnosis, Evaluation, and Characterization of Klima simulations augmented with historical simulations as well as simulations to evaluate impacts of different forcing agents. The simulated climate has many realistic features of the climate system, with notable improvements in clouds and precipitation compared to E3SMv1. E3SMv1 suffered from an excessively high equilibrium climate sensitivity (ECS) of 5.3 K. In E3SMv2, ECS is reduced to 4.0 K which is now within the plausible range based on a recent World Climate Research Program assessment. However, a number of important biases remain including a weak Atlantic Meridional Overturning Circulation, deficiencies in the characteristics and spectral distribution of tropical atmospheric variability, and a significant underestimation of the observed warming in the second half of the historical period. An analysis of single‐forcing simulations indicates that correcting the historical temperature bias would require a substantial reduction in the magnitude of the aerosol‐related forcing.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2016

Center for Integrated Nanotechnologies (CINT) Chemical Release Modeling Evaluation

This evaluation documents the methodology and results of chemical release modeling for operations at Building 518, Center for Integrated Nanotechnologies (CINT) Core Facility. This evaluation is intended to supplement an update to the CINT [Standalone] Hazards Analysis (SHA). This evaluation also updates the original [Design] Hazards Analysis (DHA) completed in 2003 during the design and construction of the facility; since the original DHA, additional toxic materials have been evaluated and modeled to confirm the continued low hazard classification of the CINT facility and operations. This evaluation addresses the potential catastrophic release of the current inventory of toxic chemicals at Building 518 based on a standard query in the Chemical Information System (CIS).

77 NANOSCIENCE AND NANOTECHNOLOGY↗
DOE OSTI2022

The E3SM Diagnostics Package (E3SM Diags v2.7): a Python-based diagnostics package for Earth system model evaluation

Abstract. The E3SM Diagnostics Package (E3SM Diags) is a modern, Python-based Earth system model (ESM) evaluation tool (with Python module name e3sm_diags), developed to support the Department of Energy (DOE) Energy Exascale Earth System Model (E3SM). E3SM Diags provides a wide suite of tools for evaluating native E3SM output, as well as ESM data on regular latitude–longitude grids, including output from Coupled Model Intercomparison Project (CMIP) class models. E3SM Diags is modeled after the National Center for Atmospheric Research (NCAR) Atmosphere Model Working Group (AMWG, 2022) diagnostics package. In its version 1 release, E3SM Diags included a set of core essential diagnostics to evaluate the mean physical climate from model simulations. As of version 2.7, more process-oriented and phenomenon-based evaluation diagnostics have been implemented, such as analysis of the quasi-biennial oscillation (QBO), the El Niño–Southern Oscillation (ENSO), streamflow, the diurnal cycle of precipitation, tropical cyclones, ozone and aerosol properties. An in situ dataset from DOE's Atmospheric Radiation Measurement (ARM) program has been integrated into the package for evaluating the representation of simulated cloud and precipitation processes. This tool is designed with enough flexibility to allow for the addition of new observational datasets and new diagnostic algorithms. Additional features include customizable figures; streamlined installation, configuration and execution; and multiprocessing for fast computation. The package uses an up-to-date observational data repository maintained by its developers, where recent datasets are added to the repository as they become available. Finally, several applications for the E3SM Diags module were introduced to fit a diverse set of use cases from the scientific community.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Model Evaluation by Measurements from Collocated Remote Sensors in Complex Terrain

Model improvement efforts involve an evaluation of changes in model skill in response to changes in model physics and parameterization. When using wind measurements from various remote sensors to determine model forecast accuracy, it is important to understand the effects of measurement-uncertainty differences among the sensors resulting from differences in the methods of measurement, the vertical and temporal resolution of the measurements, and the spatial variability of these differences. Here we quantify instrument measurement variability in 80-m wind speed during WFIP2 and its impact on the calculated errors and the change in error from one model version to another. Here the model versions tested involved updates in model physics from HRRRv1 to HRRRv4, and reductions in grid interval from 3 km to 750 m. Model errors were found to be 2–3 m s –1 . Differences in errors as determined by various instruments at each site amounted to about 10% of this value, or 0.2–0.3 m s –1 . Changes in model skill due to physics or grid-resolution updates also differed depending on the instrument used to determine the errors; most of the instrument-to-instrument differences were ~0.1 m s –1 , but some reached 0.3 m s –1 . All instruments at a given site mostly showed consistency in the sign of the change in error. In two examples, though, the sign changed, illustrating a consequence of differences in measurements: errors determined using one instrument may show improvement in model skill, whereas errors determined using another instrument may indicate degradation. This possibility underscores the importance of having accurate measurements to determine the model error.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2012

Material Model Evaluation of a Composite Honeycomb Energy Absorber

A study was conducted to evaluate four different material models in predicting the dynamic crushing response of solid-element-based models of a composite honeycomb energy absorber, designated the Deployable Energy Absorber (DEA). Dynamic crush tests of three DEA components were simulated using the nonlinear, explicit transient dynamic code, LS-DYNA . In addition, a full-scale crash test of an MD-500 helicopter, retrofitted with DEA blocks, was simulated. The four material models used to represent the DEA included: *MAT_CRUSHABLE_FOAM (Mat 63), *MAT_HONEYCOMB (Mat 26), *MAT_SIMPLIFIED_RUBBER/FOAM (Mat 181), and *MAT_TRANSVERSELY_ANISOTROPIC_CRUSHABLE_FOAM (Mat 142). Test-analysis calibration metrics included simple percentage error comparisons of initial peak acceleration, sustained crush stress, and peak compaction acceleration of the DEA components. In addition, the Roadside Safety Verification and Validation Program (RSVVP) was used to assess similarities and differences between the experimental and analytical curves for the full-scale crash test.

Jackson, Karen E.↗
DOE OSTI2021

Electric Water Heaters for Transactive Systems: Model Evaluations and Performance Quantification

Electric water heaters (EWHs) are opportune appliances for implementing demand-side control. EWH models serve as a fundamental step toward accurately estimating EWH flexibility potential and designing proper control strategies. Existing studies have adapted numerous modeling approaches in evaluating the potential of EWHs for a variety of grid applications. This paper presents an analytical study that evaluates the performance of state-of-art EWH models in terms of accuracy and computational complexity for adaptation in evaluation studies for the transactive system. The work proposes a transactive control strategy that optimally utilizes the thermal inertia of EWHs for providing grid services. Here, the performance of the control strategy and the impact of modeling accuracy is evaluated for device-level and feeder-level use cases using the IEEE 123 node test distribution system appropriately populated with EWHs. The simulation results illustrate the effectiveness of the control strategy in reducing the feeder demand during peak period by 13% and also quantities the impact of using simplified modeling approaches for determining the potential of EWHs for providing grid services.

42 ENGINEERING↗
NASA NTRSDate not supplied

An Ensemble Investigation of the Causes for Regional Air-Quality Model Critical Load Exceedances Prediction Variability in European and North American Domains Using Diagnostics From Phase 4 of the Air Quality Model Evaluation International Initiative

We summarize tentative findings from multi air quality model ensembles for the years 2009 and 2010 in Europe (EU), and 2010 and 2016 in North America (NA), under AQMEII-4. The model predictions of sulphur and nitrogen deposition were used to estimate exceedances of critical loads for acidification and eutrophication, to show the extent to which the ensemble members agree in the magnitude and the trend of ecologically meaningful impacts. Model exceedance variability was analyzed using AQMEII-4 diagnostics. Evaluation against concentration and wet deposition observations, coupled with these diagnostics, identified specific process representations as the causes for variability between model predictions and for reduced model performance. All models predicted reductions in ecosystem acidification impacts in North America between the years 2010 and 2016, in accord with SO2 emissions reduction legislation which started in 2010 (SO2 SIP) However, all models in EU and NA domains had net negative biases for wet deposition of sulphur and nitrogen relative to observations. The wet S deposition average mean bias for the NA ensemble was -0.17 eq ha-1 d-1, and for the EU ensemble -1.15 eq ha-1 d-1. The NA daily wet deposition average mean bias for NH4+ was -0.37 eq ha-1d-1; EU -1.19 eq ha-1 d-1. The daily NA wet NO3- deposition average mean bias was -0.24 eq ha-1d-1; EU -0.69 eq ha-1 d-1. The members of the ensemble diverged (factor of 10) in their North American predictions for Ndep and consequently their eutrophication exceedances. The models with the highest eutrophication predictions also predicted the highest levels of gas-phase ammonia dry deposition (standard deviation of ammonia dry deposition flux across ensemble members was larger than the ensemble average). These models also had negative biases of predicted ammonia concentrations; average mean biases of -0.63 (satellite NH3) and -0.85 ppbv (surface NH3) compared to ensemble averages of -0.30 and -0.34 ppbv. Diagnostics showed that these differences resulted from the manner in which bidirectional ammonia fluxes were parameterized within these models. The second largest source of NA eutrophication prediction variability were models with positive biases in particulate ammonium and nitrate concentrations, and higher particle nitrogen deposition levels ( particle ammonium concentration bias +0.35 ug m-3; ensemble bias +0.15 ug m-3). We believe two factors may have led to these latter overestimates: higher levels of fine mode particle nitrate formation compared to other models (due to the use of an inorganic heterogeneous chemistry algorithm which did not take base cation chemistry into account), and updates to particle dry deposition velocities carried out in the absence of concurrent updates to wet scavenging algorithms. The relative importance of dry gas, dry particulate, and wet deposition towards total sulphur and nitrogen deposition totals differed between EU and North American domains, though all models had negative biases in wet deposition as noted above. Parallel and subsequent work suggests that multiphase hydrometeor scavenging may improve model wet deposition performance. An increased research focus is recommended for four model processes: multiphase hydrometeor scavenging, ammonia bidirectional fluxes, base cation chemistry and emissions, and particle dry deposition.

regional air-quality model↗
DOE OSTI2020

Advancing Diagnostic Model Evaluation to Better Understand Water Shortage Mechanisms in Institutionally Complex River Basins

Abstract Water resources systems models enable valuable inferences on consequential system stressors by representing both the geophysical processes determining the movement of water and the human elements distributing it to its various competing uses. This study contributes a diagnostic evaluation framework that pairs exploratory modeling with global sensitivity analysis to enhance our ability to make inferences on water scarcity vulnerabilities in institutionally complex river basins. Diagnostic evaluation of models representing institutionally complex river basins with many stakeholders poses significant challenges. First, it needs to exploit a large and diverse suite of simulations to capture important human‐natural system interactions as well as institutionally aware behavioral mechanisms. Second, it needs to have performance metrics that are consequential and draw on decision‐relevant model outputs that adequately capture the multisector concerns that emerge from diverse basin stakeholders. We demonstrate the proposed model diagnostic framework by evaluating how potential interactions between changing hydrologic conditions and human demands influence the frequencies and durations of water shortages of varying magnitudes experienced by hundreds of users in a subbasin of the Colorado River. We show that the dominant factors shaping these effects vary both across users and, for an individual user, across percentiles of shortage magnitude. These differences hold even for users sharing diversion locations, demand levels or water right seniority. Our findings underline the importance of detailed institutional representation for such basins, as institutions strongly shape how dominant factors of stakeholder vulnerabilities propagate through the complex network of users.

Hadjimichael, Antonia↗
DOE OSTI2026

ARM Data-Oriented Metrics and Diagnostics Package for Climate Model Evaluation

A Python-based metrics and diagnostics package is currently being developed by the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) Infrastructure Team at Lawrence Livermore National Laboratory (LLNL) to facilitate the use of long-term, high-frequency measurements from the ARM Facility in evaluating the regional climate simulation of clouds, radiation, and precipitation. This metrics and diagnostics package computes climatological means of targeted climate model simulation and generates tables and plots for comparing the model simulation with ARM observational data. The Coupled Model Intercomparison Project (CMIP) model data sets are also included in the package to enable model intercomparison as demonstrated in Zhang et al. (2017). The mean of the CMIP model can serve as a reference for individual models. Basic performance metrics are computed to measure the accuracy of mean state and variability of climate models. The evaluated physical quantities include cloud fraction, temperature, relative humidity, cloud liquid water path, total column water vapor, precipitation, sensible and latent heat fluxes, and radiative fluxes, with plan to extend to more fields, such as aerosol and microphysics properties. Process-oriented diagnostics focusing on individual cloud- and precipitation-related phenomena are also being developed for the evaluation and development of specific model physical parameterizations. The version 1.0 package is designed based on data collected at ARM’s Southern Great Plains (SGP) Research Facility, with the plan to extend to other ARM sites. The metrics and diagnostics package is currently built upon standard Python libraries and additional Python packages developed by DOE (such as CDMS and CDAT). The ARM metrics and diagnostic package is available publicly with the hope that it can serve as an easy entry point for climate modelers to compare their models with ARM data. In this report, we first present the input data, which constitutes the core content of the metrics and diagnostics package in section 2, and a user's guide documenting the workflow/structure of the version 1.0 codes, and including step-by-step instruction for running the package in section 3.

54 ENVIRONMENTAL SCIENCES↗