Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Model Evaluation Protocol for Fire Models Involving Fuels at Liquefied Natural Gas Facilities (Version 2)

This document provides a description of the model evaluation protocol (MEP) for pool fires, jet fires, and fireballs involving liquefied natural gas (LNG) and processing fuels at LNG facilities. The purpose of the MEP is to provide procedures regarding the assessment of a model’s suitability to predict heat flux from fires. Three components, namely, a scientific assessment, model verification, and model validation comprise the MEP. The evaluation of a model satisfying these three components is to be documented in the form of a model evaluation report (MER). Discussion of models for the prediction of fire, detailed information on each of the three MEP components, the MEP procedure regarding new versions of previously approved models, and the format of the model evaluation report (MER) are provided.

03 NATURAL GAS↗
DOE OSTI2022

Model Evaluation Protocol for Fire Models Involving Fuels at Liquefied Natural Gas Facilities.

This document provides a description of the model evaluation protocol (MEP) for pool fires, jet fires, and fireballs involving liquefied natural gas (LNG), refrigerant fluids, and byproducts at LNG facilities. The purpose of the MEP is to provide procedures regarding the assessment of a model's suitability to predict heat flux from fires. Three components, namely, a scientific assessment, model verification, and model validation comprise the MEP. The evaluation of a model satisfying these three components is to be documented in the form of a model evaluation report (MER). Discussion of models for the prediction of fire, detailed information on each of the three MEP components, the MEP procedure regarding new versions of previously approved models, and the format of the model evaluation report (MER) are provided.

03 NATURAL GAS↗
DOE OSTI2015

Application of an Online-Coupled Regional Climate Model, WRF-CAM5, over East Asia for Examination of Ice Nucleation Schemes: Part I. Comprehensive Model Evaluation and Trend Analysis for 2006 and 2011

Online-coupled climate and chemistry models are necessary to realistically represent the interactions between climate variables and chemical species and accurately simulate aerosol direct and indirect effects on cloud, precipitation, and radiation. In this Part I of a two-part paper, simulations from the Weather Research and Forecasting model coupled with the physics package of Community Atmosphere Model (WRF-CAM5) are conducted with the default heterogeneous ice nucleation parameterization over East Asia for two full years: 2006 and 2011. A comprehensive model evaluation is performed using satellite and surface observations. The model shows an overall acceptable performance for major meteorological variables at the surface and in the boundary layer, as well as column variables (e.g., precipitation, cloud fraction, precipitating water vapor, downward longwave and shortwave radiation). Moderate to large biases exist for cloud condensation nuclei over oceanic areas, cloud variables (e.g., cloud droplet number concentration, cloud liquid and ice water paths, cloud optical depth, longwave and shortwave cloud forcing). These biases indicate a need to improve the model treatments for cloud processes, especially cloud droplets and ice nucleation, as well as to reduce uncertainty in the satellite retrievals. The model simulates well the column abundances of chemical species except for column SO 2 but relatively poor for surface concentrations of several species such as CO, NO 2 , SO 2 , PM2.5, and PM10. Several reasons could contribute to the underestimation of major chemical species in East Asia including underestimations of anthropogenic emissions and natural dust emissions, uncertainties in the spatial and vertical distributions of the anthropogenic emissions, as well as biases in meteorological, radiative, and cloud predictions. Despite moderate to large biases in the chemical predictions, the model performance is generally consistent with or even better than that reported for East Asia with only a few exceptions. The model generally reproduces the observed seasonal variations and the difference between 2006 and 2011 for most variables or chemical species. Overall, these results demonstrate promising skills of WRF-CAM5 for long-term simulations at a regional scale and suggest several areas of potential improvements.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

A Surface Radiation Balance Dataset from Siple Dome in West Antarctica for Atmospheric and Climate Model Evaluation

Abstract A field campaign at Siple Dome in West Antarctica during the austral summer 2019/20 offers an opportunity to evaluate climate model performance, particularly cloud microphysical simulation. Over Antarctic ice sheets and ice shelves, clouds are a major regulator of the surface energy balance, and in the warm season their presence occasionally induces surface melt that can gradually weaken an ice shelf structure. This dataset from Siple Dome, obtained using transportable and solar-powered equipment, includes surface energy balance measurements, meteorology, and cloud remote sensing. To demonstrate how these data can be used to evaluate model performance, comparisons are made with meteorological reanalysis known to give generally good performance over Antarctica (ERA5). Surface albedo measurements show expected variability with observed cloud amount, and can be used to evaluate a model’s snowpack parameterization. One case study discussed involves a squall with northerly winds, during which ERA5 fails to produce cloud cover throughout one of the days. A second case study illustrates how shortwave spectroradiometer measurements that encompass the 1.6- μ m atmospheric window reveal cloud phase transitions associated with cloud life cycle. Here, continuously precipitating mixed-phase clouds become mainly liquid water clouds from local morning through the afternoon, not reproduced by ERA5. We challenge researchers to run their various regional or global models in a manner that has the large-scale meteorology follow the conditions of this field campaign, compare cloud and radiation simulations with this Siple Dome dataset, and potentially investigate why cloud microphysical simulations or other model components might produce discrepancies with these observations. significance statement Antarctica is a critical region for understanding climate change and sea level rise, as the great ice sheets and the ice shelves are subject to increasing risk as global climate warms. Climate models have difficulties over Antarctica, particularly with simulation of cloud properties that regulate snow surface melting or refreezing. Atmospheric and climate-related field work has significant challenges in the Antarctic, due to the small number of research stations that can support state-of-the-art equipment. Here we present new data from a suite of transportable and solar-powered instruments that can be deployed to remote Antarctic sites, including regions where ice shelves are most at risk, and we demonstrate how key components of climate model simulations can be evaluated against these data.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2015

Aerosol transport and wet scavenging in deep convective clouds: a case study and model evaluation using a multiple passive tracer analysis approach

The effect of wet scavenging on ambient aerosols in deep, continental convective clouds in the mid-latitudes is studied for a severe storm case in Oklahoma during the Deep Convective Clouds and Chemistry (DC3) field campaign. A new passive-tracer based transport analysis framework is developed to characterize the convective transport based on the vertical distribution of several slowly reacting and nearly insoluble trace gases. The passive gas concentration in the upper troposphere convective outflow results from a mixture of 47% from the lower level (0-3 km), 21% entrained from the upper troposphere, and 32% from mid-atmosphere based on observations. The transport analysis framework is applied to aerosols to estimate aerosol transport and wet-scavenging efficiency. Observations yield high overall scavenging efficiencies of 81% and 68% for aerosol mass (Dp < 1μm) and aerosol number (0.03< Dp < 2.5μm), respectively. Little chemical selectivity to wet scavenging is seen among observed submicron sulfate (84%), organic (82%), and ammonium (80%) aerosols, while nitrate has a much lower scavenging efficiency of 57% likely due to the uptake of nitric acid. Observed larger size particles (0.15 - 2.5μm) are scavenged more efficiently (84%) than smaller particles (64%; 0.03 - 0.15μm). The storm is simulated using the chemistry version of the WRF model. Compared to the observation based analysis, the standard model underestimates the wet scavenging efficiency for both mass and number concentrations with low biases of 31% and 40%, respectively. Adding a new treatment of secondary activation significantly improves simulation results, so that the bias in scavenging efficiency in mass and number concentrations is reduced to <10%. This supports the hypothesis that secondary activation is an important process for wet removal of aerosols in deep convective storms.

Yang, Qing↗
DOE OSTI2026

Observational Data for Next-Generation Climate Model Evaluation: Requirements, Considerations, and Best Practices

Climate model simulations are an important source of information about our planet’s climate system and also enable informed decision-making under different future scenarios. As a new archive of results from the next generation of climate models is anticipated to become available with the Coupled Model Intercomparison Project phase 7 (CMIP7), the need to develop efficient and robust methods to evaluate models is paramount. Observations are an integral part of model evaluation, providing a means to quantify and understand the degree to which climate models can faithfully reproduce Earth system processes. Such analysis is critical for constraining climate projections, identifying areas of focus for model development, and assisting analysts in deciphering the utility of models for specific applications. Observations of Earth system come from a diversity of sources, span different space–time domains, and are produced by different communities, and each dataset features different data structures and formats, metadata standards, and its own unique uncertainties. Uncertainties in an observational dataset may stem from gaps in temporal and spatial coverage, instrumentation errors, or assumptions in retrieval and processing methods. How then does one ensure that observational data are ready for use and utilized in the most appropriate way for robust, rapid, and routine climate model evaluation? The CMIP7 Model Benchmarking Task Team with input from the broader climate modeling, model evaluation, and observational data communities present a vision and considerations for best practices toward the optimal and appropriate use of observational data to support next-generation climate model evaluation.

Climate models↗
DOE OSTI2020

Empirical Validation of Multi-Zone HVAC System Model: Evaluation of Existing Infiltration Models used in Building Energy Simulation

Infiltration can have a significant impact on building loads. Studies have shown that infiltration can account for 15-40% of annual space conditioning needs in commercial buildings (Emmerich et al. 2019; Younes et al. 2012). The driving force of infiltration is the pressure difference across the building envelope caused by wind, the stack effect (known as buoyancy effect), and the operation of ventilation equipment. Wind pressure is governed by wind direction, speed, building shape, and other structures around the building. The stack effect is a function of the building height and air density differences of ambient air (Han, 2015). The effect of wind is dominant in low-rise residential buildings, and the stack effect is dominant in highrise buildings (ASHRAE 2017). In building energy simulation programs (e.g., EnergyPlus), various empirical infiltration models (e.g., the effective leakage area model, the flow coefficient model) are available to simulate infiltration rates. To help users in selecting a proper infiltration model for modeling of the two-story Flexible Research Platform (FRP), the team evaluates the existing infiltration models in EnergyPlus based on field measurements from the FRP. The blower door and tracer gas decay tests were performed in the FRP. The blower door test result was used to estimate input parameters required in the infiltration models. The actual infiltration rates were estimated with the tracer gas decay test results.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2015

Nevada National Security Site (NNSS) Environmental Remediation Using EPA's Model Evaluation Guidance to Move to Closure - 15588

The National Nuclear Security Administration Nevada Field Office (NNSA/NFO) has adapted the Environmental Protection Agency's guidance on environmental model evaluation as a framework for evaluating corrective action investigation models. Model evaluation is an important confidence building activity, which is needed to support closure of the Nevada National Security Site (NNSS) Corrective Action Units (CAUs) associated with underground nuclear testing. Through this process, site-investigation models of groundwater flow and contaminant transport are reviewed and compared to new site-specific data to determine if the results can be used to support regulatory decisions required for closure and to achieve NNSA/NFO site management objectives. The regulatory decisions include identifying and establishing CAU regulatory boundaries; identifying institutional controls, including use-restriction boundaries; and developing a long-term closure monitoring program. A successful case study from Frenchman Flat, Nevada is used to illustrate the model evaluation process. Groundwater flow and transport from two of the 10 underground nuclear tests in Frenchman Flat had the largest potential for contaminating groundwater, and were selected for evaluation. Wells were drilled, logged, hydraulically tested, sampled, and analyzed for radionuclides and the data was evaluated with respect to previous site conceptualization and numerical model results. The approach and results of this process as applied to the PIN STRIPE test and how it helped build confidence to close the Frenchman Flat CAU is described in this case study. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2023

Incorporation of aerosol into the COSPv2 satellite lidar simulator for climate model evaluation

Abstract. Atmospheric aerosol has substantial impacts on climate, air quality and biogeochemical cycles, and its concentrations are highly variable in space and time. A key variability to evaluate within models that simulate aerosol is the vertical distribution, which influences atmospheric heating profiles and aerosol–cloud interactions, to help constrain aerosol residence time and to better represent the magnitude of simulated impacts. To ensure a consistent comparison between modeled and observed vertical distribution of aerosol, we implemented an aerosol lidar simulator within the Cloud Feedback Model Intercomparison Project (CFMIP) Observation Simulator Package version 2 (COSPv2). We assessed the attenuated total backscattered (ATB) signal and the backscatter ratios (SRs) at 532 nm in the U.S. Department of Energy's Energy Exascale Earth System Model version 1 (E3SMv1). The simulator performs the computations at the same vertical resolution as the Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP), making use of aerosol optics from the E3SMv1 model as inputs and assuming that aerosol is uniformly distributed horizontally within each model grid box. The simulator applies a cloud masking and an aerosol detection threshold to obtain the ATB and SR profiles that would be observed above clouds by CALIOP with its aerosol detection capability. Our analysis shows that the aerosol distribution simulated at a seasonal timescale is generally in good agreement with observations. Over the Southern Ocean, however, the model does not produce the SR maximum as observed in the real world. Comparison between clear-sky and all-sky SRs shows little differences, indicating that the cloud screening by potentially incorrect model clouds does not affect the mean aerosol signal averaged over a season. This indicates that the differences between observed and simulated SR values are due not to sampling errors, but to deficiencies in the representation of aerosol in models. Finally, we highlight the need for future applications of lidar observations at multiple wavelengths to provide insights into aerosol properties and distribution and their representation in Earth system models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2015

Incorporating an advanced aerosol activation parameterization into WRF-CAM5: Model evaluation and parameterization intercomparison

Aerosol activation into cloud droplets is an important process that governs aerosol indirect effects. The advanced treatment of aerosol activation by Fountoukis and Nenes (2005) and its recent updates, collectively called the FN series, have been incorporated into a newly developed regional coupled climate-air quality model based on the Weather Research and Forecasting model with the physics package of the Community Atmosphere Model version 5 (WRF-CAM5) to simulate aerosol-cloud interactions in both resolved and convective clouds. The model is applied to East Asia for two full years of 2005 and 2010. A comprehensive model evaluation is performed for model predictions of meteorological, radiative, and cloud variables, chemical concentrations, and column mass abundances against satellite data and surface observations from air quality monitoring sites across East Asia. The model performs overall well for major meteorological variables including near-surface temperature, specific humidity, wind speed, precipitation, cloud fraction, precipitable water, downward shortwave and longwave radiation, and column mass abundances of CO, SO 2 , NO 2 , HCHO, and O 3 in terms of both magnitudes and spatial distributions. Larger biases exist in the predictions of surface concentrations of CO and NO x at all sites and SO 2 , O 3 , PM 2.5 , and PM 10 concentrations at some sites, aerosol optical depth, cloud condensation nuclei over ocean, cloud droplet number concentration (CDNC), cloud liquid and ice water path, and cloud optical thickness. Compared with the default Abdul-Razzack Ghan (2000) parameterization, simulations with the FN series produce ~107–113% higher CDNC, with half of the difference attributable to the higher aerosol activation fraction by the FN series and the remaining half due to feedbacks in subsequent cloud microphysical processes. With the higher CDNC, the FN series are more skillful in simulating cloud water path, cloud optical thickness, downward shortwave radiation, shortwave cloud forcing, and precipitation. Here, the model evaluation identifies several areas of improvements including emissions and their vertical allocation as well as model formulations such as aerosol formation, cloud droplet nucleation, and ice nucleation.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Data–Driven Velocity Model Evaluation Using K–Means Clustering

In this work, we develop a data-driven clustering method to evaluate a velocity model using surface wave velocity dispersion. This is done by first computing theoretical dispersion curves for 1-D velocity profiles of all the grid locations and then splitting the resulting dispersion curves into a certain number of groups via the K-means clustering. The observed dispersion curves are also clustered following the same procedure and the velocity model is assessed by comparing the spatial patterns obtained for the observed and synthetic data sets. The method is applied to evaluate two community velocity models in southern California, CVM-S4.26 and CVM-H15.1, using phase velocity maps derived for 3–16 s Rayleigh waves. We found a good correlation in the spatial distribution of clusters between the result of CVM-S4.26 and that of the observed data, suggesting that the CVM-S4.26 fits the observed dispersion maps better than the CVM-H15.1 in terms of features extracted from the clustering analysis.

58 GEOSCIENCES↗
DOE OSTI2016

Towards improved and more routine Earth system model evaluation in CMIP

The Coupled Model Intercomparison Project (CMIP) has successfully provided the climate community with a rich collection of simulation output from Earth system models (ESMs) that can be used to understand past climate changes and make projections and uncertainty estimates of the future. Confidence in ESMs can be gained because the models are based on physical principles and reproduce many important aspects of observed climate. More research is required to identify the processes that are most responsible for systematic biases and the magnitude and uncertainty of future projections so that more relevant performance tests can be developed. At the same time, there are many aspects of ESM evaluation that are well established and considered an essential part of systematic evaluation but have been implemented ad hoc with little community coordination. Given the diversity and complexity of ESM analysis, we argue that the CMIP community has reached a critical juncture at which many baseline aspects of model evaluation need to be performed much more efficiently and consistently. We provide a perspective and viewpoint on how a more systematic, open, and rapid performance assessment of the large and diverse number of models that will participate in current and future phases of CMIP can be achieved, and announce our intention to implement such a system for CMIP6. Accomplishing this could also free up valuable resources as many scientists are frequently "re-inventing the wheel" by re-writing analysis routines for well-established analysis methods. A more systematic approach for the community would be to develop and apply evaluation tools that are based on the latest scientific knowledge and observational reference, are well suited for routine use, and provide a wide range of diagnostics and performance metrics that comprehensively characterize model behaviour as soon as the output is published to the Earth System Grid Federation (ESGF). The CMIP infrastructure enforces data standards and conventions for model output and documentation accessible via the ESGF, additionally publishing observations (obs4MIPs) and reanalyses (ana4MIPs) for model intercomparison projects using the same data structure and organization as the ESM output. This largely facilitates routine evaluation of the ESMs, but to be able to process the data automatically alongside the ESGF, the infrastructure needs to be extended with processing capabilities at the ESGF data nodes where the evaluation tools can be executed on a routine basis. Efforts are already underway to develop community-based evaluation tools, and we encourage experts to provide additional diagnostic codes that would enhance this capability for CMIP. And, at the same time, we encourage the community to contribute observations and reanalyses for model evaluation to the obs4MIPs and ana4MIPs archives. The intention is to produce through the ESGF a widely accepted quasi-operational evaluation framework for CMIP6 that would routinely execute a series of standardized evaluation tasks. Over time, as this capability matures, we expect to produce an increasingly systematic characterization of models which, compared with early phases of CMIP, will more quickly and openly identify the strengths and weaknesses of the simulations. This will also reveal whether long-standing model errors remain evident in newer models and will assist modelling groups in improving their models. Finally, this framework will be designed to readily incorporate updates, including new observations and additional diagnostics and metrics as they become available from the research community.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

The Fire and Smoke Model Evaluation Experiment—A Plan for Integrated, Large Fire–Atmosphere Field Campaigns

The Fire and Smoke Model Evaluation Experiment (FASMEE) is designed to collect integrated observations from large wildland fires and provide evaluation datasets for new models and operational systems. Wildland fire, smoke dispersion, and atmospheric chemistry models have become more sophisticated, and next-generation operational models will require evaluation datasets that are coordinated and comprehensive for their evaluation and advancement. Integrated measurements are required, including ground-based observations of fuels and fire behavior, estimates of fire-emitted heat and emissions fluxes, and observations of near-source micrometeorology, plume properties, smoke dispersion, and atmospheric chemistry. To address these requirements the FASMEE campaign design includes a study plan to guide the suite of required measurements in forested sites representative of many prescribed burning programs in the southeastern United States and increasingly common high-intensity fires in the western United States. Here we provide an overview of the proposed experiment and recommendations for key measurements. The FASMEE study provides a template for additional large-scale experimental campaigns to advance fire science and operational fire and smoke models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

CrossCheck: Rapid, Reproducible, and Interpretable Model Evaluation

Evaluation beyond aggregate performance metrics, e.g. F1-score, is crucial to both establish an appropriate level of trust in machine learning models and identify future model improvements. In this paper we demonstrate CrossCheck, an interactive visualization tool for rapid crossmodel comparison and reproducible error analysis. We describe the tool and discuss design and implementation details. We then present three use cases (named entity recognition, reading comprehension, and clickbait detection) that show the benefits of using the tool for model evaluation. CrossCheck allows data scientists to make informed decisions to choose between multiple models, identify when the models are correct and for which examples, investigate whether the models are making the same mistakes as humans, evaluate models’ generalizability and highlight models’ limitations, strengths and weaknesses. Furthermore, CrossCheck is implemented as a Jupyter widget, which allows rapid and convenient integration into data scientists’ model development workflows.

Arendt, Dustin L.↗