Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2011

Land Surface Verification Toolkit (LVT) - A Generalized Framework for Land Surface Model Evaluation

Model evaluation and verification are key in improving the usage and applicability of simulation models for real-world applications. In this article, the development and capabilities of a formal system for land surface model evaluation called the Land surface Verification Toolkit (LVT) is described. LVT is designed to provide an integrated environment for systematic land model evaluation and facilitates a range of verification approaches and analysis capabilities. LVT operates across multiple temporal and spatial scales and employs a large suite of in-situ, remotely sensed and other model and reanalysis datasets in their native formats. In addition to the traditional accuracy-based measures, LVT also includes uncertainty and ensemble diagnostics, information theory measures, spatial similarity metrics and scale decomposition techniques that provide novel ways for performing diagnostic model evaluations. Though LVT was originally designed to support the land surface modeling and data assimilation framework known as the Land Information System (LIS), it also supports hydrological data products from other, non-LIS environments. In addition, the analysis of diagnostics from various computational subsystems of LIS including data assimilation, optimization and uncertainty estimation are supported within LVT. Together, LIS and LVT provide a robust end-to-end environment for enabling the concepts of model data fusion for hydrological applications. The evolving capabilities of LVT framework are expected to facilitate rapid model evaluation efforts and aid the definition and refinement of formal evaluation procedures for the land surface modeling community.

Kumar, Sujay V.↗
DOE OSTI2026

Model Evaluation Protocol for Fire Models Involving Fuels at Liquefied Natural Gas Facilities (Version 2)

This document provides a description of the model evaluation protocol (MEP) for pool fires, jet fires, and fireballs involving liquefied natural gas (LNG) and processing fuels at LNG facilities. The purpose of the MEP is to provide procedures regarding the assessment of a model’s suitability to predict heat flux from fires. Three components, namely, a scientific assessment, model verification, and model validation comprise the MEP. The evaluation of a model satisfying these three components is to be documented in the form of a model evaluation report (MER). Discussion of models for the prediction of fire, detailed information on each of the three MEP components, the MEP procedure regarding new versions of previously approved models, and the format of the model evaluation report (MER) are provided.

03 NATURAL GAS↗
DOE OSTI2022

Model Evaluation Protocol for Fire Models Involving Fuels at Liquefied Natural Gas Facilities.

This document provides a description of the model evaluation protocol (MEP) for pool fires, jet fires, and fireballs involving liquefied natural gas (LNG), refrigerant fluids, and byproducts at LNG facilities. The purpose of the MEP is to provide procedures regarding the assessment of a model's suitability to predict heat flux from fires. Three components, namely, a scientific assessment, model verification, and model validation comprise the MEP. The evaluation of a model satisfying these three components is to be documented in the form of a model evaluation report (MER). Discussion of models for the prediction of fire, detailed information on each of the three MEP components, the MEP procedure regarding new versions of previously approved models, and the format of the model evaluation report (MER) are provided.

03 NATURAL GAS↗
DOE OSTI2015

Application of an Online-Coupled Regional Climate Model, WRF-CAM5, over East Asia for Examination of Ice Nucleation Schemes: Part I. Comprehensive Model Evaluation and Trend Analysis for 2006 and 2011

Online-coupled climate and chemistry models are necessary to realistically represent the interactions between climate variables and chemical species and accurately simulate aerosol direct and indirect effects on cloud, precipitation, and radiation. In this Part I of a two-part paper, simulations from the Weather Research and Forecasting model coupled with the physics package of Community Atmosphere Model (WRF-CAM5) are conducted with the default heterogeneous ice nucleation parameterization over East Asia for two full years: 2006 and 2011. A comprehensive model evaluation is performed using satellite and surface observations. The model shows an overall acceptable performance for major meteorological variables at the surface and in the boundary layer, as well as column variables (e.g., precipitation, cloud fraction, precipitating water vapor, downward longwave and shortwave radiation). Moderate to large biases exist for cloud condensation nuclei over oceanic areas, cloud variables (e.g., cloud droplet number concentration, cloud liquid and ice water paths, cloud optical depth, longwave and shortwave cloud forcing). These biases indicate a need to improve the model treatments for cloud processes, especially cloud droplets and ice nucleation, as well as to reduce uncertainty in the satellite retrievals. The model simulates well the column abundances of chemical species except for column SO 2 but relatively poor for surface concentrations of several species such as CO, NO 2 , SO 2 , PM2.5, and PM10. Several reasons could contribute to the underestimation of major chemical species in East Asia including underestimations of anthropogenic emissions and natural dust emissions, uncertainties in the spatial and vertical distributions of the anthropogenic emissions, as well as biases in meteorological, radiative, and cloud predictions. Despite moderate to large biases in the chemical predictions, the model performance is generally consistent with or even better than that reported for East Asia with only a few exceptions. The model generally reproduces the observed seasonal variations and the difference between 2006 and 2011 for most variables or chemical species. Overall, these results demonstrate promising skills of WRF-CAM5 for long-term simulations at a regional scale and suggest several areas of potential improvements.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

A Surface Radiation Balance Dataset from Siple Dome in West Antarctica for Atmospheric and Climate Model Evaluation

Abstract A field campaign at Siple Dome in West Antarctica during the austral summer 2019/20 offers an opportunity to evaluate climate model performance, particularly cloud microphysical simulation. Over Antarctic ice sheets and ice shelves, clouds are a major regulator of the surface energy balance, and in the warm season their presence occasionally induces surface melt that can gradually weaken an ice shelf structure. This dataset from Siple Dome, obtained using transportable and solar-powered equipment, includes surface energy balance measurements, meteorology, and cloud remote sensing. To demonstrate how these data can be used to evaluate model performance, comparisons are made with meteorological reanalysis known to give generally good performance over Antarctica (ERA5). Surface albedo measurements show expected variability with observed cloud amount, and can be used to evaluate a model’s snowpack parameterization. One case study discussed involves a squall with northerly winds, during which ERA5 fails to produce cloud cover throughout one of the days. A second case study illustrates how shortwave spectroradiometer measurements that encompass the 1.6- μ m atmospheric window reveal cloud phase transitions associated with cloud life cycle. Here, continuously precipitating mixed-phase clouds become mainly liquid water clouds from local morning through the afternoon, not reproduced by ERA5. We challenge researchers to run their various regional or global models in a manner that has the large-scale meteorology follow the conditions of this field campaign, compare cloud and radiation simulations with this Siple Dome dataset, and potentially investigate why cloud microphysical simulations or other model components might produce discrepancies with these observations. significance statement Antarctica is a critical region for understanding climate change and sea level rise, as the great ice sheets and the ice shelves are subject to increasing risk as global climate warms. Climate models have difficulties over Antarctica, particularly with simulation of cloud properties that regulate snow surface melting or refreezing. Atmospheric and climate-related field work has significant challenges in the Antarctic, due to the small number of research stations that can support state-of-the-art equipment. Here we present new data from a suite of transportable and solar-powered instruments that can be deployed to remote Antarctic sites, including regions where ice shelves are most at risk, and we demonstrate how key components of climate model simulations can be evaluated against these data.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2023

A Surface Radiation Balance Data Set from Siple Dome in West Antarctica for Atmospheric and Climate Model Evaluation

A field campaign at Siple Dome in West Antarctica during the austral summer 2019-2020 offers an opportunity to evaluate climate model performance, particularly cloud microphysical simulation. Over Antarctic ice sheets and ice shelves, clouds are a major regulator of the surface energy balance, and in the warm season their presence occasionally induces surface melt that can gradually weaken an ice shelf structure. This dataset from Siple Dome, obtained using transportable and solar-poweredequipment, includes surface energy balance measurements, meteorology and cloud remote sensing. To demonstrate how these data can be used to evaluate model performance, comparisons are made with meteorological reanalysis known to give generally good performance over Antarctica (ERA5). Surface albedo measurements show expected variability with observed cloud amount, and can be used to evaluate a model's snowpack parameterization. One case study discussed involves a squall with northerly winds, during which ERA5 fails to produce cloud cover throughout one of the days. A second case study illustrates how shortwave spectroradiometer measurements that encompass the 1.6-micron atmospheric window reveal cloud phase transitions associated with cloud lifecycle. Here, continuously precipitating mixed-phase clouds become mainly liquid water clouds from local morning through the afternoon, not reproduced by ERA5. We challenge researchers to run their various regional or global models in a manner that has the large-scale meteorology follow the conditions of this field campaign, compare cloud and radiation simulations with this Siple Dome dataset, and potentially investigate why cloud microphysical simulations or other model components might produce discrepancies with these observations.

cloud remote sensing↗
DOE OSTI2015

Aerosol transport and wet scavenging in deep convective clouds: a case study and model evaluation using a multiple passive tracer analysis approach

The effect of wet scavenging on ambient aerosols in deep, continental convective clouds in the mid-latitudes is studied for a severe storm case in Oklahoma during the Deep Convective Clouds and Chemistry (DC3) field campaign. A new passive-tracer based transport analysis framework is developed to characterize the convective transport based on the vertical distribution of several slowly reacting and nearly insoluble trace gases. The passive gas concentration in the upper troposphere convective outflow results from a mixture of 47% from the lower level (0-3 km), 21% entrained from the upper troposphere, and 32% from mid-atmosphere based on observations. The transport analysis framework is applied to aerosols to estimate aerosol transport and wet-scavenging efficiency. Observations yield high overall scavenging efficiencies of 81% and 68% for aerosol mass (Dp < 1μm) and aerosol number (0.03< Dp < 2.5μm), respectively. Little chemical selectivity to wet scavenging is seen among observed submicron sulfate (84%), organic (82%), and ammonium (80%) aerosols, while nitrate has a much lower scavenging efficiency of 57% likely due to the uptake of nitric acid. Observed larger size particles (0.15 - 2.5μm) are scavenged more efficiently (84%) than smaller particles (64%; 0.03 - 0.15μm). The storm is simulated using the chemistry version of the WRF model. Compared to the observation based analysis, the standard model underestimates the wet scavenging efficiency for both mass and number concentrations with low biases of 31% and 40%, respectively. Adding a new treatment of secondary activation significantly improves simulation results, so that the bias in scavenging efficiency in mass and number concentrations is reduced to <10%. This supports the hypothesis that secondary activation is an important process for wet removal of aerosols in deep convective storms.

Yang, Qing↗
DOE OSTI2026

Observational Data for Next-Generation Climate Model Evaluation: Requirements, Considerations, and Best Practices

Climate model simulations are an important source of information about our planet’s climate system and also enable informed decision-making under different future scenarios. As a new archive of results from the next generation of climate models is anticipated to become available with the Coupled Model Intercomparison Project phase 7 (CMIP7), the need to develop efficient and robust methods to evaluate models is paramount. Observations are an integral part of model evaluation, providing a means to quantify and understand the degree to which climate models can faithfully reproduce Earth system processes. Such analysis is critical for constraining climate projections, identifying areas of focus for model development, and assisting analysts in deciphering the utility of models for specific applications. Observations of Earth system come from a diversity of sources, span different space–time domains, and are produced by different communities, and each dataset features different data structures and formats, metadata standards, and its own unique uncertainties. Uncertainties in an observational dataset may stem from gaps in temporal and spatial coverage, instrumentation errors, or assumptions in retrieval and processing methods. How then does one ensure that observational data are ready for use and utilized in the most appropriate way for robust, rapid, and routine climate model evaluation? The CMIP7 Model Benchmarking Task Team with input from the broader climate modeling, model evaluation, and observational data communities present a vision and considerations for best practices toward the optimal and appropriate use of observational data to support next-generation climate model evaluation.

Climate models↗
NASA NTRS2012

Likelihood-Based Climate Model Evaluation

Climate models are deterministic, mathematical descriptions of the physics of climate. Confidence in predictions of future climate is increased if the physics are verifiably correct. A necessary, (but not sufficient) condition is that past and present climate be simulated well. Quantify the likelihood that a (summary statistic computed from a) set of observations arises from a physical system with the characteristics captured by a model generated time series. Given a prior on models, we can go further: posterior distribution of model given observations.

climate models↗
DOE OSTI2020

Empirical Validation of Multi-Zone HVAC System Model: Evaluation of Existing Infiltration Models used in Building Energy Simulation

Infiltration can have a significant impact on building loads. Studies have shown that infiltration can account for 15-40% of annual space conditioning needs in commercial buildings (Emmerich et al. 2019; Younes et al. 2012). The driving force of infiltration is the pressure difference across the building envelope caused by wind, the stack effect (known as buoyancy effect), and the operation of ventilation equipment. Wind pressure is governed by wind direction, speed, building shape, and other structures around the building. The stack effect is a function of the building height and air density differences of ambient air (Han, 2015). The effect of wind is dominant in low-rise residential buildings, and the stack effect is dominant in highrise buildings (ASHRAE 2017). In building energy simulation programs (e.g., EnergyPlus), various empirical infiltration models (e.g., the effective leakage area model, the flow coefficient model) are available to simulate infiltration rates. To help users in selecting a proper infiltration model for modeling of the two-story Flexible Research Platform (FRP), the team evaluates the existing infiltration models in EnergyPlus based on field measurements from the FRP. The blower door and tracer gas decay tests were performed in the FRP. The blower door test result was used to estimate input parameters required in the infiltration models. The actual infiltration rates were estimated with the tracer gas decay test results.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS1977

New model framework and structure and the commonality evaluation model

The development of a framework and structure for shuttle era unmanned spacecraft projects and the development of a commonality evaluation model is documented. The methodology developed for model utilization in performing cost trades and comparative evaluations for commonality studies is discussed. The model framework consists of categories of activities associated with the spacecraft system's development process. The model structure describes the physical elements to be treated as separate identifiable entities. Cost estimating relationships for subsystem and program-level components were calculated.

Source record↗