Search NASA⌕ Search

SEARCH · Search NASA

Results for “climate model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Observational Data for Next-Generation Climate Model Evaluation: Requirements, Considerations, and Best Practices

Climate model simulations are an important source of information about our planet’s climate system and also enable informed decision-making under different future scenarios. As a new archive of results from the next generation of climate models is anticipated to become available with the Coupled Model Intercomparison Project phase 7 (CMIP7), the need to develop efficient and robust methods to evaluate models is paramount. Observations are an integral part of model evaluation, providing a means to quantify and understand the degree to which climate models can faithfully reproduce Earth system processes. Such analysis is critical for constraining climate projections, identifying areas of focus for model development, and assisting analysts in deciphering the utility of models for specific applications. Observations of Earth system come from a diversity of sources, span different space–time domains, and are produced by different communities, and each dataset features different data structures and formats, metadata standards, and its own unique uncertainties. Uncertainties in an observational dataset may stem from gaps in temporal and spatial coverage, instrumentation errors, or assumptions in retrieval and processing methods. How then does one ensure that observational data are ready for use and utilized in the most appropriate way for robust, rapid, and routine climate model evaluation? The CMIP7 Model Benchmarking Task Team with input from the broader climate modeling, model evaluation, and observational data communities present a vision and considerations for best practices toward the optimal and appropriate use of observational data to support next-generation climate model evaluation.

Climate models↗
DOE OSTI2020

The ARM Data-oriented Metrics and Diagnostics Package for Climate Models - A New Tool for Evaluating Climate Models with Field Data

The U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) program User Facility produces ground-based long-term continuous unique measurements for atmospheric state, precipitation, turbulent fluxes, radiation, aerosol, cloud and the land surface, which are collected at multiple sites. These comprehensive datasets have been widely used to calibrate climate models and are proven to be invaluable for climate model development and improvement. This article introduces an evaluation package to facilitate the use of ground-based ARM measurements in climate model evaluation. The ARM data-oriented metrics and diagnostics package (ARM-DIAGS) includes both ARM observational datasets and a Python-based analysis toolkit for computation and visualization. The observational datasets are compiled from multiple ARM data products and specifically tailored for use in climate model evaluation. In addition, ARM-DIAGS also includes simulation data from models participating the Coupled Model Inter-comparison Project (CMIP), which will allow climate-modeling groups to compare a new, candidate version of their model to existing CMIP models. The analysis toolkit is designed to make the metrics and diagnostics quickly available to the model developers.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

A Surface Radiation Balance Dataset from Siple Dome in West Antarctica for Atmospheric and Climate Model Evaluation

Abstract A field campaign at Siple Dome in West Antarctica during the austral summer 2019/20 offers an opportunity to evaluate climate model performance, particularly cloud microphysical simulation. Over Antarctic ice sheets and ice shelves, clouds are a major regulator of the surface energy balance, and in the warm season their presence occasionally induces surface melt that can gradually weaken an ice shelf structure. This dataset from Siple Dome, obtained using transportable and solar-powered equipment, includes surface energy balance measurements, meteorology, and cloud remote sensing. To demonstrate how these data can be used to evaluate model performance, comparisons are made with meteorological reanalysis known to give generally good performance over Antarctica (ERA5). Surface albedo measurements show expected variability with observed cloud amount, and can be used to evaluate a model’s snowpack parameterization. One case study discussed involves a squall with northerly winds, during which ERA5 fails to produce cloud cover throughout one of the days. A second case study illustrates how shortwave spectroradiometer measurements that encompass the 1.6- μ m atmospheric window reveal cloud phase transitions associated with cloud life cycle. Here, continuously precipitating mixed-phase clouds become mainly liquid water clouds from local morning through the afternoon, not reproduced by ERA5. We challenge researchers to run their various regional or global models in a manner that has the large-scale meteorology follow the conditions of this field campaign, compare cloud and radiation simulations with this Siple Dome dataset, and potentially investigate why cloud microphysical simulations or other model components might produce discrepancies with these observations. significance statement Antarctica is a critical region for understanding climate change and sea level rise, as the great ice sheets and the ice shelves are subject to increasing risk as global climate warms. Climate models have difficulties over Antarctica, particularly with simulation of cloud properties that regulate snow surface melting or refreezing. Atmospheric and climate-related field work has significant challenges in the Antarctic, due to the small number of research stations that can support state-of-the-art equipment. Here we present new data from a suite of transportable and solar-powered instruments that can be deployed to remote Antarctic sites, including regions where ice shelves are most at risk, and we demonstrate how key components of climate model simulations can be evaluated against these data.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Evaluating Climate Models’ Cloud Feedbacks Against Expert Judgment

Abstract The persistent and growing spread in effective climate sensitivity (ECS) across global climate models necessitates rigorous evaluation of their cloud feedbacks. Here we evaluate several cloud feedback components simulated in 19 climate models against benchmark values determined via an expert synthesis of observational, theoretical, and high‐resolution modeling studies. We find that models with smallest feedback errors relative to these benchmark values generally have moderate total cloud feedbacks (0.4–0.6 W m −2 K −1 ) and ECS (3–4 K). Those with largest errors generally have total cloud feedback and ECS values that are too large or too small. Models tend to achieve large positive total cloud feedbacks by having several cloud feedback components that are systematically biased high rather than by having a single anomalously large component, and vice versa. In general, better simulation of mean‐state cloud properties leads to stronger but not necessarily better cloud feedbacks. The Python code base provided herein could be applied to developmental versions of models to assess cloud feedbacks and cloud errors and place them in the context of other models and of expert judgment in real‐time during model development.

58 GEOSCIENCES↗
DOE OSTI2023

A seamless approach for evaluating climate models across spatial scales

In regions of the world where topography varies significantly with distance, most global climate models (GCMs) have spatial resolutions that are too coarse to accurately simulate key meteorological variables that are influenced by topography, such as clouds, precipitation, and surface temperatures. One approach to tackle this challenge is to run climate models of sufficiently high resolution in those topographically complex regions such as the North American Regionally Refined Model (NARRM) subset of the Department of Energy’s (DOE) Energy Exascale Earth System Model version 2 (E3SM v2). Although high-resolution simulations are expected to provide unprecedented details of atmospheric processes, running models at such high resolutions remains computationally expensive compared to lower-resolution models such as the E3SM Low Resolution (LR). Moreover, because regionally refined and high-resolution GCMs are relatively new, there are a limited number of observational datasets and frameworks available for evaluating climate models with regionally varying spatial resolutions. As such, we developed a new framework to quantify the added value of high spatial resolution in simulating precipitation over the contiguous United States (CONUS). To determine its viability, we applied the framework to two model simulations and an observational dataset. We first remapped all the data into Hierarchical Equal-Area Iso-Latitude Pixelization (HEALPix) pixels. HEALPix offers several mathematical properties that enable seamless evaluation of climate models across different spatial resolutions including its equal-area and partitioning properties. The remapped HEALPix-based data are used to show how the spatial variability of both observed and simulated precipitation changes with resolution increases. This study provides valuable insights into the requirements for achieving accurate simulations of precipitation patterns over the CONUS. It highlights the importance of allocating sufficient computational resources to run climate models at higher temporal and spatial resolutions to capture spatial patterns effectively. Furthermore, the study demonstrates the effectiveness of the HEALPix framework in evaluating precipitation simulations across different spatial resolutions. This framework offers a viable approach for comparing observed and simulated data when dealing with datasets of varying spatial resolutions. By employing this framework, researchers can extend its usage to other climate variables, datasets, and disciplines that require comparing datasets with different spatial resolutions.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

gcm_eval (Global Climate Model Evaluation) [SWR-24-37]

The interplay between energy, climate, and weather is becoming more complex as our changing climate continues to affect the weather we experience which in turn drives changes in the ever increasing share of renewable energy generation and energy demand. Energy system analyses commonly rely on meteorological inputs to estimate renewable energy generation and energy demand; however, these inputs rarely represent the estimated impacts of future climate change. Climate models and publicly available climate change datasets can be used for this purpose, but the selection of inputs from the myriad of available models and datasets is a nuanced and often subjective process. There is no single perfect climate model or dataset for all applications. In this repository, we include software and the corresponding assessments of various global climate models (GCMs) from the Coupled Model Intercomparison Project (CMIP6), evaluating their skills with respect to the historical climate and comparing of their future projections of climate change. We focus on variables that directly affect the energy system including the representation of extreme values that can drive grid resilience events. The objective of this repository is not to recommend the best climate model and dataset for a given analysis, but instead to provide a reference to facilitate the selection of climate models and datasets in subsequent work.

Buster, Grant↗
DOE OSTI2021

Evaluating Climate Models with the CLIVAR 2020 ENSO Metrics Package

El Niño–Southern Oscillation (ENSO) is the dominant mode of interannual climate variability on the planet, with far-reaching global impacts. It is therefore key to evaluate ENSO simulations in state-of-the-art numerical models used to study past, present, and future climate. Recently, the Pacific Region Panel of the International Climate and Ocean: Variability, Predictability and Change (CLIVAR) Project, as a part of the World Climate Research Programme (WCRP), led a community-wide effort to evaluate the simulation of ENSO variability, teleconnections, and processes in climate models. The new CLIVAR 2020 ENSO metrics package enables model diagnosis, comparison, and evaluation to 1) highlight aspects that need improvement; 2) monitor progress across model generations; 3) help in selecting models that are well suited for particular analyses; 4) reveal links between various model biases, illuminating the impacts of those biases on ENSO and its sensitivity to climate change; and to 5) advance ENSO literacy. By interfacing with existing model evaluation tools, the ENSO metrics package enables rapid analysis of multipetabyte databases of simulations, such as those generated by the Coupled Model Intercomparison Project phases 5 (CMIP5) and 6 (CMIP6). The CMIP6 models are found to significantly outperform those from CMIP5 for 8 out of 24 ENSO-relevant metrics, with most CMIP6 models showing improved tropical Pacific seasonality and ENSO teleconnections. Only one ENSO metric is significantly degraded in CMIP6, namely, the coupling between the ocean surface and subsurface temperature anomalies, while the majority of metrics remain unchanged.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Clouds and Radiation Processes in Regional Climate Models Evaluated Using Observations Over the Ice‐free Arctic Ocean

Abstract The presence of clouds in the Arctic regulates the surface energy budget (SEB) over the sea‐ice surface and the ice‐free ocean. Following several previous field campaigns, the cloud‐radiation relationship, including cloud vertical structure and phase, has been elucidated; however, modeling of this relationship has matured slowly. In recognition of the recent decline in the Arctic sea‐ice extent, representation of the cloud system in numerical models should consider the effects of areas covered by sea ice and ice‐free areas. Using an in situ stationary meteorological observation data set obtained over the ice‐free Arctic Ocean by the Japanese Research Vessel Mirai (September 2014), coordinated evaluation of six regional climate models (RCMs) with nine model runs was performed by focusing on clouds and the SEB. The most remarkable findings were as follows: (1) reduced occurrence of unstable stratification with low‐level cloud water in all models in comparison to the observations, (2) significant differences in cloud water representations between single‐ and double‐moment cloud schemes, (3) extensive differences in partitioning of hydrometeors including solid/liquid precipitation, and (4) pronounced lower‐tropospheric air temperature biases. These issues are considered as the main sources of SEB uncertainty over ice‐free areas of the Arctic Ocean. The results from a coupled RCM imply that the SEB is constrained by both the atmosphere and the ocean (and sea ice) with considerable feedback. Coordinated improvement of both stand‐alone atmospheric and coupled RCMs would promote a more comprehensive and improved understanding of the Arctic air‐ice‐sea coupled system.

Inoue, Jun↗
DOE OSTI2019

Evaluating Climate Model Simulations of the Radiative Forcing and Radiative Response at Earth’s Surface

In this work, we analyze the radiative forcing and radiative response at Earth’s surface, where perturbations in the radiation budget regulate the atmospheric hydrological cycle. By applying a radiative kernel-regression technique to CMIP5 climate model simulations where CO 2 is instantaneously quadrupled, we evaluate the intermodel spread in surface instantaneous radiative forcing, radiative adjustments to this forcing, and radiative responses to surface warming. The cloud radiative adjustment to CO 2 forcing and the temperature-mediated cloud radiative response exhibit significant intermodel spread. In contrast to its counterpart at the top of the atmosphere, the temperature-mediated cloud radiative response at the surface is found to be positive in some models and negative in others. Also, the compensation between the temperature-mediated lapse rate and water vapor radiative responses found in top-of-atmosphere calculations is not present for surface radiative flux changes. Instantaneous radiative forcing at the surface is rarely reported for model simulations; as a result, intermodel differences have not previously been evaluated in global climate models. We demonstrate that the instantaneous radiative forcing is the largest contributor to intermodel spread in effective radiative forcing at the surface. We also find evidence of differences in radiative parameterizations in current models and argue that this is a significant, but largely overlooked, source of bias in climate change simulations.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Incorporation of aerosol into the COSPv2 satellite lidar simulator for climate model evaluation

Abstract. Atmospheric aerosol has substantial impacts on climate, air quality and biogeochemical cycles, and its concentrations are highly variable in space and time. A key variability to evaluate within models that simulate aerosol is the vertical distribution, which influences atmospheric heating profiles and aerosol–cloud interactions, to help constrain aerosol residence time and to better represent the magnitude of simulated impacts. To ensure a consistent comparison between modeled and observed vertical distribution of aerosol, we implemented an aerosol lidar simulator within the Cloud Feedback Model Intercomparison Project (CFMIP) Observation Simulator Package version 2 (COSPv2). We assessed the attenuated total backscattered (ATB) signal and the backscatter ratios (SRs) at 532 nm in the U.S. Department of Energy's Energy Exascale Earth System Model version 1 (E3SMv1). The simulator performs the computations at the same vertical resolution as the Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP), making use of aerosol optics from the E3SMv1 model as inputs and assuming that aerosol is uniformly distributed horizontally within each model grid box. The simulator applies a cloud masking and an aerosol detection threshold to obtain the ATB and SR profiles that would be observed above clouds by CALIOP with its aerosol detection capability. Our analysis shows that the aerosol distribution simulated at a seasonal timescale is generally in good agreement with observations. Over the Southern Ocean, however, the model does not produce the SR maximum as observed in the real world. Comparison between clear-sky and all-sky SRs shows little differences, indicating that the cloud screening by potentially incorrect model clouds does not affect the mean aerosol signal averaged over a season. This indicates that the differences between observed and simulated SR values are due not to sampling errors, but to deficiencies in the representation of aerosol in models. Finally, we highlight the need for future applications of lidar observations at multiple wavelengths to provide insights into aerosol properties and distribution and their representation in Earth system models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

ARM Data-Oriented Metrics and Diagnostics Package for Climate Model Evaluation

A Python-based metrics and diagnostics package is currently being developed by the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) Infrastructure Team at Lawrence Livermore National Laboratory (LLNL) to facilitate the use of long-term, high-frequency measurements from the ARM Facility in evaluating the regional climate simulation of clouds, radiation, and precipitation. This metrics and diagnostics package computes climatological means of targeted climate model simulation and generates tables and plots for comparing the model simulation with ARM observational data. The Coupled Model Intercomparison Project (CMIP) model data sets are also included in the package to enable model intercomparison as demonstrated in Zhang et al. (2017). The mean of the CMIP model can serve as a reference for individual models. Basic performance metrics are computed to measure the accuracy of mean state and variability of climate models. The evaluated physical quantities include cloud fraction, temperature, relative humidity, cloud liquid water path, total column water vapor, precipitation, sensible and latent heat fluxes, and radiative fluxes, with plan to extend to more fields, such as aerosol and microphysics properties. Process-oriented diagnostics focusing on individual cloud- and precipitation-related phenomena are also being developed for the evaluation and development of specific model physical parameterizations. The version 1.0 package is designed based on data collected at ARM’s Southern Great Plains (SGP) Research Facility, with the plan to extend to other ARM sites. The metrics and diagnostics package is currently built upon standard Python libraries and additional Python packages developed by DOE (such as CDMS and CDAT). The ARM metrics and diagnostic package is available publicly with the hope that it can serve as an easy entry point for climate modelers to compare their models with ARM data. In this report, we first present the input data, which constitutes the core content of the metrics and diagnostics package in section 2, and a user's guide documenting the workflow/structure of the version 1.0 codes, and including step-by-step instruction for running the package in section 3.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

ARM data-oriented metrics and diagnostics package for climate model evaluation (ARM-DIAGS-V3) version 3

A Python-based metrics and diagnostics package is currently being developed by the ARM Infrastructure Team at Lawrence Livermore National Laboratory to facilitate the use of long-term high frequency measurements from the ARM program in evaluating the regional climate simulation of clouds, radiation, precipitation, and aerosols. This metrics and diagnostics package computes climatological means of targeted climate model simulation and generates tables and plots for comparing the model simulation with ARM observational data. The CMIP model data sets are also included in the package to enable model inter-comparison as demonstrated in Zhang et al. (2018) and Zhang et al. (2020). The mean of the CMIP model can be served as a reference for individual models. Basic performance metrics are computed to measure the accuracy of mean state and variability of climate models. The evaluated physical quantities include cloud fraction, temperature, relative humidity, cloud liquid water path, total column water vapor, precipitation, sensible and latent heat fluxes, aerosol optical depth, and radiative fluxes, with plan to extend to more fields, such as the evaluation of model simulated aerosol physicochemical properties and cloud microphysics properties. Process-oriented diagnostics focusing on aerosol, cloud, and precipitation-related phenomena are also being developed for the evaluation and development of specific model physical parameterizations. In addition to the Southern Great Plains (SGP), North Slope of Alaska (NSA) and Tropical Western Pacific (TWP) atmospheric observatories in the ARMDIAGS version 2.0, the version 3.0 package have extended to the data collected at the ARM Eastern North Atlantic (ENA) atmospheric observatory and the Observation and Modeling of the Green Ocean Amazon (GOAMAZON) field campaign. The metrics and diagnostics package are currently built upon standard Python libraries and additional Python packages developed by DOE (CDAT). The ARM metrics and diagnostic package is available publicly with the hope that it can serve as an easy entry point for climate modelers to compare their models with ARM data. In this report, we first provide an overview of major metrics in section 2. The input data, which constitutes the core content of the metrics and diagnostics package, is summarized in section 3. A user's guide documenting the workflow/structure of the version 3.0 codes and including step-by-step instruction for running the package is described in section 4.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Taking climate model evaluation to the next level

We report that earth system models are complex and represent a large number of processes, resulting in a persistent spread across climate projections for a given future scenario. Owing to different model performances against observations and the lack of independence among models, there is now evidence that giving equal weight to each available model projection is suboptimal. This Perspective discusses newly developed tools that facilitate a more rapid and comprehensive evaluation of model simulations with observations, process-based emergent constraints that are a promising way to focus evaluation on the observations most relevant to climate projections, and advanced methods for model weighting. Lastly, these approaches are needed to distill the most credible information on regional climate changes, impacts, and risks for stakeholders and policy-makers.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

A multi-year short-range hindcast experiment with CESM1 for evaluating climate model moist processes from diurnal to interannual timescales

Abstract. We present a multi-year short-range hindcast experiment and its experimental design for better evaluation of both the mean state and variability of atmospheric moist processes in climate models from diurnal to interannual timescales and facilitate model development. We used the Community Earth System Model version 1 as the base model and performed a suite of 3 d hindcasts initialized every day starting at 00:00 Z from 1997 to 2012. Three processes – the diurnal cycle of clouds during different cloud regimes over the central US, precipitation and diabatic heating associated with the Madden–Julian Oscillation (MJO), and the response of precipitation, surface radiative and heat fluxes, as well as zonal wind stress to sea surface temperature anomalies associated with the El Niño–Southern Oscillation – are evaluated as examples to demonstrate how one can better utilize simulations from this experiment to gain insights into model errors and their connection to physical parameterizations or large-scale state. This is achieved by comparing the hindcasts with corresponding long-term observations for periods based on different phenomena. These analyses can only be done through this multi-year hindcast approach to establish robust statistics of the processes under well-controlled large-scale environment because these phenomena are either a result of interannual climate variability or only happen a few times in a given year (e.g., MJO, or cloud regime types). Furthermore, comparison of hindcasts to the typical simulations in climate mode with the same model allows one to infer what portion of a model's climate error directly comes from fast errors in the parameterizations of moist processes. As demonstrated here, model biases in the mean state and variability associated with parameterized moist processes usually develop within a few days and manifest within weeks to affect the simulations of large-scale circulation and ultimately the climate mean state and variability. Therefore, model developers can achieve additional useful understanding of the underlying problems in model physics by conducting a multi-year hindcast experiment.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

ARM Data-Oriented Metrics and Diagnostics Package for Climate Model Evaluation

A Python-based metrics and diagnostics package is currently being developed by the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) user facility Infrastructure Team at Lawrence Livermore National Laboratory to facilitate the use of long-term, high-frequency measurements from ARM in evaluating the regional climate simulation of clouds, radiation, and precipitation. This metrics and diagnostics package computes climatological means of targeted climate model simulation and generates tables and plots for comparing the model simulation with ARM observational data.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Quasi-stationary Planetary Scale Waves in Modern Climate Models

We evaluate climate model simulations of geopotential height at the largest space-time scales, considering the first Fourier components (in longitude) of long-term climatological means. Quasi-stationary planetary scale waves emerge from this analysis. Variations of these waves are important components of climate change, for example associated with regional drought. Systematic examination of their long-term climatology provides context for understanding their evolution in time. We compare four reanalysis datasets with “historical runs” from the latest version of the Coupled Model Intercomparison Project (CMIP) and find general agreement, but we also identify model errors outside the range of both observational uncertainty and the uncertainty implied by interannual variability. Errors in wave phase have possible implications for climate model projections of regional climate and of shifting weather patterns. One model in the set we examine, originally created nearly 20 years before the others, exhibits noticeably greater errors.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing Models

Experts in data and physical sciences have to regularly grapple with the problem of competing models. Be it analytical or physics-based models, a cross-cutting challenge for experts is to reliably diagnose which model outcomes appropriately predict or simulate real-world phenomena. Expert judgment involves reconciling information across many, and often, conflicting criteria that describe the quality of model outcomes. In this paper, through a design study with climate scientists, we develop a deeper understanding of the problem and solution space of model diagnostics, resulting in the following contributions: i) a problem and task characterization using which we map experts’ model diagnostics goals to multi-way visual comparison tasks, ii) a design space of comparative visual cues for letting experts quickly understand the degree of disagreement among competing models and gauge the degree of stability of model outputs with respect to alternative criteria, and iii) design and evaluation of MyriadCues, an interactive visualization interface for exploring alternative hypotheses and insights about good and bad models by leveraging comparative visual cues. Here, we present case studies and subjective feedback by experts, which validate how MyriadCues enables more transparent model diagnostic mechanisms, as compared to the state of the art.

54 ENVIRONMENTAL SCIENCES↗