Search NASA⌕ Search

SEARCH · Search NASA

Results for “multi-model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

National and subnational short-term forecasting of COVID-19 in Germany and Poland during early 2021

During the COVID-19 pandemic there has been a strong interest in forecasts of the short-term development of epidemiological indicators to inform decision makers. In this study we evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland for the period from January through April 2021. We evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland. These were issued by 15 different forecasting models, run by independent research teams. Moreover, we study the performance of combined ensemble forecasts. Evaluation of probabilistic forecasts is based on proper scoring rules, along with interval coverage proportions to assess calibration. The presented work is part of a pre-registered evaluation study. We find that many, though not all, models outperform a simple baseline model up to four weeks ahead for the considered targets. Ensemble methods show very good relative performance. The addressed time period is characterized by rather stable non-pharmaceutical interventions in both countries, making short-term predictions more straightforward than in previous periods. However, major trend changes in reported cases, like the rebound in cases due to the rise of the B.1.1.7 (Alpha) variant in March 2021, prove challenging to predict. Multi-model approaches can help to improve the performance of epidemiological forecasts. However, while death numbers can be predicted with some success based on current case and hospitalization data, predictability of case numbers remains low beyond quite short time horizons. Additional data sources including sequencing and mobility data, which were not extensively used in the present study, may help to improve performance.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2026

Diagnosing the representation of surface and layered soil moisture in Earth system models

Surface soil moisture (mrsos) and vertically integrated soil moisture (mrsol) over the top 10 cm should, by definition, be physically consistent in Earth System Models (ESMs). However, an evaluation of nine CMIP6 models reveals substantial inconsistencies: in some models, mrsos and integrated mrsol agree globally; in others, they align only in specific regions; and in a few, they diverge across all grid cells. These discrepancies arise from a combination of factors, including metadata errors, inconsistent variable definitions, or diagnostic sequencing within the model. We demonstrate how such issues can lead to significant biases, even when both variables are present and seemingly well-defined. As model complexity increases and multi-model comparisons become more common, assumptions about variable equivalence may lead to flawed conclusions. This study highlights the need for routine consistency checks, improved metadata standards, and community-wide practices that ensure reliability of derived variables across ESM outputs, particularly in preparation for CMIP7.

Earth system models↗
DOE OSTI2025

Evaluation of historical precipitation interannual variability in CMIP6 over the United States

Interannual precipitation variability profoundly influences society via its effects on agriculture, water resources, infrastructure, and disaster risks. In this study, we use daily in situ precipitation observations from the global historical climatology network-daily (GHCN-D) to assess the ability of 21 Coupled Model Intercomparison Project Phase 6 (CMIP6) models, including the 50-member fifth-generation Canadian Earth System Model single model initial-condition large ensemble (CanESM5_SMILE), to realistically simulate historical interannual precipitation variability trends within 17 regions of the contiguous United States (CONUS). We assess how accurately the CMIP6 simulations align with observational data across annual, summer, and winter periods, focusing on four key hydrometeorological metrics, including interannual precipitation variability, relative interannual precipitation variability (coefficient of variation), annual mean precipitation, and annual wet day frequency. Our findings reveal that CMIP6 ensemble members generally reproduce the spatial patterns of observed trends in annual mean precipitation. In most regions, models agree well with the signs of observed changes in annual mean precipitation, though discrepancies in trend magnitude are evident. Further, observed trends in winter mean precipitation broadly exhibit a spatial pattern similar to that of the observed annual mean. However, analysis of the CanESM5_SMILE shows that trends in precipitation variability may primarily be the result of model-simulated internal variability, suggesting caution in interpreting multi-model single-realization ensemble results. Challenges in accurately simulating interannual precipitation variability underscore the need for ongoing model refinement and validation to enhance climate projections, especially in regions vulnerable to extreme precipitation events.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

A spatial evaluation of Arctic sea ice and regional limitations in CMIP6 historical simulations

The Arctic sea ice response to a warming climate is assessed in a subset of models participating in Phase 6 of the Coupled Model Intercomparison Project (CMIP6), using several metrics in comparison with satellite observations and results from the Pan-Arctic Ice Ocean Modeling and Assimilation System and the Regional Arctic System Model. Our study examines the historical representation of sea ice extent, volume, and thickness using spatial analysis metrics, such as the integrated ice-edge error, Brier score, and Spatial Probability Score. We find that the CMIP6 multi-model mean captures the mean annual cycle and 1979-2014 sea ice trends remarkably well. However, individual models experience a wide range of uncertainty in the spatial distribution of sea ice when compared against satellite measurements and reanalysis data. Our metrics expose common and individual regional model biases, which sea ice temporal analyses alone do not capture. We identify large ice edge and ice thickness errors in Arctic sub-regions, implying possible model specific limitations in or lack of representation of some key physical processes. We postulate that many of them could be related to the oceanic forcing, especially in the marginal and shelf seas, where seasonal sea ice changes are not adequately simulated. We therefore conclude that an individual model’s ability to represent the observed/reanalysis spatial distribution still remains a challenge. We propose the spatial analysis metrics as useful tools to diagnose model limitations, narrow down possible processes affecting them, and guide future model improvements critical to the representation and projections of Arctic climate change.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

A hierarchical evaluation framework for assessing climate simulations relevant to the energy-water-land nexus (Final Report)

The overarching goal was to construct a hierarchy of new and well-tested metrics and analysis tools that support both fundamental and use-inspired research. Motivating this goal was a convergence of needs from climate scientists and stakeholders alike for a systematic, robust framework of model evaluation and diagnosis to provide scientific insights, inform model development, support best practices for the use of climate model outputs, and facilitate communication of climate information in the evolving landscapes of multi-model, multi-resolution, and large ensemble simulations that generate terabytes of data for any single climate run. Steps toward attaining these goals benefited from expertise and established capability of the project team, which included leadership of the North American Regional Climate Change Assessment Program (NARCCAP) and the Coordinated Regional Downscaling Experiment (CORDEX), development of hierarchical model evaluation approaches, and successful research in the analysis and diagnosis of climate model skill, as well as the understanding and modeling of regional climate processes in North America. As part of the overarching goal, the project worked to disseminate a suite of methodologies, algorithms, and software components that the wider community can employ to advance climate science and applications. With rigorous demonstration, the evaluation framework and the mix of standard and high risk / high reward approaches helped form the basis for future development of a computationally enabled user-friendly system for community use.

17 WIND ENERGY↗
DOE OSTI2022

Water and Climate Impacts on ERCOT Long-Term Systems Assessment

Water and climate change pose many potential challenges to the electric power system. Substantial water is withdrawn every day to support thermoelectric power generating unit operations, and changes to water supply have the potential to affect generation dispatch. Climate change can accelerate growing demand for electricity, which can necessitate additional generating capacity, often in locations with limited water supply. Drought conditions also threaten thermoelectric power plant operations due to streamflow and reservoir levels dropping below intake structures, or water temperatures exceeding a power plants' permitted operating conditions. Here we explore how future climate change might influence decisions related to electricity capacity expansion planning in Electric Reliability Council of Texas (ERCOT) using a multi-model framework. Specifically, water resource modeling is used to simulate climate impacts on the future water supply for thermoelectric and hydropower generation for four future climate projections. Separately, temperature impacts on electricity load are evaluated for these scenarios. These climate impacts are applied to five alternative electricity futures in an electricity capacity expansion model that projects future generation and transmission capacity additions in ERCOT. Results indicate that climate has a measurable influence on future generation and transmission capacity needs, with temperature-driven increases in peak and average load resulting in 5-15 GW additional generating capacity and up to 1 GW additional transmission capacity. Additional capacity is a diverse mix of PV, natural gas, and wind, depending on the makeup of economic, policy and technology assumptions. Climate impacts increase total system costs 2-5%, while the marginal cost of energy and emissions are not affected substantially by climate change effects.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2026

Cooperative Agreement To Analyze variabiLity, change and predictabilitY in the earth SysTem (CATALYST)

CATALYST proposes to perform foundational coordinated research in a team-oriented collaborative effort aimed at advancing a robust understanding of modes of Earth system variability and change using models, observations and process studies. The proposed research will address the DOE/BER mission by exploring the limits to predictability, identifying fundamental underlying mechanisms, quantifying interactions among modes of variability, and discovering tipping points in the Earth system to understand the current and future impacts of these phenomena on regional and global climate. Four fundamental gaps are identified in our knowledge of the Earth system: 1) What are the limits to predictability on various timescales? 2) What are the interactions among modes of Earth system variability? 3) How may modes of Earth system variability change in response to changes in external forcing, and what are the tipping points involved with those changes? 4) How are high impact events connected to modes of Earth system variability and how may they change in the future? Related to those gaps in our knowledge, we formulate four research objectives to address those gaps using a combination of Earth system models (ESMs) and machine learning (ML) methods. Research Objective 1 (RO1) addresses the first gap above and proposes to understand modes of variability and their limits of predictability on subseasonal to decadal timescales using ESMs and ML. Research Objective 2 (RO2) addresses the second gap and proposes to use a hierarchy of models to understand relevant processes and feedbacks related to how modes of variability interact with each other. Research Objective 3 (RO3) is designed to study the third gap and proposes to examine the role of external forcings in changes of modes of Earth system variability and their interactions, and the likelihood and predictability of tipping points and irreversible changes. Research Objective 4 (RO4) will address the fourth gap and proposes to use high resolution ESMs, regionally refined models (RRMs), and ML methods to investigate the relationships between high impact events (e.g. flash droughts and precipitation extremes, atmospheric rivers (ARs), tropical cyclones (TCs), storm surge/sea level rise), the synoptic systems that produce them, and their changes related to modes of Earth system variability. The research will involve the use of the Community Earth System Model (CESM), Energy Exascale Earth System Model (E3SM), CMIP multi-model data sets, a hierarchy of simpler models, and numerous observational data sets. In the course of the proposed research, CATALYST will contribute to metrics and diagnostics that will be integrated in Coordinated Model Evaluation Capabilities (CMEC), particularly with regards to the Quasi-biennial Oscillation (QBO) and its interactions with the Madden-Julian Oscillation (MJO), high atmospheric pressure blocking, and new precipitation metrics.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

CMIP6-based Multi-model Hydropower Projection over the Conterminous US, Version 1.1

This dataset presents a suite of hydropower projections for the conterminous United States (CONUS), derived from multiple downscaled and bias-corrected Global Climate Models (GCMs) from the Coupled Model Intercomparison Project Phase 6 (CMIP6). The CMIP6 GCMs are downscaled using either statistical (DBCCA) or dynamical (RegCM) approaches, based on two meteorological reference datasets (Daymet and Livneh). The resulting downscaled precipitation, temperature, and wind speed data are then used to drive two calibrated hydrologic models (VIC and PRMS), enabling simulations of projected future hydrologic responses across the CONUS. Simulated total runoff is subsequently employed to drive two hydropower models (WMP, now implemented as mosartwmpy-power, and WRES) to evaluate how climate change may affect future hydropower production for both federal and non-federal hydropower fleets. This dataset was developed to support the SECURE Water Act Section 9505 Assessment for the U.S. Department of Energy (DOE) Water Power Technologies Office (WPTO). For further details, see Broman et al. (2024), Thurber et al. (2024), Kao et al. (2022), and Zhou et al. (2023).

Voisin, Nathalie [Pacific Northwest National Labor↗
DOE OSTI2021

Evaluating stratospheric ozone and water vapour changes in CMIP6 models from 1850 to 2100

Abstract. Stratospheric ozone and water vapour are key components of the Earth system, and past and future changes to both have important impacts on global and regional climate. Here, we evaluate long-term changes in these species from the pre-industrial period (1850) to the end of the 21st century in Coupled Model Intercomparison Project phase 6 (CMIP6) models under a range of future emissions scenarios. There is good agreement between the CMIP multi-model mean and observations for total column ozone (TCO), although there is substantial variation between the individual CMIP6 models. For the CMIP6 multi-model mean, global mean TCO has increased from ∼ 300 DU in 1850 to ∼ 305 DU in 1960, before rapidly declining in the 1970s and 1980s following the use and emission of halogenated ozone-depleting substances (ODSs). TCO is projected to return to 1960s values by the middle of the 21st century under the SSP2-4.5, SSP3-7.0, SSP4-3.4, SSP4-6.0, and SSP5-8.5 scenarios, and under the SSP3-7.0 and SSP5-8.5 scenarios TCO values are projected to be ∼ 10 DU higher than the 1960s values by 2100. However, under the SSP1-1.9 and SSP1-1.6 scenarios, TCO is not projected to return to the 1960s values despite reductions in halogenated ODSs due to decreases in tropospheric ozone mixing ratios. This global pattern is similar to regional patterns, except in the tropics where TCO under most scenarios is not projected to return to 1960s values, either through reductions in tropospheric ozone under SSP1-1.9 and SSP1-2.6, or through reductions in lower stratospheric ozone resulting from an acceleration of the Brewer–Dobson circulation under other Shared Socioeconomic Pathways (SSPs). In contrast to TCO, there is poorer agreement between the CMIP6 multi-model mean and observed lower stratospheric water vapour mixing ratios, with the CMIP6 multi-model mean underestimating observed water vapour mixing ratios by ∼ 0.5 ppmv at 70 hPa. CMIP6 multi-model mean stratospheric water vapour mixing ratios in the tropical lower stratosphere have increased by ∼ 0.5 ppmv from the pre-industrial to the present-day period and are projected to increase further by the end of the 21st century. The largest increases (∼ 2 ppmv) are simulated under the future scenarios with the highest assumed forcing pathway (e.g. SSP5-8.5). Tropical lower stratospheric water vapour, and to a lesser extent TCO, shows large variations following explosive volcanic eruptions.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Arctic tropospheric ozone: assessment of current knowledge and model performance

As the third most important greenhouse gas (GHG) after carbon dioxide (CO 2 ) and methane (CH 4 ), tropospheric ozone (O 3 ) is also an air pollutant causing damage to human health and ecosystems. This study brings together recent research on observations and modeling of tropospheric O 3 in the Arctic, a rapidly warming and sensitive environment. At different locations in the Arctic, the observed surface O 3 seasonal cycles are quite different. Coastal Arctic locations, for example, have a minimum in the springtime due to O 3 depletion events resulting from surface bromine chemistry. In contrast, other Arctic locations have a maximum in the spring. The 12 state-of-the-art models used in this study lack the surface halogen chemistry needed to simulate coastal Arctic surface O 3 depletion in the springtime; however, the multi-model median (MMM) has accurate seasonal cycles at non-coastal Arctic locations. There is a large amount of variability among models, which has been previously reported, and we show that there continues to be no convergence among models or improved accuracy in simulating tropospheric O 3 and its precursor species. The MMM underestimates Arctic surface O 3 by 5% to 15% depending on the location. The vertical distribution of tropospheric O 3 is studied from recent ozonesonde measurements and the models. The models are highly variable, simulating free-tropospheric O 3 within a range of ±50% depending on the model and the altitude. The MMM performs best, within ±8% for most locations and seasons. However, nearly all models overestimate O 3 near the tropopause (~300 hPa or ~8 km), likely due to ongoing issues with underestimating the altitude of the tropopause and excessive downward transport of stratospheric O 3 at high latitudes. For example, the MMM is biased high by about 20% at Eureka. Observed and simulated O 3 precursors (CO, NO x , and reservoir PAN) are evaluated throughout the troposphere. Models underestimate wintertime CO everywhere, likely due to a combination of underestimating CO emissions and possibly overestimating OH. Throughout the vertical profile (compared to aircraft measurements), the MMM underestimates both CO and NO x but overestimates PAN. Perhaps as a result of competing deficiencies, the MMM O 3 matches the observed O 3 reasonably well. Our findings suggest that despite model updates over the last decade, model results are as highly variable as ever and have not increased in accuracy for representing Arctic tropospheric O 3 .

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Evaluating uncertainty in aerosol forcing of tropical precipitation shifts

Abstract. An observed southward shift in tropical rainfall over land between 1950 and 1985, followed by a weaker recovery post-1985, has been attributed to anthropogenic aerosol radiative forcing and cooling of the Northern Hemisphere relative to the Southern Hemisphere. We might therefore expect models that have a strong historic hemispheric contrast in aerosol forcing to simulate a further northward tropical rainfall shift in the near-term future when anthropogenic aerosol emission reductions will predominantly warm the Northern Hemisphere. We investigate this paradigm using a perturbed parameter ensemble (PPE) of transient coupled ocean–atmosphere climate simulations that span a range of aerosol radiative forcing comparable to multi-model studies. In the 20th century, in our single-model ensemble, we find no relationship between the magnitude of pre-industrial to 1975 inter-hemispheric anthropogenic aerosol radiative forcing and tropical precipitation shifts. Instead, tropical precipitation shifts are associated with major volcanic eruptions and are strongly affected by internal variability. However, we do find a relationship between the magnitude of pre-industrial to 2005 inter-hemispheric anthropogenic aerosol radiative forcing and future tropical precipitation shifts over 2006 to 2060 under scenario RCP8.5. Our results suggest that projections of tropical precipitation shifts will be improved by reducing aerosol radiative forcing uncertainty, but predictive gains may be offset by temporary shifts in tropical precipitation caused by future major volcanic eruptions.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Systematic and objective evaluation of Earth system models: PCMDI Metrics Package (PMP) version 3

Systematic, routine, and comprehensive evaluation of Earth system models (ESMs) facilitates benchmarking improvement across model generations and identifying the strengths and weaknesses of different model configurations. By gauging the consistency between models and observations, this endeavor is becoming increasingly necessary to objectively synthesize the thousands of simulations contributed to the Coupled Model Intercomparison Project (CMIP) to date. The Program for Climate Model Diagnosis and Intercomparison (PCMDI) Metrics Package (PMP) is an open-source Python software package that provides quick-look objective comparisons of ESMs with one another and with observations. The comparisons include metrics of large- to global-scale climatologies, tropical inter-annual and intra-seasonal variability modes such as the El Niño–Southern Oscillation (ENSO) and Madden–Julian Oscillation (MJO), extratropical modes of variability, regional monsoons, cloud radiative feedbacks, and high-frequency characteristics of simulated precipitation, including its extremes. The PMP comparison results are produced using all model simulations contributed to CMIP6 and earlier CMIP phases. An important objective of the PMP is to document the performance of ESMs participating in the recent phases of CMIP, together with providing version-controlled information for all datasets, software packages, and analysis codes being used in the evaluation process. Among other purposes, this also enables modeling groups to assess performance changes during the ESM development cycle in the context of the error distribution of the multi-model ensemble. Quantitative model evaluation provided by the PMP can assist modelers in their development priorities. In this paper, we provide an overview of the PMP, including its latest capabilities, and discuss its future direction.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Evaluation of Global Fire Simulations in CMIP6 Earth System Models

Fire is the primary form of terrestrial ecosystem disturbance on a global scale and an important Earth system process. Most Earth system models (ESMs) have incorporated fire modeling, with 19 of them submitting model outputs of fire-related variables to the Coupled Model Intercomparison Project Phase 6 (CMIP6). This study provides the first comprehensive evaluation of CMIP6 historical fire simulations by comparing them with multiple satellite-based products and charcoal-based historical reconstructions. Our results show that most CMIP6 models simulate the present-day global burned area and fire carbon emissions within the range of satellite-based products. They also capture the major features of observed spatial patterns and seasonal cycles, the relationship of fires with precipitation and population density, and the influence of the El Niño–Southern Oscillation (ENSO) on the interannual variability of tropical fires. Regional fire carbon emissions simulated by the CMIP6 models from 1850 to 2010 generally align with the charcoal-based reconstructions, although there are regional mismatches, such as in southern South America and eastern temperate North America prior to the 1910s and in temperate North America, eastern boreal North America, Europe, and boreal Asia since the 1980s. The CMIP6 simulations have addressed three critical issues identified in CMIP5: (1) the simulated global burned area being less than half of that of the observations, (2) the failure to reproduce the high burned area fraction observed in Africa, and (3) the weak fire seasonal variability. Furthermore, the CMIP6 models exhibit improved accuracy in capturing the observed relationship between fires and both climatic and socioeconomic drivers and better align with the historical long-term trends indicated by charcoal-based reconstructions in most regions worldwide. However, the CMIP6 models still fail to reproduce the decline in global burned area and fire carbon emissions observed over the past 2 decades, mainly attributed to an underestimation of anthropogenic fire suppression, and the spring peak in fires in the Northern Hemisphere midlatitudes, mainly due to an underestimation of crop fires. In addition, the model underestimates the fire sensitivity to wet–dry conditions, indicating the need to improve fuel wet-ness estimation. Based on these findings, we present specific guidance for fire scheme development and suggest a postprocessing methodology for using CMIP6 multi-model outputs to generate reliable fire projection products.

Wildfire, Earth system models↗
DOE OSTI2025

Standardising the “Gregory method” for calculating equilibrium climate sensitivity

The equilibrium climate sensitivity (ECS) – the equilibrium global mean temperature response to a doubling of atmospheric CO 2 – is a high-profile metric for quantifying the Earth system's response to human-induced climate change. A widely applied approach to estimating the ECS is the “Gregory method” (Gregory et al., 2004), which uses an ordinary least squares (OLS) regression between the net radiative flux, N, and surface air temperature anomalies, ΔT, from a 150 year experiment in which atmospheric CO 2 concentrations are quadrupled. The ECS is determined by extrapolating the linear fit to N=0, i.e. the ΔT-intercept, indicating the point at which the system is back in equilibrium. This method has been used to compare ECS estimates across the CMIP5 and CMIP6 ensembles and will likely be a key diagnostic for CMIP7. Despite its widespread application, there is little consistency or transparency between studies in how the climate model data is processed prior to the regression, leading to potential discrepancies in ECS estimates. We identify 32 alternative data processing pathways, varying by differences in global mean weighting, net radiative flux variable, anomaly calculation method, and linear regression fit. Using 44 CMIP6 models, we systematically assess the impact of these choices on ECS estimates and calculate uncertainty ranges using two bootstrap approaches. While the inter-model ECS range is insensitive to the data processing pathway, individual outlier models exhibit notable differences. Approximating a model's native grid cell area (if irregular) with cosine of the latitude can decrease the ECS by 11 %, the choice of N-variable can change the ECS by 6 %, and some anomaly calculation methods can introduce spurious temporal correlations in the processed data. Beyond data processing choices, we also evaluate an alternative linear regression method – total least squares (TLS) – which has a more statistically robust basis than OLS. However, for consistency with previous literature, and given TLS may reduce the ECS compared to OLS (by up to 24 %), thereby making a known bias in the Gregory method worse, we do not feel there is sufficient clarity to recommend a transition to TLS in all cases. To improve reproducibility and comparability in future studies, we recommend a standardised Gregory method: weighting the global mean by cell area, using the top of the atmosphere (as opposed to the top of model) N-variable, and calculating anomalies by first applying a rolling average to the preindustrial control timeseries then subtracting from the raw CO 2 quadrupling experiment. This approach accounts for model drift while reducing noise in the data to best meet the pre-conditions of the linear regression. While CMIP6 results of the multi-model mean ECS appear insensitive to these processing choices, similar assumptions may not hold for CMIP7, underscoring the need for standardised data preparation in future climate sensitivity assessments.

Geosciences↗
DOE OSTI2026

New framework for benchmarking decadal predictions leveraging the PCMDI Metric Package with interactive visualization

Reliable climate predictions across multiple timescales are increasingly critical as climate-related risks continue to rise. With the growing number and diversity of climate prediction systems, systematic intercomparison has become essential. Here, we present a comprehensive evaluation framework based on the PCMDI Metric Package to assess the performance of multiple decadal climate prediction systems. Unlike uninitialized simulations, initialized predictions exhibit bias and predictive skill that evolve with forecast lead time. To address this, we introduce (1) model-by-lead-time portrait plots, which efficiently summarize metrics of global temperature, precipitation, and Arctic/Antarctic sea-ice extent, and (2) an HTML-based interactive visualization platform that provides detailed regional and seasonal diagnostics of model bias, skill scores, and ensemble spread for each model and lead time. Comparisons with uninitialized simulations further quantify the relative impacts of initialization and external forcing on prediction skill. The proposed framework provides a scalable and transparent approach for multi-model climate prediction assessments and can be readily extended to a wide range of operational and research forecasting systems.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

A process-based evaluation of biases in extratropical stratosphere–troposphere coupling in subseasonal forecast systems

Abstract. Two-way coupling between the stratosphere and troposphere is recognized as an important source of subseasonal-to-seasonal (S2S) predictability and can open windows of opportunity for improved forecasts. Model biases can, however, lead to a poor representation of such coupling processes; drifts in a model's circulation related to model biases, resolution, and parameterizations have the potential to feed back on the circulation and affect stratosphere–troposphere coupling. We introduce a set of diagnostics using readily available data that can be used to reveal these biases and then apply these diagnostics to 22 S2S forecast systems. In the Northern Hemisphere, nearly all S2S forecast systems underestimate the strength of the observed upward coupling from the troposphere to the stratosphere, downward coupling within the stratosphere, and the persistence of lower-stratospheric temperature anomalies. While downward coupling from the lower stratosphere to the near surface is well represented in the multi-model ensemble mean, there is substantial intermodel spread likely related to how well each model represents tropospheric stationary waves. In the Southern Hemisphere, the stratospheric vortex is oversensitive to upward-propagating wave flux in the forecast systems. Forecast systems generally overestimate the strength of downward coupling from the lower stratosphere to the troposphere, even as most underestimate the radiative persistence in the lower stratosphere. In both hemispheres, models with higher lids and a better representation of tropospheric quasi-stationary waves generally perform better at simulating these coupling processes.

Garfinkel, Chaim I. (ORCID:000000017258666X)↗
DOE OSTI2021

Relative Cost-Effectiveness of Electricity and Transportation Policies as a Means to Reduce CO2 Emissions in the United States: A Multi-Model Assessment

Two common energy policy instruments in the United States are tax incentives and technology standards. Although these instruments have been shown to be less cost-effective as a means to reduce CO2 emissions than direct emissions pricing mechanisms, it can be challenging to compare the CO2 emissions reduction costs of such policies across sectors, given the wide range in estimates for any given policy and inconsistencies in how such estimates are constructed across studies. This study addresses this analytical gap by simultaneously comparing the cost-effectiveness of policies across the electricity and transportation sectors using three publicly available US energy system models (EM-NEMS, ReEDS, and GCAM-USA). Four policies are explicitly compared: wind and solar tax credits, a renewable portfolio standard (RPS), a renewable fuel standard (RFS), and an electric vehicle (EV) tax credit. An economy-wide carbon tax is used as a benchmark for cost-effectiveness. Results from this study confirm prior insights about the cost-effectiveness of economy-wide carbon pricing relative to sectoral instruments but also reveal several novel insights about particular sectoral policies. Specifically, this study finds that (1) current electricity tax incentives provide uneven support for wind and solar technologies, (2) despite known inefficiencies, renewable energy policies in the electricity sector are less expensive than earlier estimates due to technology advancement and changes in market conditions, (3) within transportation, an expanded RFS with increasing advanced biofuel targets is more cost-effective than an EV tax credit extension under plausible assumptions, (4) EV incentives lead to a rebound in conventional vehicle fuel economy that further erodes cost-effectiveness, and (5) the change in policy costs over time is not known a priori, but the relative cost ordering among these policies does not depend on the timeframe of analysis. These results are largely robust to the underlying modeling framework, increasing the confidence with which they can be applied to climate policy evaluation.

economics↗
DOE OSTI2023

Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations

Background: Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. Methods: We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1–4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models’ predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models’ forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models’ past predictive performance. Results: Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models’ forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models’ forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models’ forecasts of deaths (N=763 predictions from 20 models). Across a 1–4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. Conclusions: Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks. Funding: AA, BH, BL, LWa, MMa, PP, SV funded by National Institutes of Health (NIH) Grant 1R01GM109718, NSF BIG DATA Grant IIS-1633028, NSF Grant No.: OAC-1916805, NSF Expeditions in Computing Grant CCF-1918656, CCF-1917819, NSF RAPID CNS-2028004, NSF RAPID OAC-2027541, US Centers for Disease Control and Prevention 75D30119C05935, a grant from Google, University of Virginia Strategic Investment Fund award number SIF160, Defense Threat Reduction Agency (DTRA) under Contract No. HDTRA1-19-D-0007, and respectively Virginia Dept of Health Grant VDH-21-501-0141, VDH-21-501-0143, VDH-21-501-0147, VDH-21-501-0145, VDH-21-501-0146, VDH-21-501-0142, VDH-21-501-0148. AF, AMa, GL funded by SMIGE - Modelli statistici inferenziali per governare l'epidemia, FISR 2020-Covid-19 I Fase, FISR2020IP-00156, Codice Progetto: PRJ-0695. AM, BK, FD, FR, JK, JN, JZ, KN, MG, MR, MS, RB funded by Ministry of Science and Higher Education of Poland with grant 28/WFSN/2021 to the University of Warsaw. BRe, CPe, JLAz funded by Ministerio de Sanidad/ISCIII. BT, PG funded by PERISCOPE European H2020 project, contract number 101016233. CP, DL, EA, MC, SA funded by European Commission - Directorate-General for Communications Networks, Content and Technology through the contract LC-01485746, and Ministerio de Ciencia, Innovacion y Universidades and FEDER, with the project PGC2018-095456-B-I00. DE., MGu funded by Spanish Ministry of Health / REACT-UE (FEDER). DO, GF, IMi, LC funded by Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20200700ER. DS, ELR, GG, NGR, NW, YW funded by National Institutes of General Medical Sciences (R35GM119582; the content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS or the National Institutes of Health). FB, FP funded by InPresa, Lombardy Region, Italy. HG, KS funded by European Centre for Disease Prevention and Control. IV funded by Agencia de Qualitat i Avaluacio Sanitaries de Catalunya (AQuAS) through contract 2021-021OE. JDe, SMo, VP funded by Netzwerk Universitatsmedizin (NUM) project egePan (01KX2021). JPB, SH, TH funded by Federal Ministry of Education and Research (BMBF; grant 05M18SIA). KH, MSc, YKh funded by Project SaxoCOV, funded by the German Free State of Saxony. Presentation of data, model results and simulations also funded by the NFDI4Health Task Force COVID-19 ( https://www.nfdi4health.de/task-force-covid-19-2 ) within the framework of a DFG-project (LO-342/17-1). LP, VE funded by Mathematical and Statistical modelling project (MUNI/A/1615/2020), Online platform for real-time monitoring, analysis and management of epidemic situations (MUNI/11/02202001/2020); VE also supported by RECETOX research infrastructure (Ministry of Education, Youth and Sports of the Czech Republic: LM2018121), the CETOCOEN EXCELLENCE (CZ.02.1.01/0.0/0.0/17-043/0009632), RECETOX RI project (CZ.02.1.01/0.0/0.0/16-013/0001761). NIB funded by Health Protection Research Unit (grant code NIHR200908). SAb, SF funded by Wellcome Trust (210758/Z/18/Z).

60 APPLIED LIFE SCIENCES↗