Search NASA⌕ Search

SEARCH · Search NASA

Results for “multi-model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Health and air pollutant emission impacts of net zero CO2 by 2050 scenarios from the energy modeling forum 37 study

Carbon dioxide and non-greenhouse gas air pollutants are emitted from many of the same sources. Decarbonization actions thus typically yield air pollutant emission reductions, resulting in significant air quality benefits. Although several studies have highlighted this connection, including in the context of net zero carbon emission targets, substantial uncertainty remains regarding how alternative technological pathways to this goal will affect the spatial distribution and magnitude of air pollutants. Comprehensive multi-model and multi-scenario analyzes are needed to explore the relative impacts of alternative pathways. Here, our study begins to address this gap by leveraging the results from the recent Energy Modeling Forum 37 inter-model comparison exercise on U.S. decarbonization pathways. Comparing the results of the six teams who submitted air pollutant emissions suggests that strategies that target net zero U.S. carbon emissions would yield significant reductions in many air pollutants, and that this finding is generally robust across pathways. However, some energy sources, such as biomass and fossil fuels with carbon capture, will emit air pollutants and can potentially influence the magnitude, spatial distribution, and even sign of localized air pollutant emission changes. In the second part of this analysis, a simplified air quality and health impacts screening model is used to evaluate the air quality impacts in 2035 of sectoral emission changes from the three models that provided sectoral detail. Relative to a reference scenario, a net zero pathway is estimated to reduce fine particulate matter concentrations across the contiguous U.S., with health benefits from reduced mortality ranging from $\$$65 billion to $\$$250 billion in 2035 alone (2023$\$$s). These benefits would be expected to grow over time as the net zero trajectory becomes more stringent. Both the magnitude of potential benefits and the substantial variation of the projections across models underscore the need for an EMF-like inter-model comparison exercise focused on air quality.

Air pollutants↗
DOE OSTI2022

Third international challenge to model the medium- to long-range transport of radioxenon to four Comprehensive Nuclear-Test-Ban Treaty monitoring stations

In 2015 and 2016, atmospheric transport modeling challenges were conducted in the context of the Comprehensive Nuclear-Test-Ban Treaty (CTBT) verification, however, with a more limited scope with respect to emission inventories, simulation period and number of relevant samples (i.e., those above the Minimum Detectable Concentration (MDC)) involved. Therefore, a more comprehensive atmospheric transport modeling challenge was organized in 2019. Stack release data of Xe-133 were provided by the Institut National des Radioéléments/IRE (Belgium) and the Canadian Nuclear Laboratories/CNL (Canada) and accounted for in the simulations over a three (mandatory) or six (optional) months period. Best estimate emissions of additional facilities (radiopharmaceutical production and nuclear research facilities, commercial reactors or relevant research reactors) of the Northern Hemisphere were included as well. Model results were compared with observed atmospheric activity concentrations at four International Monitoring System (IMS) stations located in Europe and North America with overall considerable influence of IRE and/or CNL emissions for evaluation of the participants’ runs. Participants were prompted to work with controlled and harmonized model set-ups to make runs more comparable, but also to increase diversity. It was found that using the stack emissions of IRE and CNL with daily resolution does not lead to better results than disaggregating annual emissions of these two facilities taken from the literature if an overall score for all stations covering all valid observed samples is considered. A moderate benefit of roughly 10% is visible in statistical scores for samples influenced by IRE and/or CNL to at least 50% and there can be considerable benefit for individual samples. Effects of transport errors, not properly characterized remaining emitters and long IMS sampling times (12–24 h) undoubtedly are in contrast to and reduce the benefit of high-quality IRE and CNL stack data. Complementary best estimates for remaining emitters push the scores up by 18% compared to just considering IRE and CNL emissions alone. Despite the efforts undertaken the full multi-model ensemble built is highly redundant. An ensemble based on a few arbitrary runs is sufficient to model the Xe-133 background at the stations investigated. The effective ensemble size is below five. An optimized ensemble at each station has on average slightly higher skill compared to the full ensemble. However, the improvement (maximum of 20% and minimum of 3% in RMSE) in skill is likely being too small for being exploited for an independent period.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Unraveling the 2021 Central Tennessee flood event using a hierarchical multi-model inundation modeling framework

Flood prediction systems need hierarchical atmospheric, hydrologic, and hydraulic models to predict rainfall, runoff, streamflow, and floodplain inundation. The accuracy of such systems depends on the error propagation through the modeling chain, sensitivity to input data, and choice of models. In this study, we used multiple precipitation forcings (hindcast and forecast) to drive hydrologic and hydrodynamic models to analyze the impacts of various drivers on the estimates of flood inundation depth and extent. We implement this framework to unravel the August 2021 extreme flooding event that occurred in Central Tennessee, USA. We used two radar-based quantitative precipitation estimates (STAGE4 and MRMS) as well as quantitative precipitation forecasts (QPF) from the National Weather Service Weather Prediction Center (WPC) to drive a series of models in the hierarchical framework, including the Variable Infiltration Capacity (VIC) land surface model, the Routing Application for Parallel Computation of Discharge (RAPID) river routing model, and the AutoRoute and TRITON inundation models. An evaluation with observed high-water marks demonstrates that the framework can reasonably simulate flood inundation. Despite the complex error propagation mechanism of the modeling chain, we show that inundation estimates are most sensitive to rainfall estimates. Most notably, QPF significantly underestimates flood magnitudes and inundations leading to unanticipated severe flooding for all stakeholders involved in the event. Finally, we discuss the implications of the hydrodynamic modeling framework for real-time flood forecasting.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Evaluating Multimodel Ensemble Seasonal Climate Forecasts on Rangeland Plant Production in the California Annual Grassland

Seasonal precipitation and temperature directly affect total plant production in the California Annual Grassland (CAG). Technological advances have resulted in skillful seasonal climate forecasts (i.e., significant correlations between actual and forecasted climate), which could be input into plant production models to inform stocking and other rangeland management decisions. This study presents a procedure for forecasting plant production in the CAG ecosystem to predict annual plant production for grazing, restoration, or other rangeland management practices using a combination of historical gridMET climate data and seasonal hindcasts (i.e., retrospective forecasts, from the North American Multi-Model Ensemble program). Further, the results of this study first confirmed high forecast skill, throughout the growing season at all sites. We also identified skillful plant production forecasts across most of the growing season at two sites and in three of the seven forecasting months at one study site. Forecasting climate and end-of-year plant production across the growing season at three CAG sites allowed us to identify the places and times in the growing season when forecasting might be most helpful in informing management decisions. Integrating plant production forecasting into rangeland management practices could significantly improve rangeland management outcomes. These procedures provide a user guide for creating plant production forecasts for any given area of interest and may be applicable across a wide range of other agricultural and rangeland management systems.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Performance evaluation of CMIP6 models on the Arctic-Siberian Plain teleconnection affecting the East Asian heat waves

The frequency and intensity of summer heat waves in East Asia have increased sharply in recent decades, significantly impacting public health and the economy. The Arctic-Siberian Plain (ASP) teleconnection pattern has been identified as a key driver, with ASP warming amplifying atmospheric circulation patterns conducive to extreme temperatures. This study evaluates the ability of Coupled Model Inter-comparison Project phase 6 models to simulate the ASP pattern across interannual variability (IAV) and intra-seasonal variability (ISV) timescales using the Common Basis Function method. The multi-model mean shows statistically significant pattern correlations with ERA5 reanalysis, with correlation coefficients of 0.90 and 0.99 for IAV and ISV, respectively. While the ASP pattern is generally well captured, models exhibit substantial inter-model diversity in the intensity and position of anticyclonic anomalies over the ASP and East Asia. Models with ASP pattern variability similar to reanalysis better reproduce extreme East Asian temperatures, whereas those over- or underestimating ASP variability exhibit lower skill. These performance differences are related to differences in simulating key variables associated with the development of the ASP pattern. Our findings highlight the role of the ASP pattern in modulating extreme heat events, as models with improved ASP simulations align more closely with observed temperature extremes. Refining ASP representations in models could enhance seasonal heat wave predictions, improving climate adaptation strategies.

Arctic-Siberian Plain (ASP)↗
DOE OSTI2022

National and subnational short-term forecasting of COVID-19 in Germany and Poland during early 2021

During the COVID-19 pandemic there has been a strong interest in forecasts of the short-term development of epidemiological indicators to inform decision makers. In this study we evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland for the period from January through April 2021. We evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland. These were issued by 15 different forecasting models, run by independent research teams. Moreover, we study the performance of combined ensemble forecasts. Evaluation of probabilistic forecasts is based on proper scoring rules, along with interval coverage proportions to assess calibration. The presented work is part of a pre-registered evaluation study. We find that many, though not all, models outperform a simple baseline model up to four weeks ahead for the considered targets. Ensemble methods show very good relative performance. The addressed time period is characterized by rather stable non-pharmaceutical interventions in both countries, making short-term predictions more straightforward than in previous periods. However, major trend changes in reported cases, like the rebound in cases due to the rise of the B.1.1.7 (Alpha) variant in March 2021, prove challenging to predict. Multi-model approaches can help to improve the performance of epidemiological forecasts. However, while death numbers can be predicted with some success based on current case and hospitalization data, predictability of case numbers remains low beyond quite short time horizons. Additional data sources including sequencing and mobility data, which were not extensively used in the present study, may help to improve performance.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2026

Diagnosing the representation of surface and layered soil moisture in Earth system models

Surface soil moisture (mrsos) and vertically integrated soil moisture (mrsol) over the top 10 cm should, by definition, be physically consistent in Earth System Models (ESMs). However, an evaluation of nine CMIP6 models reveals substantial inconsistencies: in some models, mrsos and integrated mrsol agree globally; in others, they align only in specific regions; and in a few, they diverge across all grid cells. These discrepancies arise from a combination of factors, including metadata errors, inconsistent variable definitions, or diagnostic sequencing within the model. We demonstrate how such issues can lead to significant biases, even when both variables are present and seemingly well-defined. As model complexity increases and multi-model comparisons become more common, assumptions about variable equivalence may lead to flawed conclusions. This study highlights the need for routine consistency checks, improved metadata standards, and community-wide practices that ensure reliability of derived variables across ESM outputs, particularly in preparation for CMIP7.

Earth system models↗
DOE OSTI2025

Evaluation of historical precipitation interannual variability in CMIP6 over the United States

Interannual precipitation variability profoundly influences society via its effects on agriculture, water resources, infrastructure, and disaster risks. In this study, we use daily in situ precipitation observations from the global historical climatology network-daily (GHCN-D) to assess the ability of 21 Coupled Model Intercomparison Project Phase 6 (CMIP6) models, including the 50-member fifth-generation Canadian Earth System Model single model initial-condition large ensemble (CanESM5_SMILE), to realistically simulate historical interannual precipitation variability trends within 17 regions of the contiguous United States (CONUS). We assess how accurately the CMIP6 simulations align with observational data across annual, summer, and winter periods, focusing on four key hydrometeorological metrics, including interannual precipitation variability, relative interannual precipitation variability (coefficient of variation), annual mean precipitation, and annual wet day frequency. Our findings reveal that CMIP6 ensemble members generally reproduce the spatial patterns of observed trends in annual mean precipitation. In most regions, models agree well with the signs of observed changes in annual mean precipitation, though discrepancies in trend magnitude are evident. Further, observed trends in winter mean precipitation broadly exhibit a spatial pattern similar to that of the observed annual mean. However, analysis of the CanESM5_SMILE shows that trends in precipitation variability may primarily be the result of model-simulated internal variability, suggesting caution in interpreting multi-model single-realization ensemble results. Challenges in accurately simulating interannual precipitation variability underscore the need for ongoing model refinement and validation to enhance climate projections, especially in regions vulnerable to extreme precipitation events.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Cooperative Agreement To Analyze variabiLity, change and predictabilitY in the earth SysTem (CATALYST)

CATALYST proposes to perform foundational coordinated research in a team-oriented collaborative effort aimed at advancing a robust understanding of modes of Earth system variability and change using models, observations and process studies. The proposed research will address the DOE/BER mission by exploring the limits to predictability, identifying fundamental underlying mechanisms, quantifying interactions among modes of variability, and discovering tipping points in the Earth system to understand the current and future impacts of these phenomena on regional and global climate. Four fundamental gaps are identified in our knowledge of the Earth system: 1) What are the limits to predictability on various timescales? 2) What are the interactions among modes of Earth system variability? 3) How may modes of Earth system variability change in response to changes in external forcing, and what are the tipping points involved with those changes? 4) How are high impact events connected to modes of Earth system variability and how may they change in the future? Related to those gaps in our knowledge, we formulate four research objectives to address those gaps using a combination of Earth system models (ESMs) and machine learning (ML) methods. Research Objective 1 (RO1) addresses the first gap above and proposes to understand modes of variability and their limits of predictability on subseasonal to decadal timescales using ESMs and ML. Research Objective 2 (RO2) addresses the second gap and proposes to use a hierarchy of models to understand relevant processes and feedbacks related to how modes of variability interact with each other. Research Objective 3 (RO3) is designed to study the third gap and proposes to examine the role of external forcings in changes of modes of Earth system variability and their interactions, and the likelihood and predictability of tipping points and irreversible changes. Research Objective 4 (RO4) will address the fourth gap and proposes to use high resolution ESMs, regionally refined models (RRMs), and ML methods to investigate the relationships between high impact events (e.g. flash droughts and precipitation extremes, atmospheric rivers (ARs), tropical cyclones (TCs), storm surge/sea level rise), the synoptic systems that produce them, and their changes related to modes of Earth system variability. The research will involve the use of the Community Earth System Model (CESM), Energy Exascale Earth System Model (E3SM), CMIP multi-model data sets, a hierarchy of simpler models, and numerous observational data sets. In the course of the proposed research, CATALYST will contribute to metrics and diagnostics that will be integrated in Coordinated Model Evaluation Capabilities (CMEC), particularly with regards to the Quasi-biennial Oscillation (QBO) and its interactions with the Madden-Julian Oscillation (MJO), high atmospheric pressure blocking, and new precipitation metrics.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

CMIP6-based Multi-model Hydropower Projection over the Conterminous US, Version 1.1

This dataset presents a suite of hydropower projections for the conterminous United States (CONUS), derived from multiple downscaled and bias-corrected Global Climate Models (GCMs) from the Coupled Model Intercomparison Project Phase 6 (CMIP6). The CMIP6 GCMs are downscaled using either statistical (DBCCA) or dynamical (RegCM) approaches, based on two meteorological reference datasets (Daymet and Livneh). The resulting downscaled precipitation, temperature, and wind speed data are then used to drive two calibrated hydrologic models (VIC and PRMS), enabling simulations of projected future hydrologic responses across the CONUS. Simulated total runoff is subsequently employed to drive two hydropower models (WMP, now implemented as mosartwmpy-power, and WRES) to evaluate how climate change may affect future hydropower production for both federal and non-federal hydropower fleets. This dataset was developed to support the SECURE Water Act Section 9505 Assessment for the U.S. Department of Energy (DOE) Water Power Technologies Office (WPTO). For further details, see Broman et al. (2024), Thurber et al. (2024), Kao et al. (2022), and Zhou et al. (2023).

Voisin, Nathalie [Pacific Northwest National Labor↗
DOE OSTI2023

Arctic tropospheric ozone: assessment of current knowledge and model performance

As the third most important greenhouse gas (GHG) after carbon dioxide (CO 2 ) and methane (CH 4 ), tropospheric ozone (O 3 ) is also an air pollutant causing damage to human health and ecosystems. This study brings together recent research on observations and modeling of tropospheric O 3 in the Arctic, a rapidly warming and sensitive environment. At different locations in the Arctic, the observed surface O 3 seasonal cycles are quite different. Coastal Arctic locations, for example, have a minimum in the springtime due to O 3 depletion events resulting from surface bromine chemistry. In contrast, other Arctic locations have a maximum in the spring. The 12 state-of-the-art models used in this study lack the surface halogen chemistry needed to simulate coastal Arctic surface O 3 depletion in the springtime; however, the multi-model median (MMM) has accurate seasonal cycles at non-coastal Arctic locations. There is a large amount of variability among models, which has been previously reported, and we show that there continues to be no convergence among models or improved accuracy in simulating tropospheric O 3 and its precursor species. The MMM underestimates Arctic surface O 3 by 5% to 15% depending on the location. The vertical distribution of tropospheric O 3 is studied from recent ozonesonde measurements and the models. The models are highly variable, simulating free-tropospheric O 3 within a range of ±50% depending on the model and the altitude. The MMM performs best, within ±8% for most locations and seasons. However, nearly all models overestimate O 3 near the tropopause (~300 hPa or ~8 km), likely due to ongoing issues with underestimating the altitude of the tropopause and excessive downward transport of stratospheric O 3 at high latitudes. For example, the MMM is biased high by about 20% at Eureka. Observed and simulated O 3 precursors (CO, NO x , and reservoir PAN) are evaluated throughout the troposphere. Models underestimate wintertime CO everywhere, likely due to a combination of underestimating CO emissions and possibly overestimating OH. Throughout the vertical profile (compared to aircraft measurements), the MMM underestimates both CO and NO x but overestimates PAN. Perhaps as a result of competing deficiencies, the MMM O 3 matches the observed O 3 reasonably well. Our findings suggest that despite model updates over the last decade, model results are as highly variable as ever and have not increased in accuracy for representing Arctic tropospheric O 3 .

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Evaluating uncertainty in aerosol forcing of tropical precipitation shifts

Abstract. An observed southward shift in tropical rainfall over land between 1950 and 1985, followed by a weaker recovery post-1985, has been attributed to anthropogenic aerosol radiative forcing and cooling of the Northern Hemisphere relative to the Southern Hemisphere. We might therefore expect models that have a strong historic hemispheric contrast in aerosol forcing to simulate a further northward tropical rainfall shift in the near-term future when anthropogenic aerosol emission reductions will predominantly warm the Northern Hemisphere. We investigate this paradigm using a perturbed parameter ensemble (PPE) of transient coupled ocean–atmosphere climate simulations that span a range of aerosol radiative forcing comparable to multi-model studies. In the 20th century, in our single-model ensemble, we find no relationship between the magnitude of pre-industrial to 1975 inter-hemispheric anthropogenic aerosol radiative forcing and tropical precipitation shifts. Instead, tropical precipitation shifts are associated with major volcanic eruptions and are strongly affected by internal variability. However, we do find a relationship between the magnitude of pre-industrial to 2005 inter-hemispheric anthropogenic aerosol radiative forcing and future tropical precipitation shifts over 2006 to 2060 under scenario RCP8.5. Our results suggest that projections of tropical precipitation shifts will be improved by reducing aerosol radiative forcing uncertainty, but predictive gains may be offset by temporary shifts in tropical precipitation caused by future major volcanic eruptions.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Systematic and objective evaluation of Earth system models: PCMDI Metrics Package (PMP) version 3

Systematic, routine, and comprehensive evaluation of Earth system models (ESMs) facilitates benchmarking improvement across model generations and identifying the strengths and weaknesses of different model configurations. By gauging the consistency between models and observations, this endeavor is becoming increasingly necessary to objectively synthesize the thousands of simulations contributed to the Coupled Model Intercomparison Project (CMIP) to date. The Program for Climate Model Diagnosis and Intercomparison (PCMDI) Metrics Package (PMP) is an open-source Python software package that provides quick-look objective comparisons of ESMs with one another and with observations. The comparisons include metrics of large- to global-scale climatologies, tropical inter-annual and intra-seasonal variability modes such as the El Niño–Southern Oscillation (ENSO) and Madden–Julian Oscillation (MJO), extratropical modes of variability, regional monsoons, cloud radiative feedbacks, and high-frequency characteristics of simulated precipitation, including its extremes. The PMP comparison results are produced using all model simulations contributed to CMIP6 and earlier CMIP phases. An important objective of the PMP is to document the performance of ESMs participating in the recent phases of CMIP, together with providing version-controlled information for all datasets, software packages, and analysis codes being used in the evaluation process. Among other purposes, this also enables modeling groups to assess performance changes during the ESM development cycle in the context of the error distribution of the multi-model ensemble. Quantitative model evaluation provided by the PMP can assist modelers in their development priorities. In this paper, we provide an overview of the PMP, including its latest capabilities, and discuss its future direction.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Evaluation of Global Fire Simulations in CMIP6 Earth System Models

Fire is the primary form of terrestrial ecosystem disturbance on a global scale and an important Earth system process. Most Earth system models (ESMs) have incorporated fire modeling, with 19 of them submitting model outputs of fire-related variables to the Coupled Model Intercomparison Project Phase 6 (CMIP6). This study provides the first comprehensive evaluation of CMIP6 historical fire simulations by comparing them with multiple satellite-based products and charcoal-based historical reconstructions. Our results show that most CMIP6 models simulate the present-day global burned area and fire carbon emissions within the range of satellite-based products. They also capture the major features of observed spatial patterns and seasonal cycles, the relationship of fires with precipitation and population density, and the influence of the El Niño–Southern Oscillation (ENSO) on the interannual variability of tropical fires. Regional fire carbon emissions simulated by the CMIP6 models from 1850 to 2010 generally align with the charcoal-based reconstructions, although there are regional mismatches, such as in southern South America and eastern temperate North America prior to the 1910s and in temperate North America, eastern boreal North America, Europe, and boreal Asia since the 1980s. The CMIP6 simulations have addressed three critical issues identified in CMIP5: (1) the simulated global burned area being less than half of that of the observations, (2) the failure to reproduce the high burned area fraction observed in Africa, and (3) the weak fire seasonal variability. Furthermore, the CMIP6 models exhibit improved accuracy in capturing the observed relationship between fires and both climatic and socioeconomic drivers and better align with the historical long-term trends indicated by charcoal-based reconstructions in most regions worldwide. However, the CMIP6 models still fail to reproduce the decline in global burned area and fire carbon emissions observed over the past 2 decades, mainly attributed to an underestimation of anthropogenic fire suppression, and the spring peak in fires in the Northern Hemisphere midlatitudes, mainly due to an underestimation of crop fires. In addition, the model underestimates the fire sensitivity to wet–dry conditions, indicating the need to improve fuel wet-ness estimation. Based on these findings, we present specific guidance for fire scheme development and suggest a postprocessing methodology for using CMIP6 multi-model outputs to generate reliable fire projection products.

Wildfire, Earth system models↗
DOE OSTI2025

Standardising the “Gregory method” for calculating equilibrium climate sensitivity

The equilibrium climate sensitivity (ECS) – the equilibrium global mean temperature response to a doubling of atmospheric CO 2 – is a high-profile metric for quantifying the Earth system's response to human-induced climate change. A widely applied approach to estimating the ECS is the “Gregory method” (Gregory et al., 2004), which uses an ordinary least squares (OLS) regression between the net radiative flux, N, and surface air temperature anomalies, ΔT, from a 150 year experiment in which atmospheric CO 2 concentrations are quadrupled. The ECS is determined by extrapolating the linear fit to N=0, i.e. the ΔT-intercept, indicating the point at which the system is back in equilibrium. This method has been used to compare ECS estimates across the CMIP5 and CMIP6 ensembles and will likely be a key diagnostic for CMIP7. Despite its widespread application, there is little consistency or transparency between studies in how the climate model data is processed prior to the regression, leading to potential discrepancies in ECS estimates. We identify 32 alternative data processing pathways, varying by differences in global mean weighting, net radiative flux variable, anomaly calculation method, and linear regression fit. Using 44 CMIP6 models, we systematically assess the impact of these choices on ECS estimates and calculate uncertainty ranges using two bootstrap approaches. While the inter-model ECS range is insensitive to the data processing pathway, individual outlier models exhibit notable differences. Approximating a model's native grid cell area (if irregular) with cosine of the latitude can decrease the ECS by 11 %, the choice of N-variable can change the ECS by 6 %, and some anomaly calculation methods can introduce spurious temporal correlations in the processed data. Beyond data processing choices, we also evaluate an alternative linear regression method – total least squares (TLS) – which has a more statistically robust basis than OLS. However, for consistency with previous literature, and given TLS may reduce the ECS compared to OLS (by up to 24 %), thereby making a known bias in the Gregory method worse, we do not feel there is sufficient clarity to recommend a transition to TLS in all cases. To improve reproducibility and comparability in future studies, we recommend a standardised Gregory method: weighting the global mean by cell area, using the top of the atmosphere (as opposed to the top of model) N-variable, and calculating anomalies by first applying a rolling average to the preindustrial control timeseries then subtracting from the raw CO 2 quadrupling experiment. This approach accounts for model drift while reducing noise in the data to best meet the pre-conditions of the linear regression. While CMIP6 results of the multi-model mean ECS appear insensitive to these processing choices, similar assumptions may not hold for CMIP7, underscoring the need for standardised data preparation in future climate sensitivity assessments.

Geosciences↗
DOE OSTI2026

New framework for benchmarking decadal predictions leveraging the PCMDI Metric Package with interactive visualization

Reliable climate predictions across multiple timescales are increasingly critical as climate-related risks continue to rise. With the growing number and diversity of climate prediction systems, systematic intercomparison has become essential. Here, we present a comprehensive evaluation framework based on the PCMDI Metric Package to assess the performance of multiple decadal climate prediction systems. Unlike uninitialized simulations, initialized predictions exhibit bias and predictive skill that evolve with forecast lead time. To address this, we introduce (1) model-by-lead-time portrait plots, which efficiently summarize metrics of global temperature, precipitation, and Arctic/Antarctic sea-ice extent, and (2) an HTML-based interactive visualization platform that provides detailed regional and seasonal diagnostics of model bias, skill scores, and ensemble spread for each model and lead time. Comparisons with uninitialized simulations further quantify the relative impacts of initialization and external forcing on prediction skill. The proposed framework provides a scalable and transparent approach for multi-model climate prediction assessments and can be readily extended to a wide range of operational and research forecasting systems.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

A process-based evaluation of biases in extratropical stratosphere–troposphere coupling in subseasonal forecast systems

Abstract. Two-way coupling between the stratosphere and troposphere is recognized as an important source of subseasonal-to-seasonal (S2S) predictability and can open windows of opportunity for improved forecasts. Model biases can, however, lead to a poor representation of such coupling processes; drifts in a model's circulation related to model biases, resolution, and parameterizations have the potential to feed back on the circulation and affect stratosphere–troposphere coupling. We introduce a set of diagnostics using readily available data that can be used to reveal these biases and then apply these diagnostics to 22 S2S forecast systems. In the Northern Hemisphere, nearly all S2S forecast systems underestimate the strength of the observed upward coupling from the troposphere to the stratosphere, downward coupling within the stratosphere, and the persistence of lower-stratospheric temperature anomalies. While downward coupling from the lower stratosphere to the near surface is well represented in the multi-model ensemble mean, there is substantial intermodel spread likely related to how well each model represents tropospheric stationary waves. In the Southern Hemisphere, the stratospheric vortex is oversensitive to upward-propagating wave flux in the forecast systems. Forecast systems generally overestimate the strength of downward coupling from the lower stratosphere to the troposphere, even as most underestimate the radiative persistence in the lower stratosphere. In both hemispheres, models with higher lids and a better representation of tropospheric quasi-stationary waves generally perform better at simulating these coupling processes.

Garfinkel, Chaim I. (ORCID:000000017258666X)↗
DOE OSTI2023

Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations

Background: Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. Methods: We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1–4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models’ predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models’ forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models’ past predictive performance. Results: Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models’ forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models’ forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models’ forecasts of deaths (N=763 predictions from 20 models). Across a 1–4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. Conclusions: Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks. Funding: AA, BH, BL, LWa, MMa, PP, SV funded by National Institutes of Health (NIH) Grant 1R01GM109718, NSF BIG DATA Grant IIS-1633028, NSF Grant No.: OAC-1916805, NSF Expeditions in Computing Grant CCF-1918656, CCF-1917819, NSF RAPID CNS-2028004, NSF RAPID OAC-2027541, US Centers for Disease Control and Prevention 75D30119C05935, a grant from Google, University of Virginia Strategic Investment Fund award number SIF160, Defense Threat Reduction Agency (DTRA) under Contract No. HDTRA1-19-D-0007, and respectively Virginia Dept of Health Grant VDH-21-501-0141, VDH-21-501-0143, VDH-21-501-0147, VDH-21-501-0145, VDH-21-501-0146, VDH-21-501-0142, VDH-21-501-0148. AF, AMa, GL funded by SMIGE - Modelli statistici inferenziali per governare l'epidemia, FISR 2020-Covid-19 I Fase, FISR2020IP-00156, Codice Progetto: PRJ-0695. AM, BK, FD, FR, JK, JN, JZ, KN, MG, MR, MS, RB funded by Ministry of Science and Higher Education of Poland with grant 28/WFSN/2021 to the University of Warsaw. BRe, CPe, JLAz funded by Ministerio de Sanidad/ISCIII. BT, PG funded by PERISCOPE European H2020 project, contract number 101016233. CP, DL, EA, MC, SA funded by European Commission - Directorate-General for Communications Networks, Content and Technology through the contract LC-01485746, and Ministerio de Ciencia, Innovacion y Universidades and FEDER, with the project PGC2018-095456-B-I00. DE., MGu funded by Spanish Ministry of Health / REACT-UE (FEDER). DO, GF, IMi, LC funded by Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20200700ER. DS, ELR, GG, NGR, NW, YW funded by National Institutes of General Medical Sciences (R35GM119582; the content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS or the National Institutes of Health). FB, FP funded by InPresa, Lombardy Region, Italy. HG, KS funded by European Centre for Disease Prevention and Control. IV funded by Agencia de Qualitat i Avaluacio Sanitaries de Catalunya (AQuAS) through contract 2021-021OE. JDe, SMo, VP funded by Netzwerk Universitatsmedizin (NUM) project egePan (01KX2021). JPB, SH, TH funded by Federal Ministry of Education and Research (BMBF; grant 05M18SIA). KH, MSc, YKh funded by Project SaxoCOV, funded by the German Free State of Saxony. Presentation of data, model results and simulations also funded by the NFDI4Health Task Force COVID-19 ( https://www.nfdi4health.de/task-force-covid-19-2 ) within the framework of a DFG-project (LO-342/17-1). LP, VE funded by Mathematical and Statistical modelling project (MUNI/A/1615/2020), Online platform for real-time monitoring, analysis and management of epidemic situations (MUNI/11/02202001/2020); VE also supported by RECETOX research infrastructure (Ministry of Education, Youth and Sports of the Czech Republic: LM2018121), the CETOCOEN EXCELLENCE (CZ.02.1.01/0.0/0.0/17-043/0009632), RECETOX RI project (CZ.02.1.01/0.0/0.0/16-013/0001761). NIB funded by Health Protection Research Unit (grant code NIHR200908). SAb, SF funded by Wellcome Trust (210758/Z/18/Z).

60 APPLIED LIFE SCIENCES↗