Search NASA⌕ Search

SEARCH · Search NASA

Results for “forecast evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Updating and Evaluating Anthropogenic Emissions for NOAA’s Global Ensemble Forecast Systems for Aerosols (GEFS-Aerosols): Application of an SO 2 Bias-Scaling Method

We updated the anthropogenic emissions inventory in NOAA’s operational Global Ensemble Forecast for Aerosols (GEFS-Aerosols) to improve the model’s prediction of aerosol optical depth (AOD). We used a methodology to quickly update the pivotal global anthropogenic sulfur dioxide (SO 2 ) emissions using a speciated AOD bias-scaling method. The AOD bias-scaling method is based on the latest model predictions compared to NASA’s Modern-Era Retrospective analysis for Research and Applications, version 2 (MERRA2). The model bias was subsequently applied to the CEDS 2019 SO 2 emissions for adjustment. The monthly mean GEFS-Aerosols AOD predictions were evaluated against a suite of satellite observations (e.g., MISR, VIIRS, and MODIS), ground-based AERONET observations, and the International Cooperative for Aerosol Prediction (ICAP) ensemble results. The results show that transitioning from CEDS 2014 to CEDS 2019 emissions data led to a significant improvement in the operational GEFS-Aerosols model performance, and applying the bias-scaled SO 2 emissions could further improve global AOD distributions. The biases of the simulated AODs against the observed AODs varied with observation type and seasons by a factor of 3~13 and 2~10, respectively. The global AOD distributions showed that the differences in the simulations against ICAP, MISR, VIIRS, and MODIS were the largest in March–May (MAM) and the smallest in December–February (DJF). When evaluating against the ground-truth AERONET data, the bias-scaling methods improved the global seasonal correlation (r), Index of Agreement (IOA), and mean biases, except for the MAM season, when the negative regional biases were exacerbated compared to the positive regional biases. The effect of bias-scaling had the most beneficial impact on model performance in the regions dominated by anthropogenic emissions, such as East Asia. However, it showed less improvement in other areas impacted by the greater relative transport of natural emissions sources, such as India. The accuracies of the reference observation or assimilation data for the adjusted inputs and the model physics for outputs, and the selection of regions with less seasonal emissions of natural aerosols determine the success of the bias-scaling methods. A companion study on emission scaling of anthropogenic absorbing aerosols needs further improved aerosol prediction.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Assessment of Cloud Mask Forecasts from the WRF-Solar Ensemble Prediction System: Preprint

Numerical weather prediction (NWP) models are important tools used by government agencies and the renewable energy enterprise to forecast solar radiation. Cloud prediction, a key process of NWP models, has the highest impact on the accuracy of solar forecasting. This study uses satellite observations from the National Solar Radiation Data Base (NSRDB) to evaluate the cloud mask forecast by the WRF-Solar Ensemble Prediction System (WRF-Solar EPS). Preliminary analysis of the data in 2018 demonstrates the need of further improvement in predicting thin and low-level clouds. The information obtained from this work will be used to enhance the WRF-Solar EPS in reproducing the cloud field over the contiguous U.S. and reducing solar forecasting errors.

41 EE - Solar Energy Technologies Office (EE-4S)↗
NASA NTRS2020

Evaluation of adjoint-based observation impacts as a function of forecast length using an Observing System Simulation Experiment

Adjoints of numerical weather prediction models may be employed for Forecast Sensitivity to Observation (FSO) in order to monitor the contribution of ingested observation data on short-term forecast skill. However, the calculation of short-term forecast error is difficult due to the lack of a truly independent dataset for verification. In an Observing System Simulation Experiment framework, the Nature Run is able to provide a true and complete verification dataset and allows accurate evaluation of short term forecast errors. In this work, an OSSE developed at the National Aeronautics and Space Administration Global Modeling and Assimilation Office is used to explore the impact of observational data on forecasts in the 6 to 48 hour range. An adjoint of the Global Earth Observing System model is employed to compare the observation impacts estimated using both self-analysis verification and the true Nature Run verification. Self-analysis verification is found to inflate the estimated forecast error growth during the early forecast period, resulting in overestimations of observation impacts, particularly in the 6-12 hour forecast range. By 48 hours, the self-analysis verification estimates of forecast error and observation impacts more closely match the true values. The fraction of beneficial observations is also overinflated at short forecast times when self-analysis verification is used. The progression of impacts of an individual observation or data type depends on the character of the growth of the initial condition error that each observation affects.

numerical weather prediction↗
NASA NTRS1993

Aviation Weather Program (AWP)

The Aviation Weather Program (AWP) combines additional weather observations, improved forecast technology, and more efficient distribution of information to pilots, controllers, and automated systems to improve the weather information provided to the air traffic control system, pilots, and other users of aviation weather information. Specific objectives include the needs to: improve airport and en-route capacity by accurate, high resolution, timely forecasts of changing weather conditions affecting airport and en-route operations; improve analyses and forecasts of upper-level winds for efficient flight planning and traffic management; and increase flight safety through improved aviation weather hazard forecasting (e.g. icing, turbulence, severe storms, microbursts, or strong winds). The AWP would benefit from participation in a cooperative multiscale experiment by obtaining data for: evaluation of aviation weather forecast products, analysis of four dimensional data assimilation schemes, and experimental techniques for retrieving aerosol and other visibility parameters. A multiscale experiment would also be helpful to AWP by making it possible to evaluate the added benefit of enhanced data sets collected during the experiment on those forecast and analysis products. The goals of the Coperative Multiscale Experiment (CME) are an essential step in attaining the long-term AWP objective of providing two-to-four hour location-specific forecasts of significant weather. Although the possibility of a funding role for the AWP in the CME is presently unclear, modest involvement of Federal Aviation Administration (FAA)/AWP personnel could be expected.

Foote, Brant↗
DOE OSTI2026

Subseasonal Forecasting and MJO Teleconnections in Machine Learning Weather Prediction Models

Abstract In recent years, machine‐learning (ML) models trained on reanalysis data have rivaled physics‐based forecast models in terms of performance skill for global weather forecasting. With increased rollout stability, the question of how these models perform for subseasonal to seasonal (S2S, week 3–8) forecasting has emerged. In this study we run a large set of subseasonal hindcasts over 2004–2023 to evaluate two ML weather forecast models at the S2S time scale, SFNO‐HENS (Nvidia, fully ML) and NeuralGCM (Google Research, hybrid). Corresponding hindcasts from the European Centre for Medium‐Range Weather Forecasts (ECMWF) are used as a baseline for comparison to a physics‐based model. Because our focus is on predicting moisture transport over the Western United States between October and March, we evaluate the models' prediction skill for the Madden‐Julian Oscillation (MJO) and its associated teleconnections in the North Pacific. We find that both ML models are competitive with the ECWMF model, with comparable skill in predicting the North Pacific large‐scale circulation and the MJO at week 3 and beyond. Even though overall the mid‐latitude subseasonal prediction skill remains low, the ML models exhibit interesting behavior such as a realistic propagation of the MJO across the Maritime Continent and realistic teleconnections. A SFNO‐HENS sensitivity experiment with altered initial conditions in the tropics demonstrates the stability of the model, and it illustrates the capability of ML models to represent important physical processes of the atmosphere at the S2S time scale. Plain Language Summary Predicting weather patterns and precipitation a few weeks in advance (subseasonal time scale) is of great interest for stakeholders such as water managers in the Southwest United States (US), where arid conditions prevail. Subseasonal forecasts from traditional weather forecast models exhibit low skill in the region, limiting their applicability. Here we examine whether the recent breakthrough in weather forecasting made with machine learning/artificial intelligence models can translate to improved subseasonal forecasts. Recently‐developed machine learning models exhibit comparable skill to a state‐of‐the‐art physics‐based model for predicting weather patterns in the North Pacific/North America region, and associated moisture transport. The same applies to their skill in predicting the tropical pattern, the Madden‐Julian Oscillation, and its important remote perturbations over the midlatitude East Pacific and Southwest US. Additionally, a perturbation experiment carried out with one of the machine learning models illustrates their ability to not only predict the evolution of atmospheric fields, but also to learn and represent physical processes such as tropics‐extratropics Rossby wave propagation. Key Points Two machine learning weather forecast models exhibit state‐of‐the‐art prediction skill at the subseasonal time scale in the Pacific sector The models equal ECWMF in terms of Madden‐Julian oscillation (MJO) prediction skill, and they accurately predict the MJO propagation and associated teleconnections The two machine‐learning models represent key physical processes for subseasonal prediction, despite being trained for weather forecasting

Peings, Yannick↗
NASA NTRS2011

Evaluating the Impact of AIRS Observations on Regional Forecasts at the SPoRT Center

NASA Short-term Prediction Research and Transition (SPoRT) Center collaborates with operational partners of different sizes and operational goals to improve forecasts using targeted projects and data sets. Modeling and DA activities focus on demonstrating utility of NASA data sets and capabilities within operational systems. SPoRT has successfully assimilated the Atmospheric Infrared Sounder (AIRS) radiance and profile data. A collaborative project is underway with the Joint Center for Satellite Data Assimilation (JCSDA) to use AIRS profiles to better understand the impact of AIRS radiances assimilated within Gridpoint Statistical Interpolation (GSI) in hopes of engaging the operational DA community in a reassessment of assimilation methodologies to more effectively assimilate hyperspectral radiances.

Zavodsky, Bradley↗
NASA NTRS2016

IN31A-1734 Development and Evaluation of a Gridded CrIS/ATMS Visualization for Operational Forecasting

A collaborative effort between SPoRT, CIMSS, CIRA, GINA, and NOAA has produced a unique gridded visualization of real-time CrIS/ATMS sounding products. This product uses the NUCAPS retrieval algorithm and polar2grid software to generate plan-view and cross-section visualization for forecast challenges associated with cold air aloft and convective potential. Forecasters at select partner offices have been able to view the Gridded NUCAPS products in AWIPS alongside other operational data products with generally favorable feedback.

aviation↗
NASA NTRS2024

Northern Brazil Agriculture: Measuring Soybean Yields in Northern Brazil During El Niño-Southern Oscillation Conditions to Evaluate Trends in Agricultural Production and Support Crop Forecasting, 1984 – 2023

As one of the largest agricultural exporters in the world, Brazil’s crop productivity is highly influential to the world’s food supply. Crop productivity across Brazil is heavily influenced by climatic factors, such as the El Niño-Southern Oscillation (ENSO), which drives spatially heterogenous effects on growing conditions. To understand the relationship between ENSO conditions and staple crop productivity, the team used 1984 – 2023 climatic data and vegetation indices for four Brazilian states: Bahia, Mato Grosso, Pará, and Tocantins. The team focused on soybean growing areas in each state derived from annual land use/land cover classifications. Monthly mean Normalized Difference Vegetation Index (NDVI) per state were calculated using multispectral imagery from the Landsat 5 Thematic Mapper (TM), Landsat 7 Enhanced Thematic Mapper Plus (ETM+), and Landsat 8 Operational Land Imagery (OLI) sensors. In addition, the team used crop production indices, ENSO anomalies, temperature, and precipitation data in this study. The team found that although monthly ENSO anomaly did have a positive relationship between temperature and a slight negative relationship with precipitation, ENSO and monthly soy NDVI were not significantly correlated. Results find no association between ENSO conditions and NDVI in the study area at the state level spatial resolution during the study period. Furthermore, the team found that there was no correlation between cumulative growing season soy NDVI and detrended soy production yield for this study region at the spatial and temporal scale of the analysis. Findings from this project will be used by the United States Department of Agriculture’s Foreign Agriculture Service to inform crop productivity forecasting.

remote sensing↗
DOE OSTI2020

Evaluation of the Four-Dimensional Ensemble-Variational Hybrid Data Assimilation with Self-Consistent Regional Background Error Covariance for Improved Hurricane Intensity Forecasts

The feasibility of a hurricane initialization framework based on the Gridpoint Statistical Interpolation (GSI)-based four-dimensional ensemble-variational (GSI-4DEnVar) hybrid data assimilation system for the Hurricane Weather Research and Forecasting model (HWRF) model is evaluated in this study. The system considers the temporal evolution of error covariances via the use of four-dimensional ensemble perturbations that are provided by high-resolution, self-consistent HWRF ensemble forecasts. It is different from the configuration of the GSI-based three-dimensional ensemble-variational (GSI-3DEnVar) hybrid data assimilation system, similar to that used in the operational HWRF, which employs background error covariances provided by coarser-resolution global ensembles from the National Centers for Environmental Prediction (NCEP) Global Forecast System (GFS) ensemble Kalman filtering data assimilation system. In addition, our proposed initialization framework discards the empirical intensity correction in the vortex initialization package that is employed by the GSI-3DEnVar initialization framework in operational HWRF. Data assimilation and numerical simulation experiments for Hurricanes Joaquin (2015), Patricia (2015), and Matthew (2016) are conducted during their intensity changes. The impacts of two initialization frameworks on the HWRF analyses and forecasts are compared. It is found that GSI-4DEnVar leads to a reduction in track, minimum sea level pressure (MSLP), and maximum surface wind (MSW) forecast errors in all of the HWRF simulations, compared with the GSI-3DEnVar initialization framework. With assimilating high-resolution observations within the hurricane inner-core region, GSI-4DEnVar can produce the initial hurricane intensity reasonably well without the empirical vortex intensity correction. Further diagnoses with Hurricane Joaquin indicate that GSI-4DEnVar can significantly alleviate the imbalances in the initial conditions and enhance the performance of the data assimilation and subsequent hurricane intensity and precipitation forecasts

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Analysis of Building Model Forecasts using Autonomous HVAC Optimization System for Residential Neighborhood

Heating, ventilation, and air conditioning (HVAC) systems account for the highest share of home energy consumption in the United States. Optimized HVAC control can provide thermal improved comfort to the occupants, improve energy efficiency, reduce energy cost, and support grid services. In this paper, we discuss a multi-agent and cloud-based software framework that has been deployed in occupied residential neighborhood. This system enables automatic data collection, learning, optimization, and dispatches signals to neighborhood devices. HVAC optimization is based on model predictive control (MPC). Since the operational performance of MPC depends on model forecasting accuracy, it is crucial to evaluate the model continuously and modify or retrain it as necessary. In this research, we developed an automated workflow to evaluate the performance of temperature and power forecasts based on measured data in the real world. This will provide researchers with a deeper understanding of the model and how it can be improved.

Lebakula, Viswadeep↗
DOE OSTI2025

Performance Evaluation of Weather@home2 Simulations over West African Region

Weather and climate forecasting, using climate models, have become essential tools and life-savers in the West African region; in spite of the fact that climate models do not fully comply with attributes of forecast qualities—RASAP: reliability, association, skill, accuracy, and precision. The objective of this paper is to quantitatively evaluate, in comparison to CRU and ERA5 datasets, the RASAP compliance-level of the weather@home2 modeling system (w@h2). Findings from some statistical evaluations show that, to a moderately significant extent, w@h2 model provides useful information during the monsoon seasons; skills to capture the Little Dry Season over the Guinea zone; predictive skills for the onset season; ability to reproduce all the annual characteristics of the surface maximum air temperature over the region; as well as skill to detect heat waves that usually ravage West Africa during the boreal spring. The model displays traces of attributes that are needed for seasonal climate predictions and applications. Deficiencies in the quantitative reproducibility point to the facts that the model does provide a reliability akin to that of regional climate models. This paper further furnishes a prospective user with information on whether the model might be “useful or not” for a particular application.

West Africa↗
NASA NTRS1993

Session on validation of coupled models

The session on validation of coupled models is reviewed. The current use of a mesoscale model with a grid size of 20-km during STORM-FEST in 1992 has proven to be extremely valuable. The availability of forecast products at a much higher temporal and spatial resolution was very helpful for mesoscale forecasting, mission planning, and the guidance of research aircraft. Recent numerical simulation of ocean cyclones and mesoscsle convective systems using nonhydrostatic cloud/mesoscale models with a grid size as small as 2-km have demonstrated the potential of these models for predicting mesoscale convective systems, squall lines, hurricane rainbands, mesoscale gravity waves, and mesoscale frontal structures embedded within an extratropical cyclone. Although mesoscale/cloud scale models have demonstrated strong potential for use in operational forecasting, very limited quantitative evaluation (and verification) of these models were performed. As a result, the accuracy, the systematic biases, and the useful forecasts limits were not properly defined for these models. Also, no serious attempts were made to use these models for operational prediction of mesoscale convective systems.

Kuo, Bill↗
DOE OSTI2022

Operational Forecasting of Induced Seismicity

Operational Forecasting of Induced Seismicity (ORION) is an open-source toolkit developed as part of the US Department of Energy SMART and NRAP programs. ORION is designed to help both expert and non-expert users understand the seismic hazard at a site. This tool analyzes how the seismic hazard evolves during injection of fluid, and can be used to identify potential mitigation strategies for observed or forecasted seismic activity. ORION is built using Python, and has an optional GUI where users can easily configure model parameters, evaluate temporal and spatial forecasts, and inspect key data (seismicity, fluid injection, pressure models, etc.).

Kroll, Kayla↗
DOE OSTI2023

Machine learning models inaccurately predict current and future high-latitude C balances

The high-latitude carbon (C) cycle is a key feedback to the global climate system, yet because of system complexity and data limitations, there is currently disagreement over whether the region is a source or sink of C. Recent advances in big data analytics and computing power have popularized the use of machine learning (ML) algorithms to upscale site measurements of ecosystem processes, and in some cases forecast the response of these processes to climate change. Due to data limitations, however, ML model predictions of these processes are almost never validated with independent datasets. To better understand and characterize the limitations of these methods, we develop an approach to independently evaluate ML upscaling and forecasting. We mimic data-driven upscaling and forecasting efforts by applying ML algorithms to different subsets of regional process-model simulation gridcells, and then test ML performance using the remaining gridcells. In this study, we simulate C fluxes and environmental data across Alaska using ecosys, a process-rich terrestrial ecosystem model, and then apply boosted regression tree ML algorithms to training data configurations that mirror and expand upon existing AmeriFLUX eddy-covariance data availability. We first show that a ML model trained using ecosys outputs from currently-available Alaska AmeriFLUX sites incorrectly predicts that Alaska is presently a modeled net C source. Increased spatial coverage of the training dataset improves ML predictions, halving the bias when 240 modeled sites are used instead of 15. However, even this more accurate ML model incorrectly predicts Alaska C fluxes under 21st century climate change because of changes in atmospheric CO 2 , litter inputs, and vegetation composition that have impacts on C fluxes which cannot be inferred from the training data. Our results provide key insights to future C flux upscaling efforts and expose the potential for inaccurate ML upscaling and forecasting of high-latitude C cycle dynamics.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Influence of Local Water Vapor Analysis Uncertainty on Ensemble Forecasts of Tropical Cyclogenesis Using Hurricane Irma (2017) as a Testbed

Abstract Tropical cyclone formation is known to require abundant water vapor in the lower to middle troposphere within the incipient disturbance. In this study, we assess the impacts of local water vapor analysis uncertainty on the predictability of the formation of Hurricane Irma (2017). To this end, we reduce the magnitude of the incipient disturbance’s water vapor perturbations obtained from an ensemble-based data assimilation system that constrained moisture by assimilating all-sky infrared and microwave radiances. Five-day ensemble forecasts are initialized two days before genesis using each set of modified analysis perturbations. Growth of convective differences and intensity uncertainty are evaluated for each ensemble forecast. We observe that when initializing an ensemble forecast with only moisture uncertainty within the incipient disturbance, the resulting intensity uncertainty at every lead time exceeds half that of an ensemble containing initial perturbations to all variables throughout the domain. Although ensembles with different initial moisture uncertainty amplitudes reveal a similar pathway to genesis, uncertainty in genesis timing varies substantially across ensembles since moister members exhibit earlier spinup of the low-level vortex. These differences in genesis timing are traced back to the first 6–12 h of integration, when differences in the position and intensity of mesoscale convective systems across ensemble members develop more quickly with greater initial moisture uncertainty. In addition, the rapid growth of intensity uncertainty may be greatly modulated by the diurnal cycle. Ultimately, this study underscores the importance of targeting the incipient disturbance with high spatiotemporal water vapor observations for ingestion into data assimilation systems. Significance Statement Hurricanes form from clusters of thunderstorms that organize into a coherent system. One of the key ingredients for the formation process is an abundance of moisture. In this study, we test the sensitivity of hurricane formation to the initial moisture content in the vicinity of the cluster of thunderstorms that would become Hurricane Irma (2017). To do so, we initialize sets of forecasts each having a different variability of initial moisture content within the embryonic disturbance. Our results show that the predictability of hurricane formation is highly dependent on the uncertainty of the moisture content within the initial disturbance. Consequently, more high-quality observations of the moisture within the precursor disturbances to hurricanes are expected to improve forecasts of their formation.

Hartman, Christopher M.↗
DOE OSTI2026

3D Cloud Aerosol Precipitation Experiment at kennaook Cape Grim (3D-CAPE-k) Field Campaign Report

The objective of this campaign was to complement the U.S. Department of Energy Atmospheric Radiation Measurement (ARM) User Facility’s Cloud And Precipitation Experiment at kennaook (CAPE-k) in northwest Tasmania by deploying scanning, fixed pointing angle, and in situ aerosol, cloud, and precipitation remote-sensing instruments during the last six months of the experiment. These instruments provided a three-dimensional (3D) context to the second ARM Mobile Facility (AMF2) vertically pointing observations collected at the kennaook Cape Grim (KCG) site and captured a portion of the life cycle of the clouds before and after the passage over the AMF2 vertical column. These 3D cloud measurements will be used in the near future to 1) investigate how the vertical profiles of aerosol and cloud properties are representative of the broader area and how these properties evolve during the portion of the cloud life cycle captured by the scanning instruments, 2) quantify how accurate the cloud fraction derived with assumptions from vertically pointing observations are compared with direct 3D cloud fraction measurements, 3) evaluate cloud fraction and liquid water path in the Australian Community Climate and Earth-System Simulator–Convective-scale (ACCESS-C) forecast model, and 4) evaluate aerosol and cloud products from the European Space Agency/Japanese Aerospace Exploration Agency (ESA/JAXA) Earth Cloud Aerosol Radiation Explorer (EarthCARE) mission using the scanning measurements, offering more chances of exact collocation, thereby complementing the statistical approach we are planning to employ with AMF2 observations for that same purpose.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Short-term wind forecasting using statistical models with a fully observable wind flow

The utility of model output data from the Weather Research and Forecasting mesoscale model is explored for very short-term forecasting (5-30 minutes horizon) of wind speed to be used in large scale simulations of an autonomous electric power grid. Using this synthetic data for the development and evaluation of short-term forecasting algorithms offer many unique advantages over observational data, such as the ability to observe the full wind flow field in the surrounding region. Several short-term forecasting algorithms are implemented and evaluated using the synthetic data at several different time horizons and for three different geographic locations. Comparison is made with observational data from one location. We find that short-term forecasts of the synthetic data considering wind flow from the surrounding region perform 26% better than persistence in terms of root mean square error at the 5-minute time horizon. This improvement is comparable to studies of observational data in the literature. These results provide motivation to use synthetic data for short term forecasting in grid simulations, and open the door to future algorithmic improvements.

17 WIND ENERGY↗