Public health applications of historical smoke forecasts: An evaluation of archived BlueSky data for the coterminous United States, 2015–2018
Not Available
54 ENVIRONMENTAL SCIENCES↗
SEARCH · Search NASA
Results for “forecast evaluation”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Impact of Building Characteristics on Energy Demand Profile Forecasting: An Evaluation of Predictive Accuracy and Methods
Explore the source record for details and available documents.
Chowdhury, Shovan [ORNL]↗
Short-Term Load Forecasting Considering EV Charging Loads with Prediction Interval Evaluation
Short-term load forecasting plays a critical role in power system planning and operation. Along with the electrification of various loads, electricity demands are becoming increasingly hard to predict. Notably, the recent rise in electric vehicles (EVs) has further contributed to this unpredictability. To address this issue, this paper proposes a probabilistic load forecasting strategy utilizing Gaussian process regression, structured in a day-ahead manner. While many works focus on deterministic prediction, probabilistic forecasting offers additional insights into variability and uncertainty, enabling more flexible and reliable operation for power systems. To enhance the accuracy of the load forecasting model, the inputs include features related to EV charging habits as well as commonly used weather information. The load forecasting results are evaluated using various metrics, including conventional ones that assess the accuracy of point forecasts, as well as additional metrics that test the reliability of prediction intervals. The proposed load forecasting method is finally tested on real residential power consumption data and EV charging data sampled from real-world sources. The results prove that the new features can greatly improve the performance of the load forecasting method.
electrical vehicle↗
Open Source Evaluation Framework for Solar Forecasting
The Solar Forecast Arbiter is an open-source evaluation framework for solar forecasting. The framework enables evaluations of solar irradiance, solar power, and net-load forecasts that are impartial, repeatable and auditable. The Solar Forecast Arbiter addresses stakeholder-informed use cases including evaluation of forecast skill, comparisons to reference data sets, private forecast trials, and evaluation of probabilistic forecast skill. The framework includes a data validation toolkit, reference data sources, data privacy protocols, and benchmark forecast capabilities for intra-hour and day ahead forecast horizons. Reports and metrics communicate the relative merits of the test and benchmark forecasts. The reports are created from standardized templates and include graphics for qualitatively evaluating deterministic and probabilistic forecasts and standard metrics for quantitatively evaluating forecasts. The Solar Forecast Arbiter is designed to support all solar forecasting stakeholders, including Solar Forecasting 2 Topic Area 2 and Topic Area 3 teams.
14 SOLAR ENERGY↗
Day-Ahead Probabilistic Forecasting of Net-Load and Demand Response Potentials with High Penetration of Behind-the-Meter Solar-plus-Storage
The goal of this project is to develop advanced methods for day-ahead net-load forecasting, by leveraging the state-of-the-art machine learning techniques. The developed models produce both point and probabilistic forecasts for a variety of use cases, and are versatile to work with different types of data sets. The innovation lies in the novel design of the architectures, leveraging the most recent advances in machine learning that have not been explored in power systems, accompanied by techniques in the broader artificial intelligence fields such as fuzzy systems. This project has achieved the following accomplishments: (1) preprocessing of over 10 data sets covering varying geographical regions, time horizons, and system levels, which form a robust foundation for training and evaluating forecasting models across a wide range of realistic grid scenarios; (2) development of an interactive web app that enables exploratory analysis of load and generation data, and supports better understanding of data trends, anomalies, and correlations, facilitating model development and stakeholder engagement; (3) implementation of over 10 benchmark models for point and probabilistic forecasting, which include a mix of conventional machine learning methods and state-of-the-art deep learning approaches, providing a comprehensive baseline for performance comparison and validation of the proposed models; (4) development of a fuzzy system based gradient boosting model, tailored for small (less than 3 years) data sets, which achieves a mean absolute percentage error (MAPE) of 4% for point forecasting and a 20% improvement in average pinball loss for probabilistic forecasting; (5) development of a Transformer (a state-of-the-art deep learning architecture) based neural network model, tailored for large (3 years or more) data sets, which achieves a MAPE of 2% for point forecasting and a 20% improvement in average pinball loss for probabilistic forecasting; (6) development of a methodology for quantifying DR potential, and extensions of the previous models for multi-target forecasting of net load and DR potential, which achieve a MAPE of 10% for DR potential.
24 POWER TRANSMISSION AND DISTRIBUTION↗
On the predictability of turbulent fluxes from land: PLUMBER2 MIP experimental description and preliminary results
Accurate representation of the turbulent exchange of carbon, water, and heat between the land surface and the atmosphere is critical for modelling global energy, water, and carbon cycles in both future climate projections and weather forecasts. Evaluation of models' ability to do this is performed in a wide range of simulation environments, often without explicit consideration of the degree of observational constraint or uncertainty and typically without quantification of benchmark performance expectations. We describe a Model Intercomparison Project (MIP) that attempts to resolve these shortcomings, comparing the surface turbulent heat flux predictions of around 20 different land models provided with in situ meteorological forcing evaluated with measured surface fluxes using quality-controlled data from 170 eddy-covariance-based flux tower sites. Predictions from seven out-of-sample empirical models are used to quantify the information available to land models in their forcing data and so the potential for land model performance improvement. Sites with unusual behaviour, complicated processes, poor data quality, or uncommon flux magnitude are more difficult to predict for both mechanistic and empirical models, providing a means of fairer assessment of land model performance. When examining observational uncertainty, model performance does not appear to improve in low-turbulence periods or with energy-balance-corrected flux tower data, and indeed some results raise questions about whether the energy balance correction process itself is appropriate. In all cases the results are broadly consistent, with simple out-of-sample empirical models, including linear regression, comfortably outperforming mechanistic land models. In all but two cases, latent heat flux and net ecosystem exchange of CO 2 are better predicted by land models than sensible heat flux, despite it seeming to have fewer physical controlling processes. Land models that are implemented in Earth system models also appear to perform notably better than stand-alone ecosystem (including demographic) models, at least in terms of the fluxes examined here. The approach we outline enables isolation of the locations and conditions under which model developers can know that a land model can improve, allowing information pathways and discrete parameterisations in models to be identified and targeted for future model development.
54 ENVIRONMENTAL SCIENCES↗
Event Log / Raw Data
The WFIP3 event log is a curated record spanning 578 days of meteorological phenomena and field observations that complements the campaign’s high-frequency measurements. The log combines manually documented daily weather discussions with automatically derived indicators of key atmospheric processes, providing standardized, publicly available context to support model evaluation, forecast verification, and case-study selection for offshore boundary-layer research.
17 WIND ENERGY↗
Evaluation of FluSight influenza forecasting in the 2021–22 and 2022–23 seasons with a new target laboratory-confirmed influenza hospitalizations
Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. For the 2021–22 and 2022–23 influenza seasons, 26 forecasting teams provided national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one-to-four weeks ahead. Forecast skill is evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperform the baseline model across forecast weeks and locations in 2021–22 and 12 out of 18 models in 2022–23. Averaging across all forecast targets, the FluSight ensemble is the 2nd most accurate model measured by WIS in 2021–22 and the 5th most accurate in the 2022–23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degrade over longer forecast horizons. In this work we demonstrate that while the FluSight ensemble was a robust predictor, even ensembles face challenges during periods of rapid change.
59 BASIC BIOLOGICAL SCIENCES↗
Enduring Transuranic Liquid Waste Volume Forecast and Constituent Evaluation (FY20)
The “FY11 Radioactive Liquid Waste (RLW) Influent Boundary Condition Assessment and Forecast” report provided influent volume forecasts that were critical in establishing the design capacity for the new Radioactive Liquid Waste Transuranic (RLW-TRU) capability. The design for the new RLW-TRU capability was paused as the final design was being completed and prior to the transition of the new management and operations contract for Los Alamos National Laboratory (LANL). Recently, NNSA has directed the project to restart and complete the design effort. As part of the restart process, LANL was asked to validate or document any material changes in the influent volumes or constituents that formed the design basis for the new capability. This analysis updates the transuranic liquid waste influents volumes, incorporating the changes in the LANL program of record since the FY11 assessment, and includes a cursory validation of influent constituents. The results of the FY19 update to influent volume forecasts shows: 1) the influent volume of 29,000 L should continue to be used as the project design-basis; 2) There have been no material changes in the operational processes that would change the constituents of concern from either a treatment or material compatibility perspective.
12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
Evaluating wind speed and power forecasts for wind energy applications using an open-source and systematic validation framework
Building on the verification and validation work developed under the Second Wind Forecast Improvement Project, this work exhibits the value of a consistent procedure to evaluate wind power forecasts. We established an open-source Python code base tailored for wind speed and wind power forecast validation, WE-Validate. The code base can evaluate model forecasts with observations in a coherent manner. To demonstrate the systematic validation framework of WE-Validate, we designed and hosted a forecast evaluation benchmark exercise. We invited forecast providers in industry and academia to participate and submit forecasts for two case studies. We then evaluated the submissions with WE-Validate. Our findings suggest that ensemble means have reasonable skills in time series forecasting, whereas they are often inferior to single ensemble members in wind ramp forecasting. Adopting a voting scheme in ramp forecasting that allows ensemble members to detect ramps independently leads to satisfactory skill scores. Throughout this document, we also emphasize the importance of using statistically robust and resistant metrics as well as equitable skill scores in forecast evaluation.
17 WIND ENERGY↗
Using High-Resolution NSRDB Data to Evaluate Cloud Mask Forecast from WRF-Solar EPS
Validating spatiotemporal distributions of cloud forecasts using numerical weather prediction (NWP) models is difficult as this requires high-quality cloud-property at significantly high spatial and temporal resolution over extended periods of time. Observations of cloud properties, such as cloud mask, cloud optical thickness, and cloud type, are vital for assessing the capability of NWP models to forecast various types of clouds. Using the National Solar Radiation Database (NSRDB), this research evaluates ensemble cloud-mask predictions from the WRF-Solar ensemble prediction system (WRF-Solar EPS). From the WRF-Solar EPS, day-ahead solar forecasts for the contiguous United States (CONUS) for 2018 are simulated. Given the NSRDB data is accessible at a resolution of 2 km, we can calculate the cloud fraction across the 9-km grid of WRF-Solar EPS. This allows us to spatially assess the cloud-mask forecasts using two methods against the high-resolution NSRDB: (a) considering all 2-km NSRDB clouds in the forecast domain (EMAll), and (b) using a minimum cloud fraction threshold of 50% to designate a pixel as cloudy (EMP50). The low-resolution cloud masks from WRF-Solar EPS are evaluated directly against the cloud-resolving scale gridded observations from NSRDB using EMAll. With EMP50, we presume that scenes with less than 50% cloud cover from the 2-km NSRDB are clear. Thus, this assessment approach allows for a fair comparison with WRF-Solar EPS resolved to a 9-km grid. A method of point-by-point verification is used to evaluate dichotomous (yes/no) cloud mask predictions against the NSRDB. For each pixel of model extent, cloud frequency and traditional metrics (e.g., probability of detection, false alarm rate, and hit rate, etc.) are computed and compared with satellite-derived data sets. Mismatched cloud frequency (MCF) is computed to measure the present capability of WRF-Solar EPS in representing various types of clouds, which are categorized using three levels of cloud top height (CTH) and cloud optical depth (COD) across entire CONUS. Preliminary results show that the WRF-Solar EPS provides MCF values ranging from 9% to 46%, 16% to 33%, and 8% to 27% for low-level, middle-level, and high-level clouds, respectively, for three CTHs. The model produces MCFs ranging from 27% to 46%, 13% to 34%, and 8% to 19% for thin, medium-thickness, and thick clouds, respectively, for three CODs. The presentation will include a detailed description of the current outcomes as well as potential future extensions. The evaluation approach established in this study is readily extensible to the evaluation of cloud predictions from different ensemble NWP models. In addition, the findings of the suggested evaluation technique aid in identifying model weaknesses and will ultimately lead to advances in WRF-Solar EPS's skill in predicting clouds and solar irradiance.
cloud mask forecast↗
Review of Onsite Temperature and Solar Forecasting Models to Enable Better Building Design and Operations
Advanced building controls and energy optimization for new constructions and retrofits rely on accurate weather data. Traditionally, most studies utilize airport weather information as the decision inputs. However, most buildings are in environments that are quite different than those at the airport miles away. Tree cover, adjacent buildings, and micro-climate effects caused by the larger surrounding area can all yield deviations in air temperature, humidity, solar irradiance, and wind that are large enough to influence design and operation decisions. In order to overcome this challenge, there are many prior studies on developing weather forecasting algorithms from micro-to meso-scales. Additionally, this paper reviews and complies knowledge on common weather data resources, data processing methodologies and forecasting techniques of weather information. Commonly used statistical, machine learning and physical-based models are discussed and presented as two major categories: deterministic forecasting and probabilistic forecasting. Finally, evaluation metrics for forecasting errors are listed and discussed.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
Evaluating Recursive Blind Forecast Against API and Baseline: A Puerto Rican Case Study on Solar Irradiance for Normal and Extreme Weather
This paper leverages ongoing work in a community microgrid in Adjuntas, Puerto Rico to forecast global horizontal irradiance (GHI) and compare performance in normal and extreme weather. Given a positive correlation of 0.98 between GHI and PV power, forecasting GHI can be an effective, indirect forecast of photovoltaic (PV) power, especially in microgrids where the end-users, owners, operators, or other stakeholders are reluctant to share data for training or validation due to privacy and security concerns. A recursive one-shot (termed as "blind") forecast is, hence, formulated, wherein a gradient-boosted regression tree (GBR) is built to forecast GHI for a 7-day horizon in normal weather, and a 2-day horizon in extreme weather. To demonstrate its resilience, the architecture is trained on normal and hurricane weather GHI from 2002-2022. It is generalized on February 9-16, 2023, and on the landfall of Hurricane Nicole (Nov 4-5, 2022), respectively. Forecasts from GBR are compared against that from a satellite-based API resource and three baselines: persistence, averaging, and exponential smoothing. Results show GBR and persistence outperform sophisticated API in both types of weather for this case study.
Sundararajan, Aditya↗
Evaluation of an open forecasting challenge to assess skill of West Nile virus neuroinvasive disease prediction
Abstract Background West Nile virus (WNV) is the leading cause of mosquito-borne illness in the continental USA. WNV occurrence has high spatiotemporal variation, and current approaches to targeted control of the virus are limited, making forecasting a public health priority. However, little research has been done to compare strengths and weaknesses of WNV disease forecasting approaches on the national scale. We used forecasts submitted to the 2020 WNV Forecasting Challenge, an open challenge organized by the Centers for Disease Control and Prevention, to assess the status of WNV neuroinvasive disease (WNND) prediction and identify avenues for improvement. Methods We performed a multi-model comparative assessment of probabilistic forecasts submitted by 15 teams for annual WNND cases in US counties for 2020 and assessed forecast accuracy, calibration, and discriminatory power. In the evaluation, we included forecasts produced by comparison models of varying complexity as benchmarks of forecast performance. We also used regression analysis to identify modeling approaches and contextual factors that were associated with forecast skill. Results Simple models based on historical WNND cases generally scored better than more complex models and combined higher discriminatory power with better calibration of uncertainty. Forecast skill improved across updated forecast submissions submitted during the 2020 season. Among models using additional data, inclusion of climate or human demographic data was associated with higher skill, while inclusion of mosquito or land use data was associated with lower skill. We also identified population size, extreme minimum winter temperature, and interannual variation in WNND cases as county-level characteristics associated with variation in forecast skill. Conclusions Historical WNND cases were strong predictors of future cases with minimal increase in skill achieved by models that included other factors. Although opportunities might exist to specifically improve predictions for areas with large populations and low or high winter temperatures, areas with high case-count variability are intrinsically more difficult to predict. Also, the prediction of outbreaks, which are outliers relative to typical case numbers, remains difficult. Further improvements to prediction could be obtained with improved calibration of forecast uncertainty and access to real-time data streams (e.g. current weather and preliminary human cases). Graphical Abstract
54 ENVIRONMENTAL SCIENCES↗
Short-term electricity load forecasting: Application-driven evaluation of machine learning models across spatial and temporal scales
As we transition towards a decarbonized economy, the integration of variable renewable energy resources and new demands (e.g., electric vehicles, heat pumps) into the electricity grid places unprecedented pressure on grid operators to effectively anticipate and manage peak load. In this context, machine learning algorithms are proving to be indispensable for accurate short-term load forecasting, a crucial task to address these challenges. This study benchmarks 6 machine learning algorithms, including three neural networks and three tree-based algorithms, across various levels of spatial aggregation and time horizons (1, 4, 8, 24, and 48 h). The central contribution of this work is the comparison and analysis of load forecasting models not only based on statistical metrics, but also based on a novel error metric, which evaluates the cost implications of forecast errors for power system stakeholders. Results show that tree-based models outperform neural networks, based on statistical metrics, and yield less skewed error distributions for most spatial scales. However, through the lens of the novel error metric, neural networks are the more competitive choice, especially for forecast horizons that exceed 8 h. The study concludes with actionable recommendations to grid operators and highlights the need for the development of error metrics that link forecasting accuracy to operational costs. To promote transparency and open science, the datasets and Python code are open-sourced via a supplementary repository.
Houben, Nikolaus↗
Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States
This paper compares the probabilistic accuracy of short-term forecasts of reported deaths due to COVID-19 during the first year and a half of the pandemic in the United States. Results show high variation in accuracy between and within stand-alone models and more consistent accuracy from an ensemble model that combined forecasts from all eligible models. This demonstrates that an ensemble model provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it. This work strengthens the evidence base for synthesizing multiple models to support public-health action.
60 APPLIED LIFE SCIENCES↗
Use of Physics to Improve Solar Forecast: Part II, Machine Learning and Model Interpretability
Machine learning (ML) models have been applied to forecast solar energy; however, they often lack clarity of interpretability and underlying physics. This work addresses such challenges by developing a hierarchy of ML models that gradually introduce predictors to improve the forecast accuracy based on a physics-based framework. Three ML models (ARIMA, LSTM, and XGBoost) are examined and compared with four physics-informed persistence models reported in Part I and the simple persistence model to assess the improvement of different models. The 7-year measurements at the U.S. Department of Energy's Atmospheric Radiation Measurement's Southern Great Plains Central Facility site are used for forecasts and evaluations. The results reveal that the step-by-step introduction of predictors leads to different improvements for models at different hierarchical levels. Comparison of the ML models with persistence models shows that LSTM and XGBoost outperform all the persistence models, with LSTM having the overall best performance; however, ARIMA underperforms the four physics-informed persistence models. This study demonstrates the importance and utility of incorporating physics into ML models in improving forecast accuracy by introducing a hierarchy of physics-based predictors, distinguishing predictor contributions, and enhancing the ML interpretability. The combined use of Global Horizontal Irradiance (GHI) and Direct Normal Irradiance (DNI) significantly improves the forecast accuracy compared to using individual irradiances alone because the pair contains more information on cloud-radiation interactions.
interpretability↗
Uncertainty-Guided Prediction Horizon of Phase-Resolved Ocean Wave Forecasting Under Data Sparsity: Experimental and Numerical Evaluation
Accurate short-term wave forecasting is critical for the safe and efficient operation of marine structures that rely on real-time, phase-resolved ocean wave information for control and monitoring purposes (e.g., digital twins). These systems often depend on environmental sensors (e.g., waverider buoys, wave-sensing LIDAR). Challenges arise when upstream sensor data are missing, sparse, or phase-shifted due to drift. This study investigates the performance of two machine learning models, time-series dense encoder (TiDE) and long short-term memory (LSTM), for forecasting phase-resolved ocean surface elevations under varying degrees of data degradation. We introduce the τ-trimming algorithm, which adapts the prediction horizon based on uncertainty thresholds derived from historical forecasts. Numerical wave tank (NWT) and wave basin experiments are used to benchmark model performance under short- and long-term data masking, spatially coarse sensor grids, and upstream phase shifts. Results show under a 50% probability of upstream data loss, the τ-trimmed TiDE model achieves a 46% reduction in error at the most upstream target, compared to 22% for LSTM. Furthermore, phase misalignment in upstream data introduces a near-linear increase in forecast error. Under moderate model settings, a ±3 s misalignment increases the mean absolute error by approximately 0.5 m, while the same error is accumulated at ±4 s using the more conservative approach. These findings inform the design of resilient, uncertainty-aware wave forecasting systems suited for realistic offshore sensing environments.
42 ENGINEERING↗