Search NASA⌕ Search

SEARCH · Search NASA

Results for “forecast evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
NASA NTRS2020

Evaluation of Global Fire Weather Database Reanalysis and Short-Term Forecast Products

Daily Fire Weather Index (FWI) System components calculated from the NASA Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2), are compared to FWI calculations from a global network of weather stations over 2004–2018, and short-term, experimental (8 d) daily FWI forecasts are evaluated for their skill across the Terrestrial Ecoregions of the World for 2018. FWI components from MERRA-2 were, in general, biased low compared to station data, but this reflects a mix of coherent low and high biases of different magnitudes. Biases in different MERRA-2 FWI components were related to different biases in weather input variables for different regions, but temperature and relative humidity biases were the most important overall. FWI forecasts had high skill for 1–2 d lead times for most of the world. For longer lead times, forecast skill decreased most quickly at high latitudes and was most closely related to decreasing skill of relative humidity forecasts. These results provide a baseline for the evaluation and use of fire weather products calculated from global analysis and forecast fields.

Global Fire Weather Database↗
DOE OSTI2026

Short-term electricity load forecasting: Application-driven evaluation of machine learning models across spatial and temporal scales

As we transition towards a decarbonized economy, the integration of variable renewable energy resources and new demands (e.g., electric vehicles, heat pumps) into the electricity grid places unprecedented pressure on grid operators to effectively anticipate and manage peak load. In this context, machine learning algorithms are proving to be indispensable for accurate short-term load forecasting, a crucial task to address these challenges. This study benchmarks 6 machine learning algorithms, including three neural networks and three tree-based algorithms, across various levels of spatial aggregation and time horizons (1, 4, 8, 24, and 48 h). The central contribution of this work is the comparison and analysis of load forecasting models not only based on statistical metrics, but also based on a novel error metric, which evaluates the cost implications of forecast errors for power system stakeholders. Results show that tree-based models outperform neural networks, based on statistical metrics, and yield less skewed error distributions for most spatial scales. However, through the lens of the novel error metric, neural networks are the more competitive choice, especially for forecast horizons that exceed 8 h. The study concludes with actionable recommendations to grid operators and highlights the need for the development of error metrics that link forecasting accuracy to operational costs. To promote transparency and open science, the datasets and Python code are open-sourced via a supplementary repository.

Houben, Nikolaus↗
NASA NTRS2001

Evaluation of NCAR Icing/SLD Forecasts, Tools and Techniques Used During The 1998 NASA SLD Flight Season

Supercooled Large Droplet (SLD) icing conditions were implicated in at least one recent aircraft crash, and have been associated with other aircraft incidents. Inflight encounters with SLD can result in ice accreting on unprotected areas of the wing where it can not be removed. Because this ice can adversely affect flight characteristics of some aircraft, there has been concern about flight safety in these conditions. The FAA held a conference on in-flight icing in 1996 where the state of knowledge concerning SLD was explored. One outcome of these meetings was an identified need to acquire SLD flight research data, particularly in the Great Lakes Region. The flight research data was needed by the FAA to develop a better understanding of the meteorological characteristics associated with SLD and facilitate an assessment of existing aircraft icing certification regulations with respect to SLD. In response to this need, NASA, the Federal Aviation Administration (FAA), and the National Center for Atmospheric Research (NCAR) conducted a cooperative icing flight research program to acquire SLD flight research data. The NASA Glenn Research Center's Twin Otter icing research aircraft was flown throughout the Great Lakes region during the winters of 1996-97 and 1997-98 to acquire SLD icing and meteorological data. The NASA Twin Otter was instrumented to measure cloud microphysical properties (particle size, LWC (Liquid Water Content), temperature, etc.), capture images of wing and tail ice accretion, and then record the resultant effect on aircraft performance due to the ice accretion. A satellite telephone link enabled the researchers onboard the Twin Otter to communicate with NCAR meteorologists. who provided real-time guidance into SLD icing conditions. NCAR meteorologists also provided preflight SLD weather forecasts that were used to plan the research flights, and served as on-board researchers. This document contains an evaluation of the tools and techniques NCAR forecasters used to predict the location of SLD icing conditions during the winter of 1997-1998. The objectives of this report are to: (1) assess the tools used to forecast in-flight icing. (2) assess the success/failure rate of the forecasts, and (3) discuss suggested changes to forecast techniques.

Bernstein, Ben C.↗
NASA NTRS2009

The New Weather Radar for America's Space Program in Florida: A Temperature Profile Adaptive Scan Strategy

A new weather radar is being acquired for use in support of America s space program at Cape Canaveral Air Force Station, NASA Kennedy Space Center, and Patrick AFB on the east coast of central Florida. This new radar replaces the modified WSR-74C at Patrick AFB that has been in use since 1984. The new radar is a Radtec TDR 43-250, which has Doppler and dual polarization capability. A new fixed scan strategy was designed to best support the space program. The fixed scan strategy represents a complex compromise between many competing factors and relies on climatological heights of various temperatures that are important for improved lightning forecasting and evaluation of Lightning Launch Commit Criteria (LCC), which are the weather rules to avoid lightning strikes to in-flight rockets. The 0 C to -20 C layer is vital since most generation of electric charge occurs within it and so it is critical in evaluating Lightning LCC and in forecasting lightning. These are two of the most important duties of 45 WS. While the fixed scan strategy that covers most of the climatological variation of the 0 C to -20 C levels with high resolution ensures that these critical temperatures are well covered most of the time, it also means that on any particular day the radar is spending precious time scanning at angles covering less important heights. The goal of this project is to develop a user-friendly, Interactive Data Language (IDL) computer program that will automatically generate optimized radar scan strategies that adapt to user input of the temperature profile and other important parameters. By using only the required scan angles output by the temperature profile adaptive scan strategy program, faster update times for volume scans and/or collection of more samples per gate for better data quality is possible, while maintaining high resolution at the critical temperature levels. The temperature profile adaptive technique will also take into account earth curvature and refraction when geo-locating the radar beam (i.e., beam height and arc distance), including non-standard refraction based on the user-input temperature profile. In addition to temperature profile adaptivity, this paper will also summarize the other requirements for this scan strategy program such as detection of low-level boundaries, detection of anvil clouds, reducing the Cone Of Silence, and allowing for times when deep convective clouds will not occur. The adaptive technique will be carefully compared to and benchmarked against the new fixed scan strategy. Specific environmental scenarios in which the adaptive scan strategy is able to optimize and improve coverage and resolution at critical heights, scan time, and/or sample numbers relative to the fixed scan strategy will be presented.

Carey, L. D.↗
NASA NTRS2021

Evaluation and Intercomparison of Wildfire Smoke Forecasts from Multiple Modeling Systems for the 2019 Williams Flats Fire

Wildfire smoke is one of the most significant concerns of human and environmental health, associated with its substantial impacts on air quality, weather, and climate. However, biomass burning emissions and smoke remain among the largest sources of uncertainties in air quality forecasts. In this study, we evaluate the smoke emissions and plume forecasts from 12 state-of-the-art air quality forecasting systems during the Williams Flats fire in Washington State, US, August 2019, which was intensively observed during the Fire Influence on Regional to Global Environments and Air Quality (FIREX-AQ) field campaign. Model forecasts with lead times within 1 d are intercompared under the same framework based on observations from multiple platforms to reveal their performance regarding fire emissions, aerosol optical depth (AOD), surface PM2.5, plume injection, and surface PM2.5 to AOD ratio. The comparison of smoke organic carbon (OC) emissions suggests a large range of daily totals among the models, with a factor of 20 to 50. Limited representations of the diurnal patterns and day-to-day variations of emissions highlight the need to incorporate new methodologies to predict the temporal evolution and reduce uncertainty of smoke emission estimates. The evaluation of smoke AOD (sAOD) forecasts suggests overall underpredictions in both the magnitude and smoke plume area for nearly all models, although the high-resolution models have a better representation of the fine-scale structures of smoke plumes. The models driven by fire radiative power (FRP)-based fire emissions or assimilating satellite AOD data generally outperform the others. Additionally, limitations of the persistence assumption used when predicting smoke emissions are revealed by substantial underpredictions of sAOD on 8 August 2019, mainly over the transported smoke plumes, owing to the underestimated emissions on 7 August. In contrast, the surface smoke PM2.5 (sPM2.5) forecasts show both positive and negative overall biases for these models, with most members presenting more considerable diurnal variations of sPM2.5. Overpredictions of sPM2.5 are found for the models driven by FRP-based emissions during nighttime, suggesting the necessity to improve vertical emission allocation within and above the planetary boundary layer (PBL). Smoke injection heights are further evaluated using the NASA Langley Research Center's Differential Absorption High Spectral Resolution Lidar (DIAL-HSRL) data collected during the flight observations. As the fire became stronger over 3–8 August, the plume height became deeper, with a day-to-day range of about 2–9 km a.g.l. However, narrower ranges are found for all models, with a tendency of overpredicting the plume heights for the shallower injection transects and underpredicting for the days showing deeper injections. The misrepresented plume injection heights lead to inaccurate vertical plume allocations along the transects corresponding to transported smoke that is 1 d old. Discrepancies in model performance for surface PM2.5 and AOD are further suggested by the evaluation of their ratio, which cannot be compensated for by solely adjusting the smoke emissions but are more attributable to model representations of plume injections, besides other possible factors including the evolution of PBL depths and aerosol optical property assumptions. By consolidating multiple forecast systems, these results provide strategic insight on pathways to improve smoke forecasts.

AOD↗
DOE OSTI2022

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States

This paper compares the probabilistic accuracy of short-term forecasts of reported deaths due to COVID-19 during the first year and a half of the pandemic in the United States. Results show high variation in accuracy between and within stand-alone models and more consistent accuracy from an ensemble model that combined forecasts from all eligible models. This demonstrates that an ensemble model provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it. This work strengthens the evidence base for synthesizing multiple models to support public-health action.

60 APPLIED LIFE SCIENCES↗
NASA NTRS1981

The relative contributions of increased resolution in the data assimilation and in the forecast model to satellite data impact

Assimilation cycles were carried out with two versions of the GLAS second order GCM: a coarse version with 4 deg latitude by 5 deg longitude resolution, called the C model, and a fine version with 2.5 deg latitude by 3 deg longitude resolution called the F model. For the two DST-6 cases where the combined influence of satellite data and model resolution are at a maximum at sea level, the relative contributions of increased resolution in the data assimilation and in the forecast models were evaluated. F model forecasts were generated from the C model SAT assimilation interpolated by the F grid, and C model forecasts were generated from the F model SAT assimilation interpolated to the C grid. These forecasts were then compared with the corresponding forecasts which had utilized the same grid resolution in the data assimilation and forecast models, CS and FS.

Atlas, R.↗
NASA NTRS1988

Meteorological and Environmental Inputs to Aviation Systems

Reports on aviation meteorology, most of them informal, are presented by representatives of the National Weather Service, the Bracknell (England) Meteorological Office, the NOAA Wave Propagation Lab., the Fleet Numerical Oceanography Center, and the Aircraft Owners and Pilots Association. Additional presentations are included on aircraft/lidar turbulence comparison, lightning detection and locating systems, objective detection and forecasting of clear air turbulence, comparative verification between the Generalized Exponential Markov (GEM) Model and official aviation terminal forecasts, the evaluation of the Prototype Regional Observation and Forecast System (PROFS) mesoscale weather products, and the FAA/MIT Lincoln Lab. Doppler Weather Radar Program.

Camp, Dennis W.↗
DOE OSTI2022

Use of Physics to Improve Solar Forecast: Part II, Machine Learning and Model Interpretability

Machine learning (ML) models have been applied to forecast solar energy; however, they often lack clarity of interpretability and underlying physics. This work addresses such challenges by developing a hierarchy of ML models that gradually introduce predictors to improve the forecast accuracy based on a physics-based framework. Three ML models (ARIMA, LSTM, and XGBoost) are examined and compared with four physics-informed persistence models reported in Part I and the simple persistence model to assess the improvement of different models. The 7-year measurements at the U.S. Department of Energy's Atmospheric Radiation Measurement's Southern Great Plains Central Facility site are used for forecasts and evaluations. The results reveal that the step-by-step introduction of predictors leads to different improvements for models at different hierarchical levels. Comparison of the ML models with persistence models shows that LSTM and XGBoost outperform all the persistence models, with LSTM having the overall best performance; however, ARIMA underperforms the four physics-informed persistence models. This study demonstrates the importance and utility of incorporating physics into ML models in improving forecast accuracy by introducing a hierarchy of physics-based predictors, distinguishing predictor contributions, and enhancing the ML interpretability. The combined use of Global Horizontal Irradiance (GHI) and Direct Normal Irradiance (DNI) significantly improves the forecast accuracy compared to using individual irradiances alone because the pair contains more information on cloud-radiation interactions.

interpretability↗
NASA NTRS2003

Transition from Research to Operations: Assessing Value of Experimental Forecast Products within the NWSFO Environment

The NASA Short-term Prediction Research and Transition (SPoRT) Center seeks to accelerate the infusion of NASA Earth Science Enterprise (ESE) observations, data assimilation and modeling research into NWS forecast operations and decision-making. To meet long-term program expectations, it is not sufficient simply to give forecasters sophisticated workstations or new forecast products without fully assessing the ways in which they will be utilized. Close communication must be established between the research and operational communities so that developers have a complete understanding of user needs. In turn, forecasters must obtain a more comprehensive knowledge of the modeling and sensing tools available to them. A major goal of the SPoRT Program is to develop metrics and conduct assessment studies with NWS forecasters to evaluate the impacts and benefits of ESE experimental products on forecast skill. At a glance the task seems relatively straightforward. However, performing assessment of experimental products in an operational environment is demanding. Given the tremendous time constraints placed on NWS forecasters, it is imperative that forecaster input be obtained in a concise unobtrusive manor. Great care must also be taken to ensure that forecasters understand their participation will eventually benefit them and WFO operations in general. Two requirements of the assessment plan developed under the SPoRT activity are that it 1) Can be implemented within the WFO environment; and 2) Provide tangible results for BOTH the research and operational communities. Supplemental numerical quantitative precipitation forecasts (QPF) were chosen as the first experimental SPoRT product to be evaluated during a Pilot Assessment Program conducted 1 May 2003 within the Huntsville AL National Weather Service Forecast Office. Forecast time periods were broken up into six- hour bins ranging from zero to twenty-four hours. Data were made available for display in AWIPS on an operational basis so they could be efficiently incorporated into the forecast process. The methodology used to assess the value of experimental QPFs compared to available operational products is best described as a three-tier approach involving both forecasters and research scientists. Tier-one is a web-based survey completed by duty forecasters on the aviation and public desks. The survey compiles information on how the experimental product was used in the forecast decision making process. Up to 6 responses per twenty-four hours can be compiled during a precipitation event. Tier-two consists of an event post mortem and experimental product assessment performed daily by the NASA/NWS Liaison. Tier-three is a detailed breakdown/analysis of specific events targeted by either the NWS SO0 or SPoRT team members. The task is performed by both NWS and NASA research scientists and may be conducted once every couple of months. The findings from the Pilot Assessment Program will be reported at the meeting.

Lapenta, William M.↗
NASA NTRS1987

4-D techniques for evaluation of atmospheric model forecasts

A capability for four-dimensional display of meteorological data is being developed at the Space Science and Engineering Center of the University of Wisconsin. McIDAS is used for all aspects of the analysis, including acquiring data, running the model. storing the output, and displaying the results. A version of the Australian Regional Analysis and Forecast Models was applied to the eastern portion of the USA and adjacent Atlantic Ocean. The assimilation system is being used to analyze intensive observing periods during the GALE (Genesis of Atlantic Lows Experiment) field experiment.

Santek, David↗
NASA NTRS2003

Evaluation of Improved Pushback Forecasts Derived from Airline Ground Operations Data

Accurate and timely predictions of airline pushbacks can potentially lead to improved performance of automated decision-support tools for airport surface traffic, thus reducing the variability and average duration of costly airline delays. One factor which affects the realization of these benefits is the level of uncertainty inherent in the turn processes. To characterize this inherent uncertainty, three techniques are developed for predicting time-to-go until pushback as a function of available ground-time; elapsed ground-time; and the status (not-started/in-progress/completed) of individual turn processes (cleaning, fueling, etc.). These techniques are tested against a large and detailed dataset covering approximately l0(exp 4) real-world turn operations obtained through collaboration with Deutsche Lufthansa AG. Even after the dataset is filtered to obtain a sample of turn operations with minimal uncertainty, the standard deviation of forecast error for all three techniques is lower-bounded away from zero, indicating that turn operations have a significant stochastic component. This lower-bound result shows that decision-support tools must be designed to incorporate robust mechanisms for coping with pushback demand stochasticity, rather than treating the pushback demand process as a known deterministic input.

Carr, Francis↗
NASA NTRS1979

Subjective evaluation of the combined influence of satellite temperature sounding data and increased model resolution on numerical weather forecasting

The present evaluation is concerned with (1) the significance of prognostic differences resulting from the inclusion of satellite-derived temperature soundings, (2) how specific differences between the SAT and NOSAT prognoses evolve, and (3) comparison of two experiments using the Goddard Laboratory for Atmospheric Sciences general circulation model. The subjective evaluation indicates that the beneficial impact of sounding data is enhanced with increased resolution. It is suggested that satellite sounding data posses valuable information content which at times can correct gross analysis errors in data sparse regions.

Atlas, R.↗
NASA NTRS2017

Skillful Spring Forecasts of September Arctic Sea Ice Extent Using Passive Microwave Data

In this study, we demonstrate skillful spring forecasts of detrended September Arctic sea ice extent using passive microwave observations of sea ice concentration (SIC) and melt onset (MO). We compare these to forecasts produced using data from a sophisticated melt pond model, and find similar to higher skill values, where the forecast skill is calculated relative to linear trend persistence. The MO forecasts shows the highest skill in March-May, while the SIC forecasts produce the highest skill in June-August, especially when the forecasts are evaluated over recent years (since 2008). The high MO forecast skill in early spring appears to be driven primarily by the presence and timing of open water anomalies, while the high SIC forecast skill appears to be driven by both open water and surface melt processes. Spatial maps of detrended anomalies highlight the drivers of the different forecasts, and enable us to understand regions of predictive importance. Correctly capturing sea ice state anomalies, along with changes in open water coverage appear to be key processes in skillfully forecasting summer Arctic sea ice.

melt onset↗
DOE OSTI2025

Uncertainty-Guided Prediction Horizon of Phase-Resolved Ocean Wave Forecasting Under Data Sparsity: Experimental and Numerical Evaluation

Accurate short-term wave forecasting is critical for the safe and efficient operation of marine structures that rely on real-time, phase-resolved ocean wave information for control and monitoring purposes (e.g., digital twins). These systems often depend on environmental sensors (e.g., waverider buoys, wave-sensing LIDAR). Challenges arise when upstream sensor data are missing, sparse, or phase-shifted due to drift. This study investigates the performance of two machine learning models, time-series dense encoder (TiDE) and long short-term memory (LSTM), for forecasting phase-resolved ocean surface elevations under varying degrees of data degradation. We introduce the τ-trimming algorithm, which adapts the prediction horizon based on uncertainty thresholds derived from historical forecasts. Numerical wave tank (NWT) and wave basin experiments are used to benchmark model performance under short- and long-term data masking, spatially coarse sensor grids, and upstream phase shifts. Results show under a 50% probability of upstream data loss, the τ-trimmed TiDE model achieves a 46% reduction in error at the most upstream target, compared to 22% for LSTM. Furthermore, phase misalignment in upstream data introduces a near-linear increase in forecast error. Under moderate model settings, a ±3 s misalignment increases the mean absolute error by approximately 0.5 m, while the same error is accumulated at ±4 s using the more conservative approach. These findings inform the design of resilient, uncertainty-aware wave forecasting systems suited for realistic offshore sensing environments.

42 ENGINEERING↗
DOE OSTI2023

Seasonal forecasting skill for the High Mountain Asia region in the Goddard Earth Observing System

Seasonal variability of the global hydrologic cycle directly impacts human activities, including hazard assessment and mitigation, agricultural decisions, and water resources management. This is particularly true across the High Mountain Asia (HMA) region, where availability of water resources can change depending on local seasonality of the hydrologic cycle. Forecasting the atmospheric states and surface conditions, including hydrometeorologically relevant variables, at subseasonal-to-seasonal (S2S) lead times of weeks to months is an area of active research and development. NASA's Goddard Earth Observing System (GEOS) S2S prediction system has been developed with this research goal in mind. Here, we benchmark the forecast skill of GEOS-S2S (version 2) hydrometeorological forecasts at 1–3-month lead times in the HMA region, including a portion of the Indian subcontinent, during the retrospective forecast period, 1981–2016. To assess forecast skill, we evaluate 2 m air temperature, total precipitation, fractional snow cover, snow water equivalent, surface soil moisture, and terrestrial water storage forecasts against the Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2) and independent reanalysis data, satellite observations, and data fusion products. Anomaly correlation is highest when the forecasts are evaluated against MERRA-2 and particularly in variables with long memory in the climate system, likely due to the similar initial conditions and model architecture used in GEOS-S2S and MERRA-2. When compared to MERRA-2, results for the 1-month forecast skill range from an anomaly correlation of R anom =0.18 for precipitation to R anom =0.62 for soil moisture. Anomaly correlations are consistently lower when forecasts are evaluated against independent observations; results for the 1-month forecast skill range from R anom =0.13 for snow water equivalent to R anom =0.24 for fractional snow cover. We find that, generally, hydrometeorological forecast skill is dependent on the forecast lead time, the memory of the variable within the physical system, and the validation dataset used. Overall, these results benchmark the GEOS-S2S system's ability to forecast HMA hydrometeorology.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2014

Assessment of the Pseudo Geostationary Lightning Mapper Products at the Spring Program and Summer Experiment

Since 2010, the de facto Geostationary Lightning Mapper (GLM) demonstration product has been the Pseudo‐Geostationary Lightning Mapper (PGLM) product suite. Originally prepared for the Hazardous Weather Testbed's Spring Program (specifically the Experimental Warning Program) when only four ground‐based lightning mapping arrays were available, the effort now spans collaborations with several institutions and eight collaborative networks. For 2013, NASA's Short‐term Prediction Research and Transition (SPoRT) Center and NOAA's National Severe Storms Laboratory have worked to collaborate with each network to obtain data in real‐time. This has gone into producing the SPoRT variant of the PGLM that was demonstrated in AWIPS II for the 2013 Spring Program. Alongside the PGLM products, the SPoRT / Meteorological Development Laboratory's total lightning tracking tool also was evaluated to assess not just another visualization of future GLM data but how to best extract more information while in the operational environment. Specifically, this tool addressed the leading request by forecasters during evaluations; provide a time series trend of total lightning in real‐time. In addition to the Spring Program, SPoRT is providing the PGLM "mosaic" to the Aviation Weather Center (AWC) and Storm Prediction Center. This is the same as what is used at the Hazardous Weather Testbed, but combines all available networks into one display for use at the national centers. This year, the mosaic was evaluated during the AWC's Summer Experiment. An important distinction between this and the Spring Program is that the Summer Experiment focuses on the national center perspective and not at the local forecast office level. Specifically, the Summer Experiment focuses on aviation needs and concerns and brings together operational forecaster, developers, and FAA representatives. This presentation will focus on the evaluation of SPoRT's pseudo‐GLM products in these separate test beds. The emphasis will be on how future GLM observations can support operations at both the local and national scale and how the PGLM was used in combination with other lightning data sets. Evaluations for the PGLM were quite favorable with forecasters appreciating the high temporal resolution, the ability to look for rapid increases in lightning activity ahead of severe weather, as well as situational awareness for where convection is firing and for flight routing.

Stano, Geoffrey T.↗