A new method to evaluate, optimize, and forecast human and robot performance in 21st century space operations
Explore the source record for details and available documents.
robots autonomy↗
SEARCH · Search NASA
Results for “forecast evaluation”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Evaluation of Global Forecast System Model Accuracy for NASA Spaceflight Operations
Explore the source record for details and available documents.
Frank Leahy↗
Improving ENSO Predictions Using the GMAO S 2 S Forecast System Through Evaluation of Observation Systems
Explore the source record for details and available documents.
A Jersild↗
Evaluation of Global Fire Weather Database Reanalysis and Short-Term Forecast Products
Daily Fire Weather Index (FWI) System components calculated from the NASA Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2), are compared to FWI calculations from a global network of weather stations over 2004–2018, and short-term, experimental (8 d) daily FWI forecasts are evaluated for their skill across the Terrestrial Ecoregions of the World for 2018. FWI components from MERRA-2 were, in general, biased low compared to station data, but this reflects a mix of coherent low and high biases of different magnitudes. Biases in different MERRA-2 FWI components were related to different biases in weather input variables for different regions, but temperature and relative humidity biases were the most important overall. FWI forecasts had high skill for 1–2 d lead times for most of the world. For longer lead times, forecast skill decreased most quickly at high latitudes and was most closely related to decreasing skill of relative humidity forecasts. These results provide a baseline for the evaluation and use of fire weather products calculated from global analysis and forecast fields.
Global Fire Weather Database↗
Evaluation of NCAR Icing/SLD Forecasts, Tools and Techniques Used During The 1998 NASA SLD Flight Season
Supercooled Large Droplet (SLD) icing conditions were implicated in at least one recent aircraft crash, and have been associated with other aircraft incidents. Inflight encounters with SLD can result in ice accreting on unprotected areas of the wing where it can not be removed. Because this ice can adversely affect flight characteristics of some aircraft, there has been concern about flight safety in these conditions. The FAA held a conference on in-flight icing in 1996 where the state of knowledge concerning SLD was explored. One outcome of these meetings was an identified need to acquire SLD flight research data, particularly in the Great Lakes Region. The flight research data was needed by the FAA to develop a better understanding of the meteorological characteristics associated with SLD and facilitate an assessment of existing aircraft icing certification regulations with respect to SLD. In response to this need, NASA, the Federal Aviation Administration (FAA), and the National Center for Atmospheric Research (NCAR) conducted a cooperative icing flight research program to acquire SLD flight research data. The NASA Glenn Research Center's Twin Otter icing research aircraft was flown throughout the Great Lakes region during the winters of 1996-97 and 1997-98 to acquire SLD icing and meteorological data. The NASA Twin Otter was instrumented to measure cloud microphysical properties (particle size, LWC (Liquid Water Content), temperature, etc.), capture images of wing and tail ice accretion, and then record the resultant effect on aircraft performance due to the ice accretion. A satellite telephone link enabled the researchers onboard the Twin Otter to communicate with NCAR meteorologists. who provided real-time guidance into SLD icing conditions. NCAR meteorologists also provided preflight SLD weather forecasts that were used to plan the research flights, and served as on-board researchers. This document contains an evaluation of the tools and techniques NCAR forecasters used to predict the location of SLD icing conditions during the winter of 1997-1998. The objectives of this report are to: (1) assess the tools used to forecast in-flight icing. (2) assess the success/failure rate of the forecasts, and (3) discuss suggested changes to forecast techniques.
Bernstein, Ben C.↗
The New Weather Radar for America's Space Program in Florida: A Temperature Profile Adaptive Scan Strategy
A new weather radar is being acquired for use in support of America s space program at Cape Canaveral Air Force Station, NASA Kennedy Space Center, and Patrick AFB on the east coast of central Florida. This new radar replaces the modified WSR-74C at Patrick AFB that has been in use since 1984. The new radar is a Radtec TDR 43-250, which has Doppler and dual polarization capability. A new fixed scan strategy was designed to best support the space program. The fixed scan strategy represents a complex compromise between many competing factors and relies on climatological heights of various temperatures that are important for improved lightning forecasting and evaluation of Lightning Launch Commit Criteria (LCC), which are the weather rules to avoid lightning strikes to in-flight rockets. The 0 C to -20 C layer is vital since most generation of electric charge occurs within it and so it is critical in evaluating Lightning LCC and in forecasting lightning. These are two of the most important duties of 45 WS. While the fixed scan strategy that covers most of the climatological variation of the 0 C to -20 C levels with high resolution ensures that these critical temperatures are well covered most of the time, it also means that on any particular day the radar is spending precious time scanning at angles covering less important heights. The goal of this project is to develop a user-friendly, Interactive Data Language (IDL) computer program that will automatically generate optimized radar scan strategies that adapt to user input of the temperature profile and other important parameters. By using only the required scan angles output by the temperature profile adaptive scan strategy program, faster update times for volume scans and/or collection of more samples per gate for better data quality is possible, while maintaining high resolution at the critical temperature levels. The temperature profile adaptive technique will also take into account earth curvature and refraction when geo-locating the radar beam (i.e., beam height and arc distance), including non-standard refraction based on the user-input temperature profile. In addition to temperature profile adaptivity, this paper will also summarize the other requirements for this scan strategy program such as detection of low-level boundaries, detection of anvil clouds, reducing the Cone Of Silence, and allowing for times when deep convective clouds will not occur. The adaptive technique will be carefully compared to and benchmarked against the new fixed scan strategy. Specific environmental scenarios in which the adaptive scan strategy is able to optimize and improve coverage and resolution at critical heights, scan time, and/or sample numbers relative to the fixed scan strategy will be presented.
Carey, L. D.↗
Evaluation and Intercomparison of Wildfire Smoke Forecasts from Multiple Modeling Systems for the 2019 Williams Flats Fire
Wildfire smoke is one of the most significant concerns of human and environmental health, associated with its substantial impacts on air quality, weather, and climate. However, biomass burning emissions and smoke remain among the largest sources of uncertainties in air quality forecasts. In this study, we evaluate the smoke emissions and plume forecasts from 12 state-of-the-art air quality forecasting systems during the Williams Flats fire in Washington State, US, August 2019, which was intensively observed during the Fire Influence on Regional to Global Environments and Air Quality (FIREX-AQ) field campaign. Model forecasts with lead times within 1 d are intercompared under the same framework based on observations from multiple platforms to reveal their performance regarding fire emissions, aerosol optical depth (AOD), surface PM2.5, plume injection, and surface PM2.5 to AOD ratio. The comparison of smoke organic carbon (OC) emissions suggests a large range of daily totals among the models, with a factor of 20 to 50. Limited representations of the diurnal patterns and day-to-day variations of emissions highlight the need to incorporate new methodologies to predict the temporal evolution and reduce uncertainty of smoke emission estimates. The evaluation of smoke AOD (sAOD) forecasts suggests overall underpredictions in both the magnitude and smoke plume area for nearly all models, although the high-resolution models have a better representation of the fine-scale structures of smoke plumes. The models driven by fire radiative power (FRP)-based fire emissions or assimilating satellite AOD data generally outperform the others. Additionally, limitations of the persistence assumption used when predicting smoke emissions are revealed by substantial underpredictions of sAOD on 8 August 2019, mainly over the transported smoke plumes, owing to the underestimated emissions on 7 August. In contrast, the surface smoke PM2.5 (sPM2.5) forecasts show both positive and negative overall biases for these models, with most members presenting more considerable diurnal variations of sPM2.5. Overpredictions of sPM2.5 are found for the models driven by FRP-based emissions during nighttime, suggesting the necessity to improve vertical emission allocation within and above the planetary boundary layer (PBL). Smoke injection heights are further evaluated using the NASA Langley Research Center's Differential Absorption High Spectral Resolution Lidar (DIAL-HSRL) data collected during the flight observations. As the fire became stronger over 3–8 August, the plume height became deeper, with a day-to-day range of about 2–9 km a.g.l. However, narrower ranges are found for all models, with a tendency of overpredicting the plume heights for the shallower injection transects and underpredicting for the days showing deeper injections. The misrepresented plume injection heights lead to inaccurate vertical plume allocations along the transects corresponding to transported smoke that is 1 d old. Discrepancies in model performance for surface PM2.5 and AOD are further suggested by the evaluation of their ratio, which cannot be compensated for by solely adjusting the smoke emissions but are more attributable to model representations of plume injections, besides other possible factors including the evolution of PBL depths and aerosol optical property assumptions. By consolidating multiple forecast systems, these results provide strategic insight on pathways to improve smoke forecasts.
AOD↗
The relative contributions of increased resolution in the data assimilation and in the forecast model to satellite data impact
Assimilation cycles were carried out with two versions of the GLAS second order GCM: a coarse version with 4 deg latitude by 5 deg longitude resolution, called the C model, and a fine version with 2.5 deg latitude by 3 deg longitude resolution called the F model. For the two DST-6 cases where the combined influence of satellite data and model resolution are at a maximum at sea level, the relative contributions of increased resolution in the data assimilation and in the forecast models were evaluated. F model forecasts were generated from the C model SAT assimilation interpolated by the F grid, and C model forecasts were generated from the F model SAT assimilation interpolated to the C grid. These forecasts were then compared with the corresponding forecasts which had utilized the same grid resolution in the data assimilation and forecast models, CS and FS.
Atlas, R.↗
Meteorological and Environmental Inputs to Aviation Systems
Reports on aviation meteorology, most of them informal, are presented by representatives of the National Weather Service, the Bracknell (England) Meteorological Office, the NOAA Wave Propagation Lab., the Fleet Numerical Oceanography Center, and the Aircraft Owners and Pilots Association. Additional presentations are included on aircraft/lidar turbulence comparison, lightning detection and locating systems, objective detection and forecasting of clear air turbulence, comparative verification between the Generalized Exponential Markov (GEM) Model and official aviation terminal forecasts, the evaluation of the Prototype Regional Observation and Forecast System (PROFS) mesoscale weather products, and the FAA/MIT Lincoln Lab. Doppler Weather Radar Program.
Camp, Dennis W.↗
Transition from Research to Operations: Assessing Value of Experimental Forecast Products within the NWSFO Environment
The NASA Short-term Prediction Research and Transition (SPoRT) Center seeks to accelerate the infusion of NASA Earth Science Enterprise (ESE) observations, data assimilation and modeling research into NWS forecast operations and decision-making. To meet long-term program expectations, it is not sufficient simply to give forecasters sophisticated workstations or new forecast products without fully assessing the ways in which they will be utilized. Close communication must be established between the research and operational communities so that developers have a complete understanding of user needs. In turn, forecasters must obtain a more comprehensive knowledge of the modeling and sensing tools available to them. A major goal of the SPoRT Program is to develop metrics and conduct assessment studies with NWS forecasters to evaluate the impacts and benefits of ESE experimental products on forecast skill. At a glance the task seems relatively straightforward. However, performing assessment of experimental products in an operational environment is demanding. Given the tremendous time constraints placed on NWS forecasters, it is imperative that forecaster input be obtained in a concise unobtrusive manor. Great care must also be taken to ensure that forecasters understand their participation will eventually benefit them and WFO operations in general. Two requirements of the assessment plan developed under the SPoRT activity are that it 1) Can be implemented within the WFO environment; and 2) Provide tangible results for BOTH the research and operational communities. Supplemental numerical quantitative precipitation forecasts (QPF) were chosen as the first experimental SPoRT product to be evaluated during a Pilot Assessment Program conducted 1 May 2003 within the Huntsville AL National Weather Service Forecast Office. Forecast time periods were broken up into six- hour bins ranging from zero to twenty-four hours. Data were made available for display in AWIPS on an operational basis so they could be efficiently incorporated into the forecast process. The methodology used to assess the value of experimental QPFs compared to available operational products is best described as a three-tier approach involving both forecasters and research scientists. Tier-one is a web-based survey completed by duty forecasters on the aviation and public desks. The survey compiles information on how the experimental product was used in the forecast decision making process. Up to 6 responses per twenty-four hours can be compiled during a precipitation event. Tier-two consists of an event post mortem and experimental product assessment performed daily by the NASA/NWS Liaison. Tier-three is a detailed breakdown/analysis of specific events targeted by either the NWS SO0 or SPoRT team members. The task is performed by both NWS and NASA research scientists and may be conducted once every couple of months. The findings from the Pilot Assessment Program will be reported at the meeting.
Lapenta, William M.↗
4-D techniques for evaluation of atmospheric model forecasts
A capability for four-dimensional display of meteorological data is being developed at the Space Science and Engineering Center of the University of Wisconsin. McIDAS is used for all aspects of the analysis, including acquiring data, running the model. storing the output, and displaying the results. A version of the Australian Regional Analysis and Forecast Models was applied to the eastern portion of the USA and adjacent Atlantic Ocean. The assimilation system is being used to analyze intensive observing periods during the GALE (Genesis of Atlantic Lows Experiment) field experiment.
Santek, David↗
Evaluation of Improved Pushback Forecasts Derived from Airline Ground Operations Data
Accurate and timely predictions of airline pushbacks can potentially lead to improved performance of automated decision-support tools for airport surface traffic, thus reducing the variability and average duration of costly airline delays. One factor which affects the realization of these benefits is the level of uncertainty inherent in the turn processes. To characterize this inherent uncertainty, three techniques are developed for predicting time-to-go until pushback as a function of available ground-time; elapsed ground-time; and the status (not-started/in-progress/completed) of individual turn processes (cleaning, fueling, etc.). These techniques are tested against a large and detailed dataset covering approximately l0(exp 4) real-world turn operations obtained through collaboration with Deutsche Lufthansa AG. Even after the dataset is filtered to obtain a sample of turn operations with minimal uncertainty, the standard deviation of forecast error for all three techniques is lower-bounded away from zero, indicating that turn operations have a significant stochastic component. This lower-bound result shows that decision-support tools must be designed to incorporate robust mechanisms for coping with pushback demand stochasticity, rather than treating the pushback demand process as a known deterministic input.
Carr, Francis↗
Evaluation of Marine Heatwave Forecast Skill in GEOS-S2S Version 3 System
Explore the source record for details and available documents.
Li Ren↗
Subjective evaluation of the combined influence of satellite temperature sounding data and increased model resolution on numerical weather forecasting
The present evaluation is concerned with (1) the significance of prognostic differences resulting from the inclusion of satellite-derived temperature soundings, (2) how specific differences between the SAT and NOSAT prognoses evolve, and (3) comparison of two experiments using the Goddard Laboratory for Atmospheric Sciences general circulation model. The subjective evaluation indicates that the beneficial impact of sounding data is enhanced with increased resolution. It is suggested that satellite sounding data posses valuable information content which at times can correct gross analysis errors in data sparse regions.
Atlas, R.↗
Skillful Spring Forecasts of September Arctic Sea Ice Extent Using Passive Microwave Data
In this study, we demonstrate skillful spring forecasts of detrended September Arctic sea ice extent using passive microwave observations of sea ice concentration (SIC) and melt onset (MO). We compare these to forecasts produced using data from a sophisticated melt pond model, and find similar to higher skill values, where the forecast skill is calculated relative to linear trend persistence. The MO forecasts shows the highest skill in March-May, while the SIC forecasts produce the highest skill in June-August, especially when the forecasts are evaluated over recent years (since 2008). The high MO forecast skill in early spring appears to be driven primarily by the presence and timing of open water anomalies, while the high SIC forecast skill appears to be driven by both open water and surface melt processes. Spatial maps of detrended anomalies highlight the drivers of the different forecasts, and enable us to understand regions of predictive importance. Correctly capturing sea ice state anomalies, along with changes in open water coverage appear to be key processes in skillfully forecasting summer Arctic sea ice.
melt onset↗
Assessment of the Pseudo Geostationary Lightning Mapper Products at the Spring Program and Summer Experiment
Since 2010, the de facto Geostationary Lightning Mapper (GLM) demonstration product has been the Pseudo‐Geostationary Lightning Mapper (PGLM) product suite. Originally prepared for the Hazardous Weather Testbed's Spring Program (specifically the Experimental Warning Program) when only four ground‐based lightning mapping arrays were available, the effort now spans collaborations with several institutions and eight collaborative networks. For 2013, NASA's Short‐term Prediction Research and Transition (SPoRT) Center and NOAA's National Severe Storms Laboratory have worked to collaborate with each network to obtain data in real‐time. This has gone into producing the SPoRT variant of the PGLM that was demonstrated in AWIPS II for the 2013 Spring Program. Alongside the PGLM products, the SPoRT / Meteorological Development Laboratory's total lightning tracking tool also was evaluated to assess not just another visualization of future GLM data but how to best extract more information while in the operational environment. Specifically, this tool addressed the leading request by forecasters during evaluations; provide a time series trend of total lightning in real‐time. In addition to the Spring Program, SPoRT is providing the PGLM "mosaic" to the Aviation Weather Center (AWC) and Storm Prediction Center. This is the same as what is used at the Hazardous Weather Testbed, but combines all available networks into one display for use at the national centers. This year, the mosaic was evaluated during the AWC's Summer Experiment. An important distinction between this and the Spring Program is that the Summer Experiment focuses on the national center perspective and not at the local forecast office level. Specifically, the Summer Experiment focuses on aviation needs and concerns and brings together operational forecaster, developers, and FAA representatives. This presentation will focus on the evaluation of SPoRT's pseudo‐GLM products in these separate test beds. The emphasis will be on how future GLM observations can support operations at both the local and national scale and how the PGLM was used in combination with other lightning data sets. Evaluations for the PGLM were quite favorable with forecasters appreciating the high temporal resolution, the ability to look for rapid increases in lightning activity ahead of severe weather, as well as situational awareness for where convection is firing and for flight routing.
Stano, Geoffrey T.↗
An Examination of the Impact of Grid Spacing on WRF Simulations of Wintertime Precipitation in the Mid-Atlantic United States
A large set of deterministic and ensemble forecasts was produced to identify the optimal spacing for forecasting U.S. East Coast snowstorms. WRF forecasts were produced on cloud-allowing (~1-km grid spacing) and convection-allowing (3–4 km) grids, and compared against forecasts with parameterized convection (>~10 km). Performance diagrams were used to evaluate 19 deterministic forecasts from the winter of 2013–14. Ensemble forecasts of five disruptive snowstorms spanning the years 2015–18 were evaluated using various methods to evaluate probabilistic forecasts. While deterministic forecasts using cloud-allowing grids were not better than convection-allowing forecasts, both had lower bias and higher success ratios than forecasts with parameterized convection. All forecasts were underdispersive. Nevertheless, forecasts on the higher-resolution grids were more reliable than those with parameterized convection. Forecasts on the cloud-allowing grid were best able to discriminate areas that received heavy snow and those that did not, while the forecasts with parameterized convection were least able to do so. It is recommended to use convection-resolving and (if computationally possible) to use cloud-allowing forecast grids when predicting East Coast winter storms.
Snowstorms↗