Search NASA⌕ Search

SEARCH · Search NASA

Results for “model skill scores”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

A Numerical Study of Hypoxia in Chesapeake Bay Using an Unstructured Grid Model: Validation and Sensitivity to Bathymetry Representation

A three-dimensional unstructured-grid hydrodynamic and water quality model (Semi-implicit Cross-scale Hydroscience Intergrated System Model-Integrated Compartment Model) is applied successfully for Chesapeake Bay. The model is validated with observations of salinity, chlorophyll-a, dissolved oxygen, nutrients, and phytoplankton productions from the year 1991 to 1995 for the mainstem and some major tributaries, based on multiple model skill scores. Model experiments are conducted to test the importance of having (1) an accurate representation of bathymetry to correctly predict hypoxia and other processes and (2) a high-resolution model grid for tributaries to correctly simulate water quality variables. Further, comparison with the model experiment results with bathymetry smoothing indicates that bathymetry smoothing, as commonly used for many systems, changes the stratification and lateral circulation pattern, resulting in more salt intrusion into shallow water regions, and an increase in the freshwater age. Consequently, a model with bathymetry smoothing can lead to an unrealistic prediction of the distribution of hypoxia and phytoplankton production. Local grid refinement shows significant improvement of model simulations on local stratification and water quality variables. Overall, the use of high-resolution unstructured grid model leads to a faithful representation of the complex geometry, and thus a seamless cross-scale capability for simulating water quality processes in the Bay including tributaries and tidal creeks.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

LASSO Data Bundles

The Alpha 2 release is the second release from the LASSO Pilot Phase that builds upon the Alpha 1 release. Alpha 2 contains additional diagnostics in the data bundles and focuses on cases from spring-summer 2016. A data bundle is a unified package consisting of LASSO LES input and output, observations, evaluation diagnostics, and model skill scores. LES input include model configuration information and forcing data. LES output includes profile statistics and full domain fields of cloud and environmental variables. Model evaluation data consists of LES output and ARM observations co-registered on the same grid and sampling frequency. Model performance is quantified by skill scores and diagnostics in terms of cloud and environmental variables.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Assessing the Skill of the Improved Treatment of Riverine Freshwater in the Community Earth System Model (CESM) Relative to a New Salinity Climatology

Abstract Recent studies have explored the sensitivity of global ocean model simulations to the treatment of riverine freshwater and the representation of estuarine processes via an estuary box model applied within Community Earth System Model (CESM). This study builds on these efforts by assessing the model skill score relative to a new salinity climatology. The new climatology averages the original observational data of the World Ocean Database directly onto the CESM ocean component tracer grid cells without spatial interpolation, smoothing, or other gap‐filling techniques to mitigate coastal ocean salinity bias present in the World Ocean Atlas. The mean square error for coastal upper ocean salinity relative to climatology is reduced by up to 14%, and the mean square error of near‐surface salinity stratification is reduced by up to 28% near major river mouths in the simulations with improved treatments of river runoff. The improvement in upper ocean bulk salinity is attributed primarily to focusing runoff as point sources thereby avoiding the artificial horizontal spreading of the control run and to applying a locally varying instead of a global constant reference salinity for riverine virtual salt fluxes. The improvements in near‐surface salinity stratification are primarily attributed to adding parameterized estuarine mixing with the estuary box model. Salinity and salinity stratification skill improvements are achieved not just near large rivers but also along the global coast and skill improvements extend far offshore. Despite these improvements, many other sources of model‐climatology mismatch in coastal salinity and stratification remain and merit further attention.

58 GEOSCIENCES↗
DOE OSTI2026

New framework for benchmarking decadal predictions leveraging the PCMDI Metric Package with interactive visualization

Reliable climate predictions across multiple timescales are increasingly critical as climate-related risks continue to rise. With the growing number and diversity of climate prediction systems, systematic intercomparison has become essential. Here, we present a comprehensive evaluation framework based on the PCMDI Metric Package to assess the performance of multiple decadal climate prediction systems. Unlike uninitialized simulations, initialized predictions exhibit bias and predictive skill that evolve with forecast lead time. To address this, we introduce (1) model-by-lead-time portrait plots, which efficiently summarize metrics of global temperature, precipitation, and Arctic/Antarctic sea-ice extent, and (2) an HTML-based interactive visualization platform that provides detailed regional and seasonal diagnostics of model bias, skill scores, and ensemble spread for each model and lead time. Comparisons with uninitialized simulations further quantify the relative impacts of initialization and external forcing on prediction skill. The proposed framework provides a scalable and transparent approach for multi-model climate prediction assessments and can be readily extended to a wide range of operational and research forecasting systems.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1977

The effect of increased horizontal resolution on synoptic forecasts with the GISS model of the global atmosphere

The improved horizontal resolution global circulation model in question has a horizontal spacing of 2.5 deg latitude by 3 deg longitude for a resolution of about 250 km in midlatitude. This paper reports on experiments to determine the improvements in forecasting skill and in initial conditions that were made possible by the ultrafine resolution and ultrafine assimilation. The size of the improvement in skill score when the ultrafine model is used for forecasting and assimilation was so large that 60-hour forecasts with the ultrafine model had as good a skill score as the 48-hour forecasts with the fine model. Synoptic evaluations confirmed that ultrafine model forecasts are better than the fine model forecasts.

Quirk, W. J.↗
NASA NTRS2003

A Global Aerosol Model Forecast for the ACE-Asia Field Experiment

We present the results of aerosol forecast during the Aerosol Characterization Experiment (ACE-Asia) field experiment in spring 2001, using the Georgia Tech/Goddard Global Ozone Chemistry Aerosol Radiation and Transport (GOCART) model and the meteorological forecast fields from the Goddard Earth Observing System Data Assimilation System (GEOS DAS). The aerosol model forecast provides direct information on aerosol optical thickness and concentrations, enabling effective flight planning, while feedbacks from measurements constantly evaluate the model, making successful model improvements. We verify the model forecast skill by comparing model predicted total aerosol extinction, dust, sulfate, and SO2 concentrations with those quantities measured by the C-130 aircraft during the ACE-Asia intensive operation period. The GEOS DAS meteorological forecast system shows excellent skills in predicting winds, relative humidity, and temperature for the ACE-Asia experiment area as well as for each individual flight, with skill scores usually above 0.7. The model is also skillful in forecast of pollution aerosols, with most scores above 0.5. The model correctly predicted the dust outbreak events and their trans-Pacific transport, but it constantly missed the high dust concentrations observed in the boundary layer. We attribute this missing dust source to the desertification regions in the Inner Mongolia Province in China, which have developed in recent years but were not included in the model during forecasting. After incorporating the desertification sources, the model is able to reproduce the observed high dust concentrations at low altitudes over the Yellow Sea. Two key elements for a successful aerosol model forecast are correct source locations that determine where the emissions take place, and realistic forecast winds and convection that determine where the aerosols are transported. We demonstrate that our global model can not only account for the large-scale intercontinental transport, but also produce the small-scale spatial and temporal variations that are adequate for aircraft measurements planning.

Chin, Mian↗
NASA NTRS2021

Description of the NASA GEOS Composition Forecast Modeling System GEOS-CF v1.0

The Goddard Earth Observing System composition forecast (GEOS-CF) system is a high-resolution (0.25 degree) global constituent prediction system from NASA’s Global Modeling and Assimilation Office (GMAO). GEOS-CF offers a new tool for atmospheric chemistry research, with the goal to supplement NASA’s broad range of space-based and in-situ observation sand to support flight campaign planning, support of satellite observations, and air quality research. GEOS-CF expands on the GEOS weather and aerosol modeling system by introducing the GEOS-Chem chemistry module to provide analyses and 5-day forecasts of atmospheric constituents including ozone (O3), carbon monoxide (CO), nitrogen dioxide (NO2), and fine particulate matter (PM2.5). The chemistry module integrated in GEOS-CF is identical to the offline GEOS-Chem model and readily benefits from the innovations provided by the GEOS-Chem community.Evaluation of GEOS-CF against satellite, ozone sonde and surface observations show realistic simulated concentrations of O3, NO2, and CO, with normalized mean biases of -0.1 to -0.3, normalized root mean square errors (NRMSE) between 0.1-0.4, and correlations between 0.3-0.8. Comparisons against surface observations highlight the successful representation of air pollutants under a variety of meteorological conditions, yet also highlight current limitations, such as an over prediction of summertime ozone over the Southeast United States. GEOS-CFv1.0 generally overestimates aerosols by 20-50% due to known issues in GEOS-Chem v12.0.1 that have been addressed in later versions.The 5-day hourly forecasts have skill scores comparable to the analysis. Model skills can be improved significantly by applying a bias-correction to the surface model output using a machine-learning approach.

GEOS-CF↗
DOE OSTI2022

Convolutional neural networks for intra-hour solar forecasting based on sky image sequences

Accurate and timely solar forecasts play an increasingly critical role in power systems. Compared to longer forecasting timescales, very short-term solar forecasting has lagged behind in both research and practice. In this paper, we propose deep convolutional neural networks (CNNs) to provide operational intra-hour (10-minute-ahead to 60-minute-ahead) solar forecasts. We develop two CNN structures inspired by a widely-used CNN architecture. The CNNs are tailored to our solar forecasting regression tasks and rely solely on sky image sequences. Case studies based on six years of data (over 150,000 data points) demonstrate that the best CNN model has forecast skill scores of 20%-39% over the naive persistence of cloudiness benchmark, even at these very short timescales. The CNNs also have consistently superior performance when compared to shallow machine learning models with meteorological predictors, where the improvement averages around 7%. The sensitivity analyses show that the sky image length, resolution, and weather conditions have impacts on the deep learning model accuracy. In our intra-hour problem with specific setups, two sky images with a 10-minute 128 x 128 resolution yield the most accurate forecasts. Current limitations, future work, and deployment challenges and solutions are also discussed.

14 SOLAR ENERGY↗
DOE OSTI2020

Using Multiple Signatures to Improve Accuracy of Substorm Identification

Abstract We have developed a new procedure for combining lists of substorm onset times from multiple sources. We apply this procedure to observational data and to magnetohydrodynamic (MHD) model output from 1–31 January 2005. We show that this procedure is capable of rejecting false positive identifications and filling data gaps that appear in individual lists. The resulting combined onset lists produce a waiting time distribution that is comparable to previously published results, and superposed epoch analyses of the solar wind driving conditions and magnetospheric response during the resulting onset times are also comparable to previous results. Comparison of the substorm onset list from the MHD model to that obtained from observational data reveals that the MHD model reproduces many of the characteristic features of the observed substorms, in terms of solar wind driving, magnetospheric response, and waiting time distribution. Heidke skill scores show that the MHD model has statistically significant skill in predicting substorm onset times.

Haiducek, John D.↗
NASA NTRS2017

CEDAR-GEM Challenge for Systematic Assessment of Ionosphere/Thermosphere Models in Predicting TEC During the 2006 December Storm Event

In order to assess current modeling capability of reproducing storm impacts on total electron content (TEC), we considered quantities such as TEC, TEC changes compared to quiet time values, and the maximum value of the TEC and TEC changes during a storm. We compared the quantities obtained from ionospheric models against ground-based GPS TEC measurements during the 2006 AGU storm event (14-15 December 2006) in the selected eight longitude sectors. We used 15 simulations obtained from eight ionospheric models, including empirical, physics-based, coupled ionosphere-thermosphere, and data assimilation models. To quantitatively evaluate performance of the models in TEC prediction during the storm, we calculated skill scores such as RMS error, Normalized RMS error (NRMSE), ratio of the modeled to observed maximum increase (Yield), and the difference between the modeled peak time and observed peak time. Furthermore, to investigate latitudinal dependence of the performance of the models, the skill scores were calculated for five latitude regions. Our study shows that RMSE of TEC and TEC changes of the model simulations range from about 3 TECU (total electron content unit, 1 TECU = 1016 el m2) (in high latitudes) to about 13 TECU (in low latitudes), which is larger than latitudinal average GPS TEC error of about 2 TECU. Most model simulations predict TEC better than TEC changes in terms of NRMSE and the difference in peak time, while the opposite holds true in terms of Yield. Model performance strongly depends on the quantities considered, the type of metrics used, and the latitude considered.

Shim, J. S.↗
NASA NTRS2009

Validation of Community Models: 2. Development of a Baseline, Using the Wang-Sheeley-Arge Model

This paper is the second in a series providing independent validation of community models of the outer corona and inner heliosphere. Here I present a comprehensive validation of the Wang-Sheeley-Arge (WSA) model. These results will serve as a baseline against which to compare the next generation of comparable forecasting models. The WSA model is used by a number of agencies to predict Solar wind conditions at Earth up to 4 days into the future. Given its importance to both the research and forecasting communities, it is essential that its performance be measured systematically and independently. I offer just such an independent and systematic validation. I report skill scores for the model's predictions of wind speed and interplanetary magnetic field (IMF) polarity for a large set of Carrington rotations. The model was run in all its routinely used configurations. It ingests synoptic line of sight magnetograms. For this study I generated model results for monthly magnetograms from multiple observatories, spanning the Carrington rotation range from 1650 to 2074. I compare the influence of the different magnetogram sources and performance at quiet and active times. I also consider the ability of the WSA model to forecast both sharp transitions in wind speed from slow to fast wind and reversals in the polarity of the radial component of the IMF. These results will serve as a baseline against which to compare future versions of the model as well as the current and future generation of magnetohydrodynamic models under development for forecasting use.

MacNeice, Peter↗
NASA NTRS2008

Systematic Independent Validation of Inner Heliospheric Models

This presentation is the first in a series which will provide independent validation of community models of the outer corona and inner heliosphere. In this work we establish a set of measures to be used in validating this group of models. We use these procedures to generate a comprehensive set of results from the Wang- Sheeley-Arge (WSA) model which will be used as a baseline, or reference, against which to compare all other models. We also run a test of the validation procedures by applying them to a small set of results produced by the ENLIL Magnetohydrodynamic (MHD) model. In future presentations we will validate other models currently hosted by the Community Coordinated Modeling Center(CCMC), including a comprehensive validation of the ENLIL model. The Wang-Sheeley-Arge (WSA) model is widely used to model the Solar wind, and is used by a number of agencies to predict Solar wind conditions at Earth as much as four days into the future. Because it is so important to both the research and space weather forecasting communities, it is essential that its performance be measured systematically, and independently. In this paper we offer just such an independent and systematic validation. We report skill scores for the model's predictions of wind speed and IMF polarity for a large set of Carrington rotations. The model was run in all its routinely used configurations. It ingests line of sight magnetograms. For this study we generated model results for monthly magnetograms from the National Solar Observatory (SOLIS), Mount Wilson Observatory and the GONG network, spanning the Carrington rotation range from 1650 to 2068. We compare the influence of the different magnetogram sources, performance at quiet and active times, and estimate the effect of different empirical wind speed tunings. We also consider the ability of the WSA model to identify sharp transitions in wind speed from slow to fast wind. These results will serve as a baseline against which to compare future versions of the model

MacNeice, P. J.↗
NASA NTRS2013

Statistical and Hydrological Evaluation of TRMM-Based Multi-Satellite Precipitation Analysis over the Wangchu Basin of Bhutan: Are the Latest Satellite Precipitation Products 3B42V7 Ready for Use in Ungauged Basins?

The objective of this study is to quantitatively evaluate the successive Tropical Rainfall Measuring Mission (TRMM) Multi-satellite Precipitation Analysis (TMPA) products and further to explore the improvements and error propagation of the latest 3B42V7 algorithm relative to its predecessor 3B42V6 using the Coupled Routing and Excess Storage (CREST) hydrologic model in the mountainous Wangchu Basin of Bhutan. First, the comparison to a decade-long (2001-2010) daily rain gauge dataset reveals that: 1) 3B42V7 generally improves upon 3B42V6s underestimation both for the whole basin (bias from -41.15 to -8.38) and for a 0.250.25 grid cell with high-density gauges (bias from -40.25 to 0.04), though with modest enhancement of correlation coefficients (CC) (from 0.36 to 0.40 for basin-wide and from 0.37 to 0.41 for grid); and 2) 3B42V7 also improves its occurrence frequency across the rain intensity spectrum. Using the CREST model that has been calibrated with rain gauge inputs, the 3B42V6-based simulation shows limited hydrologic prediction NSCE skill (0.23 in daily scale and 0.25 in monthly scale) while 3B42V7 performs fairly well (0.66 in daily scale and 0.77 in monthly scale), a comparable skill score with the gauge rainfall simulations. After recalibrating the model with the respective TMPA data, significant improvements are observed for 3B42V6 across all categories, but not as much enhancement for the already well-performing 3B42V7 except for a reduction in bias (from -26.98 to -4.81). In summary, the latest 3B42V7 algorithm reveals a significant upgrade from 3B42V6 both in precipitation accuracy (i.e., correcting the underestimation) thus improving its potential hydrological utility. Forcing the model with 3B42V7 rainfall yields comparable skill scores with in-situ gauges even without recalibration of the hydrological model by the satellite precipitation, a compensating approach often used but not favored by the hydrology community, particularly in ungauged basins.

TRMM↗
DOE OSTI2019

Initial Results from the GEM Challenge on the Spacecraft Surface Charging Environment

Spacecraft surface charging during geomagnetically disturbed times is one of the most important causes of satellite anomalies. Predicting the surface charging environment is one prevalent task of the geospace environment models. Therefore, the GEM Focus Group “Inner Magnetosphere Cross-energy/Population Interactions” initiated a community-wide challenge study to assess the capability of several inner magnetosphere ring current models in determining surface charging environment for the Van Allen Probes orbits during the March 17, 2013 storm event. The integrated electron flux between 10 and 50 keV is used as the metrics. Various skill scores are applied to quantitatively measure the modeling performance against observations. Results indicate that no model consistently perform the best in all of the skill scores or for both satellites.We find that from these simulations the ring current model with observational flux boundary condition and Weimer electric potential driver generally reproduces the most realistic flux level around the spacecraft. A simple and weaker Volland-Stern electric field is not capable of effectively transporting the same plasma at the boundary towards the Earth. On the other hand, if the ring current model solves the electric field self-consistently and obtains similar strength and pattern in the equatorial plane as the Weimer model, the boundary condition plays another crucial role in determining the electron flux level in the inner region. Finally, when the boundary flux spectra based on MHD model/empirical model deviate from the shape or magnitude of the observed distribution function, the simulation produces poor skill scores along RBSP orbits.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2019

Initial Results from the GEM Challenge on the Spacecraft Surface Charging Environment

Spacecraft surface charging during geomagnetically disturbed times is one of the most important causes of satellite anomalies. Predicting the surface charging environment is one prevalent task of the geospace environment models. Therefore, the Geospace Environment Modeling (GEM) Focus Group "Inner Magnetosphere Crossenergy/Population Interactions" initiated a communitywide challenge study to assess the capability of several inner magnetosphere ring current models in determining surface charging environment for the Van Allen Probes orbits during the 17 March 2013 storm event. The integrated electron flux between 10 and 50 keV is used as the metrics. Various skill scores are applied to quantitatively measure the modeling performance against observations. Results indicate that no model consistently perform the best in all of the skill scores or for both satellites. We find that from these simulations the ring current model with observational flux boundary condition and Weimer electric potential driver generally reproduces the most realistic flux level around the spacecraft. A simple and weaker VollandStern electric field is not capable of effectively transporting the same plasma at the boundary toward the Earth. On the other hand, if the ring current model solves the electric field selfconsistently and obtains similar strength and pattern in the equatorial plane as the Weimer model, the boundary condition plays another crucial role in determining the electron flux level in the inner region. When the boundary flux spectra based on magnetohydrodynamics (MHD) model/empirical model deviate from the shape or magnitude of the observed distribution function, the simulation produces poor skill scores along Van Allen Probes orbits.

Yu, Yiqun↗
NASA NTRS2019

Comprehensive Assessment of Models and Events Using Library Tools (CAMEL) Framework: Time Series Comparisons

The Comprehensive Assessment of Models and Events using Library Tools (CAMEL) framework leverages existing Community Coordinated Modeling Center services: Run on Request post processing tools that generate model time series outputs and the new Community Coordinated Modeling Center Metadata Registry that describes simulation runs using Space Physics Archive Search and Extract metadata. The new CAMEL visualization tool compares the modeled time series with observational data and computes a suite of skill scores such as Prediction Efficiency, Root Mean Square Error, and Symmetric Signed Percentage Bias. Model data pairs used for skill calculations are obtained considering a user selected maximum difference between the time of observation and the nearest model output. The system renders available data for all locations and time periods selected using interactive visualizations that allow the user to zoom, pan, and pick data values along traces. Skill scores are reported for each selected event or aggregated over all events for all participating model runs. Separately, scores are reported for all locations (satellites or stations) and for each location individually. We are building on past experiences with model data comparisons of magnetosphere and ionosphere model outputs from GEM2008, GEMCEDAR Electrodynamics Thermosphere Ionosphere, and the SWPC Operational Space Weather Model challenges. The CAMEL visualization tool is demonstrated using three validation studies: (a) Wang Sheeley Arge heliosphere simulations compared against OMNI solar wind data, (b) ground magnetic perturbations from several magnetosphere and ionosphere electrodynamics models as observed by magnetometers, and (c) electron fluxes from several ring current simulations compared to Radiation Belt Storm Probes Helium Oxygen Proton Electron instrument measurements, integrated over different energy ranges.

Rastätter, Lutz↗
NASA NTRS2022

Multi-Variate LSTM Prediction of Alaska Magnetometer Chain Utilizing a Coupled Model Approach

During periods of rapidly changing geomagnetic conditions electric fields form within the Earth’s surface and induce currents known as geomagnetically induced currents(GICs), which interact with unprotected electrical systems our society relies on. In this study, we train multi-variate Long-Short Term Memory neural networks to predict magnitude of north-south component of the geomagnetic field (|BN|) at multiple ground magnetometer stations across Alaska provided by the SuperMAG database with a future goal of predicting geomagnetic field disturbances. Each neural network is driven by solar wind and interplanetary magnetic field inputs from the NASA OMNI database spanning from 2000–2015 and is fine tuned for each station to maximize the effectiveness in predicting |BN|. The neural networks are then compared against multivariate linear regression models driven with the same inputs at each station using Heidke skill scores with thresholds at the 50, 75, 85, and 99 percentiles for |BN|. The neural network models show significant increases over the linear regression models for |BN| thresholds. We also calculate the Heidke skill scores for d|BN|/dt by deriving d|BN|/dt from |BN| predictions. However, neural network models do not show clear outperformance compared to the linear regression models. To retain the sign information and thus predict BN instead of |BN|, a secondary so-called polarity model is utilized. The polarity model is run in tandem with the neural networks predicting geomagnetic field in a coupled model approach and results in a high correlation between predicted and observed values for all stations. We find this model a promising starting point for a machine learned geomagnetic field model to be expanded upon through increased output time history and fast turnaround times.

Matthew Blandin↗