Search NASA⌕ Search

SEARCH · Search NASA

Results for “ensemble prediction”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

A Spatiotemporal-Aware Weighting Scheme for Improving Climate Model Ensemble Predictions

Multimodel ensembling has been widely used to improve climate model predictions, and the improvement strongly depends on the ensembling scheme. In this work, we propose a Bayesian neural network (BNN) ensembling method, which combines climate models within a Bayesian model averaging framework, to improve the predictive capability of model ensembles. Our proposed BNN approach calculates spatiotemporally varying model weights and biases by leveraging individual models' simulation skill, calibrates the ensemble prediction against observations by considering observation data uncertainty, and quantifies epistemic uncertainty when extrapolating to new conditions. More importantly, the BNN method provides interpretability about which climate model contributes more to the ensemble prediction at which locations and times. Thus, beyond its predictive capability, the method also brings insights and understanding of the models to guide further model and data development. In this study, we design experiments using an ensemble of CMIP6 climate model simulations to illustrate the BNN ensembling method's capability with respect to prediction accuracy, interpretability, and uncertainty quantification (UQ). We demonstrate that BNN can correctly assign larger weights to the regions and seasons where the individual model fits the observation better. Moreover, its offered interpretability is consistent with our understanding of localized climate model performance. Additionally, BNN shows an increasing uncertainty when the prediction is farther away from the period with constrained data, which appropriately reflects our trustworthiness of the models in the changing climate.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

A Gridded Solar Irradiance Ensemble Prediction System Based on WRF-Solar EPS and the Analog Ensemble

The WRF-Solar Ensemble Prediction System (WRF-Solar EPS) and a calibration method, the analog ensemble (AnEn), are used to generate calibrated gridded ensemble forecasts of solar irradiance over the contiguous United States (CONUS). Global horizontal irradiance (GHI) and direct normal irradiance (DNI) retrievals, based on geostationary satellites from the National Solar Radiation Database (NSRDB) are used for both calibrating and verifying the day-ahead GHI and DNI predictions (GDIP). A 10-member ensemble of WRF-Solar EPS is run in a re-forecast mode to generate day-ahead GDIP for three years. The AnEn is used to calibrate GDIP at each grid point independently using the NSRDB as the “ground truth”. Performance evaluations of deterministic and probabilistic attributes are carried out over the whole CONUS. The results demonstrate that using the AnEn calibrated ensemble forecast from WRF-Solar EPS contributes to improving the overall quality of the GHI predictions with respect to an AnEn calibrated system based only on the deterministic run of WRF-Solar. In fact, the calibrated WRF-Solar EPS’s mean exhibits a lower bias and RMSE than the calibrated deterministic WRF-Solar. Moreover, using the ensemble mean and spread as predictors for the AnEn allows a more effective calibration than using variables only from the deterministic runs. Finally, it has been shown that the recently introduced algorithm of correction for rare events is of paramount importance to obtain the lowest values of GHI from the calibrated ensemble (WRF-Solar EPS AnEn), qualitatively consistent with those observed from the NSRDB.

14 SOLAR ENERGY↗
DOE OSTI2022

Multimodel ensemble predictions of precipitation using bayesian neural networks

Multimodel ensembling improves predictions and considers model uncertainties. In this study, we present a Bayesian Neural Network (BNN) ensemble approach for large-scale precipitation predictions based on a set of climate models. BNN infers spatiotemporally varying model weights and biases through the calibration against observations. This ensemble scheme of BNN sufficiently leverages individual model skill for accurate predictions as well as provides interpretability about which models contribute more to the ensemble prediction at which locations and times to inform model development. Additionally, BNN accurately quantifies epistemic uncertainty to avoid overconfident projections. We demonstrate BNN’s superior prediction performance to three state-of-the-art ensemble approaches and discuss its interpretability and uncertainty quantification.

Fan, Patrick↗
DOE OSTI2022

Multimodel Ensemble Predictions of Precipitation using Bayesian Neural Networks

Multimodel ensembling improves predictions and considers model uncertainties. In this study, we present a Bayesian Neural Network (BNN) ensemble approach for large-scale precipitation predictions based on a set of climate models. BNN infers spatiotemporally varying model weights and biases through the calibration against observations. This ensemble scheme of BNN sufficiently leverages individual model skill for accurate predictions as well as provides interpretability about which models contribute more to the ensemble prediction at which locations and times to inform model development. Additionally, BNN accurately quantifies epistemic uncertainty to avoid overconfident projections. We demonstrate BNN’s superior prediction performance to three state-of-the-art ensemble approaches and discuss its interpretability and uncertainty quantification.

Fan, Patrick↗
DOE OSTI2024

Evaluating Ensemble Predictions of South Asian Monsoon Low Pressure System Genesis

Abstract Synoptic-scale vortices known as monsoon low pressure systems (LPSs) frequently produce intense precipitation and hydrological disasters in South Asia, so accurately forecasting LPS genesis is crucial for improving disaster preparedness and response. However, the accuracy of LPS genesis forecasts by numerical weather prediction models has remained unknown. Here, we evaluate the performance of two global ensemble models—the U.S. Global Ensemble Forecast System (GEFS) and the Ensemble Prediction System of the European Centre for Medium-Range Weather Forecasts (ECMWF)—in predicting LPS genesis during the years 2021–22. The GEFS successfully predicted about half the observed LPS genesis events 1–2 days in advance; the ECMWF model captured an additional 10% of observed genesis events. Both models had a false alarm ratio (FAR) of around 50% for 1–2-day lead times. In both ensembles, the control run typically exhibited a higher probability of detection (POD) of observed events and a lower FAR compared to the perturbed ensemble members. However, a consensus forecast, in which genesis is predicted when at least 20% of ensemble members forecast LPS formation, had POD values surpassing those of the control run for all lead times. Moreover, probabilistic predictions of genesis over the Bay of Bengal, where most LPSs form, were skillful, with the fraction of ensemble members predicting LPS formation over a 5-day lead time approximating the observed frequency of genesis, without any adjustment or bias correction.

Suhas, D. L.↗
DOE OSTI2021

The WRF-Solar Ensemble Prediction System to Provide Solar Irradiance Probabilistic Forecasts

In this study, we introduce the recently developed WRF-solar ensemble prediction system and a calibration method. The performances of forecast models are evaluated using the National Solar Radiation Database observational analysis for day-ahead solar irradiance predictions. The results demonstrate that the ensemble forecast improves the quality of the forecasts by considering the uncertainty of each ensemble member. The analog ensemble calibration contributed to the reduction of positive bias and an overall improvement in the probabilistic attributes, such as reliability and statistical consistency.

14 SOLAR ENERGY↗
DOE OSTI2023

Using Ultrasound Image Augmentation and Ensemble Predictions to Prevent Machine-Learning Model Overfitting

Deep learning predictive models have the potential to simplify and automate medical imaging diagnostics by lowering the skill threshold for image interpretation. However, this requires predictive models that are generalized to handle subject variability as seen clinically. Here, we highlight methods to improve test accuracy of an image classifier model for shrapnel identification using tissue phantom image sets. Using a previously developed image classifier neural network—termed ShrapML—blind test accuracy was less than 70% and was variable depending on the training/test data setup, as determined by a leave one subject out (LOSO) holdout methodology. Introduction of affine transformations for image augmentation or MixUp methodologies to generate additional training sets improved model performance and overall accuracy improved to 75%. Further improvements were made by aggregating predictions across five LOSO holdouts. This was done by bagging confidences or predictions from all LOSOs or the top-3 LOSO confidence models for each image prediction. Top-3 LOSO confidence bagging performed best, with test accuracy improved to greater than 85% accuracy for two different blind tissue phantoms. This was confirmed by gradient-weighted class activation mapping to highlight that the image classifier was tracking shrapnel in the image sets. Overall, data augmentation and ensemble prediction approaches were suitable for creating more generalized predictive models for ultrasound image analysis, a critical step for real-time diagnostic deployment.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2022

Process-oriented stochastic perturbations applied to the parametrization of turbulence and shallow convection for ensemble prediction

Weather forecasting nowadays often requires some estimation of uncertainties associated with the output of meteorological models, in order to better inform decision making, especially in the context of intense weather events. Ensemble prediction systems provide such information through sets of possible scenarios which are designed to represent various uncertainty sources, including model uncertainties. A wide variety of methods have been proposed to estimate model uncertainties, among which perturbation methods targeting uncertain processes are a promising research field. Here, we focus on the representation of small-scale variability by process-oriented perturbation schemes applied to two key physical processes, namely turbulence and shallow convection. The perturbations are applied to a single-column version of the convection-permitting AROME model, in three idealized boundary-layer cases. Large-eddy simulations (LESs) of the same cases serve as a reference for the subgrid variability that has to be represented, and the results are also compared to those given by the Stochastically Perturbed Parametrization Tendencies (SPPT) method, which is a method commonly used by weather forecast centres to represent model uncertainty. The spread produced by our process-oriented perturbations of turbulence and shallow convection does not represent all the small-scale variability implied by the LESs for temperature and humidity. However, it is of a similar order of magnitude for the wind, thanks to perturbations generated by the stochastic turbulence scheme. The dispersion is structurally different from what is obtained with SPPT. It is non-negligible in the lower levels, where SPPT perturbations are usually suppressed because of numerical instabilities, indicating a possible complementarity between the schemes.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

The WRF-Solar Ensemble Prediction System: Development, Test, and Validation

Providing reliable probabilistic solar radiation information is needed to improve management of the uncertainty and variability of solar generation. Thus, guidance on how to develop skillful and accurate ensemble forecasts is essential and it will ultimately contribute to integration of high amounts of solar energy on the grid. A team from the National Renewable Energy Laboratory and the National Center for Atmospheric Research had been collaborating to develop the WRF-Solar ensemble prediction system (WRF-Solar EPS) in the past three years to produce probabilistic solar irradiance forecasts and better predict solar energy by quantifying forecast uncertainty. The WRF-Solar EPS basically generates ensemble members for solar irradiance based on stochastic perturbations to provide intraday and day-ahead probabilistic forecasts. This study will present main research steps in developing the WRF-Solar EPS including: (a) tangent linear analysis for identifying key input variables of six WRF-Solar modules significantly related to predicting of cloud and solar irradiance, (b) combining stochastic perturbation technique with the WRF-Solar model, and (c) ensemble calibration method to decrease error and uncertainty of ensemble-based solar forecasts. The capability of WRF-Solar EPS is now updated to the most recent version of standard WRF model. This presentation will summarize comprehensive results from the evaluation of forecasts against the National Solar Radiation Data Base as well as ground-measured observations. Moreover, we will introduce the user's guide for WRF-Solar EPS (e.g., parameters to configure stochastic perturbations) and future extension of this research.

day-ahead forecast↗
DOE OSTI2021

EPIsembleVis: A geo-visual analysis and comparison of the prediction ensembles of multiple COVID-19 models

In this work, we present EPIsembleVis, a web-based comparative visual analysis tool for evaluating the consistency of multiple COVID-19 prediction models. Our approach analyzes a collection of COVID-19 predictions from different epidemiological models as an ensemble and utilizes two metrics to quantify model performance. These metrics include (a) prediction uncertainty (represented as the dispersion of predictions in each ensemble) and (b) prediction error (calculated by comparing individual model predictions with the recorded data). Through an interactive visual interface, our approach provides a data-driven workflow for (a) selecting and constructing the COVID-19 model prediction ensemble based on the spatiotemporal overlap of available predictions of multiple epidemiological models, (b) quantifying the model performance using both the uncertainty of each model prediction ensemble, and the error of each ensemble member that represents individual model predictions, and (c) visualizing the spatiotemporal variability in the projection performance of individual models using a suite of novel ensemble visualization techniques, such as the data availability map, a spatiotemporal textured-tile calendar, multivariate rose chart, and time-series leaflet glyph. We demonstrate the capability of our ensemble visual interface through a case study that investigates the performance of weekly COVID-19 predictions, which are provided through the COVID-19 Forecast Hub UMass-Amherst Influenza Forecasting Center of Excellence [47] for the United States and United States Territories. The EPIsembleVis tool is implemented using open-source web technologies and adaptive system design, rendering it interoperable with Elasticsearch and Kibana for automatically ingesting COVID-19 predictions from online repositories, and it is generalizable for analyzing worldwide projections from more epidemiological models.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2021

Assessment of Cloud Mask Forecasts from the WRF-Solar Ensemble Prediction System: Preprint

Numerical weather prediction (NWP) models are important tools used by government agencies and the renewable energy enterprise to forecast solar radiation. Cloud prediction, a key process of NWP models, has the highest impact on the accuracy of solar forecasting. This study uses satellite observations from the National Solar Radiation Data Base (NSRDB) to evaluate the cloud mask forecast by the WRF-Solar Ensemble Prediction System (WRF-Solar EPS). Preliminary analysis of the data in 2018 demonstrates the need of further improvement in predicting thin and low-level clouds. The information obtained from this work will be used to enhance the WRF-Solar EPS in reproducing the cloud field over the contiguous U.S. and reducing solar forecasting errors.

41 EE - Solar Energy Technologies Office (EE-4S)↗
NASA NTRS2021

Dynamic Ensemble Prediction of Cognitive Performance in Space

Astronauts are exposed to a unique set of stressors in spaceflight. Microgravity, isolation, confinement, and environmental and operational hazards: all of these can impact sleep, vigilant attention, and alertness, which are critical to mission success. In this paper, we seek to understand the most important predictors of alertness over the course of a space mission, using self-reported, cognitive, and environmental data collected from 24 astronauts on 6-month missions to the International Space Station (ISS). Alertness was repeatedly and objectively assessed on the ISS with a brief 3-minute Psychomotor Vigilance Test (PVT) that is highly sensitive to sleep deprivation. To relate PVT performance to time-varying and sparsely-measured environmental, operational, and psychological covariates, we propose a n ensemble prediction model comprising of linear mixed effects regression, random forest, and functional concurrent regression models. An extensive cross-validation procedure reveals that this ensemble outperforms any one of its components alone. We also discover that a participant’s past performance, reported fatigue and stress, and temperature and radiation exposure were among the most important variables associated with alertness. This method is broadly applicable to environmental studies where the main goal is accurate, individualized prediction involving a mixture of person-level traits and irregularly measured time series.

Danni Tu↗
DOE OSTI2021

The WRF-Solar Ensemble Prediction System to Provide Solar Irradiance Probabilistic Forecasts

In this study, we introduce the recently developed WRF-Solar Ensemble System (WRF-Solar EPS) and a calibration method. The performances of forecast models are evaluated using the National Solar Radiation Data Base (NSRDB) observational analysis for day-ahead solar irradiance predictions. The results demonstrate that the ensemble forecast improves the quality of the forecasts by taking into account the uncertainty of each ensemble member. The Analog Ensemble (AnEn) calibration contributed to the reduction of positive bias and an overall improvement in the probabilistic attributes such as reliability and statistical consistency.

analog ensemble↗
NASA NTRS2012

Optimal Initial Perturbations for Ensemble Prediction of the Madden-Julian Oscillation during Boreal Winter

An initialization strategy, tailored to the prediction of the Madden-Julian oscillation (MJO), is evaluated using the Goddard Earth Observing System Model, version 5 (GEOS-5), coupled general circulation model (CGCM). The approach is based on the empirical singular vectors (ESVs) of a reduced-space statistically determined linear approximation of the full nonlinear CGCM. The initial ESV, extracted using 10 years (1990-99) of boreal winter hindcast data, has zonal wind anomalies over the western Indian Ocean, while the final ESV (at a forecast lead time of 10 days) reflects a propagation of the zonal wind anomalies to the east over the Maritime Continent an evolution that is characteristic of the MJO. A new set of ensemble hindcasts are produced for the boreal winter season from 1990 to 1999 in which the leading ESV provides the initial perturbations. The results are compared with those from a set of control hindcasts generated using random perturbations. It is shown that the ESV-based predictions have a systematically higher bivariate correlation skill in predicting the MJO compared to those using the random perturbations. Furthermore, the improvement in the skill depends on the phase of the MJO. The ESV is particularly effective in increasing the forecast skill during those phases of the MJO in which the control has low skill (with correlations increasing by as much as 0.2 at 20 25-day lead times), as well as during those times in which the MJO is weak.

Ham, Yoo-Geun↗
DOE OSTI2024

HydraGNN_Predictive_GFM_2024 - Ensemble of predictive graph foundation models for ground state atomistic materials modeling

We provide the ensemble of fifteen pre-trained graph foundation models (GFMs) for atomistic materials modeling applications. Each one of the fifteen GFMs has been trained on five open-source datasets that (once aggregated) amount to over 154 million atomistic structures, which cover over two-thirds of the natural elements of the periodic table and that comprises a broad set of organic and inorganic compounds. This vast set of atomistic structures comprises ground state configurations that are dynamically stable (i.e., equilibrated structures with atomic forces approximately close to zero values) as well as dynamically unstable structures (i.e., non-equilibrium structures with non-negligible non-zero values of atomic forces). The ensemble of datasets aggregated does NOT include excited states. The datasets have been curated to remove atomistic structures with spectral norm of the force tensor above 100 eV/angstrom. Moreover, a linear term of the energy was computed for each dataset using a linear regression model that uses the chemical concentration of each natural element as regressor. The linear term predicted by the linear regression model has been subtracted from each original energy value to perform a re-alignment of the energy values across different electronic structures approximation theories performed to generate the diverse multi-source, multi-fidelity datasets. The folder "ADIOS_files" contains the set of pre-processed datasets in Adaptable I/O System (ADIOS) format (https://www.exascaleproject.org/research-project/adios/) that have been used for the development and training of GFMs in this work. The "ADIOS_files" directory contains 6 sub-directories named as follows: - ANI1x-v3.bp - MPTrj-v3.bp - OC2020-20M-v3.bp - OC2020-v3.bp - OC2022-v3.bp - qm7x-v3.bp Each sub-directory contains the pre-processed datasets converted in Adaptable I/O System (ADIOS) format (https://www.exascaleproject.org/research-project/adios/) that have been used to the development, training, and performance testing of the ensemble go predictive graph foundation models. Each GFM was developed using HydraGNN (https://github.com/ORNL/HydraGNN) as underlying graph neural network (GNN) architecture. The multi-task learning (MTL) capability of HydraGNN was used to simultaneously train the GFMs on labeled values for direct predictions of energy (a total system property of an atomistic structure that measures the chemical stability) and atomic forces (an atomic level property of an atomistic structure that measures the dynamical stability). The hyper parameters of the GFM have been tuned using scalable hyperparameter optimization (HPO) algorithms implemented in the software DeepHyper (https://github.com/deephyper/deephyper). The pre-training of each HPO trial was performed using distributed data parallelism (DDP) to scale the training across 128 compute nodes of the exascale OLCF supercomputer Frontier. Each HPO trial was trained only for 10 epochs and an early stopping was performed to avoid wasting significant computational resources on GNN architectures that were clearly underperforming. For each HPO trial, the 'omnistat' tool developed by (AMD Research - Advanced Micro Device) was used to measure the total energy consumption in kWh. The ensemble of GFMs was obtained by selecting the fifteen best performing HPO trials. Four models have been selected for their clear advantage in accuracy, and these are the GFMs with IDs 229, 156, 147, 260. Additional eleven models have been selected based on judicious balance between accuracy and energy consumption needed for training, and these are the GFMs with IDs 165, 78, 137, 1, 175, 171, 181, 67, 179, 167, 351. Each selected GFM of the ensemble was continued to cumulate a total of at most 30 epochs. In some cases, the total number of epochs actually performed was les than 30 due to two combined factors: (1) the size of the GFM (i.e., the number of model parameters to train) and (2) the total wall-clock time for which the computational resources could be allocated on OLCF-Frontier. The "Ensemble_of_models" directory contains 15 sub-directories named as follows: - gfm_0.229 - gfm_0.156 - gfm_0.147 - gfm_0.260 - gfm_0.165 - gfm_0.78 - gfm_0.137 - gfm_0.1 - gfm_0.175 - gfm_0.171 - gfm_0.181 - gfm_0.67 - gfm_0.179 - gfm_0.167 - gfm_0.351 Each one of these sub-directories refers to one of the fifteen HPO trials that have been selected to continue the pre-training with at most 30 epochs. With each sub-directory associated with a specific HPO trial, the following files can be found: - config.json: file for argument parsing to develop and train an HydraGNN architecture - gfm_0.ID_epoch_N.pk: file with model parameters for HPO ID trial after N epochs of training The ensemble of fifteen GFM architectures was used for (1) ensemble averaging to stabilize the predictions of energy and atomic forces after pre-training for post-processing analysis and (2) ensemble uncertainty quantification (UQ). The code used to develop, pre-train, and load the pre-trained models for post-processing analysis is available on the ORNL-GitHub at the following link: https://github.com/ORNL/HydraGNN/tree/Predictive_GFM_2024

36 MATERIALS SCIENCE↗