Search NASA⌕ Search

SEARCH · Search NASA

Results for “prediction interval”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

A comparison of material flow strength models using Bayesian cross-validation

Predicting material flow strength over a range of conditions, such as temperature and strain rate, is necessary for many engineering applications. In this paper, we consider how to compare the predictiveness of several different strength models using a statistical technique called Bayesian cross-validation. Given a dataset of flow strength measurements obtained from mechanictesting experiments, the procedure consists of performing a Bayesian calibration of each strength model on a subset of the data and evaluating how well the trained models predict the remaining data. The predictiveness of a calibrated strength model is quantifiable probabilistically, which provides an interpretable metric for comparing the different models. As an illustrative example, we compare the Johnson-Cook (JC), Zerilli-Armstrong (ZA), Preston-Tonks-Wallace (PTW), and Mechanical Threshold Stress (MTS) flow strength models for the tantalum stress-strain curve data from Chen and Gray (1996). We show that prediction intervals for the four strength models cover the held-out data at most experimental conditions, but also that prediction interval coverage and prediction uncertainty varies by model and experimental condition. Finally, the analysis further allows us to identify experimental regimes for which one of the strength models predicts better than the other three.

36 MATERIALS SCIENCE↗

Statistical summaries of fatigue data for design purposes

Two methods are discussed for constructing a design curve on the safe side of fatigue data. Both the tolerance interval and equivalent prediction interval (EPI) concepts provide such a curve while accounting for both the distribution of the estimators in small samples and the data scatter. The EPI is also useful as a mechanism for providing necessary statistics on S-N data for a full reliability analysis which includes uncertainty in all fatigue design factors. Examples of statistical analyses of the general strain life relationship are presented. The tolerance limit and EPI techniques for defining a design curve are demonstrated. Examples usng WASPALOY B and RQC-100 data demonstrate that a reliability model could be constructed by considering the fatigue strength and fatigue ductility coefficients as two independent random variables. A technique given for establishing the fatigue strength for high cycle lives relies on an extrapolation technique and also accounts for "runners." A reliability model or design value can be specified.

Wirsching, P. H.↗

Conformalized-KANs: Uncertainty Quantification with Coverage Guarantees for Kolmogorov-Arnold Networks (KANs) in Scientific Machine Learning

This paper explores uncertainty quantification (UQ) methods in the context of Kolmogorov–Arnold Networks (KANs). We apply an ensemble approach to KANs to obtain a heuristic measure of UQ, enhancing interpretability and robustness in modeling complex functions. Building on this, we introduce Conformalized-KANs, which integrate conformal prediction, a distribution-free UQ technique, with KAN ensembles to generate calibrated prediction intervals with guaranteed coverage.} Extensive numerical experiments are conducted to evaluate the effectiveness of these methods, focusing particularly on the robustness and accuracy of the prediction intervals under various hyperparameter settings. We show that the conformal KAN predictions can be applied to recent extensions of KANs, including Finite Basis KANs (FBKANs) and multifideilty KANs (MFKANs). The results demonstrate the potential of our approaches to significantly improve the reliability and applicability of KANs in scientific machine learning.

• Artificial intelligence (AI) / machine learning ↗

Comparisons of Performance Metrics and Machine Learning Methods on an Entry Descent and Landing Database

This work focuses on evaluating machine learning methods and their applicability to the generation of an aerodynamic database, particularly for trajectory analysis of a capsule during entry, descent, and landing with a focus on uncertainty quantification. The source data to be used is the wind tunnel and computational data for the Integrated Design Assessment Team (IDAT) configuration of the Orion project, which has been publicly released. The methods used to generate the proposed databases are designed to naturally include a prediction interval, which will be evaluated both for their mean response as well as how well the prediction interval performs. These machine learning methods are compared to a traditionally generated database used by the Orion team as a baseline. It is found that while these machine learning methods perform well, the Orion database tends to still outperform them showing that engineering experience is still needed to make the best database possible. However, these methods still provide comparable results with significantly less effort.

Orion↗

Comparisons of Performance Metrics and Machine Learning Methods on an Entry, Descent, and Landing Database

This work focuses on evaluating machine learning methods and their applicability to the generation of an aerodynamic database, particularly for trajectory analysis of a capsule during entry, descent, and landing with a focus on uncertainty quantification. The source data to be used is the wind tunnel and computational data for the Integrated Design Assessment Team (IDAT) configuration of the Orion project, which has been publicly released. The methods used to generate the proposed databases are designed to naturally include a prediction interval, which will be evaluated both for their mean response as well as how well the prediction interval performs. These machine learning methods are compared to a traditionally generated database used by the Orion team as a baseline. It is found that while these machine learning methods perform well, the Orion database tends to still outperform them showing that engineering experience is still needed to make the best database possible. However, these methods still provide comparable results with significantly less effort.

Orion↗

A Comparison of Metamodeling Techniques via Numerical Experiments

This paper presents a comparative analysis of a few metamodeling techniques using numerical experiments for the single input-single output case. These experiments enable comparing the models' predictions with the phenomenon they are aiming to describe as more data is made available. These techniques include (i) prediction intervals associated with a least squares parameter estimate, (ii) Bayesian credible intervals, (iii) Gaussian process models, and (iv) interval predictor models. Aspects being compared are computational complexity, accuracy (i.e., the degree to which the resulting prediction conforms to the actual Data Generating Mechanism), reliability (i.e., the probability that new observations will fall inside the predicted interval), sensitivity to outliers, extrapolation properties, ease of use, and asymptotic behavior. The numerical experiments describe typical application scenarios that challenge the underlying assumptions supporting most metamodeling techniques.

Crespo, Luis G.↗

Bayesian hierarchical models for soil CO{sub 2} flux and leak detection at geologic sequestration sites

Proper characterizations of background soil CO{sub 2} respiration rates are critical for interpreting CO{sub 2} leakage monitoring results at geologic sequestration sites. In this paper, a method is developed for determining temperature-dependent critical values of soil CO{sub 2} flux for preliminary leak detection inference. The method is illustrated using surface CO{sub 2} flux measurements obtained from the AmeriFlux network fit with alternative models for the soil CO{sub 2} flux versus soil temperature relationship. The models are fit first to determine pooled parameter estimates across the sites, then using a Bayesian hierarchical method to obtain both global and site-specific parameter estimates. Model comparisons are made using the deviance information criterion (DIC), which considers both goodness of fit and model complexity. The hierarchical models consistently outperform the corresponding pooled models, demonstrating the need for site-specific data and estimates when determining relationships for background soil respiration. A hierarchical model that relates the square root of the CO{sub 2} flux to a quadratic function of soil temperature is found to provide the best fit for the AmeriFlux sites among the models tested. This model also yields effective prediction intervals, consistent with the upper envelope of the flux data across the modeled sites and temperature ranges. Calculation of upper prediction intervals using the proposed method can provide a basis for setting critical values in CO{sub 2} leak detection monitoring at sequestration sites.

54 ENVIRONMENTAL SCIENCES↗

Predicting the Size and Timing of Sunspot Maximum for Cycle 24

For cycle 24, the minimum value of the 12-month moving average (12-mma) of the AA-geomagnetic index in the vicinity of sunspot minimum (AAm) appears to have occurred in September 2009, measuring about 8.4 nT and following sunspot minimum by 9 months. This is the lowest value of AAm ever recorded, falling below that of 8.9 nT, previously attributed to cycle 14, which also is the smallest maximum amplitude (RM) cycle of the modern era (RM = 64.2). Based on the method of Ohl (the preferential association between RM and AAm for an ongoing cycle), one expects cycle 24 to have RM = 55+/-17 (the +/-1 - sigma prediction interval). Instead, using a variation of Ohl's method, one based on using 2-cycle moving averages (2-cma), one expects cycle 23's 2-cma of RM to be about 115.5+/-8.7 (the +/-1 - sigma prediction interval), inferring an RM of about 62+/-35 for cycle 24. Hence, it seems clear that cycle 24 will be smaller in size than was seen in cycle 23 (RM = 120.8) and, likely, will be comparable in size to that of cycle 14. From the Waldmeier effect (the preferential association between the ascent duration (ASC) and RM for an ongoing cycle), one expects cycle 24 to be a slow-rising cycle (ASC > or equal to 48 months), having RM occurrence after December 2012, unless it turns out to be a statistical outlier.

Wilson, Robert M.↗

On the Period-Amplitude and Amplitude-Period Relationships

Examined are Period-Amplitude and Amplitude-Period relationships based on the cyclic behavior of the 12-month moving averages of monthly mean sunspot numbers for cycles 0.23, both in terms of Fisher's exact tests for 2x2 contingency tables and linear regression analyses. Concerning the Period-Amplitude relationship (same cycle), because cycle 23's maximum amplitude is known to be 120.8, the inferred regressions (90-percent prediction intervals) suggest that its period will be 131 +/- 24 months (using all cycles) or 131 +/- 18 months (ignoring cycles 2 and 4, which have the extremes of period, 108 and 164 months, respectively). Because cycle 23 has already persisted for 142 months (May 1996 through February 2008), based on the latter prediction, it should end before September 2008. Concerning the Amplitude-Period relationship (following cycle maximum amplitude versus preceding cycle period), because cycle 23's period is known to be at least 142 months, the inferred regressions (90-percent prediction intervals) suggest that cycle 24's maximum amplitude will be about less than or equal to 96.1 +/- 55.0 (using all cycle pairs) or less than or equal to 91.0 +/- 36.7 (ignoring statistical outlier cycle pairs). Hence, cycle 24's maximum amplitude is expected to be less than 151, perhaps even less than 128, unless cycle pair 23/24 proves to be a statistical outlier.

Wilson, Robert M.↗

Uncertainty Reduction in Power Generation Forecast Using Coupled Wavelet-ARIMA

In this paper, we introduce a new approach without implying normal distributions and stationarity of power generation forecast errors. In addition, it is desired to more accurately quantify the forecast uncertainty by reducing prediction intervals of forecasts. We use automatically coupled wavelet transform and autoregressive integrated moving-average (ARIMA) forecasting to reflect multi-scale variability of forecast errors. The proposed analysis reveals slow-changing “quasi-deterministic” components of forecast errors. This helps improve forecasts produced by other means, e.g., using weather-based models, and reduce forecast errors prediction intervals.

Power system planning, uncertainty reduction, fore↗

Forecasting of municipal solid waste quantity in a developing country using multivariate grey models

Highlights: • Grey model can be used to forecast MSW quantity accurately with the limited data. • Prediction interval overcomes the uncertainty of MSW forecast effectively. • A multivariate model gives accuracy associated with factors affecting MSW quantity. • Population, urbanization, employment and household size play role for MSW quantity. - Abstract: In order to plan, manage and use municipal solid waste (MSW) in a sustainable way, accurate forecasting of MSW generation and composition plays a key role. It is difficult to carry out the reliable estimates using the existing models due to the limited data available in the developing countries. This study aims to forecast MSW collected in Thailand with prediction interval in long term period by using the optimized multivariate grey model which is the mathematical approach. For multivariate models, the representative factors of residential and commercial sectors affecting waste collected are identified, classified and quantified based on statistics and mathematics of grey system theory. Results show that GMC (1, 5), the grey model with convolution integral, is the most accurate with the least error of 1.16% MAPE. MSW collected would increase 1.40% per year from 43,435–44,994 tonnes per day in 2013 to 55,177–56,735 tonnes per day in 2030. This model also illustrates that population density is the most important factor affecting MSW collected, followed by urbanization, proportion employment and household size, respectively. These mean that the representative factors of commercial sector may affect more MSW collected than that of residential sector. Results can help decision makers to develop the measures and policies of waste management in long term period.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Challenges of COVID-19 Case Forecasting in the US, 2020–2021

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub ( https://covid19forecasthub.org ). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1–4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.

59 BASIC BIOLOGICAL SCIENCES↗

Precision Interval Estimation of the Response Surface by Means of an Integrated Algorithm of Neural Network and Linear Regression

The integration of Radial Basis Function Networks and Back Propagation Neural Networks with the Multiple Linear Regression has been accomplished to map nonlinear response surfaces over a wide range of independent variables in the process of the Modem Design of Experiments. The integrated method is capable to estimate the precision intervals including confidence and predicted intervals. The power of the innovative method has been demonstrated by applying to a set of wind tunnel test data in construction of response surface and estimation of precision interval.

Lo, Ching F.↗

Characterization and Uncertainty Analysis of a Reference Pressure Measurement System for Wind Tunnels

This paper presents the calibration results and uncertainty analysis of a high-precision reference pressure measurement system currently used in wind tunnels at the NASA Langley Research Center (LaRC). Sensors, calibration standards, and measurement instruments are subject to errors due to aging, drift with time, environment effects, transportation, the mathematical model, the calibration experimental design, and other factors. Errors occur at every link in the chain of measurements and data reduction from the sensor to the final computed results. At each link of the chain, bias and precision uncertainties must be separately estimated for facility use, and are combined to produce overall calibration and prediction confidence intervals for the instrument, typically at a 95% confidence level. The uncertainty analysis and calibration experimental designs used herein, based on techniques developed at LaRC, employ replicated experimental designs for efficiency, separate estimation of bias and precision uncertainties, and detection of significant parameter drift with time. Final results, including calibration confidence intervals and prediction intervals given as functions of the applied inputs, not as a fixed percentage of the full-scale value are presented. System uncertainties are propagated beginning with the initial reference pressure standard, to the calibrated instrument as a working standard in the facility. Among the several parameters that can affect the overall results are operating temperature, atmospheric pressure, humidity, and facility vibration. Effects of factors such as initial zeroing and temperature are investigated. The effects of the identified parameters on system performance and accuracy are discussed.

Amer, Tahani↗

Using the Modified Precursor Method to Estimate the Size of Cycle 24

Modified geomagnetic precursor techniques for predicting the size of the following sunspot cycle are developed, where these techniques use the 12-month moving averages of the number of disturbed days (when Ap greater than or equals 25), the Ap index, the aa index, and the aaI index at about 4 yr during the declining portion of the preceding sunspot cycle. For cycle 24, these techniques suggest that its RM will measure about 130 +/- 14, a value outside the consensus prediction interval of the low prediction (90 +/- 10) given by the NOAA Solar Cycle 24 Prediction Panel. Furthermore, cycle 24 is predicted to be a fast-rising cycle (ASC = 44 +/- 5 months), peaking before April 2012, presuming the official start of cycle 24 in March 2008. Also discussed are the variation of solar cycle lengths and Hale cycle effects, as related to cycles 23 and 24.

Wilson, Robert M.↗

Predicting the Size of Sunspot Cycle 24 on the Basis of Single- and Bi-Variate Geomagnetic Precursor Methods

Examined are single- and bi-variate geomagnetic precursors for predicting the maximum amplitude (RM) of a sunspot cycle several years in advance. The best single-variate fit is one based on the average of the ap index 36 mo prior to cycle minimum occurrence (E(Rm)), having a coefficient of correlation (r) equal to 0.97 and a standard error of estimate (se) equal to 9.3. Presuming cycle 24 not to be a statistical outlier and its minimum in March 2008, the fit suggests cycle 24 s RM to be about 69 +/- 20 (the 90% prediction interval). The weighted mean prediction of 11 statistically important single-variate fits is 116 +/- 34. The best bi-variate fit is one based on the maximum and minimum values of the 12-mma of the ap index; i.e., APM# and APm*, where # means the value post-E(RM) for the preceding cycle and * means the value in the vicinity of cycle minimum, having r = 0.98 and se = 8.2. It predicts cycle 24 s RM to be about 92 +/- 27. The weighted mean prediction of 22 statistically important bi-variate fits is 112 32. Thus, cycle 24's RM is expected to lie somewhere within the range of about 82 to 144. Also examined are the late-cycle 23 behaviors of geomagnetic indices and solar wind velocity in comparison to the mean behaviors of cycles 2023 and the geomagnetic indices of cycle 14 (RM = 64.2), the weakest sunspot cycle of the modern era.

Wilson, Robert M.↗

Advanced surrogate model for electron-scale turbulence in tokamak pedestals

We derive an advanced surrogate model for predicting turbulent transport at the edge of tokamaks driven by electron temperature gradient (ETG) modes. Our derivation is based on a recently developed sensitivity-driven sparse grid interpolation approach for uncertainty quantification and sensitivity analysis at scale, which informs the set of parameters that define the surrogate model as a scaling law. Our model reveals that ETG-driven electron heat flux is influenced by the safety factor q, electron beta β e and normalized electron Debye length λ D , in addition to well-established parameters such as the electron temperature and density gradients. To assess the trustworthiness of our model's predictions beyond training, we compute prediction intervals using bootstrapping. The surrogate model's predictive power is tested across a wide range of parameter values, including within-distribution testing parameters (to verify our model) as well as out-of-bounds and out-of-distribution testing (to validate the proposed model). Overall, validation efforts show that our model competes well with, or can even outperform, existing scaling laws in predicting ETG-driven transport.

fusion plasma↗

Uncertainty quantification of machine learning models to improve streamflow prediction under changing climate and environmental conditions

Machine learning (ML) models, and Long Short-Term Memory (LSTM) networks in particular, have demonstrated remarkable performance in streamflow prediction and are increasingly being used by the hydrological research community. However, most of these applications do not include uncertainty quantification (UQ). ML models are data driven and can suffer from large extrapolation errors when applied to changing climate/environmental conditions. UQ is required to quantify the influence of data noises on model predictions and avoid overconfident projections in extrapolation. In this work, we integrate a novel UQ method, called PI3NN, with LSTM networks for streamflow prediction. PI3NN calculates Prediction Intervals by training 3 Neural Networks. It can precisely quantify the predictive uncertainty caused by the data noise and identify out-of-distribution (OOD) data in a non-stationary condition to avoid overconfident predictions. We apply the PI3NN-LSTM method in the snow-dominant East River Watershed in the western US and in the rain-driven Walker Branch Watershed in the southeastern US. Results indicate that for the prediction data which have similar features as the training data, PI3NN precisely quantifies the predictive uncertainty with the desired confidence level; and for the OOD data where the LSTM network fails to make accurate predictions, PI3NN produces a reasonably large uncertainty indicating that the results are not trustworthy and should avoid overconfidence. PI3NN is computationally efficient, robust in performance, and generalizable to various network structures and data with no distributional assumptions. It can be broadly applied in ML-based hydrological simulations for credible prediction.

54 ENVIRONMENTAL SCIENCES↗