Search NASA⌕ Search

SEARCH · Search NASA

Results for “state reconstruction and forecasting”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Reduced-order autodifferentiable ensemble Kalman filters

This paper introduces a computational framework to reconstruct and forecast a partially observed state that evolves according to an unknown or expensive-to-simulate dynamical system. Our reduced-order autodifferentiable ensemble Kalman filters (ROAD-EnKFs) learn a latent low-dimensional surrogate model for the dynamics and a decoder that maps from the latent space to the state space. The learned dynamics and decoder are then used within an EnKF to reconstruct and forecast the state. Numerical experiments show that if the state dynamics exhibit a hidden low-dimensional structure, ROAD-EnKFs achieve higher accuracy at lower computational cost compared to existing methods. If such structure is not expressed in the latent state dynamics, ROAD-EnKFs achieve similar accuracy at lower cost, making them a promising approach for surrogate state reconstruction and forecasting.

Mathematics↗

A Hybrid Framework Combining Model-Based and Data-Driven Methods for Hierarchical Decentralized Robust Dynamic State Estimation

This paper combines model-based and data-driven methods to develop a hierarchical, decentralized, robust dynamic state estimator (DSE). A two-level hierarchy is proposed, where the lower level consists of robust, model-based, decentralized DSEs. The state estimates sent from the lower level are received at the upper level, where they are filtered by a robust data-driven DSE after a principled sparse selection. This selection allows us to shrink the dimension of the problem at the upper level and hence significantly speed up the computational time. The proposed hybrid framework does not depend on the centralized infrastructure of the control centers; thus it can be completely embedded into the wide-area measurement systems. This feature will ultimately facilitate the placement of hierarchical decentralized control schemes at the phasor data concentrator locations. Also, the network model is not necessary; thus, a topology processor is not required. Finally, there is no assumption on the dynamics of the electric loads. The proposed framework is tested on the 2,000-bus synthetic Texas system, and shown to be capable of reconstructing the dynamic states of the generators with high accuracy, and of forecasting in the advent of missing data.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Error Estimation of An Ensemble Statistical Seasonal Precipitation Prediction Model

This NASA Technical Memorandum describes an optimal ensemble canonical correlation forecasting model for seasonal precipitation. Each individual forecast is based on the canonical correlation analysis (CCA) in the spectral spaces whose bases are empirical orthogonal functions (EOF). The optimal weights in the ensemble forecasting crucially depend on the mean square error of each individual forecast. An estimate of the mean square error of a CCA prediction is made also using the spectral method. The error is decomposed onto EOFs of the predictand and decreases linearly according to the correlation between the predictor and predictand. Since new CCA scheme is derived for continuous fields of predictor and predictand, an area-factor is automatically included. Thus our model is an improvement of the spectral CCA scheme of Barnett and Preisendorfer. The improvements include (1) the use of area-factor, (2) the estimation of prediction error, and (3) the optimal ensemble of multiple forecasts. The new CCA model is applied to the seasonal forecasting of the United States (US) precipitation field. The predictor is the sea surface temperature (SST). The US Climate Prediction Center's reconstructed SST is used as the predictor's historical data. The US National Center for Environmental Prediction's optimally interpolated precipitation (1951-2000) is used as the predictand's historical data. Our forecast experiments show that the new ensemble canonical correlation scheme renders a reasonable forecasting skill. For example, when using September-October-November SST to predict the next season December-January-February precipitation, the spatial pattern correlation between the observed and predicted are positive in 46 years among the 50 years of experiments. The positive correlations are close to or greater than 0.4 in 29 years, which indicates excellent performance of the forecasting model. The forecasting skill can be further enhanced when several predictors are used.

Shen, Samuel S. P.↗

Neural net forecasting for geomagnetic activity

We use neural nets to construct nonlinear models to forecast the AL index given solar wind and interplanetary magnetic field (IMF) data. We follow two approaches: (1) the state space reconstruction approach, which is a nonlinear generalization of autoregressive-moving average models (ARMA) and (2) the nonlinear filter approach, which reduces to a moving average model (MA) in the linear limit. The database used here is that of Bargatze et al. (1985).

Hernandez, J. V.↗

Learning turbulent flows with generative models for super resolution and sparse flow reconstruction

Neural operators are promising surrogates for dynamical systems but when trained with standard L 2 losses they tend to oversmooth fine-scale turbulent structures. Here, we show that combining operator learning with generative modeling overcomes this limitation. We consider three practical turbulent-flow challenges where conventional neural operators fail: spatio-temporal super-resolution, forecasting, and sparse flow reconstruction. For Schlieren jet super-resolution, an adversarially trained neural operator (adv-NO) reduces the energy-spectrum error by 15 × while preserving sharp gradients at neural operator-like inference cost. For 3D homogeneous isotropic turbulence, adv-NO trained on only 160 timesteps from a single trajectory forecasts accurately for five eddy-turnover times and offers 114 × wall-clock speed-up at inference than the baseline diffusion-based forecasters, enabling near-real-time rollouts. For reconstructing cylinder wake flows from highly sparse Particle Tracking Velocimetry-like inputs, a conditional generative model infers full 3D velocity and pressure fields with correct phase alignment and statistics. These advances enable accurate reconstruction and forecasting at low compute cost, bringing near-real-time analysis and control within reach in experimental and computational fluid mechanics.

Fluid dynamics↗

United States benefits of improved worldwide wheat crop information from a LANDSAT system

The value of worldwide information improvements on wheat crops, promised by LANDSAT, is measured in the context of world wheat markets. These benefits are based on current LANDSAT technical goals and assume that information is made available to all (United States and other countries) at the same time. A detailed empirical sample demonstration of the effect of improved information is given; the history of wheat commodity prices for 1971-72 is reconstructed and the price changes from improved vs. historical information are compared. The improved crop forecasting from a LANDSAT system assumed include wheat crop estimates of 90 percent accuracy for each major wheat producing region. Accurate, objective worldwide wheat crop information using space systems may have a very stabilizing influence on world commodity markets, in part making possible the establishment of long-term, stable trade relationships.

Heiss, K. P.↗

Toward Drilling the Perfect Geothermal Well: An International Research Coordination Network for Geothermal Drilling Optimization Supported by Deep Machine Learning and Cloud Based Data Aggregation

The EDGE project, supported by the U.S. Department of Energy Geothermal Technologies Office under award DE-EE0008793, established a data-driven framework for improving the efficiency, cost-effectiveness, and reliability of geothermal well drilling. The project focused on developing scalable data infrastructure, advanced machine learning and probabilistic models, and integrated analytics tools to support continuous drilling optimization. A central objective was to reduce geothermal drilling costs by up to seventy percent while minimizing the risk of well failure through predictive diagnostics and adaptive planning. Over the project period, a comprehensive data repository was designed and deployed, incorporating records from over one hundred geothermal wells across varied geological settings. This repository supported both structured and unstructured data and adhered to FAIR data principles, enabling provenance tracking, quality control, and standardized metadata. The project introduced automated ingestion pipelines and a cloud-hosted platform that facilitated access to raw, processed, and derived datasets. This infrastructure served as the foundation for model development and analysis. Machine learning workflows were developed to predict key drilling metrics including rate of penetration, non-productive time, and total drilling costs. Self-organizing maps and dimensionality reduction methods were used to uncover operational patterns and outliers, while supervised learning algorithms such as random forests and deep neural networks were applied to forecast performance outcomes. The models were validated on heterogeneous datasets from both U.S. and Icelandic fields, demonstrating variable but significant predictive accuracy. The results indicated that finer temporal resolution, inclusion of lithological data, and consistency in operational annotations could substantially improve model performance. The project also implemented process mining techniques to reconstruct state-transition models from drilling event logs. These models enabled the identification of deviations from optimal workflows and provided insights into recurring failure modes. Analysis of non-productive time highlighted the impact of equipment failures, geological challenges, and human factors, offering opportunities for targeted mitigation strategies. The EDGE Dashboard was developed as a web-based expert system integrating data visualization, model outputs, and user-driven queries. It provided an accessible interface for operators to explore historical data, evaluate predicted outcomes, and compare drilling scenarios. Initial feedback from project partners suggested that the dashboard could serve as a foundation for more advanced advisory and optimization tools. Overall, the EDGE project demonstrated the feasibility and value of applying modern data science techniques to geothermal drilling. It delivered a set of interoperable tools and models that can support more efficient, lower-risk well development. The findings point toward a viable path for transitioning from advisory analytics to semi-autonomous drilling systems, contingent on continued collaboration, expanded datasets, and field validation. The project results have immediate relevance for drilling operations, data management practices, and future geothermal R&D efforts aimed at achieving reliable, cost-competitive geothermal energy at scale.

15 GEOTHERMAL ENERGY↗

Rao-Blackwellization for Adaptive Gaussian Sum Nonlinear Model Propagation

When dealing with imperfect data and general models of dynamic systems, the best estimate is always sought in the presence of uncertainty or unknown parameters. In many cases, as the first attempt, the Extended Kalman filter (EKF) provides sufficient solutions to handling issues arising from nonlinear and non-Gaussian estimation problems. But these issues may lead unacceptable performance and even divergence. In order to accurately capture the nonlinearities of most real-world dynamic systems, advanced filtering methods have been created to reduce filter divergence while enhancing performance. Approaches, such as Gaussian sum filtering, grid based Bayesian methods and particle filters are well-known examples of advanced methods used to represent and recursively reproduce an approximation to the state probability density function (pdf). Some of these filtering methods were conceptually developed years before their widespread uses were realized. Advanced nonlinear filtering methods currently benefit from the computing advancements in computational speeds, memory, and parallel processing. Grid based methods, multiple-model approaches and Gaussian sum filtering are numerical solutions that take advantage of different state coordinates or multiple-model methods that reduced the amount of approximations used. Choosing an efficient grid is very difficult for multi-dimensional state spaces, and oftentimes expensive computations must be done at each point. For the original Gaussian sum filter, a weighted sum of Gaussian density functions approximates the pdf but suffers at the update step for the individual component weight selections. In order to improve upon the original Gaussian sum filter, Ref. [2] introduces a weight update approach at the filter propagation stage instead of the measurement update stage. This weight update is performed by minimizing the integral square difference between the true forecast pdf and its Gaussian sum approximation. By adaptively updating each component weight during the nonlinear propagation stage an approximation of the true pdf can be successfully reconstructed. Particle filtering (PF) methods have gained popularity recently for solving nonlinear estimation problems due to their straightforward approach and the processing capabilities mentioned above. The basic concept behind PF is to represent any pdf as a set of random samples. As the number of samples increases, they will theoretically converge to the exact, equivalent representation of the desired pdf. When the estimated qth moment is needed, the samples are used for its construction allowing further analysis of the pdf characteristics. However, filter performance deteriorates as the dimension of the state vector increases. To overcome this problem Ref. [5] applies a marginalization technique for PF methods, decreasing complexity of the system to one linear and another nonlinear state estimation problem. The marginalization theory was originally developed by Rao and Blackwell independently. According to Ref. [6] it improves any given estimator under every convex loss function. The improvement comes from calculating a conditional expected value, often involving integrating out a supportive statistic. In other words, Rao-Blackwellization allows for smaller but separate computations to be carried out while reaching the main objective of the estimator. In the case of improving an estimator's variance, any supporting statistic can be removed and its variance determined. Next, any other information that dependents on the supporting statistic is found along with its respective variance. A new approach is developed here by utilizing the strengths of the adaptive Gaussian sum propagation in Ref. [2] and a marginalization approach used for PF methods found in Ref. [7]. In the following sections a modified filtering approach is presented based on a special state-space model within nonlinear systems to reduce the dimensionality of the optimization problem in Ref. [2]. First, the adaptive Gaussian sum propagation is explained and then the new marginalized adaptive Gaussian sum propagation is derived. Finally, an example simulation is presented.

state estimation↗

Abundance and properties of dark radiation from the cosmic microwave background

We study the cosmological signatures of new light relics that are collisionless like standard neutrinos or are strongly interacting. We provide a simple and succinct rephrasing of their physical effects in the cosmic microwave background, as well as the resulting parameter degeneracies with other cosmological parameters, in terms of the total radiation abundance and the fraction thereof that freely streams. In these more general terms, interacting and noninteracting light relics are differentiated by their respective decrease and increase of the free-streaming fraction, and, moreover, the scale-dependent interplay thereof with a common, correlated reduction of the fraction of matter in baryons. We then derive updated constraints on various dark-radiation scenarios with the latest cosmological observations, employing this language to identify the physical origin of the impact of each dataset. The “PR4” reanalyses of Planck CMB data prefer a larger primordial helium yield and therefore also slightly more radiation than the 2018 analysis; we investigate the differences between the two releases that drives these shifts. Smaller free-streaming fractions are disfavored by the excess lensing of the CMB measured in lensing reconstruction data from Planck and the Atacama Cosmology Telescope. On the other hand, baryon acoustic oscillation measurements from the Dark Energy Spectroscopic Instrument drive marginal detections of new, strongly interacting light relics due to that data's preference for lower matter fractions. Finally, we forecast measurements from the CMB-S4 experiment.

cosmological parameters from CMBR↗

Lessons Learned from Assimilating Altimeter Data into a Coupled General Circulation Model with the GMAO Augmented Ensemble Kalman Filter

Satellite altimetry measurements have provided global, evenly distributed observations of the ocean surface since 1993. However, the difficulties introduced by the presence of model biases and the requirement that data assimilation systems extrapolate the sea surface height (SSH) information to the subsurface in order to estimate the temperature, salinity and currents make it difficult to optimally exploit these measurements. This talk investigates the potential of the altimetry data assimilation once the biases are accounted for with an ad hoc bias estimation scheme. Either steady-state or state-dependent multivariate background-error covariances from an ensemble of model integrations are used to address the problem of extrapolating the information to the sub-surface. The GMAO ocean data assimilation system applied to an ensemble of coupled model instances using the GEOS-5 AGCM coupled to MOM4 is used in the investigation. To model the background error covariances, the system relies on a hybrid ensemble approach in which a small number of dynamically evolved model trajectories is augmented on the one hand with past instances of the state vector along each trajectory and, on the other, with a steady state ensemble of error estimates from a time series of short-term model forecasts. A state-dependent adaptive error-covariance localization and inflation algorithm controls how the SSH information is extrapolated to the sub-surface. A two-step predictor corrector approach is used to assimilate future information. Independent (not-assimilated) temperature and salinity observations from Argo floats are used to validate the assimilation. A two-step projection method in which the system first calculates a SSH increment and then projects this increment vertically onto the temperature, salt and current fields is found to be most effective in reconstructing the sub-surface information. The performance of the system in reconstructing the sub-surface fields is particularly impressive for temperature, but not as satisfactory for salt.

Keppenne, Christian↗

Latent Twins

Over the past decade, scientific machine learning has transformed the development of mathematical and computational frameworks for analyzing, modeling, and predicting complex systems. From inverse problems to numerical partial differential equations (PDEs), dynamical systems, and model reduction, these advances have pushed the boundaries of what can be simulated. Yet they have often progressed in parallel, with representation learning and algorithmic solution methods evolving largely as separate pipelines. With Latent Twins, we propose a unifying mathematical framework that creates a hidden surrogate in latent space for the underlying equations. Whereas digital twins mirror physical systems in the digital world, Latent Twins mirror mathematical systems in a learned latent space governed by operators. Through this lens, classical modeling, inversion, model reduction, and operator approximation all emerge as special cases of a single principle. We establish the fundamental approximation properties of Latent Twins for both ordinary differential equations (ODEs) and PDEs and demonstrate the framework across three representative settings: (i) canonical ODEs, capturing diverse dynamical regimes; (ii) a PDE benchmark using the shallow-water equations, contrasting Latent Twin simulations with deep operator network and forecasts with a four-dimensional variational method baseline; and (iii) a challenging real-data geopotential reanalysis dataset, reconstructing and forecasting from sparse, noisy observations. Latent Twins provide a compact, interpretable surrogate for solution operators that evaluate across arbitrary time gaps in a single-shot, while remaining compatible with scientific pipelines such as assimilation, control, and uncertainty quantification. Looking forward, this framework offers scalable, theory-grounded surrogates that bridge data-driven representation learning and classical scientific modeling across disciplines.

Latent Twins↗

Moving lens effect: Simulations, forecasts, and foreground mitigation

The peculiar motion of massive objects across the line of sight imprints a dipolar temperature anisotropy pattern on the cosmic microwave background known as the moving lens effect. This effect provides a unique probe of the transverse components of the peculiar velocity field, but has not yet been detected due to its small size. We implement and validate a stacking estimator for the moving lens signal using a galaxy catalog as a tracer of massive haloes combined with reconstructed velocities from the galaxy number density field. Using simulations, we forecast detection prospects for the moving lens signal from current and upcoming microwave background and galaxy surveys. Here, we demonstrate a new foreground mitigation strategy likely sufficient for current datasets, and discuss various sources of systematic error and noise. Upcoming galaxy surveys will provide high-significance statistical detections of the moving lens effect.

Astrophysical & cosmological simulations↗

A deep learning-based workflow for fast prediction of 3D state variables in geological carbon storage: A dimension reduction approach

Deep learning (DL) models are extensively used as surrogate models for high-fidelity simulations of multiphase fluid flow in porous media at large scales, enabling fast forecasts of the spatial–temporal evolution of three-dimensional (3D) state variables in geological carbon storage (GCS). However, training these models in high-dimensional space remains computationally demanding and prone to overfitting because of limited training data. This paper presents a novel workflow to address these challenges by integrating dimension reduction (DR) methods. Here, the proposed workflow employed pre-trained DR models to extract the latent variables of geological models and state variables and utilized the multi-layer perceptron (MLP) for constructing mapping functions between the input and output variables in latent spaces. Subsequently, the pre-trained reconstruction models converted the MLP-predicted latent state variables to their original high-dimensional form. Furthermore, we proposed a novel strategy for the DR and reconstruction of 3D saturation fields to account for the unique data characteristics of sparsity, nonuniformity, and discontinuity. The proposed strategy applied PCA and inverse PCA for 2D average saturation fields and developed a DL-based 3D reconstruction model, leveraging three 2D average saturation fields as input to produce a 3D saturation field as output. The pre-training of DR and reconstruction models and training of MLP models were conducted on 84 Gulf of Mexico (GoM) simulations and evaluated on 12 testing simulations. Each simulation contained 720 monthly time steps, with the first 360 months as the injection period and the rest as the post-injection period. The proposed workflow, incorporating DR and DL models, accurately predicts the normalized 3D pressure fields, achieving mean square error (MSE) of 2.92 × 10 -7 compared to the ground truth obtained from a full-physics simulator. Furthermore, the proposed strategy outperformed PCA and convolutional autoencoder (CAE) models on 3D saturation fields, resulting in minor workflow prediction errors with an MSE of 2.93 × 10 -5 . The results suggest the proposed workflow provides sufficient predictive fidelity across temporal and spatial scales, and enables a speedup of 160 times compared to the full-physics simulator, facilitating improved decision-making and risk assessment for large-scale GCS management in real-time scenarios.

3D reconstruction model↗

Using Wind Observations from Nearby Aircraft to Update the Optimal Descent Trajectory in Real-Time

The ability to meet a controlled time of arrival during a continuous descent operation will enable environmentally friendly and fuel efficient descent operations while simultaneously maintaining airport throughput. However, if the wind forecast used to compute the initial trajectory plan is not accurate enough, the guidance system will need to correct time deviations from the plan during the execution of the descent. Previous work proposed an on-board guidance strategy based on model predictive control, which repeatedly updates the trajectory plan in real-time from the current aircraft state and for the remainder of the descent. However, the wind conditions downstream, at altitudes not explored yet, were difficult to predict due to the lack of data. This paper shows the potential benefits of using wind observations, broadcast by nearby aircraft, to reconstruct the wind profile downstream. The wind profile in the trajectory optimization problem is modeled as a spline, which control points are updated to fit the observations before re-planning the trajectory. Results from simulations using realistic wind data show that the performance of model predictive control significantly improves when including up-to-date wind observations, in terms of time and energy errors at the metering fix and fuel consumption.

Dalmau, Ramon↗

How do North American weather regimes drive wind energy at the sub-seasonal to seasonal timescales?

Abstract There has been an increasing need for forecasting power generation at the subseasonal to seasonal (S2S) timescales to support the operation, management, and planning of the wind-energy system. At the S2S timescales, atmospheric variability is largely related to recurrent and persistent weather patterns, referred to as weather regimes (WRs). In this study, we identify four WRs that influence wind resources over North America using a universal two-stage procedure approach. These WRs are responsible for large-scale wind and power production anomalies over the CONUS at the S2S timescales. The WR-based reconstruction explains up to 40% of the monthly variance of power production over the western United States, and the explanatory power of WRs generally increases with the increase of timescales. The identified relationship between WRs and power production reveals the potential and limitations of the regional WR-based wind resource assessment over different regions of the CONUS across multiple timescales.

54 ENVIRONMENTAL SCIENCES↗

Spatiotemporal forecasting of the edge localized modes in tokamak plasmas using neural networks

Artificial intelligence techniques have been increasingly adopted by the plasma and fusion science to address problems like plasma reconstruction, surrogate modeling, and tokamak/stellarator optimization. A key focus in sustained fusion research is the prediction and mitigation of edge-localized-modes (ELMs), instabilities that occur in short, periodic bursts and can cause erosion to the tokamak vessel wall. Recent research has demonstrated the power of neural networks in approximating continuous functions. In this work, we build spatiotemporal forecasting models that can predict the onset of ELMs and their evolution at early stages. We leverage recent advances in generative modeling, sequence-to-sequence modeling, and Fourier neural operators to propose architectures and training strategies that can learn to forecast short to long term dynamics of the noisy signals due to ELMs. We benchmark the developed model against a state-of-the-art foundation model using the beam emission spectroscopy (BES) data that captures the plasma fluctuations due to ELMs over a 8 x 8 spatial grid. Our models demonstrate high accuracy, outperforming the baselines, in predicting the evolution of BES signals during ELM events. Furthermore, the developed models exhibit high accuracy in predicting the rapid rise and relaxation of the signals due to ELMs within 30–80 µs.

edge localized modes↗

Towards Physics Guided Optical Flow for Tracking Atmospheric Motion

Observations of atmospheric 3D winds are critical for improving short-range and long-range forecasting. Such advancement in forecasting directly applies to research in a number of areas including convective processes, wildfire plumes and tornado prediction. Atmospheric Motion Vectors (AMVs) provide a passively sensed approach to quantifying horizontal motion and cloud heights, which are typically sourced from geostationary sensors due to the availability of high frequency observations. Recent work has shown that estimating AMVs by tracking individual pixels with dense optical flow is a promising new direction. In this work, we compare state-of-the-art convolutional neural networks for optical flow in a physics-guided deep learning framework for predicting AMVs. The approach is semi-supervised and uses physically informed wind vectors from high-resolution numerical simulations for supervised learning followed by fine-tuning though warping and reconstruction of full-disk geostationary images (GOES-16). In the vertical direction, we use labels from the CALIPSO low-earth orbit satellite to predict cloud height from 16-band geostationary images with a neural network. We present results for both tasks on held-out time periods and secondary datasets.

Geostationary↗

Data and code repository for "How do the weather regimes drive wind speed and power production at the sub-seasonal to seasonal timescales over the CONUS?"

There has been an increasing need for forecasting power generation at the sub-seasonal to seasonal (S2S) timescales to support the operation, management, and planning of the wind-energy system. At the S2S timescales, atmospheric variability is largely related to recurrent and persistent weather patterns, referred to as weather regimes (WRs). In the study "How do the weather regimes drive wind speed and power production at the sub-seasonal to seasonal timescales over the CONUS?", we identify four WRs that influence wind resources over North America using a self-organizing map (SOM) algorithm. These WRs are responsible for large-scale wind and power production anomalies over the CONUS at the S2S timescales. The WR-based reconstruction explains up to 50% of the monthly variance of power production over the western United States, and the explanatory power generally increases with the increase of timescales. The identified relationship between WRs and power production reveals the potential and limitations of the regional WR-based wind resource assessment over different regions of the CONUS across multiple timescales. This repository includes all the data and codes we use for analyses in this study. Users may use them to reproduce the results of this study on their end.

17 WIND ENERGY↗