Building Physics-Based Solar Cycle Forecasts Using the Ensemble Kalman Filter Method
Explore the source record for details and available documents.
SMD↗
SEARCH · Search NASA
Results for “Ensemble Kalman methods”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Physics-Based Solar Cycle Forecast Using the Ensemble Kalman Filter Method
Explore the source record for details and available documents.
SMD↗
Non-asymptotic analysis of ensemble Kalman updates: effective dimension and localization
Many modern algorithms for inverse problems and data assimilation rely on ensemble Kalman updates to blend prior predictions with observed data. Ensemble Kalman methods often perform well with a small ensemble size, which is essential in applications where generating each particle is costly. This paper develops a non-asymptotic analysis of ensemble Kalman updates, which rigorously explains why a small ensemble size suffices if the prior covariance has moderate effective dimension due to fast spectrum decay or approximate sparsity. Here, we present our theory in a unified framework, comparing everal implementations of ensemble Kalman updates that use perturbed observations, square root filtering and localization. As part of our analysis, we develop new dimension-free covariance estimation bounds for approximately sparse matrices that may be of independent interest.
Mathematics↗
Analysis of a Computational Framework for Bayesian Inverse Problems: Ensemble Kalman Updates and MAP Estimators under Mesh Refinement
This paper analyzes a popular computational framework to solve infinite-dimensional Bayesian inverse problems, discretizing the prior and the forward model in a finite-dimensional weighted inner product space. We demonstrate the benefit of working on a weighted space by establishing operator-norm bounds for finite element and graph-based discretizations of Matérn-type priors and deconvolution forward models. For linear-Gaussian inverse problems, we develop a general theory to characterize the error in the approximation to the posterior. We also embed the computational framework into ensemble Kalman methods and MAP estimators for nonlinear inverse problems. Furthermore, our operator-norm bounds for prior discretizations guarantee the scalability and accuracy of these algorithms under mesh refinement.
Bayesian inverse problem↗
The Principle of Energetic Consistency
A basic result in estimation theory is that the minimum variance estimate of the dynamical state, given the observations, is the conditional mean estimate. This result holds independently of the specifics of any dynamical or observation nonlinearity or stochasticity, requiring only that the probability density function of the state, conditioned on the observations, has two moments. For nonlinear dynamics that conserve a total energy, this general result implies the principle of energetic consistency: if the dynamical variables are taken to be the natural energy variables, then the sum of the total energy of the conditional mean and the trace of the conditional covariance matrix (the total variance) is constant between observations. Ensemble Kalman filtering methods are designed to approximate the evolution of the conditional mean and covariance matrix. For them the principle of energetic consistency holds independently of ensemble size, even with covariance localization. However, full Kalman filter experiments with advection dynamics have shown that a small amount of numerical dissipation can cause a large, state-dependent loss of total variance, to the detriment of filter performance. The principle of energetic consistency offers a simple way to test whether this spurious loss of variance limits ensemble filter performance in full-blown applications. The classical second-moment closure (third-moment discard) equations also satisfy the principle of energetic consistency, independently of the rank of the conditional covariance matrix. Low-rank approximation of these equations offers an energetically consistent, computationally viable alternative to ensemble filtering. Current formulations of long-window, weak-constraint, four-dimensional variational methods are designed to approximate the conditional mode rather than the conditional mean. Thus they neglect the nonlinear bias term in the second-moment closure equation for the conditional mean. The principle of energetic consistency implies that, to precisely the extent that growing modes are important in data assimilation, this term is also important.
Cohn, Stephen E.↗
Global Evolution of Solar Magnetic Fields and Prediction of Solar Activity Cycles
Prediction of solar activity cycles is challenging because the physical processes inside the Sun involve a broad range of multiscale dynamics that no model can reproduce, and the available observations are highly limited and cover mostly surface layers. Helioseismology makes it possible to probe solar dynamics in the convective zone, but variations in the differential rotation and meridional circulation are currently available for only two solar activity cycles. It has been demonstrated that sunspot observations, which cover over 400 years, can be used to calibrate the Parker-Kleeorin-Ruzmaikin model and that the Ensemble Kalman Filter (EnKF) method can be used to link the model magnetic fields to sunspot observations to make reliable predictions of a following cycle. However, for more accurate predictions, it is necessary to use actual observations of the solar magnetic fields, which are available for only four solar cycles. This raises the question of how limitations in observational data and model uncertainties affect predictive capabilities and implies the need for the development of new forecast methodologies and validation criteria. In this presentation, I will discuss the influence of the limited number of available observations on the accuracy of EnKF estimates of solar cycle parameters.
Kitiashvili, Irina N.↗
Application of Synoptic Magnetograms for Prediction of Solar Activity Using Ensemble Kalman Filter
Solar activity predictions using the data assimilation approach have demonstrated great potential to build reliable long-term forecasts of solar activity. In particular, it has been shown that the Ensemble Kalman Filter (EnKF) method applied to a non-linear dynamo model is capable of predicting solar activity up to one sunspot cycle ahead in time, as well as estimating the properties of the next cycle a few years before it begins. These developments assume an empirical relationship between the mean toroidal magnetic field flux and the sunspot number. Estimated from the sunspot number series, variations of the toroidal field have been used to assimilate the data into the Parker-Kleeorin-Ruzmakin (PKR) dynamo model by applying the EnKF method. The dynamo model describes the evolution of the toroidal and poloidal components of the magnetic field and the magnetic helicity. Full-disk magnetograms provide more accurate and complete input data by constraining both the toroidal and poloidal global field components, but these data are available only for the last four solar cycles. In this presentation, using the available magnetogram data, we discuss development of the methodology and forecast quality criteria (including forecast uncertainties and sources of errors). We demonstrate the influence of limited time series observations on the accuracy of solar activity predictions. We present EnKF predictions of the upcoming Solar Cycle 25 based on both the sunspot number series and observed magnetic fields and discuss the uncertainties and potential of the data assimilation approach.
Kitiashvili, Irina N.↗
Solar Activity Modeling: From Subgranular Dynamical Scales to the Solar Cycles
Dynamical effects of solar magnetoconvection span a wide range spatial and temporal scales that extends from the interior to the corona and from fast turbulent motions to the global-Sun magnetic activity. To study the solar activity on short temporal scales (from minutes to hours), we use 3D radiative MHD simulations that allow us to investigate complex turbulent interactions that drive various phenomena, such as plasma eruptions, spontaneous formation of magnetic structures, funnel-like structures and magnetic loops in the corona, and others. In particular, we focus on multi-scale processes of energy exchange across the different layers, which contribute to the corona heating and eruptive dynamics, as well as interlinks between different layers of the solar interior and atmosphere. For modeling the global-scale activity we use the data assimilation approach that has demonstrated great potential for building reliable long-term forecasts of solar activity. In particular, it has been shown that the Ensemble Kalman Filter (EnKF) method applied to the Parker-Kleeorin-Ruzmakin dynamo model is capable of predicting solar activity up to one sunspot cycle ahead in time, as well as estimating the properties of the next cycle a few years before it begins. In this presentation, using the available magnetogram data, we discuss development of the methodology and forecast quality criteria (including forecast uncertainties and sources of errors). We demonstrate the influence of observational limitation on the prediction accuracy. We present the EnKF predictions of the upcoming Solar Cycle 25 based on both the sunspot number series and observed magnetic fields, and discuss the uncertainties and potential of the data assimilation approach for modeling and forecasting the solar activity.
Kitiashvili, I. N.↗
Solar Activity Modeling: From Subgranular Dynamical Scales to the Solar Cycles
The dynamical effects of solar magnetoconvection span a wide range spatial and temporal scales that extend from the interior to the corona and from fast turbulent motions to global magnetic activity. To study the solar activity on short temporal scales (from minutes to hours), we use 3D radiative MHD simulations that allow us to investigate complex turbulent interactions that drive various phenomena, such as plasma eruptions, spontaneous formation of magnetic structures, funnel-like structures and magnetic loops in the corona, and others. In particular, we focus on multi-scale processes of energy exchange across layers of the solar interior and atmosphere, which contribute to coronal heating and eruptive dynamics. For modeling global-scale activity, we use a data assimilation approach that has demonstrated great potential for building reliable long-term forecasts of solar activity. In particular, it has been shown that the Ensemble Kalman Filter (EnKF) method applied to the Parker-Kleeorin-Ruzmakin dynamo model is capable of predicting solar activity up to one sunspot cycle ahead in time, as well as estimating the properties of the next cycle a few years before it begins. In this presentation, using the available magnetogram data, we discuss development of the methodology and forecast quality criteria (including forecast uncertainties and sources of errors). We demonstrate the influence of observational limitations on prediction accuracy, and we present the EnKF predictions of the upcoming Solar Cycle (25) based on both the sunspot number series and observed magnetic fields and discuss the uncertainties and potential of the data assimilation approach for modeling and forecasting solar activity.
Kitiashvili, I. N.↗
Early Estimation of Solar Activity Cycle: Potential Capability and Limits
The variable solar magnetic activity known as the 11-year solar cycle has the longest history of solar observations. These cycles dramatically affect conditions in the heliosphere and the Earth's space environment. Our current understanding of the physical processes that make up global solar dynamics and the dynamo that generates the magnetic fields is sketchy, resulting in unrealistic descriptions in theoretical and numerical models of the solar cycles. The absence of long-term observations of solar interior dynamics and photospheric magnetic fields hinders development of accurate dynamo models and their calibration. In such situations, mathematical data assimilation methods provide an optimal approach for combining the available observational data and their uncertainties with theoretical models in order to estimate the state of the solar dynamo and predict future cycles. In this presentation, we will discuss the implementation and performance of an Ensemble Kalman Filter data assimilation method based on the Parker migratory dynamo model, complemented by the equation of magnetic helicity conservation and longterm sunspot data series. This approach has allowed us to reproduce the general properties of solar cycles and has already demonstrated a good predictive capability for the current cycle, 24. We will discuss further development of this approach, which includes a more sophisticated dynamo model, synoptic magnetogram data, and employs the DART Data Assimilation Research Testbed.
Kitiashvili, Irina N.↗
Using Data Assimilation Methods of Prediction of Solar Activity
The variable solar magnetic activity known as the 11-year solar cycle has the longest history of solar observations. These cycles dramatically affect conditions in the heliosphere and the Earth's space environment. Our current understanding of the physical processes that make up global solar dynamics and the dynamo that generates the magnetic fields is sketchy, resulting in unrealistic descriptions in theoretical and numerical models of the solar cycles. The absence of long-term observations of solar interior dynamics and photospheric magnetic fields hinders development of accurate dynamo models and their calibration. In such situations, mathematical data assimilation methods provide an optimal approach for combining the available observational data and their uncertainties with theoretical models in order to estimate the state of the solar dynamo and predict future cycles. In this presentation, we will discuss the implementation and performance of an Ensemble Kalman Filter data assimilation method based on the Parker migratory dynamo model, complemented by the equation of magnetic helicity conservation and long-term sunspot data series. This approach has allowed us to reproduce the general properties of solar cycles and has already demonstrated a good predictive capability for the current cycle, 24. We will discuss further development of this approach, which includes a more sophisticated dynamo model, synoptic magnetogram data, and employs the DART Data Assimilation Research Testbed.
Prediction↗
Using Data Assimilation Methods for Prediction of Solar Activity
The variability of solar magnetic activity known as the 11-‐year solar cycles has the longest history of observations. These solar cycles dramatically affect conditions in the heliosphere and the Earth’s space environment. Our current understanding of the physical processes that make up global solar dynamics and the dynamo that generates the magnetic fields is sketchy, resulting in unrealistic descriptions in theoretical and numerical models of the solar cycles. The absence of long-‐term observations of solar interior dynamics and photospheric magnetic fields hinders development of accurate dynamo models and their calibration. In such situations, mathematical data assimilation methods provide an optimal approach for combining the available observational data and their uncertainties with theoretical models in order to estimate the state of the solar dynamo and predict future cycles. In this presentation, we will discuss the implementation and performance of an Ensemble Kalman Filter data assimilation method based on the Parker migratory dynamo model complemented by the equation of magnetic helicity conservation and long-‐term sunspot data series. This approach has allowed us to reproduce the general properties of the solar cycles and has already demonstrated a good predictive capability for the current cycle, 24. We will discuss further development of this approach, which includes a more sophisticated dynamo model, synoptic magnetogram data, and employs the DART Data Assimilation Research Testbed.
Solar Activity↗
A Particle Batch Smoother Approach to Snow Water Equivalent Estimation
This paper presents a newly proposed data assimilation method for historical snow water equivalent SWE estimation using remotely sensed fractional snow-covered area fSCA. The newly proposed approach consists of a particle batch smoother (PBS), which is compared to a previously applied Kalman-based ensemble batch smoother (EnBS) approach. The methods were applied over the 27-yr Landsat 5 record at snow pillow and snow course in situ verification sites in the American River basin in the Sierra Nevada (United States). This basin is more densely vegetated and thus more challenging for SWE estimation than the previous applications of the EnBS. Both data assimilation methods provided significant improvement over the prior (modeling only) estimates, with both able to significantly reduce prior SWE biases. The prior RMSE values at the snow pillow and snow course sites were reduced by 68%-82% and 60%-68%, respectively, when applying the data assimilation methods. This result is encouraging for a basin like the American where the moderate to high forest cover will necessarily obscure more of the snow-covered ground surface than in previously examined, less-vegetated basins. The PBS generally outperformed the EnBS: for snow pillows the PBSRMSE was approx.54%of that seen in the EnBS, while for snow courses the PBSRMSE was approx.79%of the EnBS. Sensitivity tests show relative insensitivity for both the PBS and EnBS results to ensemble size and fSCA measurement error, but a higher sensitivity for the EnBS to the mean prior precipitation input, especially in the case where significant prior biases exist.
EnBS↗
Scalable Bayesian Physics-Informed Kolmogorov-Arnold Networks
Uncertainty quantification (UQ) plays a pivotal role in scientific machine learning, especially when surrogate models are used to approximate complex systems. Although multilayer perceptions (MLPs) are commonly employed as surrogates, they often suffer from overfitting due to their large number of parameters. Kolmogorov-Arnold networks (KANs) offer an alternative solution with fewer parameters. However, gradient-based inference methods, such as Hamiltonian Monte Carlo (HMC), may result in computational inefficiency when applied to KANs, especially for large-scale datasets, due to the high cost of back-propagation. To address these challenges, we propose a novel approach, combining the dropout Tikhonov ensemble Kalman inversion (DTEKI) with Chebyshev KANs. This gradient-free method effectively mitigates overfitting and enhances numerical stability. In addition, we incorporate the active subspace method to reduce the parameter-space dimensionality, allowing us to improve the accuracy of predictions and obtain more reliable uncertainty estimates. Extensive experiments demonstrate the efficacy of our approach in various test cases, including scenarios with large datasets and high noise levels. Our results show that the new method achieves comparable or better accuracy, much higher efficiency as well as stability compared to HMC, in addition to scalability. Moreover, by leveraging the low-dimensional parameter subspace, our method preserves prediction accuracy while substantially reducing further the computational cost.
97 MATHEMATICS AND COMPUTING↗
Ensemble Data Assimilation Without Ensembles: Methodology and Application to Ocean Data Assimilation
Two methods to estimate background error covariances for data assimilation are introduced. While both share properties with the ensemble Kalman filter (EnKF), they differ from it in that they do not require the integration of multiple model trajectories. Instead, all the necessary covariance information is obtained from a single model integration. The first method is referred-to as SAFE (Space Adaptive Forecast error Estimation) because it estimates error covariances from the spatial distribution of model variables within a single state vector. It can thus be thought of as sampling an ensemble in space. The second method, named FAST (Flow Adaptive error Statistics from a Time series), constructs an ensemble sampled from a moving window along a model trajectory. The underlying assumption in these methods is that forecast errors in data assimilation are primarily phase errors in space and/or time.
Data Assimilation↗
Sequential Kalman tuning of the t -preconditioned Crank-Nicolson algorithm: efficient, adaptive and gradient-free inference for Bayesian inverse problems
Ensemble Kalman Inversion (EKI) has been proposed as an efficient method for the approximate solution of Bayesian inverse problems with expensive forward models. However, when applied to the Bayesian inverse problem EKI is only exact in the regime of Gaussian target measures and linear forward models. Here, in this work we propose embedding EKI and Flow Annealed Kalman Inversion, its normalizing flow (NF) preconditioned variant, within a Bayesian annealing scheme as part of an adaptive implementation of the t-preconditioned Crank-Nicolson (tpCN) sampler. The tpCN sampler differs from standard pCN in that its proposal is reversible with respect to the multivariate t-distribution. The more flexible tail behaviour allows for better adaptation to sampling from non-Gaussian targets. Within our Sequential Kalman Tuning (SKT) adaptation scheme, EKI is used to initialize and precondition the tpCN sampler for each annealed target. The subsequent tpCN iterations ensure particles are correctly distributed according to each annealed target, avoiding the accumulation of errors that would otherwise impact EKI. We demonstrate the performance of SKT for tpCN on three challenging numerical benchmarks, showing significant improvements in the rate of convergence compared to adaptation within standard SMC with importance weighted resampling at each temperature level, and compared to similar adaptive implementations of standard pCN. The SKT scheme applied to tpCN offers an efficient, practical solution for solving the Bayesian inverse problem when gradients of the forward model are not available. Code implementing the SKT schemes for tpCN is available at https://github.com/RichardGrumitt/KalmanMC.
97 MATHEMATICS AND COMPUTING↗
Joint state-parameter estimation for the reduced fracture model via the united filter
Here, in this paper, we introduce an effective United Filter method for jointly estimating the solution state and physical parameters in flow and transport problems within fractured porous media. Fluid flow and transport in fractured porous media are critical in subsurface hydrology, geophysics, and reservoir geomechanics. Reduced fracture models, which represent fractures as lower-dimensional interfaces, enable efficient multi-scale simulations. However, reduced fracture models also face accuracy challenges due to modeling errors and uncertainties in physical parameters such as permeability and fracture geometry. To address these challenges, we propose a United Filter method, which integrates the Ensemble Score Filter (EnSF) for state estimation with the Direct Filter for parameter estimation. EnSF, based on a score-based diffusion model framework, produces ensemble representations of the state distribution without deep learning. Meanwhile, the Direct Filter, a recursive Bayesian inference method, estimates parameters directly from state observations. The United Filter combines these methods iteratively: EnSF estimates are used to refine parameter values, which are then fed back to improve state estimation. Numerical experiments demonstrate that the United Filter method surpasses the state-of-the-art Augmented Ensemble Kalman Filter, delivering more accurate state and parameter estimation for reduced fracture models. This framework also provides a robust and efficient solution for PDE-constrained inverse problems with uncertainties and sparse observations.
Bayesian inference↗
Advances in Land Data Assimilation at the NASA Goddard Space Flight Center
Research in land surface data assimilation has grown rapidly over the last decade. In this presentation we provide a brief overview of key research contributions by the NASA Goddard Space Flight Center (GSFC). The GSFC contributions to land assimilation primarily include the continued development and application of the Land Information System (US) and the ensemble Kalman filter (EnKF). In particular, we have developed a method to generate perturbation fields that are correlated in space, time, and across variables and that permit the flexible modeling of errors in land surface models and observations, along with an adaptive filtering approach that estimates observation and model error input parameters. A percentile-based scaling method that addresses soil moisture biases in model and observational estimates opened the path to the successful application of land data assimilation to satellite retrievals of surface soil moisture. Assimilation of AMSR-E surface soil moisture retrievals into the NASA Catchment model provided superior surface and root zone assimilation products (when validated against in situ measurements and compared to the model estimates or satellite observations alone). The multi-model capabilities of US were used to investigate the role of subsurface physics in the assimilation of surface soil moisture observations. Results indicate that the potential of surface soil moisture assimilation to improve root zone information is higher when the surface to root zone coupling is stronger. Building on this experience, GSFC leads the development of the Level 4 Surface and Root-Zone Soil Moisture (L4_SM) product for the planned NASA Soil-Moisture-Active-Passive (SMAP) mission. A key milestone was the design and execution of an Observing System Simulation Experiment that quantified the contribution of soil moisture retrievals to land data assimilation products as a function of retrieval and land model skill and yielded an estimate of the error budget for the SMAP L4_SM product. Terrestrial water storage observations from GRACE satellite system were also successfully assimilated into the NASA Catchment model and provided improved estimates of groundwater variability when compared to the model estimates alone. Moreover, satellite-based land surface temperature (LST) observations from the ISCCP archive were assimilated using a bias estimation module that was specifically designed for LST assimilation. As with soil moisture, LST assimilation provides modest yet statistically significant improvements when compared to the model or satellite observations alone. To achieve the improvement, however, the LST assimilation algorithm must be adapted to the specific formulation of LST in the land model. An improved method for the assimilation of snow cover observations was also developed. Finally, the coupling of LIS to the mesoscale Weather Research and Forecasting (WRF) model enabled investigations into how the sensitivity of land-atmosphere interactions to the specific choice of planetary boundary layer scheme and land surface model varies across surface moisture regimes, and how it can be quantified and evaluated against observations. The on-going development and integration of land assimilation modules into the Land Information System will enable the use of GSFC software with a variety of land models and make it accessible to the research community.
Reichle, Rolf↗