Search NASA⌕ Search

SEARCH · Search NASA

Results for “sparsity estimation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Covariance operator estimation: Sparsity, lengthscale, and ensemble Kalman filters

This paper investigates covariance operator estimation via thresholding. For Gaussian random fields with approximately sparse covariance operators, we establish non-asymptotic bounds on the estimation error in terms of the sparsity level of the covariance and the expected supremum of the field. We prove that thresholded estimators enjoy an exponential improvement in sample complexity compared with the standard sample covariance estimator if the field has a small correlation lengthscale. As an application of the theory, we study thresholded estimation of covariance operators within ensemble Kalman filters.

Covariance operator estimation↗
DOE OSTI2020

Enhanced Tensor Completion Based Approaches for State Estimation in Distribution Systems

Grid state estimation is essential for effective control and management of distribution systems. While weighted least squares has been the conventional method for state estimation, sparsity-aware methods have become popular due to their superior performance with limited data. Matrix completion and compressed sensing-based state estimation approaches exploit the underlying smoothness in the state variables. However, classic matrix completion methods do not take into account the temporal correlation of system states. Compressed sensing methods, on the other hand, require an appropriate choice of sparsifying basis that may not be easy to identify. This paper proposes a blocktensor completion based framework which uses an alternative approach to estimate voltage phasor, power injections and branch currents. This approach utilizes the temporal correlation of the system states in a tensor trace-norm minimization formulation with power flow equations as constraints. Herein, feature scaling is introduced in the problem formulation to benefit from the improved sensitivity of the tensor trace norm to the matrix columns in the scaled unfoldings of the tensor. Weighted tensor norm is utilized to exploit the structures of the different unfoldings of the state measurement tensor to improve the voltage estimation. The estimation accuracy is further improved by alternatively estimating the tensor columns and increasing the available data at each stage in the tensor completion process. The proposed methods are evaluated on the IEEE-33, 37 test systems and a 100- node test system. The proposed methods are shown to provide significant performance gains relative to the classic matrix and tensor completion based approaches.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Bayesian Framework for Multi-Timescale State Estimation in Low-Observable Distribution Systems

To support the smart grid paradigm, there has been a significant increase in sensor deployments and metering infrastructure in distribution systems. However, the measurements provided by these sensors and metering devices are typically sampled at different rates and could suffer from losses during the aggregation process. It is crucial to effectively reconcile the time-series measurements for a reliable state estimation. While weighted least squares has been the traditional approach for state estimation, sparsity-based approaches like matrix completion have become popular due to their superior performance in low-observability conditions. This paper proposes a Bayesian framework for both multi-timescale data aggregation and matrix completion based state estimation. Specifically, the multiscale time-series data aggregated from heterogenous sources are reconciled using a multitask Gaussian process that exploits the spatio-temporal correlations. Here, the resulting consistent timeseries alongwith the confidence bound on the imputations are fed into a Bayesian matrix completion method augmented with linearized power-flow constraints to accurately estimate the states in low-observability conditions. Results on three phase unbalanced IEEE 37 and IEEE 123 bus test systems reveal the superior performance of the proposed Bayesian framework. The computational complexity for the proposed Bayesian framework is also quantified.

42 ENGINEERING↗
DOE OSTI2021

Joint Matrix Completion and Compressed Sensing for State Estimation in Low-observable Distribution System

Limited measurement availability at the distribution grid presents challenges for state estimation and situational awareness. This paper combines the advantages of two sparsity-based state estimation approaches (matrix completion and compressive sensing) that have been proposed recently to address the challenge of unobservability. The proposed approach exploits both the low rank structure and a suitable transform domain representation to leverage the correlation structure of the spatio-temporal data matrix while incorporating the powerflow constraints of the distribution grid. Simulations are carried out on three phase unbalanced IEEE 37 test system to verify the effectiveness of the proposed approach. The performance results reveal - (1) the superiority over traditional matrix completion and (2) very low state estimation errors for high compression ratios representing very low observability.

Dahale, Shweta↗
DOE OSTI2021

A Consensus Equilibrium Approach for 3-D Land Seismic Shots Recovery

Physical and budget constraints often result in inadequate sampling for accurate subsurface imaging. Preprocessing approaches, such as missing trace interpolation, are typically employed to enhance seismic data in such cases. The compressed sensing (CS) framework has been applied for modeling missing seismic data, which is estimated by sparsity-based computational algorithms. While existing work mainly focuses on recovering missing traces resulting from receiver subsampling, source subsampling has greater economical advantages, as sources are more expensive than receivers. Moreover, stronger image models different from sparsity have not been explored for source recovery. This work presents a consensus equilibrium (CE) approach to recover missing seismic shots, which enables to incorporate various regularization operators modeling different data priors. Here, simulation results from a real 3-D land seismic dataset demonstrate that the CE approach provides more accurate estimations of the linear and hyperbolic events in the recovered shots, compared with pure sparsity-based reconstructions.

58 GEOSCIENCES↗
DOE OSTI2022

Dynamic Matrix Completion Based State Estimation in Distribution Grids

The power distribution network is undergoing tremendous transformation due to an increase in the penetration of renewable energy resources and electric vehicles. These changes have resulted in greater uncertainty and dynamics in the distribution grid states. Therefore, the ability to track and monitor system states has become a critical need for accurate and timely control actions. In this paper, we propose two dynamic sparsity-based state estimation approaches for distribution systems: (1) locally weighted matrix completion (LW-MC) and (2) Bayesian matrix completion with Kalman filter prediction (BMC-KF). The performance of the proposed dynamic state estimation strategies is compared with the classic/static matrix completion (static-MC) approach using the IEEE 37 and IEEE 123 bus test systems. Finally, results indicate that BMC-KF approach outperforms both LW-MC as well as static-MC even when 30% of the measurement data is available. Computational complexity associated with both approaches is quantified.

42 ENGINEERING↗
DOE OSTI2025

Covariance operator estimation via adaptive thresholding

This paper studies sparse covariance operator estimation for nonstationary processes with sharply varying marginal variance and small correlation lengthscale. We introduce a covariance operator estimator that adaptively thresholds the sample covariance function using an estimate of the variance component. Building on recent results from empirical process theory, we derive an operator norm bound on the estimation error in terms of the sparsity level of the covariance and the expected supremum of a normalized process. Furthermore, our theory and numerical simulations demonstrate the advantage of adaptive threshold estimators over universal threshold and sample covariance estimators in nonstationary settings.

Al-Ghattas, Omar [University of Chicago, IL (Unite↗
DOE OSTI2025

Data Assimilation with Machine Learning Surrogate Models: A Case Study with FourCastNet

Modern data-driven surrogate models for weather forecasting provide accurate short-term predictions but inaccurate and nonphysical long-term forecasts. This paper investigates online weather prediction using machine learning surrogates supplemented with partial and noisy observations. We empirically demonstrate and theoretically justify that, despite the long-time instability of the surrogates and the sparsity of the observations, filtering estimates can remain accurate in the long-time horizon. As a case study, we integrate the Fourier Forecasting Neural Network (FourCastNet), a weather surrogate model, within a variational data assimilation framework using partial, noisy ERA5 global reanalysis data from the European Centre for Medium-Range Weather Forecasts (ECMWF). Here, our results show that filtering estimates remain accurate over a year-long assimilation window and provide effective initial conditions for forecasting tasks, including extreme event prediction.

Data assimilation↗
DOE OSTI2022

Compressed Natural Gas Component Leak Frequency Estimation

The frequency of unintended releases in a compressed natural gas system is an important aspect of the system quantitative risk assessment. The frequencies for possible release scenarios, along with engineering models, are utilized to quantify the risks for compressed natural gas facilities. This report documents component leakage frequencies representative of compressed natural gas components that were estimated as a function of the normalized leak size. A Bayesian statistical method was used which results in leak frequency distributions for each component which represent variation and uncertainty in the leak frequency. The analysis shows that there is high uncertainty in the estimated leak frequencies due to sparsity in compressed natural gas data. These leak frequencies may still be useful in compressed natural gas system risk assessments, as long as this high uncertainty is acknowledged and considered appropriately.

03 NATURAL GAS↗
DOE OSTI2025

Data Assimilation with Machine Learning Surrogate Models: A Case Study with FourCastNet

Modern data-driven surrogate models for weather forecasting provide accurate short-term predictions but inaccurate and nonphysical long-term forecasts. This paper investigates online weather prediction using machine learning surrogates supplemented with partial and noisy observations. We empirically demonstrate and theoretically justify that, despite the long-time instability of the surrogates and the sparsity of the observations, filtering estimates can remain accurate in the long-time horizon. As a case study, we integrate FourCastNet, a weather surrogate model, within a variational data assimilation framework using partial, noisy ERA5 data. Our results show that filtering estimates remain accurate over a year-long assimilation window and provide effective initial conditions for forecasting tasks, including extreme event prediction.

Adrian, Melissa [Univ. of Chicago, IL (United Stat↗
DOE OSTI2026

Deep nonparametric estimation of operators between infinite dimensional spaces

Learning operators between infinitely dimensional spaces is an important learning task arising in machine learning, imaging science, mathematical modeling and simulations, etc. This paper studies the nonparametric estimation of Lipschitz operators using deep neural networks. Non-asymptotic upper bounds are derived for the generalization error of the empirical risk minimizer over a properly chosen network class. Under the assumption that the target operator exhibits a low dimensional structure, our error bounds decay as the training sample size increases, with an attractive fast rate depending on the intrinsic dimension in our estimation. Our assumptions cover most scenarios in real applications and our results give rise to fast rates by exploiting low dimensional structures of data in operator estimation. We also investigate the influence of network structures (e.g., network width, depth, and sparsity) on the generalization error of the neural network estimator and propose a general suggestion on the choice of network structures to maximize the learning efficiency quantitatively.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces

Learning operators between infinitely dimensional spaces is an important learning task arising in machine learning, imaging science, mathematical modeling and simulations, etc. This paper studies the nonparametric estimation of Lipschitz operators using deep neural networks. Non-asymptotic upper bounds are derived for the generalization error of the empirical risk minimizer over a properly chosen network class. Under the assumption that the target operator exhibits a low dimensional structure, our error bounds decay as the training sample size increases, with an attractive fast rate depending on the intrinsic dimension in our estimation. Our assumptions cover most scenarios in real applications and our results give rise to fast rates by exploiting low dimensional structures of data in operator estimation. We also investigate the influence of network structures (e.g., network width, depth, and sparsity) on the generalization error of the neural network estimator and propose a general suggestion on the choice of network structures to maximize the learning efficiency quantitatively.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Enabling Cybersecurity, Situational Awareness and Resilience in Distribution Grids with High Penetration of Photovoltaics (CARE-PV) (Final Report)

Since legacy distribution systems have very limited visibility beyond the substation, high penetration of PV at the grid edge presents some unique operational challenges. One approach to address these challenges is to use information from advanced metering infrastructure (AMI) and µPMUs. However, exploiting this information is impacted by a number of factors, including multi-timescale measurements, volume of data generated, communication network impairments (e.g., information loss and latency) and susceptibility to cyber-attacks. Therefore, one of the critical tasks involved in the management of a distribution grid is to develop complete situational awareness by integrating cyber-security mechanisms with state estimation strategies and leveraging this situational awareness to assure energy services at strategic locations while exploiting AMI/PV inverter/ µPMU data. This CARE-PV project addresses the fundamental challenges in situational awareness and resilience to cyber and physical vectors by exploiting the synergy between innovative modeling, estimation, data analytics, testing and validation using smart PV inverters designed at K-State and facilities at NREL. Specifically, the project involved the development, testing and validation of the following novel enabling technologies: (Thrust 1) Resilience to cyber vectors that impact data integrity was addressed via a two-level defense strategy that combines cyber intrusion detection using self-learning, cooperative smart PV inverters, and a novel moving target defense framework to combat data integrity attacks. (Thrust 2) Resilience to cyber-physical vectors that impact situational awareness by limiting data availability was addressed via novel centralized and decentralized, sparsity-based static and dynamic state estimation approaches that enhance observability even when the underlying system is unobservable. (Thrust 3) Leveraging a unique probabilistic sensitivity analysis approach accompanied by one-of-a-kind dominant influencer set computation, the vulnerability of critical infrastructure at strategic locations was evaluated so that proactive PV-based control strategies can be used to support operations under normal/outage scenarios. These CARE-PV project innovations were demonstrated on both small-scale IEEE and larger utility-scale testbeds (Thrust 4). Feedback from Industry Advisory Board members was used to formulate a commercialization pathway for a subset of CARE-PV technologies. These CARE-PV technologies will ultimately lead to reliable and secure, large-scale integration of renewable energy and mitigate the risk of energy disruption resulting from cyber incidents and other emerging threats within the energy environment.

14 SOLAR ENERGY↗
DOE OSTI2022

Dynamic signal recovery in distribution grids using compressive lossy measurements

Distribution system state estimation requires reliable aggregation of the measured data. However, the large volume of the measured data imposes a significant stress on the underlying communication infrastructure. With the challenges associated with measurement availability, current distribution systems are typically unobservable. To cope with the unobservability issue, compressive sensing theory allows us to recover system state information from a small number of measurements provided the states of the distribution system exhibit sparsity. In this paper, we evaluate the robustness of an updated Kalman filtered modified compressive sensing (KF-ModCS) technique that dynamically estimates the grid states using a small fraction of measured data. In practice, measurements used for sparsity based state estimation may also be intermittent due to communication network induced losses. Further, to understand the effect of packet losses on KF-ModCS, we provide an upper bound for the expected variances of the state estimation error for a given rate of information loss. This upper bound is further improved if the support set of the sparse signal that characterizes the state dynamics does not change over time and/or the reduced model is observable. Simulations based on two practical data sets collected from actual customers in a distribution grid validate the theoretical results.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2019

Resilient Reinforcement in Secure State Estimation against Sensor Attacks with a priori Information

This paper deals with the state estimation problem in the presence of malicious sensor attacks. Unlike the existing work, in this paper, we address the problem with a priori information of the state to be estimated. Specifically, we address three prior knowledge: the sparsity information, (alpha, n_0)-sparsity information, and side information, and in each scenario, we show that the state can be reconstructed even if more sensors are compromised. This implies that the prior information reinforces the system resilience against sensor attacks. Then, an estimator under sensor attacks considering the information is developed and, under a certain condition, the estimator can be relaxed into a tractable convex optimization problem. Further, we extend this analysis to systems in presence of measurement noises, and it is shown that the prior information reduces the state-estimation error caused by the noise. Here, the numerical simulations in a diffusion process finally illustrate the reinforcement and error-reduction results with the information.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Numerical characterization of support recovery in sparse regression with correlated design

Sparse regression is employed in diverse scientific settings as a feature selection method. A pervasive aspect of scientific data is the presence of correlations between predictive features. These correlations hamper both feature selection and estimation and jeopardize conclusions drawn from estimated models. On the other hand, theoretical results on sparsity-inducing regularized regression have largely addressed conditions for selection consistency via asymptotics, and disregard the problem of model selection, whereby regularization parameters are chosen. In this numerical study, we address these issues through exhaustive characterization of the performance of several regression estimators, coupled with a range of model selection strategies. These estimators and selection criteria were examined across correlated regression problems with varying degrees of signal to noise, distributions of non-zero model coefficients, and model sparsity. Our results reveal a fundamental tradeoff between false positive and false negative control in all regression estimators and model selection criteria examined. Additionally, we numerically explore a transition point modulated by the signal-to-noise ratio and spectral properties of the design covariance matrix at which the selection accuracy of all considered algorithms degrades. Overall, we find that SCAD coupled with BIC or empirical Bayes model selection performs the best feature selection across the regression problems considered.

97 MATHEMATICS AND COMPUTING↗