Search NASA⌕ Search

SEARCH · Search NASA

Results for “Kullback-Leibler”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

Kullback-Leibler information function and the sequential selection of experiments to discriminate among several linear models

The error variance of the process prior multivariate normal distributions of the parameters of the models are assumed to be specified, prior probabilities of the models being correct. A rule for termination of sampling is proposed. Upon termination, the model with the largest posterior probability is chosen as correct. If sampling is not terminated, posterior probabilities of the models and posterior distributions of the parameters are computed. An experiment was chosen to maximize the expected Kullback-Leibler information function. Monte Carlo simulation experiments were performed to investigate large and small sample behavior of the sequential adaptive procedure.

Sidik, S. M.↗

Kullback-Leibler information function and the sequential selection of experiments to discriminate among several linear models

A sequential adaptive experimental design procedure for a related problem is studied. It is assumed that a finite set of potential linear models relating certain controlled variables to an observed variable is postulated, and that exactly one of these models is correct. The problem is to sequentially design most informative experiments so that the correct model equation can be determined with as little experimentation as possible. Discussion includes: structure of the linear models; prerequisite distribution theory; entropy functions and the Kullback-Leibler information function; the sequential decision procedure; and computer simulation results. An example of application is given.

Sidik, S. M.↗

Sparse Cholesky factorization for solving nonlinear PDEs via Gaussian processes

In recent years, there has been widespread adoption of machine learning-based approaches to automate the solving of partial differential equations (PDEs). Among these approaches, Gaussian processes (GPs) and kernel methods have garnered considerable interest due to their flexibility, robust theoretical guarantees, and close ties to traditional methods. They can transform the solving of general nonlinear PDEs into solving quadratic optimization problems with nonlinear, PDE-induced constraints. However, the complexity bottleneck lies in computing with dense kernel matrices obtained from pointwise evaluations of the covariance kernel, and its partial derivatives, a result of the PDE constraint and for which fast algorithms are scarce. The primary goal of this paper is to provide a near-linear complexity algorithm for working with such kernel matrices. We present a sparse Cholesky factorization algorithm for these matrices based on the near-sparsity of the Cholesky factor under a novel ordering of pointwise and derivative measurements. The near-sparsity is rigorously justified by directly connecting the factor to GP regression and exponential decay of basis functions in numerical homogenization. We then employ the Vecchia approximation of GPs, which is optimal in the Kullback-Leibler divergence, to compute the approximate factor. This enables us to compute ϵ-approximate inverse Cholesky factors of the kernel matrices with complexity O(N log d (N/ϵ)) in space and O(N log 2d (N/ϵ)) in time. We integrate sparse Cholesky factorizations into optimization algorithms to obtain fast solvers of the nonlinear PDE. We numerically illustrate our algorithm’s near-linear space/time complexity for a broad class of nonlinear PDEs such as the nonlinear elliptic, Burgers, and Monge-Ampère equations. In summary, we provide a fast, scalable, and accurate method for solving general PDEs with GPs and kernel methods.

97 MATHEMATICS AND COMPUTING↗

Resilience Measurement Framework For Post-deployment Artificial Intelligence (ai) Integrated Systems

Resilience is largely defined as the ability to adapt or recover from adverse conditions, stresses, attacks, or compromises on systems that use or are enabled by digital resources. In Artificial Intelligence Management and Research for Advanced Networked Testbed Hub (AMARANTH), resilience is measured in the amount of time it took from the beginning of a testing period for the model to reach predictions outside of the original 95% confidence interval or using the Kullback-Leibler (KL) divergence theorem, the Population Stability Index (PSI), and traditional methods such as root mean squared error (RMSE) threshold. Artificial Intelligence (AI) model drift is of significant concern when deploying AI-integrated systems into critical and/or secure environments. Drift can impact resilience of the AI-integrated system post-deployment and requires consistent maintenance and upkeep to ensure the model is accurate and precise. To quantify model drift and predict the point when a model's drift becomes unacceptable, we describe using Kullback-Leibler (KL) divergence, Population Stability Index (PSI) and/or confidence interval width estimations to determine the point of failure and time to failure of a model post-deployment. Through simple code functions, the KL-divergence, PSI, confidence interval, and root mean squared (RMSE) point of failures can be used to derive when a model needs to be maintained as well as the impact of adversarial action through statistical means.

Yockey, Patience [Idaho National Laboratory (INL),↗

Gaussian processes for inferring parton distributions

The extraction of parton distribution functions (PDFs) from experimental or lattice QCD data is an ill-posed inverse problem, where regularization strongly impacts both systematic uncertainties and the reliability of the results. We study a framework based on Gaussian Process Regression (GPR) to reconstruct PDFs from lattice QCD matrix elements. Within a Bayesian framework, Gaussian processes serve as flexible priors that encode uncertainties, correlations, and constraints without imposing rigid functional forms. We investigate a wide range of kernel choices, mean functions, and hyperparameter treatments. We quantify information gained from the data using the Kullback-Leibler divergence. Synthetic data tests demonstrate the consistency and robustness of the method. Our study establishes GPR as a systematic and non-parametric approach to PDF reconstruction, offering controlled uncertainty estimates and reduced model bias in lattice QCD analyses.

hadronic spectroscopy↗

The effective number of parameters in kernel density estimation

We devise a new formula for measuring the effective degrees of freedom (EDoF) in kernel density estimation (KDE). Starting from the orthogonal polynomial sequence (OPS) expansion for the ratio of the empirical to the oracle density, we show how convolution with the kernel leads to a new OPS with respect to which one may express the resulting KDE. The expansion coefficients of the two OPS systems can then be related via a kernel sensitivity matrix, which leads to a natural oracle definition of EDoF through the trace operator. Asymptotic properties of the (empirical) plug-in EDoF are worked out through influence functions, and connections with other empirical EDoFs are established. Minimization of Kullback-Leibler divergence is investigated as an alternative to integrated squared error based bandwidth selection rules, yielding a new normal scale rule. The methodology, which arises from a proper oracle formulation and is not restricted to convolution kernels, suggests the possibility of a new bandwidth selection rule based on an information criterion such as AIC.

bandwidth selection↗

Quantum entropy as a harbinger of factorizability

Deeply inelastic scattering (DIS) is a powerful probe for investigating the QCD structure of hadronic matter and testing the standard model (SM). DIS can be described through QCD factorization theorems which separate contributions to the scattering interaction arising from disparate scales — e.g ., with nonperturbative matrix elements associated with long distances and a perturbative hard scattering kernel applying to short-distance parton-level interactions. The fundamental underpinnings of factorization may be recast in the quantum-theoretic terms of entanglement, (de)coherence, and system localization in a fashion which sheds complementary light on the dynamics at work in DIS from QCD bound states. In this Letter, we propose and quantitatively test such a quantum-information theoretic approach for dissecting factorization in DIS and its domain of validity; we employ metrics associated with quantum entanglement such as a differential quantum entropy and associated Kullback-Leibler (KL) divergences in numerical tests. We deploy these methods on an archetypal quark-spectator model of the proton, for which we monitor quantum decoherence in DIS as underlying model parameters are varied. On this basis, we demonstrate quantitatively how factorization-breaking effects may be imprinted on quantum entropies in a kinematic regime where leading-twist factorization increasingly receives large corrections from finite- Q 2 effects; our findings suggest potential applications of quantum simulation to QCD systems and their interactions.

Deep inelastic scattering↗

Information-theoretic astrophysical uncertainties in the effective theory of dark matter direct detection

The impact of astrophysical uncertainties in direct detection searches can vary significantly across particle dark matter models and detector targets, due to the different velocity and momentum dependencies of the scattering cross section. We address these uncertainties for all operators of the nonrelativistic effective field theory of dark-matter/nucleon interactions, making use of the Kullback-Leibler (KL) information divergence to measure the deviation of the true dark matter velocity distribution from the Maxwell-Boltzmann form. This approach quantifies how astrophysical uncertainties affect each operator in the effective theory, without assuming any specific functional form for the velocity distribution. While for some operators the uncertainties are smaller than 1 order of magnitude for entropically motivated deviations from the Maxwell-Boltzmann form, for other operators, these uncertainties can be as large as three orders of magnitude near threshold. Furthermore, we identify the dependence of the scattering rate for various operators of the effective theory with different velocity-weighted moments of the velocity distribution, functionally analogous to the mean, variance, or skewness. This provides new analytic insight into which features of the velocity distribution are most relevant to detect a given particle dark matter model. Our technique is general and could be applied to a broader class of physics problems where a physical observable depends on the statistical moments of an uncertain theoretical distribution.

Herrera, Gonzalo [MIT, MKI; Harvard U.; Virginia T↗

Quantifying Quantum Chaos through Microcanonical Distributions of Entanglement

A characteristic feature of “quantum chaotic” systems is that their eigenspectra and eigenstates display universal statistical properties described by random matrix theory (RMT). However, eigenstates of local systems also encode structure beyond RMT. To capture this feature, we introduce a framework that allows us to compare the properties of eigenstates in local systems with those of pure random states. In particular, our framework defines a notion of distance between quantum state ensembles that utilizes the Kullback-Leibler divergence to compare the microcanonical distribution of entanglement entropy (EE) of eigenstates with a reference RMT distribution generated by pure random states (with appropriate constraints). This notion gives rise to a quantitative metric for quantum chaos that not only accounts for averages of the distributions but also higher moments. The differences in moments are compared on a highly resolved scale set by the standard deviation of the RMT distribution, which is exponentially small in system size. As a result, the metric can distinguish between chaotic and integrable behaviors and, in addition, quantify and compare the of chaos (in terms of proximity to RMT behavior) between two systems that are assumed to be chaotic. We implement our framework in local, minimally structured, Floquet random circuits, as well as a canonical family of many-body Hamiltonians, the mixed-field Ising model (MFIM). Importantly, for Hamiltonian systems, we find that the reference random distribution must be appropriately constrained to incorporate the effect of energy conservation in order to describe the ensemble properties of midspectrum eigenstates. The metric captures deviations from RMT across all models and parameters, including those that have been previously identified as strongly chaotic, and for which other diagnostics of chaos such as level spacing statistics look strongly thermal. In Floquet circuits, the dominant source of deviations is the second moment of the distribution, and this persists for all system sizes. For the MFIM, we find significant variation of the KL divergence in parameter space. Notably, we find a small region where deviations from RMT are minimized, suggesting that “maximally chaotic” Hamiltonians may exist in fine-tuned pockets of parameter space. Published by the American Physical Society 2024

Physics↗

Towards the Flexibility of HVDC-Interconnected Systems: A Novel Emergency Freqeuncy Response Model

In this paper, we propose a novel multi-time scale emergency frequency response model by unlocking the flexibility of High Voltage Direct Current (HVDC) systems. Unlike assigning power ramping rates for FACTS to regulate frequency in traditional methods, this paper designs a step-change electromagnetic power frequency response (EPFR) scheme, by leveraging the temporal over/under DC voltage capability of HVDC. Wherein the Kullback-Leibler Divergence is adopted to convexify the modified swing equation after integrating the step-change power. Further, to avoid the complicated differential equations, we equivalently reformulate the duration limits of DC voltage deviation into the HVDC decreasing power ramping rates, which participate in the system primary frequency response. Finally, the new steady-state operation level of HVDC is involved with the secondary frequency response. As a result, an improved three-level algorithm is developed to solve the model, wherein the instant step-change, primary, and secondary frequency response are coordinated together. After applying the proposed frequency response scheme on the test system, the EPFR is validated to effectively provide the most instantaneous supports when faced with bulk power loss due to extreme contingencies, and the resilience is ensured within acceptable expenditures.

Jiang, Sufan↗

Trust-Based Detection and Mitigation of Cyber Attacks in Distributed Cooperative Control of Islanded AC Microgrids

In this study, we address the challenge of detecting and mitigating cyber attacks in the distributed cooperative control of islanded AC microgrids, with a particular focus on detecting False Data Injection Attacks (FDIAs), a significant threat to the Smart Grid (SG). The SG integrates traditional power systems with communication networks, creating a complex system with numerous vulnerable links, making it a prime target for cyber attacks. These attacks can lead to the disclosure of private data, control network failures, and even blackouts. Unlike machine learning-based approaches that require extensive datasets and mathematical models dependent on accurate system modeling, our method is free from such dependencies. To enhance the microgrid’s resilience against these threats, we propose a resilient control algorithm by introducing a novel trustworthiness parameter into the traditional cooperative control algorithm. Our method evaluates the trustworthiness of distributed energy resources (DERs) based on their voltage measurements and exchanged information, using Kullback-Leibler (KL) divergence to dynamically adjust control actions. We validated our approach through simulations on both the IEEE-34 bus feeder system with eight DERs and a larger microgrid with twenty-two DERs. The results demonstrated a detection accuracy of around 100%, with millisecond range mitigation time, ensuring rapid system recovery. Additionally, our method improved system stability by up to almost 100% under attack scenarios, showcasing its effectiveness in promptly detecting attacks and maintaining system resilience. These findings highlight the potential of our approach to enhance the security and stability of microgrid systems in the face of cyber threats.

Computer Science↗

The minimum distance approach to classification

The work to advance the state-of-the-art of miminum distance classification is reportd. This is accomplished through a combination of theoretical and comprehensive experimental investigations based on multispectral scanner data. A survey of the literature for suitable distance measures was conducted and the results of this survey are presented. It is shown that minimum distance classification, using density estimators and Kullback-Leibler numbers as the distance measure, is equivalent to a form of maximum likelihood sample classification. It is also shown that for the parametric case, minimum distance classification is equivalent to nearest neighbor classification in the parameter space.

Wacker, A. G.↗

Adaptive Metropolis Sampling with Product Distributions

The Metropolis-Hastings (MH) algorithm is a way to sample a provided target distribution pi(z). It works by repeatedly sampling a separate proposal distribution T(x,x') to generate a random walk {x(t)}. We consider a modification of the MH algorithm in which T is dynamically updated during the walk. The update at time t uses the {x(t' less than t)} to estimate the product distribution that has the least Kullback-Leibler distance to pi. That estimate is the information-theoretically optimal mean-field approximation to pi. We demonstrate through computer experiments that our algorithm produces samples that are superior to those of the conventional MH algorithm.

Wolpert, David H.↗

Topic Modeling Tool for PeTaL (Periodic Table of Life)

A topic modeling tool is constructed for the purpose of providing insights from biology to the engineer within the framework of PeTaL (Periodic Table of Life). The machine learning text mining tools–latent Dirichlet allocation (LDA) and nonnegative matrix factorization (NMF) with Kullback-Leibler (KL) divergence—are used to provide topic clusters to the user. Topic clusters are the underlying themes of a paper. For the text modeling problem, NMF-KL is the equivalent of probabilistic latent semantic analysis. Both LDA and NMF-KL are top-performing modeling tools. These tools are used to identify biological specimens relevant to the user. Various organisms solve a particular survival problem in nature differently. The topic clusters allow people without domain expertise to find these cross-topic themes in the body of documents and then branch out and examine papers whose target organisms solve the engineer’s problem. Abstracts from the Journal of Experimental Biology were used as input for the clustering tool in addition to a curated set of articles for validation. The tool is able to accept alternate input sources.

Machine learning↗

Quantum-Assisted Variational Segmentation for Image-to-Image Wildfire Detection Using Satellite Data

The quantum computing community has been searching for suitable applications to demonstrate the potential of near-term quantum devices. Quantum machine learning is a potential candidate, particularly using models that cannot be efficiently simulated with classical computers [1, 2]. This work focuses on a transition phase of quantum computers where the quantum machine learning model is still simulable classically but projected not to be simulable as the size of the model grows. Ultimately quantum computers may have advantages for high-dimensional real-world problems. Due to the limited number of qubits in current noisy intermediate-scale quantum (NISQ) devices, the direct application of quantum computers in high dimensional data is not feasible. To remedy this problem, an encoder-decoder architecture can be utilized. The encoder model would transform the high-dimensional data into a compact representation, to a level that small quantum computers can be used today (or in the near future), and the decoder would take the quantum processed outputs back to the high-dimensional space. Addressing the two challenges of quantum machine learning, this work investigates a hybrid supervised generative model with a quantum Ising Born machine embedded as the latent distribution. The model contains four main parts (Figure 1.a.): (1) a U-NET architecture responsible for learning segmentation flow, (2) a Prior network responsible for learning an encoded latent distribution of the input data, (3) a Born machine which represents the latent distribution, and (4) a Posterior network in charge of learning the joint encoded latent distribution of inputs and target data. The initial model, proposed by [3], is optimized by (1) maximizing the overlap of the prior and posterior latent distributions, and (2) minimizing the segmentation loss. The proposed model is designed to be investigated in a simulation environment applied to the real-world application of wildfire segmentation. Specifically, the model is designed to solve the patchy wildfire segmentations of Moderate Resolution Imaging Spectroradiometer (MODIS) by taking the MODIS observations and using Visible Infrared Imaging Radiometer Suite’s (VIIRS) consistent wildfire product as the target. The model solves patchy wildfire segmentations and provides insight into the epistemic errors sourced from model variation. The model utilizes the Born machine as a QUBO solver to represent the latent space as a Bernoulli distribution. The proposed configuration allows the variational segmentation model to leverage the true quantum probabilistic nature and derive a more expressive latent configuration, increasing the model performance in describing wildfire segmentations. The quantum probabilistic information of the Born machine is directly incorporated in the Kullback-Leibler divergence loss in the prior and posterior distributions, forcing the Bernoulli latent distribution to maximize the overlap of input and joint input-target distributions. The proposed model is then trained and compared with a baseline only consisting of direct Bernoulli latent distribution with no Born machine representing the latent space. The models are evaluated based on the segmentation metrics, such as precision, recall, intersect of union, with uncertainty boundaries accounting for the stochastic nature of the model. Our findings show that even in low latent-dimensional space (due to the limit in computational power of the classical quantum simulator), we are able to effectively capture the latent representation and hence the model performs better than the baseline. The findings are a projection for scaling the model into higher dimensional latent space with the Born machine surpassing the baseline performance. Figure 1. Sub-figure (a) demonstrates the architecture for the training phase. The model consists of a Prior and Posterior network that encode inputs and joint input-target data into compact representations, respectively. The Born machine represents the latent distribution, and the U-NET branch learns the segmentation patterns of the data. The stochasticity is introduced to the U-NET through its last layer to create meaningful but stochastic segmentations. Sub-figure (b) represents the inference phase where the model takes the stochastic behavior from the prior network and injects that into the U-NET. Each attempt of inference will generate different but similar segmentations from the same distribution of the wildfire event. REFERENCES [1] Coyle, B., Mills, D., Danos, V., & Kashefi, E. (2020). The Born supremacy: quantum advantage and training of an Ising Born machine. npj Quantum Information, 6(1), 1-11. [2] Liu, J. G., & Wang, L. (2018). Differentiable learning of quantum circuit born machines. Physical Review A, 98(6), 062324. [3] Kohl, S., Romera-Paredes, B., Meyer, C., De Fauw, J., Ledsam, J. R., Maier-Hein, K., ... & Ronneberger, O. (2018). A probabilistic u-net for segmentation of ambiguous images. Advances in neural information processing systems, 31.

quantum machine learning↗