Search NASA⌕ Search

SEARCH · Search NASA

Results for “Markov process”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15

Optical communication with two-photon coherent states. II - Photoemissive detection and structured receiver performance

In a previous paper (1978), the authors developed a method of analyzing the performance of two-photon coherent state (TCS) systems for free-space optical communications. General theorems permitting application of classical point process results to detection and estimation of signals in arbitrary quantum states were derived. The present paper examines the general problem of photoemissive detection statistics. On the basis of the photocounting theory of Kelley and Kleiner (1964) it is shown that for arbitrary pure state illumination, the resulting photocurrent is in general a self-exciting point process. The photocount statistics for first-order coherent fields reduce to those of a special class of Markov birth processes, which the authors term single-mode birth processes. These general results are applied to the structure of TCS radiation, and it is shown that the use of TCS radiation with direct or heterodyne detection results in minimal performance increments over comparable coherent-state systems. However, significant performance advantages are offered by use of TCS radiation with homodyne detection. The abstract quantum descriptions of homodyne and heterodyne detection are derived and a synthesis procedure for obtaining quantum measurements described by arbitrary TCS is given.

Shapiro, J. H.↗

Reinforcement learning for adaptive maintenance policy optimization under imperfect knowledge of the system degradation model and partial observability of system states

Maintenance policy optimization usually is faced with challenges that arise from an imperfect knowledge of system degradation models and from the partial observability of system degradation states. Here, this paper proposes a reinforcement learning method to address these two challenges for a class of maintenance problems with Markov degradation processes. The reinforcement learning approach consists of a learning component and a planning component. Using sequentially collected observations, at each step of decision-making the learning component improves the knowledge of system degradation in terms of the probability distributions of the transition rates based on sequential Bayesian inference. Using the updated transition rates, at each step of decision-making the maintenance policy optimization problem is then formulated as a partially observable Markov decision problem, and the planning component computes the optimal maintenance policy that maximizes the expected cumulative reward. The proposed method is illustrated using a numerical example with repair and inspection maintenance actions. The result shows that as more observations are collected, the learning component progressively learns the true system degradation process, and the planning component adjusts the optimal maintenance policy accordingly as well, which leads to increased reward.

42 ENGINEERING↗

Markov Tracking for Agent Coordination

Partially observable Markov decision processes (POMDPs) axe an attractive representation for representing agent behavior, since they capture uncertainty in both the agent's state and its actions. However, finding an optimal policy for POMDPs in general is computationally difficult. In this paper we present Markov Tracking, a restricted problem of coordinating actions with an agent or process represented as a POMDP Because the actions coordinate with the agent rather than influence its behavior, the optimal solution to this problem can be computed locally and quickly. We also demonstrate the use of the technique on sequential POMDPs, which can be used to model a behavior that follows a linear, acyclic trajectory through a series of states. By imposing a "windowing" restriction that restricts the number of possible alternatives considered at any moment to a fixed size, a coordinating action can be calculated in constant time, making this amenable to coordination with complex agents.

Washington, Richard↗

Back to the Future: Consistency-Based Trajectory Tracking

Given a model of a physical process and a sequence of commands and observations received over time, the task of an autonomous controller is to determine the likely states of the process and the actions required to move the process to a desired configuration. We introduce a representation and algorithms for incrementally generating approximate belief states for a restricted but relevant class of partially observable Markov decision processes with very large state spaces. The algorithm presented incrementally generates, rather than revises, an approximate belief state at any point by abstracting and summarizing segments of the likely trajectories of the process. This enables applications to efficiently maintain a partial belief state when it remains consistent with observations and revisit past assumptions about the process' evolution when the belief state is ruled out. The system presented has been implemented and results on examples from the domain of spacecraft control are presented.

Kurien, James↗

A Mechanical Model of Brownian Motion for One Massive Particle Including Slow Light Particles

We provide a connection between Brownian motion and a classical mechanical system. Precisely, we consider a system of one massive particle interacting with an ideal gas, evolved according to non-random mechanical principles, via interaction potentials, without any assumption requiring that the initial velocities of the environmental particles should be restricted to be “fast enough”. We prove the convergence of the (position, velocity)-process of the massive particle under a certain scaling limit, such that the mass of the environmental particles converges to 0 while the density and the velocities of them go to infinity, and give the precise expression of the limiting process, a diffusion process.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Positive contraction mappings for classical and quantum Schrödinger systems

The classical Schrödinger bridge seeks the most likely probability law for a diffusion process, in path space, that matches marginals at two end points in time; the likelihood is quantified by the relative entropy between the sought law and a prior. Jamison proved that the new law is obtained through a multiplicative functional transformation of the prior. This transformation is characterised by an automorphism on the space of endpoints probability measures, which has been studied by Fortet, Beurling, and others. A similar question can be raised for processes evolving in a discrete time and space as well as for processes defined over non-commutative probability spaces. The present paper builds on earlier work by Pavon and Ticozzi and begins by establishing solutions to Schrödinger systems for Markov chains. Our approach is based on the Hilbert metric and shows that the solution to the Schrödinger bridge is provided by the fixed point of a contractive map. We approach, in a similar manner, the steering of a quantum system across a quantum channel. We are able to establish existence of quantum transitions that are multiplicative functional transformations of a given Kraus map for the cases where the marginals are either uniform or pure states. As in the Markov chain case, and for uniform density matrices, the solution of the quantum bridge can be constructed from the fixed point of a certain contractive map. For arbitrary marginal densities, extensive numerical simulations indicate that iteration of a similar map leads to fixed points from which we can construct a quantum bridge. For this general case, however, a proof of convergence remains elusive.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Non-Markovian dynamics of a damped driven two-state system

We study a driven two-state system interacting with a generic structured environment. We outline the derivation of the time-local microscopic non-Markovian master equation, in the limit of weak coupling between the system and the reservoir, and we derive its analytic solution for general reservoir spectra in the regime of validity of the secular approximation. We also consider the non-Markovian master equation without the secular approximation and study the effect of nonsecular terms on the system dynamics for two classes of reservoir spectra: the Ohmic and the Lorentzian reservoir. Finally, we derive the analytic conditions for complete positivity of the dynamical map, with and without the secular approximation. Interestingly, the complete positivity conditions have a transparent physical interpretation in terms of the characteristic time scales of phase diffusion and relaxation processes.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Entanglement and entangling power of the dynamics in light-harvesting complexes

We study the evolution of quantum entanglement during exciton energy transfer (EET) in a network model of the Fenna-Matthews-Olson (FMO) complex, a biological pigment-protein complex involved in the early steps of photosynthesis in sulfur bacteria. The influence of Markovian as well as spatially and temporally correlated (non-Markovian) noise on the generation of entanglement across distinct chromophores (site entanglement) and different excitonic eigenstates (mode entanglement) is studied for different injection mechanisms, including thermal and coherent laser excitation. Additionally, we study the entangling power of the FMO complex under natural operating conditions. While quantum information processing tends to favor maximal entanglement, near unit EET is achieved as the result of an intricate interplay between coherent and noisy processes where the initial part of the evolution displays intermediate values of both forms of entanglement.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Data-based Aggregate Model of Refrigerators for Electric Power Grid Services

Electrical power system stability under high penetration of intermittent renewable energy generation, such as wind and solar energy, has become a high concern. It is unpredictable and hard to control due to little inertial behavior characteristic to these generation resources. To help mitigate possibly undesirable frequency excursions, demand-side frequency control presents an emerging solution. By observing the frequency deviation, devices could switch operating modes for a short time to rebalance generation. To analyze device level control algorithms, aggregate models for their behavior are needed. Refrigerators are household devices that could help with load modulation. This paper uses real measured data to analyze the refrigerator behavior, learn about its operating mode and power consumption ranges. A Markov chain process is proposed to model the transition of a refrigerator population from one operating mode to another, such that the overall load is modulated according to the control input. The Markov-chain model can be used to design control so that the overall load is modulated according to the control input.

Markov chain model, demand response, refrigerator,↗

Unsupervised Parsing of Gaze Data with a Beta-process Vector Auto-regressive Hidden Markov Model

The first stage of analyzing eye-tracking data is commonly to code the data into sequences of fixations and saccades. This process is usually automated using simple, predetermined rules for classifying ranges of the time series into events, such as “if the dispersion of gaze samples is lower than the threshold, then code as a fixation; otherwise code as a saccade.” More recent approaches incorporate additional eye-movement categories in automated parsing algorithms, particularly glissades, by using time-varying, data-driven thresholds. We describe an alternative approach using the beta-process auto-regressive hidden Markov model (BP-AR-HMM). The BP-AR-HMM offers two main advantages over existing frameworks. First, it provides a statistical model for eye movement classification rather than a single estimate. Second, the BP-AR-HMM uses a latent process to model the number and nature of the types of eye-movements and hence is not constrained to predetermined categories. We applied the BP-AR-HMM both to high-sampling rate gaze data from Andersson, Larsson, Holmqvist, Stridh, and Nyström (2016) and to low-sampling rate data from the DIEM project (Mital, Smith, Hill, & Henderson, 2011). Driven by the data properties, the BP-AR-HMM identified clusters of samples resembling saccades, fixations and smooth pursuit, and a category that approximately corresponds to glissades (as well as blinks, errors, etc.). The BP-AR-HMM serves as an effective algorithm for data-driven event parsing alone or as an initial step in exploring the characteristics of gaze data sets.

eye tracking, fixation, fixation, saccade, smooth ↗

On the Coupling Time of the Heat-Bath Process for the Fortuin–Kasteleyn Random–Cluster Model

We consider the coupling from the past implementation of the random–cluster heat-bath process, and study its random running time, or coupling time. We focus on hypercubic lattices embedded on tori, in dimensions one to three, with cluster fugacity at least one. We make a number of conjectures regarding the asymptotic behaviour of the coupling time, motivated by rigorous results in one dimension and Monte Carlo simulations in dimensions two and three. Amongst our findings, we observe that, for generic parameter values, the distribution of the appropriately standardized coupling time converges to a Gumbel distribution, and that the standard deviation of the coupling time is asymptotic to an explicit universal constant multiple of the relaxation time. Perhaps surprisingly, we observe these results to hold both off criticality, where the coupling time closely mimics the coupon collector’s problem, and also at the critical point, provided the cluster fugacity is below the value at which the transition becomes discontinuous. Finally, we consider analogous questions for the single-spin Ising heat-bath process.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Nonadiabatic transition paths from quantum jump trajectories

We present a means of studying rare reactive pathways in open quantum systems using transition path theory and ensembles of quantum jump trajectories. This approach allows for the elucidation of reactive paths for dissipative, nonadiabatic dynamics when the system is embedded in a Markovian environment. Here, we detail the dominant pathways and rates of thermally activated processes and the relaxation pathways and photoyields following vertical excitation in a minimal model of a conical intersection. We find that the geometry of the conical intersection affects the electronic character of the transition state as defined through a generalization of a committor function for a thermal barrier crossing event. Similarly, the geometry changes the mechanism of relaxation following a vertical excitation. Relaxation in models resulting from small diabatic coupling proceeds through pathways dominated by pure dephasing, while those with large diabatic coupling proceed through pathways limited by dissipation. The perspective introduced here for the nonadiabatic dynamics of open quantum systems generalizes classical notions of reactive paths to fundamentally quantum mechanical processes.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Deep Reinforcement Learning for Distribution System Restoration Using Distributed Energy Resources and Tie-Switches

Distributed energy resources (DERs), such as solar PVs and energy storage, can be used to restore distribution system critical loads after the extreme weather events to increase grid resilience. However, coordinating multiple DERs together with tie-switches for multi-step restoration process under renewable uncertainty is challenging. This paper proposes a deep reinforcement learning to control discrete actions of switching on/off tie switches and DERs for critical load restoration. The restoration problem is first cast into the Markov decision process suitable for DRL. Then, the original soft actor critic (SAC) method for continuous actions has been extended to handle discrete and continuous actions. Numerical comparison results with other stochastic optimization-based approaches on the modified IEEE 33-bus system show that the proposed method can achieve fast critical load restoration in the presence of substation power outage while maintaining system voltage limit throughout the restoration process.

active distribution systems↗

Reinforcement Learning-Based Oscillation Dampening: Scaling Up Single-Agent Reinforcement Learning Algorithms to a 100-Autonomous-Vehicle Highway Field Operational Test

In this article, we explore the technical details of the reinforcement learning (RL) algorithms that were deployed in the largest field test of automated vehicles designed to smooth traffic flow in history as of 2023, uncovering the challenges and breakthroughs that come with developing RL controllers for automated vehicles. We delve into the fundamental concepts behind RL algorithms and their application in the context of self-driving cars, discussing the developmental process from simulation to deployment in detail, from designing simulators to reward function shaping. We present the results in both simulation and deployment, discussing the flow-smoothing benefits of the RL controller. From understanding the basics of Markov decision processes to exploring advanced techniques such as deep RL, our article offers a comprehensive overview and deep dive of the theoretical foundations and practical implementations driving this rapidly evolving field. We also showcase real-world case studies and alternative research projects that highlight the impact of RL controllers in revolutionizing autonomous driving. From tackling complex urban environments to dealing with unpredictable traffic scenarios, these intelligent controllers are pushing the boundaries of what automated vehicles can achieve. Furthermore, we examine the safety considerations and hardware-focused technical details surrounding deployment of RL controllers into automated vehicles. As these algorithms learn and evolve through interactions with the environment, ensuring their behavior aligns with safety standards becomes crucial. Here, we explore the methodologies and frameworks being developed to address these challenges, emphasizing the importance of building reliable control systems for automated vehicles.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

"Reinforcement Learning Based Oscillation Dampening: Scaling up Single-Agent RL algorithms to a 100 AV highway field operational test"

In this article, we explore the technical details of the reinforcement learning (RL) algorithms that were deployed in the largest field test of automated vehicles designed to smooth traffic flow in history as of 2023, uncovering the challenges and breakthroughs that come with developing RL controllers for automated vehicles. We delve into the fundamental concepts behind RL algorithms and their application in the context of self-driving cars, discussing the developmental process from simulation to deployment in detail, from designing simulators to reward function shaping. We present the results in both simulation and deployment, discussing the flow-smoothing benefits of the RL controller. From understanding the basics of Markov decision processes to exploring advanced techniques such as deep RL, our article offers a comprehensive overview and deep dive of the theoretical foundations and practical implementations driving this rapidly evolving field. We also showcase real-world case studies and alternative research projects that highlight the impact of RL controllers in revolutionizing autonomous driving. From tackling complex urban environments to dealing with unpredictable traffic scenarios, these intelligent controllers are pushing the boundaries of what automated vehicles can achieve. Furthermore, we examine the safety considerations and hardware-focused technical details surrounding deployment of RL controllers into automated vehicles. As these algorithms learn and evolve through interactions with the environment, ensuring their behavior aligns with safety standards becomes crucial. We explore the methodologies and frameworks being developed to address these challenges, emphasizing the importance of building reliable control systems for automated vehicles.

Jang, Kathy↗

Biorthonormal eigenbasis of a Markovian master equation for the quantum Brownian motion

The solution to a quantum Markovian master equation of a harmonic oscillator weakly coupled to a thermal reservoir is investigated as a non-Hermitian eigenvalue problem in space coordinates. In terms of a pair of quantum action-angle variables, the equation becomes separable and a complete set of biorthogonal eigenfunctions can be constructed. Properties of quantum states, such as the change in the quantum coherence length, damping in the motion, and disappearance of the spatial interference pattern, can then be described as the decay of the nonequilibrium modes in the eigenbasis expansion. It is found that the process of gaining quantum coherence from the environment takes a longer time than the opposite process of losing quantum coherence to the environment. An estimate of the time scales of these processes is obtained.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Automated Adversary Emulation for Cyber-Physical Systems via Reinforcement Learning

Adversary emulation is an offensive exercise that provides a comprehensive assessment of a system’s resilience against cyber attacks. However, adversary emulation is typically a manual process, making it costly and hard to deploy in cyber-physical systems (CPS) with complex dynamics, vulnerabilities, and operational uncertainties. In this paper, we develop an automated, domain-aware approach to adversary emulation for CPS. We formulate a Markov Decision Process (MDP) model to determine an optimal attack sequence over a hybrid attack graph with cyber (discrete) and physical (continuous) components and related physical dynamics. We apply model-based and model-free reinforcement learning (RL) methods to solve the discrete-continuous MDP in a tractable fashion. As a baseline, we also develop a greedy attack algorithm and compare it with the RL procedures. We summarize our findings through a numerical study on sensor deception attacks in buildings to compare the performance and solution quality of the proposed algorithms.

Bhattacharya, Arnab↗