Search NASA⌕ Search

SEARCH · Search NASA

Results for “MARKOV PROCESS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

Cooperation and Coordination Between Fuzzy Reinforcement Learning Agents in Continuous State Partially Observable Markov Decision Processes

Successful operations of future multi-agent intelligent systems require efficient cooperation schemes between agents sharing learning experiences. We consider a pseudo-realistic world in which one or more opportunities appear and disappear in random locations. Agents use fuzzy reinforcement learning to learn which opportunities are most worthy of pursuing based on their promise rewards, expected lifetimes, path lengths and expected path costs. We show that this world is partially observable because the history of an agent influences the distribution of its future states. We consider a cooperation mechanism in which agents share experience by using and-updating one joint behavior policy. We also implement a coordination mechanism for allocating opportunities to different agents in the same world. Our results demonstrate that K cooperative agents each learning in a separate world over N time steps outperform K independent agents each learning in a separate world over K*N time steps, with this result becoming more pronounced as the degree of partial observability in the environment increases. We also show that cooperation between agents learning in the same world decreases performance with respect to independent agents. Since cooperation reduces diversity between agents, we conclude that diversity is a key parameter in the trade off between maximizing utility from cooperation when diversity is low and maximizing utility from competitive coordination when diversity is high.

Berenji, Hamid R.↗

Large deviations in stochastic heat-conduction processes provide a gradient-flow structure for heat conduction

We consider three one-dimensional continuous-time Markov processes on a lattice, each of which models the conduction of heat: the family of Brownian Energy Processes with parameter m (BEP(m)), a Generalized Brownian Energy Process, and the Kipnis-Marchioro-Presutti (KMP) process. The hydrodynamic limit of each of these three processes is a parabolic equation, the linear heat equation in the case of the BEP(m) and the KMP, and a nonlinear heat equation for the Generalized Brownian Energy Process with parameter a (GBEP(a)). We prove the hydrodynamic limit rigorously for the BEP(m), and give a formal derivation for the GBEP(a). We then formally derive the pathwise large-deviation rate functional for the empirical measure of the three processes. These rate functionals imply gradient-flow structures for the limiting linear and nonlinear heat equations. We contrast these gradient-flow structures with those for processes describing the diffusion of mass, most importantly the class of Wasserstein gradient-flow systems. The linear and nonlinear heat-equation gradient-flow structures are each driven by entropy terms of the form -log ρ; they involve dissipation or mobility terms of order ρ² for the linear heat equation, and a nonlinear function of ρ for the nonlinear heat equation.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Transitions in genetic toggle switches driven by dynamic disorder in rate coefficients

In biochemical systems, intrinsic noise may drive the system switch from one stable state to another. We investigate how kinetic switching between stable states in a bistable network is influenced by dynamic disorder, i.e., fluctuations in the rate coefficients. Using the geometric minimum action method, we first investigate the optimal transition paths and the corresponding minimum actions based on a genetic toggle switch model in which reaction coefficients draw from a discrete probability distribution. For the continuous probability distribution of the rate coefficient, we then consider two models of dynamic disorder in which reaction coefficients undergo different stochastic processes with the same stationary distribution. In one, the kinetic parameters follow a discrete Markov process and in the other they follow continuous Langevin dynamics. We find that regulation of the parameters modulating the dynamic disorder, as has been demonstrated to occur through allosteric control in bistable networks in the immune system, can be crucial in shaping the statistics of optimal transition paths, transition probabilities, and the stationary probability distribution of the network.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Hybrid framework for the simulation of stochastic chemical kinetics

Stochasticity plays a fundamental role in various biochemical processes, such as cell regulatory networks and enzyme cascades. Isothermal, well-mixed systems can be modelled as Markov processes, typically simulated using the Gillespie Stochastic Simulation Algorithm (SSA) [25]. While easy to implement and exact, the computational cost of using the Gillespie SSA to simulate such systems can become prohibitive as the frequency of reaction events increases. This has motivated numerous coarse-grained schemes, where the “fast” reactions are approximated either using Langevin dynamics or deterministically. While such approaches provide a good approximation when all reactants are abundant, the approximation breaks down when one or more species exist only in small concentrations and the fluctuations arising from the discrete nature of the reactions become significant. This is particularly problematic when using such methods to compute statistics of extinction times for chemical species, as well as simulating non-equilibrium systems such as cell-cycle models in which a single species can cycle between abundance and scarcity. In this paper, a hybrid jump-diffusion model for simulating well-mixed stochastic kinetics is derived. It acts as a bridge between the Gillespie SSA and the chemical Langevin equation. For low reactant reactions the underlying behaviour is purely discrete, while purely diffusive when the concentrations of all species are large, with the two different behaviours coexisting in the intermediate region. A bound on the weak error in the classical large volume scaling limit is obtained, and three different numerical discretisations of the jump-diffusion model are described. The benefits of such a formalism are illustrated using computational examples.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Quantum Markov States and Quantum Hidden Markov States

In a previous paper (Funct. Anal. Appl., 3 (2015), 205–209), we defined quantum Markov states. Here we recall this definition and present a proof of the results from that paper (which are given there without proofs). We give a definition of a quantum hidden Markov state generated by a function of a quantum Markov process and show how it is related to other definitions of such states. Our definitions work for quantum Markov fields on ℤ{sup N} and on graphs. We consider an example with the Cayley tree.

97 MATHEMATICS AND COMPUTING↗

A master equation and moment approach for biochemical systems with creation-time-dependent bimolecular rate functions

Noise and stochasticity are fundamental to biology and derive from the very nature of biochemical reactions where thermal motion of molecules translates into randomness in the sequence and timing of reactions. This randomness leads to cell-to-cell variability even in clonal populations. Stochastic biochemical networks have been traditionally modeled as continuous-time discrete-state Markov processes whose probability density functions evolve according to a chemical master equation (CME). In diffusion reaction systems on membranes, the Markov formalism, which assumes constant reaction propensities is not directly appropriate. This is because the instantaneous propensity for a diffusion reaction to occur depends on the creation times of the molecules involved. In this work, we develop a chemical master equation for systems of this type. While this new CME is computationally intractable, we make rational dimensional reductions to form an approximate equation, whose moments are also derived and are shown to yield efficient, accurate results. This new framework forms a more general approach than the Markov CME and expands upon the realm of possible stochastic biochemical systems that can be efficiently modeled.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

On the advantages of exploiting memory in Markov state models for biomolecular dynamics

Biomolecular dynamics play an important role in numerous biological processes. Markov State Models (MSMs) provide a powerful approach to study these dynamic processes by predicting long time scale dynamics based on many short molecular dynamics (MD) simulations. In an MSM, protein dynamics are modeled as a kinetic process consisting of a series of Markovian transitions between different conformational states at discrete time intervals (called “lag time”). To achieve this, a master equation must be constructed with a sufficiently long lag time to allow interstate transitions to become truly Markovian. This imposes a major challenge for MSM studies of proteins since the lag time is bound by the length of relatively short MD simulations available to estimate the frequency of transitions. Here, we show how one can employ the generalized master equation formalism to obtain an exact description of protein conformational dynamics both at short and long time scales without the time resolution restrictions imposed by the MSM lag time. Using a simple kinetic model, alanine dipeptide, and WW domain, we demonstrate that it is possible to construct these quasi-Markov State Models (qMSMs) using MD simulations that are 5–10 times shorter than those required by MSMs. These qMSMs only contain a handful of metastable states and, thus, can greatly facilitate the interpretation of mechanisms associated with protein dynamics. A qMSM opens the door to the study of conformational changes of complex biomolecules where a Markovian model with a few states is often difficult to construct due to the limited length of available MD simulations.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Brownian motion on [0,∞) with linear drift, reflected at zero: exact asymptotics for ergodic means

For the Brownian motion X{sub μ}(t) on the half-axis [0,∞) with linear drift μ, reflected at zero and for fixed numbers p>0, δ>0, d>0, a≥0, we calculate the exact asymptotics as T→∞ of the mathematical expectations and probabilities as well as of their conditional versions. For p=1 we give explicit formulae for the emerging constants via the Airy function. We consider an application of the results obtained to the problem of studying the behaviour of a Brownian particle in a gravitational field in a container bounded below by an impenetrable wall when μ=−mg/(2kT{sub K}), where m is the mass of the Brownian particle, g is the gravitational acceleration, k is the Boltzmann constant, T{sub K} is the temperature in the Kelvin scale. The analysis is conducted by the Laplace method for the sojourn time of homogeneous Markov processes. Bibliography: 31 titles. (paper)

97 MATHEMATICS AND COMPUTING↗

Systems of Semilinear Parabolic Variational Inequalities with Time-Dependent Convex Obstacles

We consider a system of seminlinear parabolic variational inequalities with time-dependent convex obstacles. We prove the existence and uniqueness of its solution. We also provide a stochastic representation of the solution and show that it can be approximated by the penalization method. Our proofs are based upon probabilistic methods from the theory of Markov processes and the theory of backward stochastic differential equations.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗