SEARCH · Search NASA

Results for “MARKOV”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2026

Hierarchical semi-Markov models with duration-aware dynamics for activity sequences

Residential electricity demand at granular scales is driven by what people do and for how long. Accurately forecasting this demand for applications like microgrid management and demand response therefore requires generative models for activities that can produce realistic daily activity sequences, capturing both the timing and duration of human behavior. This paper develops a generative model of human activity sequences using nationally representative time-use diaries at a 10-min resolution. We use this model to quantify which demographic factors are most critical for improving predictive performance. We propose a hierarchical semi-Markov framework that addresses two key modeling challenges. First, a time-inhomogeneous Markov router learns the patterns of “which activity comes next.” Second, a semi-Markov hazard component explicitly models activity durations, capturing “how long” activities realistically last. To ensure statistical stability when data are sparse, the model pools information across related demographic groups and time blocks. The entire framework is trained and evaluated using survey design weights to ensure our findings are representative of the U.S. population. On a held-out test set, we demonstrate that explicitly modeling durations with the hazard component provides a substantial and statistically significant improvement over purely Markovian models. Furthermore, our analysis reveals a clear hierarchy of demographic factors: Sex, Day-Type, and Household Size provide the largest predictive gains, while Region and Season, though important for energy calculations, contribute little to predicting the activity sequence itself. The result is an interpretable and robust generator of synthetic activity traces, providing a high-fidelity foundation for downstream energy systems modeling.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2023

NeuralRW-Loihi: Spiking Discrete Time Markov Chain Simulator for Intel Loihi v

SAND2024-01302O NeuralRW-Loihi: Spiking Discrete Time Markov Chain Simulator for Intel Loihi is a neuromorphic script that generates a random walk simulation of a discrete-time Markov Chain for the Loihi neuromorphic hardware. The code was used to produce the paper "Neuromorphic scaling advantages for energy-efficient random walk computations" by Smith et al., 2021. It was published in Nature Electronics. The software provides code for implementing the discrete time Markov chain random walks on Intel's Loihi Neuromorphic platform. In addition, it provides an example problem through a mini app.

Aimone, James↗

NASA NTRS1986

Towards automatic Markov reliability modeling of computer architectures

The analysis and evaluation of reliability measures using time-varying Markov models is required for Processor-Memory-Switch (PMS) structures that have competing processes such as standby redundancy and repair, or renewal processes such as transient or intermittent faults. The task of generating these models is tedious and prone to human error due to the large number of states and transitions involved in any reasonable system. Therefore model formulation is a major analysis bottleneck, and model verification is a major validation problem. The general unfamiliarity of computer architects with Markov modeling techniques further increases the necessity of automating the model formulation. This paper presents an overview of the Automated Reliability Modeling (ARM) program, under development at NASA Langley Research Center. ARM will accept as input a description of the PMS interconnection graph, the behavior of the PMS components, the fault-tolerant strategies, and the operational requirements. The output of ARM will be the reliability of availability Markov model formulated for direct use by evaluation programs. The advantages of such an approach are (a) utility to a large class of users, not necessarily expert in reliability analysis, and (b) a lower probability of human error in the computation.

Liceaga, C. A.↗

NASA NTRS1988

Markov chains for testing redundant software

A preliminary design for a validation experiment has been developed that addresses several problems unique to assuring the extremely high quality of multiple-version programs in process-control software. The procedure uses Markov chains to model the error states of the multiple version programs. The programs are observed during simulated process-control testing, and estimates are obtained for the transition probabilities between the states of the Markov chain. The experimental Markov chain model is then expanded into a reliability model that takes into account the inertia of the system being controlled. The reliability of the multiple version software is computed from this reliability model at a given confidence level using confidence intervals obtained for the transition probabilities during the experiment. An example demonstrating the method is provided.

White, Allan L.↗

NASA NTRS1990

Performability analysis using semi-Markov reward processes

Beaudry (1978) proposed a simple method of computing the distribution of performability in a Markov reward process. Two extensions of Beaudry's approach are presented. The method is generalized to a semi-Markov reward process by removing the restriction requiring the association of zero reward to absorbing states only. The algorithm proceeds by replacing zero-reward nonabsorbing states by a probabilistic switch; it is therefore related to the elimination of vanishing states from the reachability graph of a generalized stochastic Petri net and to the elimination of fast transient states in a decomposition approach to stiff Markov chains. The use of the approach is illustrated with three applications.

Ciardo, Gianfranco↗

NASA NTRS1994

Hierarchical modeling for reliability analysis using Markov models

Markov models represent an extremely attractive tool for the reliability analysis of many systems. However, Markov model state space grows exponentially with the number of components in a given system. Thus, for very large systems Markov modeling techniques alone become intractable in both memory and CPU time. Often a particular subsystem can be found within some larger system where the dependence of the larger system on the subsystem is of a particularly simple form. This simple dependence can be used to decompose such a system into one or more subsystems. A hierarchical technique is presented which can be used to evaluate these subsystems in such a way that their reliabilities can be combined to obtain the reliability for the full system. This hierarchical approach is unique in that it allows the subsystem model to pass multiple aggregate state information to the higher level model, allowing more general systems to be evaluated. Guidelines are developed to assist in the system decomposition. An appropriate method for determining subsystem reliability is also developed. This method gives rise to some interesting numerical issues. Numerical error due to roundoff and integration are discussed at length. Once a decomposition is chosen, the remaining analysis is straightforward but tedious. However, an approach is developed for simplifying the recombination of subsystem reliabilities. Finally, a real world system is used to illustrate the use of this technique in a more practical context.

Fagundo, Arturo↗

NASA NTRS1994

On the utility of the multi-level algorithm for the solution of nearly completely decomposable Markov chains

Recently the Multi-Level algorithm was introduced as a general purpose solver for the solution of steady state Markov chains. In this paper, we consider the performance of the Multi-Level algorithm for solving Nearly Completely Decomposable (NCD) Markov chains, for which special-purpose iteractive aggregation/disaggregation algorithms such as the Koury-McAllister-Stewart (KMS) method have been developed that can exploit the decomposability of the the Markov chain. We present experimental results indicating that the general-purpose Multi-Level algorithm is competitive, and can be significantly faster than the special-purpose KMS algorithm when Gauss-Seidel and Gaussian Elimination are used for solving the individual blocks.

Leutenegger, Scott T.↗

NASA NTRS1994

Hidden Markov Models for Fault Detection in Dynamic Systems

Continuous monitoring of complex dynamic systems is an increasingly important issue in diverse areas such as nuclear plant safety, production line reliability, and medical health monitoring systems. Recent advances in both sensor technology and computational capabilities have made on-line permanent monitoring much more feasible than it was in the past. In this paper it is shown that a pattern recognition system combined with a finite-state hidden Markov model provides a particularly useful method for modelling temporal context in continuous monitoring. The parameters of the Markov model are derived from gross failure statistics such as the mean time between failures. The model is validated on a real-world fault diagnosis problem and it is shown that Markov modelling in this context offers significant practical benefits.

Smyth, Padhraic↗

NASA NTRS2012

Markov Modeling of Component Fault Growth over a Derived Domain of Feasible Output Control Effort Modifications

This paper introduces a novel Markov process formulation of stochastic fault growth modeling, in order to facilitate the development and analysis of prognostics-based control adaptation. A metric representing the relative deviation between the nominal output of a system and the net output that is actually enacted by an implemented prognostics-based control routine, will be used to define the action space of the formulated Markov process. The state space of the Markov process will be defined in terms of an abstracted metric representing the relative health remaining in each of the system s components. The proposed formulation of component fault dynamics will conveniently relate feasible system output performance modifications to predictions of future component health deterioration.

Bole, Brian↗

NASA NTRSDate not supplied

Adding GPU Support to the Markov Chain Monte Carlo Code Catmip

In geophysics, we are confronted with many under-determined inverse problems. For example, all of our observations of earthquakes are made at the Earth’s surface. So, when we try to infer how slip during an earthquake evolves in space and time, we find that there are many potential slip histories that are consistent with our limited observations and our understanding of earthquake physics. One way to approach these problems is with Bayesian analysis which allows us to infer the ensemble of all potential slip models that satisfy the observations and our prior knowledge of earthquake physics. In Bayesian analysis, our prior knowledge is known as the prior probability density function or prior PDF, the fit to the data is known as the data likelihood, and the target PDF that satisfies both the prior PDF and data likelihood is known as the posterior PDF. However, simulating the posterior PDF typically requires using Markov Chain Monte Carlo (MCMC) to draw tens of billions of random realizations of earthquake slip models, which may not be computationally feasible. To make this and similar geophysical inversions computationally tractable, we developed the Cascading Adaptive Transitional Metropolis In Parallel (CATMIP) algorithm. CATMIP is an efficient parallel Markov Chain Monte Carlo (MCMC) sampler that is used for model fitting and uncertainty quantification in geophysics. Example use cases are earthquake rupture modeling, determining mineral composition on Mars, reconstructing the history of ocean salinity, and historical earthquake relocation. CATMIP employs many parallel instances of the Metropolis algorithm for sampling in a transitioning framework. Transitioning is a process in which a set of random samples at equilibrium with a known probability density function (PDF) are used as seeds for the Markov chains to sample successive target PDFs that incrementally move the distribution from the starting seeds to the final desired PDF that describes the relative plausibility of potential values for the model parameters. The algorithm is implemented as a Master-Worker model employing MPI for communication. The worker processes are loosely coupled with global parameters periodically optimized by the master process. This provides a very high amount of parallelism with little communication between updates. During the presentation we will discuss the history of the algorithm and elaborate the earthquake rupture modeling use case for the CATMIP package. Our first step toward GPU optimization was to optimize the code for the CPU. CPU profiling revealed that most of the compute time is spent in calls to level 2 BLAS routines and calls to GSL random number generators. We revised the algorithm to employ level 3 BLAS routines instead. In our presentation we will describe how this was accomplished. Adding GPU support to CATMIP consisted mostly of replacing the calls to GSL with calls to GPU vendor-provided library routines. A small number of loops were directly implemented in CUDA. In the presentation will provide implementation details. Finally, we will discuss methods for profiling and opportunities for further optimizing GPU execution. By creating a code with the flexibility to run on either a CPU or GPU architecture, CATMIP can be used on systems ranging from large CPU-based HPC environments to single servers with GPU acceleration and everything in between.

HECC↗

DOE OSTI2021

On the Stochastic Stability of Deep Markov Models

Deep Markov models (DMM) are generative models which are scalable and expressive generalization of Markov models for representation, learning, and inference problems. DMMs using deep neural networks to parametrize the transition of Markov probability distributions have recently been shown to provide more expressiveness in modeling sequential data and dynamical system responses. However, the fundamental stochastic stability guarantees of such models have not been thoroughly investigated. In this paper, we present a rigorous analytical method to prove the necessary and sufficient conditions of DMM's stochastic stability. This task is achieved by spectral analysis of the efficiently computed Jacobians of probabilistic maps modeled by deep neural networks. We make theoretical connections between the eigenvalues of neural network's weights and the different activation function types used on the stability and overall dynamic behavior of DMMs with Gaussian distributions. We empirically substantiate our theoretical results on stochastic stability and eigenvalue spectra via several numerical experiments. Formal stability guarantees of DMMs can substantially improve their robustness and trustworthiness, necessary for reliable use in safety-critical real-world applications.

Drgona, Jan↗

DOE OSTI2020

K-Spin Hamiltonian for Quantum-Resolvable Markov Decision Processes

The Markov decision process is the mathematical formalization underlying the modern field of reinforcement learning when transition and reward functions are unknown. We derive a pseudo-Boolean cost function that is equivalent to a K-spin Hamiltonian representation of the discrete, finite, discounted Markov decision process with infinite horizon. This K-spin Hamiltonian furnishes a starting point from which to solve for an optimal policy using heuristic quantum algorithms such as adiabatic quantum annealing and the quantum approximate optimization algorithm on near-term quantum hardware. In arguing that the variational minimization of our Hamiltonian is approximately equivalent to the Bellman optimality condition for a prevalent class of environments we establish an interesting analogy with classical field theory. Along with proof-of-concept calculations to corroborate our formulation by simulated and quantum annealing against classical Q-Learning, we analyze the scaling of physical resources required to solve our Hamiltonian on quantum hardware.

Hamiltonian↗

DOE OSTI2020

Markovian transport processes in a heterogeneous, variably saturated watershed: A multi-domain spatial Markov model

Transport processes in watersheds remain challenging to describe accurately with upscaled models because of their intricate complexities at multiple scales that can lead to multi-modal, non-Fickian breakthrough curves. One of the recent advances in solute transport modeling has been the Lagrangian spatial Markov model (SMM), which describes transport using velocity increments and models the correlated transitions between them. The method has been applied previously to saturated porous media and fractured media, but these successful applications suggest it may be useful for describing watershed scale processes as well. An existing, 3-D, heterogeneous variably saturated model of a headwaters catchment and classical random walk particle tracking were used to generate a forward simulation and the resulting transport was analyzed with a spatial Markov framework. Existing SMM methods were used to analyze the transport paths through the complex, unconfined domain and we found that the vadose zone plays an important role in the correlation structure. As such, a new kind of SMM is proposed, termed a multi-domain SMM (MD-SMM), that uses separate correlation models for auto-transitions in the vadose zone and the saturated zone, and for the cross-transitions between them. The MD-SMM improves representation of transport relative to the single domain model and can naturally delineate times when a particle is in the vadose zone or the saturated zone, adding an additional degree of realism to the upscaled model. Lastly, we show that the transition probability matrix for transport in the saturated portion of this unconfined system is approximately Markovian, which is an important validation of the proposed framework.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2020

Series of semi-Markov processes to model infrastructure resilience under multihazards

Civil infrastructure systems are subjected to multiple hazards, including natural and anthropogenic, that disrupt their function or the level of service offered. Estimating the function recovery of these systems (or how soon normalcy of operations will be restored) when subjected to repeated hazard events by considering the inter-event dependencies is an important problem in multihazard infrastructure resilience. However, this problem has been less addressed in the field. This paper proposes a series of semi-Markov processes model to capture the inter-event dependencies in infrastructure recovery when subjected to successive hazard events. Recovery after each new hazard event is represented by a unique semi-Markov process that models the reduced recovery rates and the increased recovery times caused by the system’s incomplete recovery from the preceding event. Two novel formulations of the inter-event dependency modeling, namely Maximal Effects Dependency (considers the worst impact of two successive hazard events) and Cumulative Effects Dependency (considers the aggregated impacts of two successive hazard events), are proposed and discussed. The model is demonstrated by considering the following applications: Three-state system subjected to deterministic and random occurrences of identical hazard events; and Multihazard resilience of a building in Charleston, SC, considering earthquake and hurricane hazards. Results indicate that considering inter-event dependencies in recovery modeling can lead to lesser-predicted resilience, thereby affecting resilience-based decision-making.

42 ENGINEERING↗

DOE OSTI2021

Sequential Bayesian inference of transition rates in the hidden Markov model for multi-state system degradation

The more easily available system performance data and advances in data analytics have provided us with opportunities to optimize maintenance programs for engineered systems, for example nuclear power plants. One key task in maintenance optimization is to obtain an accurate model for system degradation. Here, in this research, we propose a Bayesian method to address this problem. Noting that systems usually exhibit multiple states and that the actual state of a system usually is not directly observable, in the method we first model the system degradation process and the observation process based on a hidden Markov model. Then we develop a sequential Bayesian inference algorithm based on importance sampling and the forward algorithm to infer the posterior distributions of the transition rates in the hidden Markov model based on available observations. The proposed Bayesian method allows us to take advantage of evidence from multiple sources, and also allows us to perform Bayesian inference sequentially, without the need to use the entire history of observations every time new observations are collected. We demonstrate the proposed method using both synthetic data for a nuclear power plant feedwater pump and realistic data for a nuclear power plant chemistry analytical device.

42 ENGINEERING↗

DOE OSTI2020

On the advantages of exploiting memory in Markov state models for biomolecular dynamics

Biomolecular dynamics play an important role in numerous biological processes. Markov State Models (MSMs) provide a powerful approach to study these dynamic processes by predicting long time scale dynamics based on many short molecular dynamics (MD) simulations. In an MSM, protein dynamics are modeled as a kinetic process consisting of a series of Markovian transitions between different conformational states at discrete time intervals (called “lag time”). To achieve this, a master equation must be constructed with a sufficiently long lag time to allow interstate transitions to become truly Markovian. This imposes a major challenge for MSM studies of proteins since the lag time is bound by the length of relatively short MD simulations available to estimate the frequency of transitions. Here, we show how one can employ the generalized master equation formalism to obtain an exact description of protein conformational dynamics both at short and long time scales without the time resolution restrictions imposed by the MSM lag time. Using a simple kinetic model, alanine dipeptide, and WW domain, we demonstrate that it is possible to construct these quasi-Markov State Models (qMSMs) using MD simulations that are 5–10 times shorter than those required by MSMs. These qMSMs only contain a handful of metastable states and, thus, can greatly facilitate the interpretation of mechanisms associated with protein dynamics. A qMSM opens the door to the study of conformational changes of complex biomolecules where a Markovian model with a few states is often difficult to construct due to the limited length of available MD simulations.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2020

Probabilistic Power Consumption Modeling for Commercial Buildings Using Logistic Regression Markov Chain

The total energy consumed by buildings takes up to 40% of U.S. energy use, in which a large portion is contributed by commercial buildings. Building performance optimization is desirable but requires accurate building models with uncertainties taken into account. This paper proposes a novel probabilistic modeling method using Logistic Regression Markov Chain (LRMC). The LRMC model enhances the performance of traditional Markov Chain (MC) models by adopting time-variant transition matrices calibrated using logistic regression with exogenous inputs. Compared with existing building models, the proposed model produces accurate multi-step modeling results with full probability distribution. The proposed probabilistic building model is tested using actual commercial building measurements and modeling performance is evaluated with two probabilisitc metrics. The results show that the LRMC model has higher accuracy than traditional MC model and Logistic Regression (LR) model in that it yields lower error scores under both evaluation metrics.

Building modeling↗

DOE OSTI2022

Challenges in Markov Chain Monte Carlo for Bayesian Neural Networks

Markov chain Monte Carlo (MCMC) methods have not been broadly adopted in Bayesian neural networks (BNNs). This paper initially reviews the main challenges in sampling from the parameter posterior of a neural network via MCMC. Such challenges culminate to lack of convergence to the parameter posterior. Nevertheless, this paper shows that a nonconverged Markov chain, generated via MCMC sampling from the parameter space of a neural network, can yield via Bayesian marginalization a valuable posterior predictive distribution of the output of the neural network. Further, classification examples based on multilayer perceptrons showcase highly accurate posterior predictive distributions. The postulate of limited scope for MCMC developments in BNNs is partially valid; an asymptotically exact parameter posterior seems less plausible, yet an accurate posterior predictive distribution is a tenable research avenue.

97 MATHEMATICS AND COMPUTING↗