Multi-armed bandit with sub-exponential rewards
Not provided.
Operations Research & Management Science↗
SEARCH · Search NASA
Results for “multi-arm bandit”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Finite-Time Analysis of Whittle Index based Q-Learning for Restless Multi-Armed Bandits with Neural Network Function Approximation
Whittle index policy is a heuristic to the intractable restless multi-armed bandits (RMAB) problem. Although it is provably asymptotically optimal, finding Whittle indices remains difficult. In this paper, we present Neural-Q-Whittle, a Whittle index based Q-learning algorithm for RMAB with neural network function approximation, which is an example of nonlinear two-timescale stochastic approximation with Q-function values updated on a faster timescale and Whittle indices on a slower timescale. Despite the empirical success of deep Q-learning, the non-asymptotic convergence rate of Neural-Q-Whittle, which couples neural networks with two-timescale Q-learning largely remains unclear. This paper provides a finite-time analysis of Neural-Q-Whittle, where data are generated from a Markov chain, and Q-function is approximated by a ReLU neural network. Our analysis leverages a Lyapunov drift approach to capture the evolution of two coupled parameters, and the nonlinearity in value function approximation further requires us to characterize the approximation error. Combing these provide Neural-Q-Whittle with convergence rate, where is the number of iterations.
reinforcement learning, structured learning, conve↗
Output-weighted sampling for multi-armed bandits with extreme payoffs
We present a new type of acquisition function for online decision-making in multi-armed and contextual bandit problems with extreme payoffs. Specifically, we model the payoff function as a Gaussian process and formulate a novel type of upper confidence bound acquisition function that guides exploration towards the bandits that are deemed most relevant according to the variability of the observed rewards. This is achieved by computing a tractable likelihood ratio that quantifies the importance of the output relative to the inputs and essentially acts as an attention mechanism that promotes exploration of extreme rewards. Our formulation is supported by asymptotic zero-regret guarantees, and its performance is demonstrated across several synthetic benchmarks, as well as two realistic examples involving noisy sensor network data. Finally, we provide a JAX library for efficient bandit optimization using Gaussian processes.
Science & Technology - Other Topics↗
Online eco-routing for electric vehicles using combinatorial multi-armed bandit with estimated covariance
Identifying energy-efficient routes in real-time has significant implications for the energy-optimal operations of electric vehicles (EVs). Here, this study proposes a novel model for EV online eco-routing problem, which obtains the minimal expected energy consumption paths (MECPs) for multiple origin-destination (OD) pairs simultaneously. Specifically, we formulate the routing problem as a bandit problem and solve it with online algorithms. We extend the algorithms by implementing a path elimination mechanism to reduce the candidate path set and introducing the variance and covariance of the energy consumption to reduce the uncertainties. The numerical results show that the proposed algorithms can efficiently obtain near-optimal MECPs, and the solution is significantly better than the widely used shortest trip time path algorithm (STTP) and shortest trip distance path algorithm (SDP). The variation considering link energy covariance and path elimination generates paths that save 4.1% of energy compared to the SDP and 5.4% to the STTP.
33 ADVANCED PROPULSION SYSTEMS↗
Adaptive PPO With Multi-Armed Bandit Clipping and Meta-Control for Robust Power Grid Operation Under Adversarial Attacks
Not Available
Massaoudi, Mohamed [Department of Electrical and C↗
An Intelligent Distributed Ledger Construction Algorithm for IoT
Blockchain is the next generation of secure data management that creates near-immutable decentralized storage. Secure cryptography created a niche for blockchain to provide alternatives to well-known security compromises. However, design bottlenecks with traditional blockchain data structures scale poorly with increased network usage and are extremely computation-intensive. This made the technology difficult to combine with limited devices, like those in Internet of Things networks. In protocols like IOTA, replacement of blockchain's linked-list queue processing with a lightweight dynamic ledger showed remarkable throughput performance increase. However, current stochastic algorithms for ledger construction suffer distinct trade-offs between efficiency and security. This work proposed a machine-learning approach with a multi-arm bandit that resolved these issues and was designed for auditing on limited devices. This algorithm was tested in a reinforcement-learning environment simulating the IOTA ledger's construction with a decision tree. This study showed through regret analysis and experimentation that this approach was secure against impulse manipulation attacks while remaining energy-efficient. Although the IOTA protocol was a pioneer for lightweight distributed ledgers, it is expected that future blockchain protocols will adopt techniques similar to those presented in this work.
multi-arm bandit↗