Multi-armed bandit with sub-exponential rewards
Not provided.
Operations Research & Management Science↗
SEARCH · Search NASA
Results for “bandits”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
BANDIT in NASTRAN
BANDIT has been implemented into the NASTRAN April '83 release. It is now a permanent feature in NASTRAN and will be included in all future releases for all four computing machines (IBM, CDC, UNIVAC, and VAX). Originally BANDIT operated as a preprocessor of NASTRAN. It read the NASTRAN input cards and produced a set of resequencing (SEOGP) cards that would greatly reduce the computational time required by the matrix decomposition module of NASTRAN for a large structure. In the past, many computer centers had installed BANDIT in their systems together with the NASTRAN program. The user would run the BANDIT program and NASTRAN as if they were one program (or two separated programs depending on how the two programs were actually tied together in the computer systems). In some cases, the user was required to pass the input cards and the output SEQGP cards between the two programs, and n others, the data was manipulated through the use of cataloged disc files. Although there is nothing wrong with this BANDIT-NASTRAN arrangement, there are, however, several shortcomings which are mentioned in this paper. The BANDIT in the NASTRAN April '83 release has removed all the deficiencies, and it comes in one version applicable to the four computing machines.
Chan, G. C.↗
Output-weighted sampling for multi-armed bandits with extreme payoffs
We present a new type of acquisition function for online decision-making in multi-armed and contextual bandit problems with extreme payoffs. Specifically, we model the payoff function as a Gaussian process and formulate a novel type of upper confidence bound acquisition function that guides exploration towards the bandits that are deemed most relevant according to the variability of the observed rewards. This is achieved by computing a tractable likelihood ratio that quantifies the importance of the output relative to the inputs and essentially acts as an attention mechanism that promotes exploration of extreme rewards. Our formulation is supported by asymptotic zero-regret guarantees, and its performance is demonstrated across several synthetic benchmarks, as well as two realistic examples involving noisy sensor network data. Finally, we provide a JAX library for efficient bandit optimization using Gaussian processes.
Science & Technology - Other Topics↗
The BANDIT computer program for the reduction of matrix bandwidth for NASTRAN
A computer program, BANDIT, is described which was developed for use as a preprocessor to NASTRAN to automatically resequence the grid point numbers for reduced matrix bandwidth. The BANDIT program accepts a standard NASTRAN data deck as input, resequences the grid point numbers for reduced bandwidth, and generates a set of SEQGP cards for insertion into the NASTRAN bulk data deck. BANDIT uses the Cuthill-McKee strategy for resequencing grid points. The program is operational on the CDC 6000 series of computers, the IBM 360, and the Univac 1108.
Everstine, G. C.↗
Recent improvements to BANDIT
The NASTRAN preprocessor BANDIT, which improves NASTRAN's computer efficiency by resequencing grid point labels for reduced matrix bandwidth is described. The addition of (1) the Gibbs-Poole-Stockmeyer (GPS) algorithm, and (2) the user option to reduce matrix profile rather than matrix bandwidth is also described. It is shown that, compared to the Cuthill-McKee algorithm on which BANDIT was originally based, GPS is faster and achieves similar results. Current capabilities and options of BANDIT are summarized.
Everstine, G. C.↗
Learning infinite-horizon average-reward restless multi-action bandits via index awareness
We consider the online restless bandits with average-reward and multiple actions, where the state of each arm evolves according to a Markov decision process (MDP), and the reward of pulling an arm depends on both the current state of the corresponding MDP and the action taken. Since finding the optimal control is typically intractable for restless bandits, existing learning algorithms are often computationally expensive or with a regret bound that is exponential in the number of arms and states. In this paper, we advocate \textit{index-aware reinforcement learning} (RL) solutions to design RL algorithms operating on a much smaller dimensional subspace by exploiting the inherent structure in restless bandits. Specifically, we first propose novel index policies to address dimensionality concerns, which are provably optimal. We then leverage the indices to develop two low-complexity index-aware RL algorithms, namely, (i) GM-R2MAB, which has access to a generative model; and (ii) UC-R2MAB, which learns the model using an upper confidence style online exploitation method. We prove that both algorithms achieve a sub-linear regret that is only polynomial in the number of arms and states. A key differentiator between our algorithms and existing ones stems from the fact that our RL algorithms contain a novel exploitation that leverages our proposed provably optimal index policies for decision-makings.
Xiong, Guojun↗
Multi-fidelity Gaussian Process Bandit Optimisation
In many scientific and engineering applications, we are tasked with the maximisation of an expensive to evaluate black box function f. Traditional settings for this problem assume just the availability of this single function. However, in many cases, cheap approximations to f may be obtainable. For example, the expensive real world behaviour of a robot can be approximated by a cheap computer simulation. We can use these approximations to eliminate low function value regions cheaply and use the expensive evaluations of f in a small but promising region and speedily identify the optimum. We formalise this task as a multi-fidelity bandit problem where the target function and its approximations are sampled from a Gaussian process. We develop MF-GP-UCB, a novel method based on upper confidence bound techniques. In our theoretical analysis we demonstrate that it exhibits precisely the above behaviour and achieves better bounds on the regret than strategies which ignore multi-fidelity information. Empirically, MF-GP-UCB outperforms such naive strategies and other multi-fidelity methods on several synthetic and real experiments.
97 MATHEMATICS AND COMPUTING↗
Online eco-routing for electric vehicles using combinatorial multi-armed bandit with estimated covariance
Identifying energy-efficient routes in real-time has significant implications for the energy-optimal operations of electric vehicles (EVs). Here, this study proposes a novel model for EV online eco-routing problem, which obtains the minimal expected energy consumption paths (MECPs) for multiple origin-destination (OD) pairs simultaneously. Specifically, we formulate the routing problem as a bandit problem and solve it with online algorithms. We extend the algorithms by implementing a path elimination mechanism to reduce the candidate path set and introducing the variance and covariance of the energy consumption to reduce the uncertainties. The numerical results show that the proposed algorithms can efficiently obtain near-optimal MECPs, and the solution is significantly better than the widely used shortest trip time path algorithm (STTP) and shortest trip distance path algorithm (SDP). The variation considering link energy covariance and path elimination generates paths that save 4.1% of energy compared to the SDP and 5.4% to the STTP.
33 ADVANCED PROPULSION SYSTEMS↗
Augmenting subspace optimization methods with linear bandits
In this work, we consider the framework of methods for unconstrained minimization that are, in each iteration, restricted to a model that is only a valid approximation to the objective function on some affine subspace containing an incumbent point. These methods are of practical interest in computational settings where derivative information is either expensive or impossible to obtain. Recent attention has been paid in the literature to employing randomized matrix sketching for generating the affine subspaces within this framework. We consider a relatively straightforward, deterministic augmentation of such a generic subspace optimization method. In particular, we consider a sequential optimization framework where actions consist of one-dimensional linear subspaces and rewards consist of (approximations to) the magnitudes of directional derivatives computed in the direction of the action subspace. Reward maximization in this context is consistent with maximizing lower bounds on descent guaranteed by first-order Taylor models. This sequential optimization problem can be analysed through the lens of dynamic regret. We modify an existing linear upper confidence bound (UCB) bandit method and prove sublinear dynamic regret in the subspace optimization setting. We demonstrate the efficacy of employing this linear UCB method in a setting where forward-mode algorithmic differentiation can provide directional derivatives in arbitrary directions and in a derivative-free setting. For the derivative-free setting, we propose SS-POUNDers, an extension of the derivative-free optimization method POUNDers that employs the linear UCB mechanism to identify promising subspaces. Our numerical experiments suggest a preference, in either computational setting, for employing a linear UCB mechanism within a subspace optimization method.
97 MATHEMATICS AND COMPUTING↗
Online Energy-optimal Routing for Electric Vehicles with Combinatorial Multi-arm Semi-Bandit
We have seen a rapid growth in the adoption of electric vehicle in today’s commercial mobility service market, from carrying passengers to the delivery of goods. One cardinal issue concerning the operation of EVs is the optimal routing of EV fleets with limited battery capacity. In this study, we investigate the energy-optimal online routing problem for the fleet of EVs, which focuses on identifying real-time minimum electricity consumption paths (MECP) for multiple OD pairs with limited information. We develop a multi-OD combinatorial multi-arm semi-bandit model (MCMAB) that uses the fleet of EVs as sensors in the transportation network and promotes the utilization of common information shared by different OD pairs. We further enrich the model with the path elimination policy to obtain MECP of high confidence while significantly reducing the number of learning iterations and the number of explorations needed. We demonstrate the effectiveness of the MCMAB and the efficiency of the path elimination policy with comprehensive numerical experiments in Manhattan, NYC. The results show that the proposed online routing algorithms can achieve near-optimal MECPs efficiently, and the quality of the solutions is significantly better than using the shortest travel time paths as approximate MECPs.
Chen, Xiaowei↗
Finite-Time Analysis of Whittle Index based Q-Learning for Restless Multi-Armed Bandits with Neural Network Function Approximation
Whittle index policy is a heuristic to the intractable restless multi-armed bandits (RMAB) problem. Although it is provably asymptotically optimal, finding Whittle indices remains difficult. In this paper, we present Neural-Q-Whittle, a Whittle index based Q-learning algorithm for RMAB with neural network function approximation, which is an example of nonlinear two-timescale stochastic approximation with Q-function values updated on a faster timescale and Whittle indices on a slower timescale. Despite the empirical success of deep Q-learning, the non-asymptotic convergence rate of Neural-Q-Whittle, which couples neural networks with two-timescale Q-learning largely remains unclear. This paper provides a finite-time analysis of Neural-Q-Whittle, where data are generated from a Markov chain, and Q-function is approximated by a ReLU neural network. Our analysis leverages a Lyapunov drift approach to capture the evolution of two coupled parameters, and the nonlinearity in value function approximation further requires us to characterize the approximation error. Combing these provide Neural-Q-Whittle with convergence rate, where is the number of iterations.
reinforcement learning, structured learning, conve↗
A reliability-aware multi-armed bandit approach to learn and select users in demand response
Not Available
Li, Yingying↗
Adaptive PPO With Multi-Armed Bandit Clipping and Meta-Control for Robust Power Grid Operation Under Adversarial Attacks
Not Available
Massaoudi, Mohamed [Department of Electrical and C↗
Reinforcement Learning for Dynamic Dimensioning of Cloud Caches: A Restless Bandit Approach
Not provided.
Computer Science↗
Proximity Navigation of Highly Constrained Spacecraft
Bandit is a 3-kg automated spacecraft in development at Washington University in St. Louis. Bandit's primary mission is to demonstrate proximity navigation, including docking, around a 25-kg student-built host spacecraft. However, because of extreme constraints in mass, power and volume, traditional sensing and actuation methods are not available. In particular, Bandit carries only 8 fixed-magnitude cold-gas thrusters to control its 6 DOF motion. Bandit lacks true inertial sensing, and the ability to sense position relative to the host has error bounds that approach the size of the Bandit itself. Some of the navigation problems are addressed through an extremely robust, error-tolerant soft dock. In addition, we have identified a control methodology that performs well in this constrained environment: behavior-based velocity potential functions, which use a minimum-seeking method similar to Lyapunov functions. We have also adapted the discrete Kalman filter for use on Bandit for position estimation and have developed a similar measurement vs. propagation weighting algorithm for attitude estimation. This paper provides an overview of Bandit and describes the control and estimation approach. Results using our 6DOF flight simulator are provided, demonstrating that these methods show promise for flight use.
Scarritt, S.↗
Improved performance in NASTRAN (R)
Three areas of improvement in COSMIC/NASTRAN, 1989 release, were incorporated recently that make the analysis program run faster on large problems. Actual log files and actual timings on a few test samples that were run on IBM, CDC, VAX, and CRAY computers were compiled. The speed improvement is proportional to the problem size and number of continuation cards. Vectorizing certain operations in BANDIT, makes BANDIT run twice as fast in some large problems using structural elements with many node points. BANDIT is a built-in NASTRAN processor that optimizes the structural matrix bandwidth. The VAX matrix packing routine BLDPK was modified so that it is now packing a column of a matrix 3 to 9 times faster. The denser and bigger the matrix, the greater is the speed improvement. This improvement makes a host of routines and modules that involve matrix operation run significantly faster, and saves disc space for dense matrices. A UNIX version, converted from 1988 COSMIC/NASTRAN, was tested successfully on a Silicon Graphics computer using the UNIX V Operating System, with Berkeley 4.3 Extensions. The Utility Modules INPUTT5 and OUTPUT5 were expanded to handle table data, as well as matrices. Both INPUTT5 and OUTPUT5 are general input/output modules that read and write FORTRAN files with or without format. More user informative messages are echoed from PARAMR, PARAMD, and SCALAR modules to ensure proper data values and data types being handled. Two new Utility Modules, GINOFILE and DATABASE, were written for the 1989 release. Seven rigid elements are added to COSMIC/NASTRAN. They are: CRROD, CRBAR, CRTRPLT, CRBE1, CRBE2, CRBE3, and CRSPLINE.
Chan, Gordon C.↗
An Intelligent Distributed Ledger Construction Algorithm for IoT
Blockchain is the next generation of secure data management that creates near-immutable decentralized storage. Secure cryptography created a niche for blockchain to provide alternatives to well-known security compromises. However, design bottlenecks with traditional blockchain data structures scale poorly with increased network usage and are extremely computation-intensive. This made the technology difficult to combine with limited devices, like those in Internet of Things networks. In protocols like IOTA, replacement of blockchain's linked-list queue processing with a lightweight dynamic ledger showed remarkable throughput performance increase. However, current stochastic algorithms for ledger construction suffer distinct trade-offs between efficiency and security. This work proposed a machine-learning approach with a multi-arm bandit that resolved these issues and was designed for auditing on limited devices. This algorithm was tested in a reinforcement-learning environment simulating the IOTA ledger's construction with a decision tree. This study showed through regret analysis and experimentation that this approach was secure against impulse manipulation attacks while remaining energy-efficient. Although the IOTA protocol was a pioneer for lightweight distributed ledgers, it is expected that future blockchain protocols will adopt techniques similar to those presented in this work.
multi-arm bandit↗
Artificial intelligence for materials research at extremes
Abstract Materials development is slow and expensive, taking decades from inception to fielding. For materials research at extremes, the situation is even more demanding, as the desired property combinations such as strength and oxidation resistance can have complex interactions. Here, we explore the role of AI and autonomous experimentation (AE) in the process of understanding and developing materials for extreme and coupled environments. AI is important in understanding materials under extremes due to the highly demanding and unique cases these environments represent. Materials are pushed to their limits in ways that, for example, equilibrium phase diagrams cannot describe. Often, multiple physical phenomena compete to determine the material response. Further, validation is often difficult or impossible. AI can help bridge these gaps, providing heuristic but valuable links between materials properties and performance under extreme conditions. We explore the potential advantages of AE along with decision strategies. In particular, we consider the problem of deciding between low-fidelity, inexpensive experiments and high-fidelity, expensive experiments. The cost of experiments is described in terms of the speed and throughput of automated experiments, contrasted with the human resources needed to execute manual experiments. We also consider the cost and benefits of modeling and simulation to further materials understanding, along with characterization of materials under extreme environments in the AE loop. Graphical abstract AI sequential decision-making methods for materials research: Active learning, which focuses on exploration by sampling uncertain regions, Bayesian and bandit optimization as well as reinforcement learning (RL), which trades off exploration of uncertain regions with exploitation of optimum function value. Bayesian and bandit optimization focus on finding the optimal value of the function at each step or cumulatively over the entire steps, respectively, whereas RL considers cumulative value of the labeling function, where the latter can change depending on the state of the system (blue, orange, or green).
36 MATERIALS SCIENCE↗