Low-complexity algorithm for restless bandits with imperfect observations

We consider a class of restless bandit problems that finds a broad application area in reinforcement learning and stochastic optimization. We consider N independent discrete-time Markov processes, each of which had two possible states: 1 and 0 (‘good’ and ‘bad’). Only if a process is both in state 1...

Full description

Saved in:
Bibliographic Details
Published in:Mathematical methods of operations research (Heidelberg, Germany) Vol. 100; no. 2; pp. 467 - 508
Main Authors: Liu, Keqin, Weber, Richard, Zhang, Chengzhong
Format: Journal Article
Language:English
Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.10.2024
Springer Nature B.V
Subjects:
Algorithms
Business and Management
Calculus of Variations and Optimal Control; Optimization
Complexity
Dynamic programming
Markov processes
Mathematics
Mathematics and Statistics
Multi-armed bandit problems
Operations Research/Decision Theory
Optimization
Original Research
93E20
Continuous state space
90B36
93E35
Observation errors
Restless bandits
Index policy
ISSN:1432-2994, 1432-5217
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We consider a class of restless bandit problems that finds a broad application area in reinforcement learning and stochastic optimization. We consider N independent discrete-time Markov processes, each of which had two possible states: 1 and 0 (‘good’ and ‘bad’). Only if a process is both in state 1 and observed to be so does reward accrue. The aim is to maximize the expected discounted sum of returns over the infinite horizon subject to a constraint that only M ( < N ) processes may be observed at each step. Observation is error-prone: there are known probabilities that state 1 (0) will be observed as 0 (1). From this one knows, at any time t , a probability that process i is in state 1. The resulting system may be modeled as a restless multi-armed bandit problem with an information state space of uncountable cardinality. Restless bandit problems with even finite state spaces are PSPACE-HARD in general. We propose a novel approach for simplifying the dynamic programming equations of this class of restless bandits and develop a low-complexity algorithm that achieves a strong performance and is readily extensible to the general restless bandit model with observation errors. Under certain conditions, we establish the existence (indexability) of Whittle index and its equivalence to our algorithm. When those conditions do not hold, we show by numerical experiments the near-optimal performance of our algorithm in the general parametric space. Furthermore, we theoretically prove the optimality of our algorithm for homogeneous systems.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1432-2994
1432-5217
DOI:10.1007/s00186-024-00868-x