Sequential Decision Making With Limited Observation Capability: Application to Wireless Networks

This paper studies a generalized class of restless multi-armed bandits with hidden states and allow cumulative feedback, as opposed to the conventional instantaneous feedback. We call them lazy restless bandits (LRBs) as the events of decision making are sparser than the events of state transition....

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on cognitive communications and networking Vol. 5; no. 2; pp. 237 - 251
Main Authors: Kaza, Kesav, Meshram, Rahul, Mehta, Varun, Merchant, Shabbir N.
Format: Journal Article
Language:English
Published: Piscataway IEEE 01.06.2019
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithms
Computer simulation
cumulative feedback
Decision making
dynamic programming
Economic models
Fading channels
Feedback
Indexes
Markov processes
Optimization
Policies
Productivity
relay selection
Relays
restless bandits
Sequential decision making
Upper bounds
weakly coupled partially observable Markov decision processes
Whittle index
Wireless networks
ISSN:2332-7731, 2332-7731
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper studies a generalized class of restless multi-armed bandits with hidden states and allow cumulative feedback, as opposed to the conventional instantaneous feedback. We call them lazy restless bandits (LRBs) as the events of decision making are sparser than the events of state transition. Hence, feedback after each decision event is the cumulative effect of the following state transition events. The states of arms are hidden from the decision maker and rewards for actions are state dependent. The decision maker needs to choose one arm in each decision interval, such that the long-term cumulative reward is maximized. As the states are hidden, the decision maker maintains and updates its belief about them. It is shown that LRBs admit an optimal policy which has threshold structure in belief space. The Whittle-index policy for solving the LRB problem is analyzed; indexability of LRBs is shown. Further, the closed-form index expressions are provided for two sets of special cases; for more general cases, an algorithm for index computation is provided. An extensive simulation study is presented; Whittle-index, modified Whittle-index, and myopic policies are compared. The Lagrangian relaxation of the problem provides an upper bound on the optimal value function; it is used to assess the degree of sub-optimality various policies.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2332-7731
2332-7731
DOI:10.1109/TCCN.2019.2898000