A Strongly Polynomial Algorithm for Controlled Queues

We consider the problem of computing optimal policies of finite-state finite-action Markov decision processes (MDPs). A reduction to a continuum of constrained MDPs (CMDPs) is presented such that the optimal policies for these CMDPs constitute a path in a graph defined over the deterministic policie...

Full description

Saved in:
Bibliographic Details
Published in:Mathematics of operations research Vol. 34; no. 4; pp. 992 - 1007
Main Authors: Zadorojniy, Alexander, Even, Guy, Shwartz, Adam
Format: Journal Article
Language:English
Published: Linthicum INFORMS 01.11.2009
Institute for Operations Research and the Management Sciences
Subjects:
1 queue
Algorithms
Analysis
Average cost
constrained Markov decision process
controlled queues
Determinism
Linear programming
M/M
Markov analysis
Markov decision process
Markov processes
Mathematical vectors
Matrices
Optimal policy
optimization
Polynomials
Queuing theory
Studies
Uniqueness
United States
ISSN:0364-765X, 1526-5471
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We consider the problem of computing optimal policies of finite-state finite-action Markov decision processes (MDPs). A reduction to a continuum of constrained MDPs (CMDPs) is presented such that the optimal policies for these CMDPs constitute a path in a graph defined over the deterministic policies. This path contains, in particular, an optimal policy of the original MDP. We present an algorithm based on this new approach that finds this path, and thus an optimal policy. In the general case, this path might be exponentially long in the number of states and actions. We prove that the length of this path is polynomial if the MDP satisfies a coupling property. Thus we obtain a strongly polynomial algorithm for MDP s that satisfies the coupling property. We prove that discrete time versions of controlled M/M /1 queues induce MDP s that satisfy the coupling property. The only previously known polynomial algorithm for controlled M/M /1 queues in the expected average cost model is based on linear programming (and is not known to be strongly polynomial). Our algorithm works both for the discounted and expected average cost models, and the running time does not depend on the discount factor.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0364-765X
1526-5471
DOI:10.1287/moor.1090.0415