A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updat...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:2012 50th Annual Allerton Conference on Communication, Control, and Computing S. 400 - 405
Hauptverfasser: Lakshmanan, K., Bhatnagar, S.
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: IEEE 01.10.2012
Schlagworte:
Approximation algorithms
Constrained MDP
Function approximation
Lagrange multiplier method
Markov processes
Minimization
multi-stage stochastic shortest path problem
Q-learning with linear function approximation
reinforcement learning
Routing
Vectors
Zinc
ISBN:9781467345378, 1467345377
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.
ISBN:9781467345378
1467345377
DOI:10.1109/Allerton.2012.6483246