An Online Actor–Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

We develop an online actor–critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable po...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Journal of optimization theory and applications Jg. 153; H. 3; S. 688 - 708
Hauptverfasser: Bhatnagar, Shalabh, Lakshmanan, K.
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Boston Springer US 01.06.2012
Springer Nature B.V
Schlagworte:
Algorithms
Applications of Mathematics
Approximation
Asymptotic properties
Calculus of Variations and Optimal Control; Optimization
Costs
Engineering
Lagrange multiplier
Markov analysis
Markov processes
Mathematical analysis
Mathematical models
Mathematics
Mathematics and Statistics
On-line systems
Online
Operations Research/Decision Theory
Optimization
Queues
Random variables
Theory of Computation
Constrained Markov decision processes
Function approximation
Actor–critic algorithm
Long-run average cost criterion
ISSN:0022-3239, 1573-2878
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:We develop an online actor–critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable policy-dependent long-run averages of certain sample path functions. The Lagrange multiplier method is used to handle the inequality constraints. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal solution. We also provide the results of numerical experiments on a problem of routing in a multi-stage queueing network with constraints on long-run average queue lengths. We observe that our algorithm exhibits good performance on this setting and converges to a feasible point.
Bibliographie:SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ObjectType-Article-2
content type line 23
ISSN:0022-3239
1573-2878
DOI:10.1007/s10957-012-9989-5