An Online Actor–Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

We develop an online actor–critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable po...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Journal of optimization theory and applications Ročník 153; číslo 3; s. 688 - 708
Hlavní autoři: Bhatnagar, Shalabh, Lakshmanan, K.
Médium: Journal Article
Jazyk:angličtina
Vydáno: Boston Springer US 01.06.2012
Springer Nature B.V
Témata:
Algorithms
Applications of Mathematics
Approximation
Asymptotic properties
Calculus of Variations and Optimal Control; Optimization
Costs
Engineering
Lagrange multiplier
Markov analysis
Markov processes
Mathematical analysis
Mathematical models
Mathematics
Mathematics and Statistics
On-line systems
Online
Operations Research/Decision Theory
Optimization
Queues
Random variables
Theory of Computation
Constrained Markov decision processes
Function approximation
Actor–critic algorithm
Long-run average cost criterion
ISSN:0022-3239, 1573-2878
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:We develop an online actor–critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable policy-dependent long-run averages of certain sample path functions. The Lagrange multiplier method is used to handle the inequality constraints. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal solution. We also provide the results of numerical experiments on a problem of routing in a multi-stage queueing network with constraints on long-run average queue lengths. We observe that our algorithm exhibits good performance on this setting and converges to a feasible point.
Bibliografie:SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ObjectType-Article-2
content type line 23
ISSN:0022-3239
1573-2878
DOI:10.1007/s10957-012-9989-5