Natural actor–critic algorithms

We present four new reinforcement learning algorithms based on actor–critic, natural-gradient and function-approximation ideas, and we provide their convergence proofs. Actor–critic reinforcement learning methods are online approximations to policy iteration in which the value-function parameters ar...

Full description

Saved in:

Bibliographic Details
Published in:	Automatica (Oxford) Vol. 45; no. 11; pp. 2471 - 2482
Main Authors:	Bhatnagar, Shalabh, Sutton, Richard S., Ghavamzadeh, Mohammad, Lee, Mark
Format:	Journal Article
Language:	English
Published:	Kidlington Elsevier Ltd 01.11.2009 Elsevier
Subjects:	Actor–critic reinforcement learning algorithms Algorithms Applied sciences Approximate dynamic programming Artificial intelligence Cognitive science Computer science Computer science; control theory; systems Convergence Exact sciences and technology Function approximation Learning Natural gradient Parametrization Policies Policy-gradient methods Reinforcement Temporal difference learning Temporal logic Two-timescale stochastic approximation Variance Two-timescale stochastic approximation Temporal difference learning Approximate dynamic programming Policy-gradient methods Actor–critic reinforcement learning algorithms Function approximation Natural gradient algorithms Probabilistic approach Reinforcement learning Empirical method Stochastic approximation State space method Parameterization Variance Interest Gradient descent Value function Actor-critic reinforcement learning Dynamic programming Compatibility Learning algorithm Artificial intelligence Gradient method
ISSN:	0005-1098, 1873-2836
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	We present four new reinforcement learning algorithms based on actor–critic, natural-gradient and function-approximation ideas, and we provide their convergence proofs. Actor–critic reinforcement learning methods are online approximations to policy iteration in which the value-function parameters are estimated using temporal difference learning and the policy parameters are updated by stochastic gradient descent. Methods based on policy gradients in this way are of special interest because of their compatibility with function-approximation methods, which are needed to handle large or infinite state spaces. The use of temporal difference learning in this way is of special interest because in many applications it dramatically reduces the variance of the gradient estimates. The use of the natural gradient is of interest because it can produce better conditioned parameterizations and has been shown to further reduce variance in some cases. Our results extend prior two-timescale convergence results for actor–critic methods by Konda and Tsitsiklis by using temporal difference learning in the actor and by incorporating natural gradients. Our results extend prior empirical studies of natural actor–critic methods by Peters, Vijayakumar and Schaal by providing the first convergence proofs and the first fully incremental algorithms.
Bibliography:	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23
ISSN:	0005-1098 1873-2836
DOI:	10.1016/j.automatica.2009.07.008