Kernel-Based Least Squares Policy Iteration for Reinforcement Learning

In this paper, we present a kernel-based least squares policy iteration (KLSPI) algorithm for reinforcement learning (RL) in large or continuous state spaces, which can be used to realize adaptive feedback control of uncertain dynamic systems. By using KLSPI, near-optimal control policies can be obt...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE transactions on neural networks Ročník 18; číslo 4; s. 973 - 992
Hlavní autoři: Xu, Xin, Hu, Dewen, Lu, Xicheng
Médium: Journal Article
Jazyk:angličtina
Vydáno: United States IEEE 01.07.2007
Témata:
Adaptive control
Algorithms
Approximate dynamic programming
Artificial Intelligence
Biomimetics - methods
Computer Simulation
Convergence
Decision Support Techniques
Dynamical systems
Feedback
Feedback control
Iterative methods
Kernel
kernel methods
Kernels
Learning
least squares
Least squares approximation
Least squares method
Least squares methods
Least-Squares Analysis
Linear approximation
Markov Chains
Markov decision problems (MDPs)
Mathematical models
Models, Theoretical
Policies
Programmable control
Reinforcement (Psychology)
reinforcement learning (RL)
State-space methods
ISSN:1045-9227
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:In this paper, we present a kernel-based least squares policy iteration (KLSPI) algorithm for reinforcement learning (RL) in large or continuous state spaces, which can be used to realize adaptive feedback control of uncertain dynamic systems. By using KLSPI, near-optimal control policies can be obtained without much a priori knowledge on dynamic models of control plants. In KLSPI, Mercer kernels are used in the policy evaluation of a policy iteration process, where a new kernel-based least squares temporal-difference algorithm called KLSTD-Q is proposed for efficient policy evaluation. To keep the sparsity and improve the generalization ability of KLSTD-Q solutions, a kernel sparsification procedure based on approximate linear dependency (ALD) is performed. Compared to the previous works on approximate RL methods, KLSPI makes two progresses to eliminate the main difficulties of existing results. One is the better convergence and (near) optimality guarantee by using the KLSTD-Q algorithm for policy evaluation with high precision. The other is the automatic feature selection using the ALD-based kernel sparsification. Therefore, the KLSPI algorithm provides a general RL method with generalization performance and convergence guarantee for large-scale Markov decision problems (MDPs). Experimental results on a typical RL task for a stochastic chain problem demonstrate that KLSPI can consistently achieve better learning efficiency and policy quality than the previous least squares policy iteration (LSPI) algorithm. Furthermore, the KLSPI method was also evaluated on two nonlinear feedback control problems, including a ship heading control problem and the swing up control of a double-link underactuated pendulum called acrobot. Simulation results illustrate that the proposed method can optimize controller performance using little a priori information of uncertain dynamic systems. It is also demonstrated that KLSPI can be applied to online learning control by incorporating an initial controller to ensure online performance.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ObjectType-Article-2
ObjectType-Feature-1
ISSN:1045-9227
DOI:10.1109/TNN.2007.899161