MEC-A Near-Optimal Online Reinforcement Learning Algorithm for Continuous Deterministic Systems

In this paper, the first probably approximately correct (PAC) algorithm for continuous deterministic systems without relying on any system dynamics is proposed. It combines the state aggregation technique and the efficient exploration principle, and makes high utilization of online observed samples....

Full description

Saved in:
Bibliographic Details
Published in:IEEE transaction on neural networks and learning systems Vol. 26; no. 2; pp. 346 - 356
Main Authors: Zhao, Dongbin, Zhu, Yuanheng
Format: Journal Article
Language:English
Published: United States IEEE 01.02.2015
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithm design and analysis
Algorithms
Approximation algorithms
Computer simulation
Dynamical systems
Dynamics
Efficient exploration
Exploration
Heuristic algorithms
Learning
Learning systems
Mathematical analysis
Neural networks
Partitioning algorithms
Policies
Polynomials
probably approximately correct (PAC)
reinforcement learning (RL)
state aggregation
Upper bound
Efficient exploration
probably approximately correct (PAC)
state aggregation
reinforcement learning (RL)
ISSN:2162-237X, 2162-2388, 2162-2388
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, the first probably approximately correct (PAC) algorithm for continuous deterministic systems without relying on any system dynamics is proposed. It combines the state aggregation technique and the efficient exploration principle, and makes high utilization of online observed samples. We use a grid to partition the continuous state space into different cells to save samples. A near-upper Q operator is defined to produce a near-upper Q function using samples in each cell. The corresponding greedy policy effectively balances between exploration and exploitation. With the rigorous analysis, we prove that there is a polynomial time bound of executing nonoptimal actions in our algorithm. After finite steps, the final policy reaches near optimal in the framework of PAC. The implementation requires no knowledge of systems and has less computation complexity. Simulation studies confirm that it is a better performance than other similar PAC algorithms.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:2162-237X
2162-2388
2162-2388
DOI:10.1109/TNNLS.2014.2371046