An Equivalence Between Adaptive Dynamic Programming With a Critic and Backpropagation Through Time

We consider the adaptive dynamic programming technique called Dual Heuristic Programming (DHP), which is designed to learn a critic function, when using learned model functions of the environment. DHP is designed for optimizing control problems in large and continuous state spaces. We extend DHP int...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transaction on neural networks and learning systems Vol. 24; no. 12; pp. 2088 - 2100
Main Authors: Fairbank, Michael, Alonso, Eduardo, Prokhorov, Danil
Format: Journal Article
Language:English
Published: New York, NY IEEE 01.12.2013
Institute of Electrical and Electronics Engineers
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Adaptive dynamic programming (ADP)
Algorithm design and analysis
Algorithms
Applied sciences
Approximation algorithms
Artificial intelligence
Back propagation
backpropagation through time
Computer science; control theory; systems
Control theory. Systems
Convergence
dual heuristic programming (DHP)
Dynamic programming
Dynamical systems
Equations
Exact sciences and technology
Learning
Learning and adaptive systems
Mathematical models
Neural networks
Optimal control
Policies
Trajectory
value-gradient learning
Vectors
Backpropagation
Gradient
dual heuristic programming (DHP)
neural networks
backpropagation through time
Neural network
value-gradient learning
Function approximation
Modeling
Backpropagation algorithm
Adaptive dynamic programming (ADP)
Smooth function
Heuristic method
Greedy algorithm
Dynamic programming
Learning algorithm
ISSN:2162-237X, 2162-2388, 2162-2388
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We consider the adaptive dynamic programming technique called Dual Heuristic Programming (DHP), which is designed to learn a critic function, when using learned model functions of the environment. DHP is designed for optimizing control problems in large and continuous state spaces. We extend DHP into a new algorithm that we call Value-Gradient Learning, VGL(λ), and prove equivalence of an instance of the new algorithm to Backpropagation Through Time for Control with a greedy policy. Not only does this equivalence provide a link between these two different approaches, but it also enables our variant of DHP to have guaranteed convergence, under certain smoothness conditions and a greedy policy, when using a general smooth nonlinear function approximator for the critic. We consider several experimental scenarios including some that prove divergence of DHP under a greedy policy, which contrasts against our proven-convergent algorithm.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ObjectType-Article-2
ObjectType-Feature-1
content type line 23
ISSN:2162-237X
2162-2388
2162-2388
DOI:10.1109/TNNLS.2013.2271778