A Universal Empirical Dynamic Programming Algorithm for Continuous State MDPs

We propose universal randomized function approximation-based empirical value learning (EVL) algorithms for Markov decision processes. The "empirical" nature comes from each iteration being done empirically from samples available from simulations of the next state. This makes the Bellman op...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE transactions on automatic control Ročník 65; číslo 1; s. 115 - 129
Hlavní autoři: Haskell, William B., Jain, Rahul, Sharma, Hiteshi, Yu, Pengqian
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York IEEE 01.01.2020
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Témata:
Algorithms
Approximation
Approximation algorithms
Basis functions
Complexity theory
Computer simulation
Continuous state-space Markov decision processes (MDPs)
Convergence
Dynamic programming
dynamic programming (DP)
Empirical analysis
Function approximation
Function space
Heuristic algorithms
Hilbert space
Iterative methods
Machine learning
Markov processes
Mathematical analysis
Operators (mathematics)
Probabilistic logic
reinforcement learning (RL)
ISSN:0018-9286, 1558-2523
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:We propose universal randomized function approximation-based empirical value learning (EVL) algorithms for Markov decision processes. The "empirical" nature comes from each iteration being done empirically from samples available from simulations of the next state. This makes the Bellman operator a random operator. A parametric and a nonparametric method for function approximation using a parametric function space and a reproducing kernel Hilbert space respectively are then combined with EVL. Both function spaces have the universal function approximation property. Basis functions are picked randomly. Convergence analysis is performed using a random operator framework with techniques from the theory of stochastic dominance. Finite time sample complexity bounds are derived for both universal approximate dynamic programming algorithms. Numerical experiments support the versatility and computational tractability of this approach.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0018-9286
1558-2523
DOI:10.1109/TAC.2019.2907414