Policy gradient in Lipschitz Markov Decision Processes

This paper is about the exploitation of Lipschitz continuity properties for Markov Decision Processes to safely speed up policy-gradient algorithms. Starting from assumptions about the Lipschitz continuity of the state-transition model, the reward function, and the policies considered in the learnin...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Machine learning Jg. 100; H. 2-3; S. 255 - 283
Hauptverfasser: Pirotta, Matteo, Restelli, Marcello, Bascetta, Luca
Format: Journal Article
Sprache:Englisch
Veröffentlicht: New York Springer US 01.09.2015
Springer Nature B.V
Schlagworte:
Algorithms
Artificial Intelligence
Computer Science
Continuity
Control
Control systems
Decision making models
Learning
Machine learning
Markov analysis
Markov processes
Mathematical models
Mechatronics
Natural Language Processing (NLP)
Policies
Robotics
Simulation and Modeling
Policy gradient algorithm
Markov Decision Process
Reinforcement learning
Lipschitz continuity
ISSN:0885-6125, 1573-0565
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:This paper is about the exploitation of Lipschitz continuity properties for Markov Decision Processes to safely speed up policy-gradient algorithms. Starting from assumptions about the Lipschitz continuity of the state-transition model, the reward function, and the policies considered in the learning process, we show that both the expected return of a policy and its gradient are Lipschitz continuous w.r.t. policy parameters. By leveraging such properties, we define policy-parameter updates that guarantee a performance improvement at each iteration. The proposed methods are empirically evaluated and compared to other related approaches using different configurations of three popular control scenarios: the linear quadratic regulator, the mass-spring-damper system and the ship-steering control.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:0885-6125
1573-0565
DOI:10.1007/s10994-015-5484-1