Deterministic policy gradient algorithms for semi‐Markov decision processes

A large class of sequential decision‐making problems under uncertainty, with broad applications from preventive maintenance to event‐triggered control can be modeled in the framework of semi‐Markov decision processes (SMDPs). Unlike Markov decision processes (MDPs), SMDPs are underexplored in the on...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:International journal of intelligent systems Jg. 37; H. 7; S. 4008 - 4019
Hauptverfasser: Hosseinloo, Ashkan Haji, Dahleh, Munther A.
Format: Journal Article
Sprache:Englisch
Veröffentlicht: New York John Wiley & Sons, Inc 01.07.2022
Schlagworte:
Algorithms
average reward
deterministic policy
Intelligent systems
Markov analysis
Markov processes
policy gradient theorem
Preventive maintenance
reinforcement learning
SMDP
Theorems
ISSN:0884-8173, 1098-111X
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:A large class of sequential decision‐making problems under uncertainty, with broad applications from preventive maintenance to event‐triggered control can be modeled in the framework of semi‐Markov decision processes (SMDPs). Unlike Markov decision processes (MDPs), SMDPs are underexplored in the online and reinforcement learning (RL) settings. In this paper, we extend the well‐known deterministic policy gradient (DPG) theorem in MDPs to SMDPs under average‐reward criterion. The existing stochastic policy gradient methods not only require, in general, a large number of samples for training, but they also suffer from high variance in the gradient estimation when applied to problems with deterministic optimal policy. Our DPG method can potentially remedy these issues. On the basis of this method and depending on the choice of a critic, different actor–critic algorithms can easily be developed in the RL setup. We present two example actor–critic algorithms. Both algorithms employ our developed policy gradient theorem for their actors, but use two different critics; one uses a simple SARSA update while the other one uses the same on‐policy update but with compatible function approximators. We demonstrate the efficacy of our method both mathematically and via simulations.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0884-8173
1098-111X
DOI:10.1002/int.22709