Relative Entropy of Correct Proximal Policy Optimization Algorithms with Modified Penalty Factor in Complex Environment

In the field of reinforcement learning, we propose a Correct Proximal Policy Optimization (CPPO) algorithm based on the modified penalty factor β and relative entropy in order to solve the robustness and stationarity of traditional algorithms. Firstly, In the process of reinforcement learning, this...

Celý popis

Uloženo v:

Podrobná bibliografie
Vydáno v:	Entropy (Basel, Switzerland) Ročník 24; číslo 4; s. 440
Hlavní autoři:	Chen, Weimin, Wong, Kelvin Kian Loong, Long, Sifan, Sun, Zhili
Médium:	Journal Article
Jazyk:	angličtina
Vydáno:	Switzerland MDPI AG 22.03.2022 MDPI
Témata:	Algorithms Approximation approximation theory Complexity Convergence correct proximal policy optimization Deep learning Distribution functions Entropy Information theory Iterative methods Kernel functions Machine learning Mathematical models Neural networks Optimization Optimization algorithms policy gradient Random variables reinforcement learning Research methodology Teaching methods approximation theory correct proximal policy optimization entropy optimization policy gradient reinforcement learning
ISSN:	1099-4300, 1099-4300
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Popis
Shrnutí:	In the field of reinforcement learning, we propose a Correct Proximal Policy Optimization (CPPO) algorithm based on the modified penalty factor β and relative entropy in order to solve the robustness and stationarity of traditional algorithms. Firstly, In the process of reinforcement learning, this paper establishes a strategy evaluation mechanism through the policy distribution function. Secondly, the state space function is quantified by introducing entropy, whereby the approximation policy is used to approximate the real policy distribution, and the kernel function estimation and calculation of relative entropy is used to fit the reward function based on complex problem. Finally, through the comparative analysis on the classic test cases, we demonstrated that our proposed algorithm is effective, has a faster convergence speed and better performance than the traditional PPO algorithm, and the measure of the relative entropy can show the differences. In addition, it can more efficiently use the information of complex environment to learn policies. At the same time, not only can our paper explain the rationality of the policy distribution theory, the proposed framework can also balance between iteration steps, computational complexity and convergence speed, and we also introduced an effective measure of performance using the relative entropy concept.
Bibliografie:	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 content type line 23
ISSN:	1099-4300 1099-4300
DOI:	10.3390/e24040440