Relative Entropy of Correct Proximal Policy Optimization Algorithms with Modified Penalty Factor in Complex Environment

In the field of reinforcement learning, we propose a Correct Proximal Policy Optimization (CPPO) algorithm based on the modified penalty factor β and relative entropy in order to solve the robustness and stationarity of traditional algorithms. Firstly, In the process of reinforcement learning, this...

Full description

Saved in:
Bibliographic Details
Published in:Entropy (Basel, Switzerland) Vol. 24; no. 4; p. 440
Main Authors: Chen, Weimin, Wong, Kelvin Kian Loong, Long, Sifan, Sun, Zhili
Format: Journal Article
Language:English
Published: Switzerland MDPI AG 22.03.2022
MDPI
Subjects:
Algorithms
Approximation
approximation theory
Complexity
Convergence
correct proximal policy optimization
Deep learning
Distribution functions
Entropy
Information theory
Iterative methods
Kernel functions
Machine learning
Mathematical models
Neural networks
Optimization
Optimization algorithms
policy gradient
Random variables
reinforcement learning
Research methodology
Teaching methods
approximation theory
correct proximal policy optimization
entropy
optimization
policy gradient
reinforcement learning
ISSN:1099-4300, 1099-4300
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In the field of reinforcement learning, we propose a Correct Proximal Policy Optimization (CPPO) algorithm based on the modified penalty factor β and relative entropy in order to solve the robustness and stationarity of traditional algorithms. Firstly, In the process of reinforcement learning, this paper establishes a strategy evaluation mechanism through the policy distribution function. Secondly, the state space function is quantified by introducing entropy, whereby the approximation policy is used to approximate the real policy distribution, and the kernel function estimation and calculation of relative entropy is used to fit the reward function based on complex problem. Finally, through the comparative analysis on the classic test cases, we demonstrated that our proposed algorithm is effective, has a faster convergence speed and better performance than the traditional PPO algorithm, and the measure of the relative entropy can show the differences. In addition, it can more efficiently use the information of complex environment to learn policies. At the same time, not only can our paper explain the rationality of the policy distribution theory, the proposed framework can also balance between iteration steps, computational complexity and convergence speed, and we also introduced an effective measure of performance using the relative entropy concept.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:1099-4300
1099-4300
DOI:10.3390/e24040440