Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors

In reinforcement learning (RL), function approximation errors are known to easily lead to the <inline-formula> <tex-math notation="LaTeX">Q </tex-math></inline-formula>-value overestimations, thus greatly reducing policy performance. This article presents a distribu...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transaction on neural networks and learning systems Vol. 33; no. 11; pp. 6584 - 6598
Main Authors: Duan, Jingliang, Guan, Yang, Li, Shengbo Eben, Ren, Yangang, Sun, Qi, Cheng, Bo
Format: Journal Article
Language:English
Published: Piscataway IEEE 01.11.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithms
Approximation algorithms
Artificial neural networks
Control tasks
Distribution functions
Distributional soft actor–critic (DSAC)
Embedding
Entropy
Errors
Estimation
Iterative methods
Learning
Maximum entropy
overestimation
Probability distribution
Reinforcement
Reinforcement learning
reinforcement learning (RL)
ISSN:2162-237X, 2162-2388, 2162-2388
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In reinforcement learning (RL), function approximation errors are known to easily lead to the <inline-formula> <tex-math notation="LaTeX">Q </tex-math></inline-formula>-value overestimations, thus greatly reducing policy performance. This article presents a distributional soft actor-critic (DSAC) algorithm, which is an off-policy RL method for continuous control setting, to improve the policy performance by mitigating <inline-formula> <tex-math notation="LaTeX">Q </tex-math></inline-formula>-value overestimations. We first discover in theory that learning a distribution function of state-action returns can effectively mitigate <inline-formula> <tex-math notation="LaTeX">Q </tex-math></inline-formula>-value overestimations because it is capable of adaptively adjusting the update step size of the <inline-formula> <tex-math notation="LaTeX">Q </tex-math></inline-formula>-value function. Then, a distributional soft policy iteration (DSPI) framework is developed by embedding the return distribution function into maximum entropy RL. Finally, we present a deep off-policy actor-critic variant of DSPI, called DSAC, which directly learns a continuous return distribution by keeping the variance of the state-action returns within a reasonable range to address exploding and vanishing gradient problems. We evaluate DSAC on the suite of MuJoCo continuous control tasks, achieving the state-of-the-art performance.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:2162-237X
2162-2388
2162-2388
DOI:10.1109/TNNLS.2021.3082568