Error controlled actor-critic

Xingen Gao, Fei Chao*, Changle Zhou, Zhen Ge, Longzhi Yang, Xiang Chang, Changjing Shang, Qiang Shen

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

The approximation inaccuracy of the value function in reinforcement learning (RL) algorithms unavoidably leads to an overestimation phenomenon, which has negative effects on the convergence of the algorithms. To limit the negative effects of the approximation error, we propose error controlled actor-critic (ECAC) which ensures the approximation error is limited within the value function. We present an investigation of how approximation inaccuracy can impair the optimization process of actor-critic approaches. In addition, we derive an upper bound for the approximation error of the Q function approximator and discover that the error can be reduced by limiting the KL- divergence between every two consecutive policies during policy training. Experiments on a variety of continuous control tasks demonstrate that the proposed actor-critic approach decreases approximation error and outperforms previous model-free RL algorithms by a significant margin.

Original languageEnglish
Pages (from-to)62-74
Number of pages13
JournalInformation Sciences
Volume612
Early online date27 Aug 2022
DOIs
Publication statusE-pub ahead of print - 27 Aug 2022

Fingerprint

Dive into the research topics of 'Error controlled actor-critic'. Together they form a unique fingerprint.

Cite this