Fully asynchronous policy evaluation in distributed reinforcement learning over networks

This paper proposes a fully asynchronous scheme for the policy evaluation problem of distributed reinforcement learning (DisRL) over directed peer-to-peer networks. Without waiting for any other node of the network, each node can locally update its value function at any time using (possibly delayed)...

Celý popis

Uloženo v:

Podrobná bibliografie
Vydáno v:	Automatica (Oxford) Ročník 136; s. 110092
Hlavní autoři:	Sha, Xingyu, Zhang, Jiaqi, You, Keyou, Zhang, Kaiqing, Başar, Tamer
Médium:	Journal Article
Jazyk:	angličtina
Vydáno:	Elsevier Ltd 01.02.2022
Témata:	Distributed reinforcement learning Fully asynchronous updates Multi-agent networks Policy evaluation Policy evaluation Multi-agent networks Distributed reinforcement learning Fully asynchronous updates
ISSN:	0005-1098, 1873-2836
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Popis
Shrnutí:	This paper proposes a fully asynchronous scheme for the policy evaluation problem of distributed reinforcement learning (DisRL) over directed peer-to-peer networks. Without waiting for any other node of the network, each node can locally update its value function at any time using (possibly delayed) information from its neighbors. This is in sharp contrast to the gossip-based scheme where a pair of nodes concurrently update. Even though the fully asynchronous setting involves a difficult multi-timescale decision problem, we design a novel incremental aggregated gradient (IAG) based distributed algorithm and develop a push–pull augmented graph approach to prove its exact convergence at a linear rate of O(ck) where c∈(0,1) and k is the total number of updates within the entire network. Finally, numerical experiments validate that our method speeds up linearly with respect to the number of nodes, and is robust to straggler nodes.
ISSN:	0005-1098 1873-2836
DOI:	10.1016/j.automatica.2021.110092