Counterfactual value decomposition for cooperative multi-agent reinforcement learning

Value decomposition has become a central focus in Multi-Agent Reinforcement Learning (MARL) in recent years. The key challenge lies in the construction and updating of the factored value function (FVF). Traditional methods rely on FVFs with restricted representational capacity, rendering them inadeq...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Neural networks Ročník 190; s. 107692
Hlavní autoři: Liu, Kai, Zhang, Tianxian, Xu, Xiangliang, Zhao, Yuyang
Médium: Journal Article
Jazyk:angličtina
Vydáno: United States Elsevier Ltd 01.10.2025
Témata:
Algorithms
Attention mechanism
Cooperative Behavior
Counterfactual network
Humans
Multi-agent reinforcement learning
Neural Networks, Computer
Reinforcement, Psychology
Value decomposition
Multi-agent reinforcement learning
Attention mechanism
Counterfactual network
Value decomposition
ISSN:0893-6080, 1879-2782, 1879-2782
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Value decomposition has become a central focus in Multi-Agent Reinforcement Learning (MARL) in recent years. The key challenge lies in the construction and updating of the factored value function (FVF). Traditional methods rely on FVFs with restricted representational capacity, rendering them inadequate for tasks with non-monotonic payoffs. Recent approaches address this limitation by designing FVF update mechanisms that enable applicability to non-monotonic scenarios. However, these methods typically depend on the true optimal joint action value to guide FVF updates. Since the true optimal joint action is computationally infeasible in practice, these methods approximate it using the greedy joint action and update the FVF with the corresponding greedy joint action value. We observe that although the greedy joint action may be close to the true optimal joint action, its associated greedy joint action value can be substantially biased relative to the true optimal joint action value. This makes the approximation unreliable and can lead to incorrect update directions for the FVF, hindering the learning process. To overcome this limitation, we propose Comix, a novel off-policy MARL method based on a Sandwich Value Decomposition Framework. Comix constrains and guides FVF updates using both upper and lower bounds. Specifically, it leverages orthogonal best responses to construct the upper bound, thus overcoming the drawbacks introduced by the optimal approximation. Furthermore, an attention mechanism is incorporated to ensure that the upper bound can be computed with linear time complexity and high accuracy. Theoretical analyses show that Comix satisfies the IGM. Experiments on the asymmetric One-Step Matrix Game, discrete Predator-Prey, and StarCraft Multi-Agent Challenge show that Comix achieves higher learning efficiency and outperforms several state-of-the-art methods. •We propose a novel counterfactual value decomposition method.•We establish a Sandwich Value Decomposition Framework.•We develop a NN scheme to transform non-monotonic value spaces into monotonic ones.•The proposed method outperforms several state-of-the-art MARL algorithms.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:0893-6080
1879-2782
1879-2782
DOI:10.1016/j.neunet.2025.107692