Hierarchical Deep Reinforcement Learning for Multi-robot Cooperation in Partially Observable Environment

Many real-world applications require multi-robot coordination in partially-observable domains such as package delivery, search, and rescue. One typical way to address partial observability is to enable information sharing among robots via dedicated communication protocols. However, designing commu-n...

Full description

Saved in:
Bibliographic Details
Published in:2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) pp. 272 - 281
Main Authors: Liang, Zhixuan, Cao, Jiannong, Lin, Wanyu, Chen, Jinlin, Xu, Huafeng
Format: Conference Proceeding
Language:English
Published: IEEE 01.12.2021
Subjects:
Information sharing
Multi-robot Cooperation
Multi-robot Reinforcement Learning
Multi-robot systems
Protocols
Reinforcement learning
Robot kinematics
Task analysis
Training
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Many real-world applications require multi-robot coordination in partially-observable domains such as package delivery, search, and rescue. One typical way to address partial observability is to enable information sharing among robots via dedicated communication protocols. However, designing commu-nication protocols is difficult due to the dynamic environments and complex interactions among robots. Existing broadcasting-based approaches are communication-inefficient, and they usually introduce redundant information that might impair the learning process and action selection. In this paper, we propose a hierar-chical reinforcement learning approach, called COM-cooperative HRL for multi-robot cooperation in a partially observable en-vironment. Specifically, COM-cooperative HRL addresses the above gaps by introducing a partner selector to learn high-level communication strategy using short-term task-execution rewards. Besides, a low-level controller is trained to select actions based on shared information and individual observation. Extensive empirical results show a faster convergence rate and higher team performance over alternative baselines. Our approach can not only improve learning efficiency but also be adaptive to large-scale multi-robot systems.
AbstractList Many real-world applications require multi-robot coordination in partially-observable domains such as package delivery, search, and rescue. One typical way to address partial observability is to enable information sharing among robots via dedicated communication protocols. However, designing commu-nication protocols is difficult due to the dynamic environments and complex interactions among robots. Existing broadcasting-based approaches are communication-inefficient, and they usually introduce redundant information that might impair the learning process and action selection. In this paper, we propose a hierar-chical reinforcement learning approach, called COM-cooperative HRL for multi-robot cooperation in a partially observable en-vironment. Specifically, COM-cooperative HRL addresses the above gaps by introducing a partner selector to learn high-level communication strategy using short-term task-execution rewards. Besides, a low-level controller is trained to select actions based on shared information and individual observation. Extensive empirical results show a faster convergence rate and higher team performance over alternative baselines. Our approach can not only improve learning efficiency but also be adaptive to large-scale multi-robot systems.
Author Chen, Jinlin
Liang, Zhixuan
Cao, Jiannong
Lin, Wanyu
Xu, Huafeng
Author_xml – sequence: 1
  givenname: Zhixuan
  surname: Liang
  fullname: Liang, Zhixuan
  email: zhixuan.liang@connect.polyu.hk
  organization: The Hong Kong Polytechnic University,Department of Computing,HongKong,China
– sequence: 2
  givenname: Jiannong
  surname: Cao
  fullname: Cao, Jiannong
  email: jiannong.cao@polyu.edu.hk
  organization: The Hong Kong Polytechnic University,Department of Computing,HongKong,China
– sequence: 3
  givenname: Wanyu
  surname: Lin
  fullname: Lin, Wanyu
  email: wan-yu.lin@polyu.edu.hk
  organization: The Hong Kong Polytechnic University,Department of Computing,HongKong,China
– sequence: 4
  givenname: Jinlin
  surname: Chen
  fullname: Chen, Jinlin
  email: csjlchen@comp.polyu.edu.hk
  organization: The Hong Kong Polytechnic University,Department of Computing,HongKong,China
– sequence: 5
  givenname: Huafeng
  surname: Xu
  fullname: Xu, Huafeng
  email: huafxu@polyu.edu.hk
  organization: The Hong Kong Polytechnic University,Department of Computing,HongKong,China
BookMark eNotkM1KAzEURiPowlafQIS8wIy5-ZmfpYzVFqZURNflzsxNG5gmJY2Fvr0juvrgwDmLb8auffDE2COIHEDUT03YrVdG1qXJpZCQCyG0vGIzKAqjoZAAt2y_dBQx9nvX48hfiI78g5y3IfZ0IJ94Sxi98zs-Ib7-HpPLYuhC4k0Ix0lNLnjuPH_HmByO44VvuhPFM3Yj8YU_uxj8b-iO3VgcT3T_v3P29br4bJZZu3lbNc9t5qRQKRsKKkGTpooqABJVZwZhVCdB90YRYldrY3VZVKh7aXEAI0QlQQ6DtTXWas4e_rqOiLbH6A4YL9vpA6FAqh8QW1aJ
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/CogMI52975.2021.00042
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 1665416211
9781665416214
EndPage 281
ExternalDocumentID 9750312
Genre orig-research
GroupedDBID 6IE
6IL
CBEJK
RIE
RIL
ID FETCH-LOGICAL-i203t-d6e714e4e8e811e08b5d053b214c53eaab945f4768a4c2fad15008212ddff9a93
IEDL.DBID RIE
ISICitedReferencesCount 6
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000835349000033&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Dec 10 09:44:45 EST 2025
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i203t-d6e714e4e8e811e08b5d053b214c53eaab945f4768a4c2fad15008212ddff9a93
PageCount 10
ParticipantIDs ieee_primary_9750312
PublicationCentury 2000
PublicationDate 2021-Dec.
PublicationDateYYYYMMDD 2021-12-01
PublicationDate_xml – month: 12
  year: 2021
  text: 2021-Dec.
PublicationDecade 2020
PublicationTitle 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)
PublicationTitleAbbrev COGMI
PublicationYear 2021
Publisher IEEE
Publisher_xml – name: IEEE
Score 1.8518332
Snippet Many real-world applications require multi-robot coordination in partially-observable domains such as package delivery, search, and rescue. One typical way to...
SourceID ieee
SourceType Publisher
StartPage 272
SubjectTerms Information sharing
Multi-robot Cooperation
Multi-robot Reinforcement Learning
Multi-robot systems
Protocols
Reinforcement learning
Robot kinematics
Task analysis
Training
Title Hierarchical Deep Reinforcement Learning for Multi-robot Cooperation in Partially Observable Environment
URI https://ieeexplore.ieee.org/document/9750312
WOSCitedRecordID wos000835349000033&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA61ePCk0opvcvBobPPYR861pR6sRRR6K8lmUheW3VK3gv_eJLu0CF68hbkEZhLm-c2H0F3EIrDuKRAqMkOEiSWRQDlJdaQ4t4mLEWwgm0hms3SxkPMOut9hYQAgDJ_Bgz-GXr6psq0vlQ2kb7p5SuGDJIkbrFYLyqFDORhVq-enyCNFXdrHaNjDyX6RpgSfMTn-320nqL8H3-H5zq2cog6UPfQxzT1SOBCXFPgRYI1fISw9zUJ9D7d7UlfYiXBA1ZJNpasaj6pqDY2ZcV7iuX8qqii-8YsOBVldAB7v0W599D4Zv42mpCVJIDkb8pqYGBIqQEAKKaUwdEo27mNp5pQfcVBKSxFZ4bIKJTJmlXERoHP7lBljrVSSn6FuWZVwjrAR1rhojFmegYgpaCNiC1rHlnNFM3mBel5Ly3WzB2PZKujyb_EVOvJmaEY_rlG33mzhBh1mX3X-ubkNxvsBKHOfiw
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LawIxEA5iC-2pLVr6bg49NtVssuvmbBWlaqVY8CbJZmIXll2xa6H_vklclEIvvYW5BGYS5vnNh9BDGIRg7FMglCeacB0JIoAyEqtQMmY6NkYwnmyiM5nE87mY1tDjDgsDAH74DJ7c0ffydZFsXKmsJVzTzVEKHzjmrAqtVcFyaFu0usVyPAwdVtQmfgH1mziDX7Qp3mv0T_533ylq7uF3eLpzLGeoBnkDfQxShxX21CUZfgZY4Tfwa08TX-HD1abUJbYi7HG1ZF2oosTdoljB1tA4zfHUPRaZZd_4VfmSrMoA9_Z4tyZ67_dm3QGpaBJIGrRZSXQEHcqBQwwxpdC2atb2a6nAqj9kIKUSPDTc5hWSJ4GR2saA1vHTQGtjhBTsHNXzIocLhDU32sZjgWEJ8IiC0jwyoFRkGJM0EZeo4bS0WG03YSwqBV39Lb5HR4PZeLQYDScv1-jYmWQ7CHKD6uV6A7foMPkq08_1nTfkD6JzotQ
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2021+IEEE+Third+International+Conference+on+Cognitive+Machine+Intelligence+%28CogMI%29&rft.atitle=Hierarchical+Deep+Reinforcement+Learning+for+Multi-robot+Cooperation+in+Partially+Observable+Environment&rft.au=Liang%2C+Zhixuan&rft.au=Cao%2C+Jiannong&rft.au=Lin%2C+Wanyu&rft.au=Chen%2C+Jinlin&rft.date=2021-12-01&rft.pub=IEEE&rft.spage=272&rft.epage=281&rft_id=info:doi/10.1109%2FCogMI52975.2021.00042&rft.externalDocID=9750312