MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models

Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials, a property known as sample efficiency. This research thus investigates the use of learned world models to improve sample efficiency. We present a novel multi-agent model-bas...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings of the ... IEEE/RSJ International Conference on Intelligent Robots and Systems pp. 5635 - 5640
Main Authors: Willemsen, Daniel, Coppola, Mario, de Croon, Guido C.H.E.
Format: Conference Proceeding
Language:English
Published: IEEE 27.09.2021
Subjects:
Intelligent robots
Multi-robot systems
Optimization
Reinforcement learning
Task analysis
Training
ISSN:2153-0866
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials, a property known as sample efficiency. This research thus investigates the use of learned world models to improve sample efficiency. We present a novel multi-agent model-based RL algorithm: Multi-Agent Model-Based Policy Optimization (MAMBPO), utilizing the Centralized Learning for Decentralized Execution (CLDE) framework. CLDE algorithms allow a group of agents to act in a fully decentralized manner after training. This is a desirable property for many systems comprising of multiple robots. MAMBPO uses a learned world model to improve sample efficiency compared to model-free Multi-Agent Soft Actor-Critic (MASAC). We demonstrate this on two simulated multi-robot tasks, where MAMBPO achieves a similar performance to MASAC, but requires far fewer samples to do so. Through this, we take an important step towards making real-life learning for multi-robot systems possible.
AbstractList Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials, a property known as sample efficiency. This research thus investigates the use of learned world models to improve sample efficiency. We present a novel multi-agent model-based RL algorithm: Multi-Agent Model-Based Policy Optimization (MAMBPO), utilizing the Centralized Learning for Decentralized Execution (CLDE) framework. CLDE algorithms allow a group of agents to act in a fully decentralized manner after training. This is a desirable property for many systems comprising of multiple robots. MAMBPO uses a learned world model to improve sample efficiency compared to model-free Multi-Agent Soft Actor-Critic (MASAC). We demonstrate this on two simulated multi-robot tasks, where MAMBPO achieves a similar performance to MASAC, but requires far fewer samples to do so. Through this, we take an important step towards making real-life learning for multi-robot systems possible.
Author de Croon, Guido C.H.E.
Willemsen, Daniel
Coppola, Mario
Author_xml – sequence: 1
  givenname: Daniel
  surname: Willemsen
  fullname: Willemsen, Daniel
  email: j.daniel.willemsen@gmail.com
  organization: Delft University of Technology,MAVLab, Control & Operations Department, Faculty of Aerospace Engineering,Delft,The Netherlands,2628 HS
– sequence: 2
  givenname: Mario
  surname: Coppola
  fullname: Coppola, Mario
  email: m.coppola@tudelft.nl
  organization: Delft University of Technology,MAVLab, Control & Operations Department, Faculty of Aerospace Engineering,Delft,The Netherlands,2628 HS
– sequence: 3
  givenname: Guido C.H.E.
  surname: de Croon
  fullname: de Croon, Guido C.H.E.
  email: g.c.h.e.decroon@tudelft.nl
  organization: Delft University of Technology,MAVLab, Control & Operations Department, Faculty of Aerospace Engineering,Delft,The Netherlands,2628 HS
BookMark eNotkNtKw0AURUdRsK39AkHyA6nnZJK5-FZL1UJLxKqvZTJzRkZyKZMU8e-l2pe9WWxYD3vMLtquJcZuEWaIoO9Wr-W2QBRqlkGGMy14obg4Y2MUoshRYl6cs1GGBU9BCXHFpn3_BQAIUistRuxjM988vJT3ydY0-5pS8j7YQO2QNId6CGnsqm5IIoXWd9FSc1xqMrEN7Wdy6I_5h-SS7y7WLmk6R3V_zS69qXuannrC3h-Xb4vndF0-rRbzdRoy4ENKYL3zTisvrefSeMy8IYvSYC50lRe596qSkBF3mrgGowmdJZKikhw4n7Cbf28got0-hsbEn93pBv4LPqVVtQ
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/IROS51168.2021.9635836
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Xplore
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISBN 1665417145
9781665417143
EISSN 2153-0866
EndPage 5640
ExternalDocumentID 9635836
Genre orig-research
GroupedDBID 6IE
6IF
6IH
6IL
6IN
AAJGR
AAWTH
ABLEC
ACGFS
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
M43
OCL
RIE
RIL
RIO
RNS
ID FETCH-LOGICAL-i203t-e0cfdfd98f7cf37af12faec17a1469b454ff8b702e3d9e390a9e1dcee76b73033
IEDL.DBID RIE
ISICitedReferencesCount 14
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000755125504071&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Tue May 06 03:33:40 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i203t-e0cfdfd98f7cf37af12faec17a1469b454ff8b702e3d9e390a9e1dcee76b73033
PageCount 6
ParticipantIDs ieee_primary_9635836
PublicationCentury 2000
PublicationDate 2021-Sept.-27
PublicationDateYYYYMMDD 2021-09-27
PublicationDate_xml – month: 09
  year: 2021
  text: 2021-Sept.-27
  day: 27
PublicationDecade 2020
PublicationTitle Proceedings of the ... IEEE/RSJ International Conference on Intelligent Robots and Systems
PublicationTitleAbbrev IROS
PublicationYear 2021
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0001079896
Score 2.375676
Snippet Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials, a property known as sample...
SourceID ieee
SourceType Publisher
StartPage 5635
SubjectTerms Intelligent robots
Multi-robot systems
Optimization
Reinforcement learning
Task analysis
Training
Title MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models
URI https://ieeexplore.ieee.org/document/9635836
WOSCitedRecordID wos000755125504071&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3JTsMwELXaigNcWFrELh844tZJnDjmBogKJLqIAuqt8jKueklQmvL9xG7UgsSFm20ptjXjTZM37yF0rQJlQXNJmE6BMBCcyJTGxChruROZFMZ7-oUPh-l0KsYNdLPJhQEADz6Driv6f_km1ysXKutViyVOo6SJmpwn61ytbTyFcpGKpE4CDqjoPb-OJtVzInEArjDo1h__UlHxl0h__3_DH6DONhsPjzf3zCFqQHaE9n4QCbbRx-BucD8e3eKJdGy_BDwxRNUb9oBBUuQqL3EBniZV-4ggrvUi5thB3-frKhjsKVSxF8hZdtB7__Ht4YnUiglkEdKoJEC1NdaI1HJtIy5tEFoJOuCyOhCFYjGzNlWchhAZAZGgUkBgqvnzRFVbPYqOUSvLMzhBmBkjAwpKWMEY40JSpwQTxrIys02sOEVtZ6HZ55oUY1Yb5-zv5nO065zggBYhv0CtsljBJdrRX-ViWVx5T34DoNChhQ
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LTwIxEG4QTdSLDzS-7cGjhe5ud7v1pkYCkVcEDTfSbaeEC2uWxd_vtmxAEy_e2iZtmpm-Mv3m-xC6S7zEgOKSMBUDYSA4kTENiU6M4VZkUmjn6Q7v9eLxWAwq6H6dCwMADnwGdVt0f_k6VUsbKmsUiyWMg2gLbYeM-XSVrbWJqFAuYhGVacAeFY32W39YPCgiC-HyvXrZ_ZeOirtGmgf_m8AhOtnk4-HB-qY5QhWYH6P9H1SCNfTRfew-DfoPeCgt3y8BRw1RjIYdZJBkaZLmOANHlKpcTBCXihFTbMHv01UVNHYkqthJ5CxO0HvzZfTcIqVmApn5NMgJUGW00SI2XJmAS-P5RoLyuCyORJGwkBkTJ5z6EGgBgaBSgKeL-fMoKTZ7EJyi6jydwxnCTGvpUUiEEYwxLiS1WjB-KAszm8iIc1SzFpp8rmgxJqVxLv5uvkW7rVG3M-m0e6-XaM86xMIufH6Fqnm2hGu0o77y2SK7cV79BjUfpMw
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+of+the+...+IEEE%2FRSJ+International+Conference+on+Intelligent+Robots+and+Systems&rft.atitle=MAMBPO%3A+Sample-efficient+multi-robot+reinforcement+learning+using+learned+world+models&rft.au=Willemsen%2C+Daniel&rft.au=Coppola%2C+Mario&rft.au=de+Croon%2C+Guido+C.H.E.&rft.date=2021-09-27&rft.pub=IEEE&rft.eissn=2153-0866&rft.spage=5635&rft.epage=5640&rft_id=info:doi/10.1109%2FIROS51168.2021.9635836&rft.externalDocID=9635836