MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models
Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials, a property known as sample efficiency. This research thus investigates the use of learned world models to improve sample efficiency. We present a novel multi-agent model-bas...
Saved in:
| Published in: | Proceedings of the ... IEEE/RSJ International Conference on Intelligent Robots and Systems pp. 5635 - 5640 |
|---|---|
| Main Authors: | , , |
| Format: | Conference Proceeding |
| Language: | English |
| Published: |
IEEE
27.09.2021
|
| Subjects: | |
| ISSN: | 2153-0866 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials, a property known as sample efficiency. This research thus investigates the use of learned world models to improve sample efficiency. We present a novel multi-agent model-based RL algorithm: Multi-Agent Model-Based Policy Optimization (MAMBPO), utilizing the Centralized Learning for Decentralized Execution (CLDE) framework. CLDE algorithms allow a group of agents to act in a fully decentralized manner after training. This is a desirable property for many systems comprising of multiple robots. MAMBPO uses a learned world model to improve sample efficiency compared to model-free Multi-Agent Soft Actor-Critic (MASAC). We demonstrate this on two simulated multi-robot tasks, where MAMBPO achieves a similar performance to MASAC, but requires far fewer samples to do so. Through this, we take an important step towards making real-life learning for multi-robot systems possible. |
|---|---|
| AbstractList | Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials, a property known as sample efficiency. This research thus investigates the use of learned world models to improve sample efficiency. We present a novel multi-agent model-based RL algorithm: Multi-Agent Model-Based Policy Optimization (MAMBPO), utilizing the Centralized Learning for Decentralized Execution (CLDE) framework. CLDE algorithms allow a group of agents to act in a fully decentralized manner after training. This is a desirable property for many systems comprising of multiple robots. MAMBPO uses a learned world model to improve sample efficiency compared to model-free Multi-Agent Soft Actor-Critic (MASAC). We demonstrate this on two simulated multi-robot tasks, where MAMBPO achieves a similar performance to MASAC, but requires far fewer samples to do so. Through this, we take an important step towards making real-life learning for multi-robot systems possible. |
| Author | de Croon, Guido C.H.E. Willemsen, Daniel Coppola, Mario |
| Author_xml | – sequence: 1 givenname: Daniel surname: Willemsen fullname: Willemsen, Daniel email: j.daniel.willemsen@gmail.com organization: Delft University of Technology,MAVLab, Control & Operations Department, Faculty of Aerospace Engineering,Delft,The Netherlands,2628 HS – sequence: 2 givenname: Mario surname: Coppola fullname: Coppola, Mario email: m.coppola@tudelft.nl organization: Delft University of Technology,MAVLab, Control & Operations Department, Faculty of Aerospace Engineering,Delft,The Netherlands,2628 HS – sequence: 3 givenname: Guido C.H.E. surname: de Croon fullname: de Croon, Guido C.H.E. email: g.c.h.e.decroon@tudelft.nl organization: Delft University of Technology,MAVLab, Control & Operations Department, Faculty of Aerospace Engineering,Delft,The Netherlands,2628 HS |
| BookMark | eNotkNtKw0AURUdRsK39AkHyA6nnZJK5-FZL1UJLxKqvZTJzRkZyKZMU8e-l2pe9WWxYD3vMLtquJcZuEWaIoO9Wr-W2QBRqlkGGMy14obg4Y2MUoshRYl6cs1GGBU9BCXHFpn3_BQAIUistRuxjM988vJT3ydY0-5pS8j7YQO2QNId6CGnsqm5IIoXWd9FSc1xqMrEN7Wdy6I_5h-SS7y7WLmk6R3V_zS69qXuannrC3h-Xb4vndF0-rRbzdRoy4ENKYL3zTisvrefSeMy8IYvSYC50lRe596qSkBF3mrgGowmdJZKikhw4n7Cbf28got0-hsbEn93pBv4LPqVVtQ |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/IROS51168.2021.9635836 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISBN | 1665417145 9781665417143 |
| EISSN | 2153-0866 |
| EndPage | 5640 |
| ExternalDocumentID | 9635836 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IF 6IH 6IL 6IN AAJGR AAWTH ABLEC ACGFS ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP M43 OCL RIE RIL RIO RNS |
| ID | FETCH-LOGICAL-i203t-e0cfdfd98f7cf37af12faec17a1469b454ff8b702e3d9e390a9e1dcee76b73033 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 14 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000755125504071&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Tue May 06 03:33:40 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i203t-e0cfdfd98f7cf37af12faec17a1469b454ff8b702e3d9e390a9e1dcee76b73033 |
| PageCount | 6 |
| ParticipantIDs | ieee_primary_9635836 |
| PublicationCentury | 2000 |
| PublicationDate | 2021-Sept.-27 |
| PublicationDateYYYYMMDD | 2021-09-27 |
| PublicationDate_xml | – month: 09 year: 2021 text: 2021-Sept.-27 day: 27 |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings of the ... IEEE/RSJ International Conference on Intelligent Robots and Systems |
| PublicationTitleAbbrev | IROS |
| PublicationYear | 2021 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0001079896 |
| Score | 2.3757644 |
| Snippet | Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials, a property known as sample... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 5635 |
| SubjectTerms | Intelligent robots Multi-robot systems Optimization Reinforcement learning Task analysis Training |
| Title | MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models |
| URI | https://ieeexplore.ieee.org/document/9635836 |
| WOSCitedRecordID | wos000755125504071&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1JTwIxFG6AeNCLCxj39ODRQmc60069qZFoIktEDTfS5ZVwATMM_n6nZQKaePHWNumS97q-fu97CF0bDjzh3BBRTheSWL_mpLaEO6kh1dxqCDyzL6Lfz8ZjOayhm40vDAAE8Bm0fTL85duFWXlTWaecLGnGeB3VheBrX62tPYUKmUleOQFHVHaeXwej8jrBPYArjtpV5V9RVMIh0t3_X_cHqLX1xsPDzTlziGowP0J7P4gEm-ijd9e7Hw5u8Uh5tl8CgRiibA0HwCDJF3pR4BwCTaoJFkFcxYuYYg99n66zYHGgUMUhQM6yhd67j28PT6SKmEBmMWUFAWqcdVZmThjHhHJR7BSYSKhyQ5Q6SRPnMi1oDMxKYJIqCZEtxy-4Lpc6Y8eoMV_M4QRhy7hV3Dln0vLFKLlmQAUYJ1XqFBXqFDW9hCafa1KMSSWcs7-Lz9GuV4IHWsTiAjWKfAWXaMd8FbNlfhU0-Q0jUaI9 |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LTwIxEG4QTdSLDzS-7cGjC93tbrv1pkYCkVcEDTfSx5RwAbOAv99t2YAmXry1TdpMZtpOM_3mG4TuNAMWM6YDnm-XIDbuzAllAmaFgkQxo8DzzLZ4p5MOh6JXQvfrXBgA8OAzqLqm_8s3M710obJavlmSlLIttJ3EcURW2VqbiArhIhWsSAMOiag137r9_EHBHIQrCqvF9F91VLwbqR_8T4BDdLLJx8O9tac5QiWYHqP9H1SCFfTRfmw_9boPuC8d328AnhoiXw17yGCQzdRsgTPwRKnaxwRxUTFijB34fbzqgsGeRBX7EjnzE_Refxk8N4KiZkIwiQhdBEC0NdaI1HJtKZc2jKwEHXKZX4lCxUlsbao4iYAaAVQQKSA0ufycqfywU3qKytPZFM4QNpQZyay1Otc2E0xRIBy0FTKxknB5jipOQ6PPFS3GqFDOxd_Dt2i3MWi3Rq1m5_US7TmDONhFxK9QeZEt4Rrt6K_FZJ7deKt-A79rpYQ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+of+the+...+IEEE%2FRSJ+International+Conference+on+Intelligent+Robots+and+Systems&rft.atitle=MAMBPO%3A+Sample-efficient+multi-robot+reinforcement+learning+using+learned+world+models&rft.au=Willemsen%2C+Daniel&rft.au=Coppola%2C+Mario&rft.au=de+Croon%2C+Guido+C.H.E.&rft.date=2021-09-27&rft.pub=IEEE&rft.eissn=2153-0866&rft.spage=5635&rft.epage=5640&rft_id=info:doi/10.1109%2FIROS51168.2021.9635836&rft.externalDocID=9635836 |