DeepGait: Planning and Control of Quadrupedal Gaits Using Deep Reinforcement Learning

This letter addresses the problem of legged locomotion in non-flat terrain. As legged robots such as quadrupeds are to be deployed in terrains with geometries which are difficult to model and predict, the need arises to equip them with the capability to generalize well to unforeseen situations. In t...

Ausführliche Beschreibung

Gespeichert in:

Bibliographische Detailangaben
Veröffentlicht in:	IEEE robotics and automation letters Jg. 5; H. 2; S. 3699 - 3706
Hauptverfasser:	Tsounis, Vassilios, Alge, Mitja, Lee, Joonho, Farshidian, Farbod, Hutter, Marco
Format:	Journal Article
Sprache:	Englisch
Veröffentlicht:	Piscataway IEEE 01.04.2020 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Schlagworte:	Computer simulation Deep learning deep learning in robotics and automation Legged locomotion Legged robots Locomotion Machine learning Markov processes motion and path planning Motion planning Optimization Path planning Physical simulation Policies Robot dynamics Terrain
ISSN:	2377-3766, 2377-3766
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

Abstract	This letter addresses the problem of legged locomotion in non-flat terrain. As legged robots such as quadrupeds are to be deployed in terrains with geometries which are difficult to model and predict, the need arises to equip them with the capability to generalize well to unforeseen situations. In this work, we propose a novel technique for training neural-network policies for terrain-aware locomotion, which combines state-of-the-art methods for model-based motion planning and reinforcement learning. Our approach is centered on formulating Markov decision processes using the evaluation of dynamic feasibility criteria in place of physical simulation. We thus employ policy-gradient methods to independently train policies which respectively plan and execute foothold and base motions in 3D environments using both proprioceptive and exteroceptive measurements. We apply our method within a challenging suite of simulated terrain scenarios which contain features such as narrow bridges, gaps and stepping-stones, and train policies which succeed in locomoting effectively in all cases.
AbstractList	This letter addresses the problem of legged locomotion in non-flat terrain. As legged robots such as quadrupeds are to be deployed in terrains with geometries which are difficult to model and predict, the need arises to equip them with the capability to generalize well to unforeseen situations. In this work, we propose a novel technique for training neural-network policies for terrain-aware locomotion, which combines state-of-the-art methods for model-based motion planning and reinforcement learning. Our approach is centered on formulating Markov decision processes using the evaluation of dynamic feasibility criteria in place of physical simulation. We thus employ policy-gradient methods to independently train policies which respectively plan and execute foothold and base motions in 3D environments using both proprioceptive and exteroceptive measurements. We apply our method within a challenging suite of simulated terrain scenarios which contain features such as narrow bridges, gaps and stepping-stones, and train policies which succeed in locomoting effectively in all cases.
Author	Alge, Mitja Tsounis, Vassilios Lee, Joonho Hutter, Marco Farshidian, Farbod
Author_xml	– sequence: 1 givenname: Vassilios orcidid: 0000-0003-3428-8455 surname: Tsounis fullname: Tsounis, Vassilios email: tsounisv@ethz.ch organization: Robotic Systems Lab, ETH Zürich, Zürich, Switzerland – sequence: 2 givenname: Mitja surname: Alge fullname: Alge, Mitja email: algem@ethz.ch organization: Robotic Systems Lab, ETH Zürich, Zürich, Switzerland – sequence: 3 givenname: Joonho orcidid: 0000-0002-5072-7385 surname: Lee fullname: Lee, Joonho email: jolee@ethz.ch organization: Robotic Systems Lab, ETH Zürich, Zürich, Switzerland – sequence: 4 givenname: Farbod orcidid: 0000-0001-8269-6272 surname: Farshidian fullname: Farshidian, Farbod email: farshidian@mavt.ethz.ch organization: Robotic Systems Lab, ETH Zürich, Zürich, Switzerland – sequence: 5 givenname: Marco orcidid: 0000-0002-4285-4990 surname: Hutter fullname: Hutter, Marco email: mahutter@ethz.ch organization: Robotic Systems Lab, ETH Zürich, Zürich, Switzerland
BookMark	eNp9kMFLwzAUh4MoOOfugpeA582XpE0Tb2PqFArqcOeQtq-S0aU1bQ_-97ZsiHjw9B6P3_d78F2QU197JOSKwYIx0LfpZrngwGHBdaKlhBMy4SJJ5iKR8vTXfk5mbbsDABbzROh4Qrb3iM3auu6OvlbWe-c_qPUFXdW-C3VF65K-9bYIfYOFreiYbOm2HWMjSTfofFmHHPfoO5qiDWPFJTkrbdXi7DinZPv48L56mqcv6-fVMp3nQutuLmWsY2Aolc5FZqOkzFQsIeLcguW5UjKLZF4AyIzZ4R7Zoix4hiLPWIJoxZTcHHqbUH_22HZmV_fBDy8NF0pCLBSPhhQcUnmo2zZgaZrg9jZ8GQZm9GcGf2b0Z47-BkT-QXLX2c6NVqyr_gOvD6BDxJ8_GrhiSolv6px-CA
CODEN	IRALC6
CitedBy_id	crossref_primary_10_1002_rob_22575 crossref_primary_10_1177_02783649221102473 crossref_primary_10_1109_LRA_2022_3188108 crossref_primary_10_1109_TAI_2023_3316637 crossref_primary_10_3390_electronics13010116 crossref_primary_10_1109_TRO_2021_3106832 crossref_primary_10_3390_machines12120902 crossref_primary_10_1016_j_conengprac_2024_105879 crossref_primary_10_1177_02783649241312698 crossref_primary_10_1177_01423312231152936 crossref_primary_10_7746_jkros_2022_17_4_500 crossref_primary_10_1109_LRA_2021_3133610 crossref_primary_10_1109_TII_2023_3240758 crossref_primary_10_1038_s42256_022_00576_3 crossref_primary_10_1109_LRA_2022_3143567 crossref_primary_10_3390_biomimetics9100592 crossref_primary_10_1038_s41467_025_60982_0 crossref_primary_10_1016_j_robot_2023_104468 crossref_primary_10_3390_molecules28052232 crossref_primary_10_1109_ACCESS_2025_3590587 crossref_primary_10_3390_electronics14071431 crossref_primary_10_1007_s12043_023_02612_2 crossref_primary_10_1109_TRO_2022_3172469 crossref_primary_10_1088_1748_3190_adf385 crossref_primary_10_1109_LRA_2021_3062342 crossref_primary_10_1109_LRA_2021_3088797 crossref_primary_10_1126_scirobotics_abk2822 crossref_primary_10_1002_aisy_202300172 crossref_primary_10_1049_csy2_12020 crossref_primary_10_1007_s42235_022_00269_y crossref_primary_10_1126_scirobotics_adh5401 crossref_primary_10_1061_JAEEEZ_ASENG_5198 crossref_primary_10_3390_app131911045 crossref_primary_10_3390_s21175907 crossref_primary_10_1109_TNNLS_2021_3112718 crossref_primary_10_1109_LRA_2025_3595037 crossref_primary_10_1109_TRO_2021_3084374 crossref_primary_10_1109_TASE_2024_3503277 crossref_primary_10_1109_ACCESS_2024_3425837 crossref_primary_10_3233_ICA_230724 crossref_primary_10_1007_s00034_021_01675_z crossref_primary_10_1016_j_neunet_2021_09_017 crossref_primary_10_1109_LRA_2021_3066833 crossref_primary_10_1126_scirobotics_adi9641 crossref_primary_10_3389_frobt_2022_793512 crossref_primary_10_1016_j_robot_2023_104550 crossref_primary_10_1002_rob_22197 crossref_primary_10_1007_s11465_022_0742_y crossref_primary_10_1109_TRO_2022_3222958 crossref_primary_10_1016_j_robot_2024_104862 crossref_primary_10_1038_s41598_022_19599_2 crossref_primary_10_1109_LRA_2022_3185387 crossref_primary_10_1126_scirobotics_ads6192 crossref_primary_10_3389_fnbot_2021_627157 crossref_primary_10_1088_1361_665X_ac5317 crossref_primary_10_1109_TRO_2023_3302239 crossref_primary_10_1016_j_ast_2022_107980 crossref_primary_10_3389_fevo_2022_954838 crossref_primary_10_1109_ACCESS_2024_3371579 crossref_primary_10_1080_01691864_2024_2442718 crossref_primary_10_1016_j_birob_2021_100029 crossref_primary_10_1109_LRA_2024_3519908 crossref_primary_10_3390_app15063356 crossref_primary_10_7746_jkros_2023_18_2_143 crossref_primary_10_1109_TCDS_2023_3345539 crossref_primary_10_1109_ACCESS_2023_3311141 crossref_primary_10_20965_jrm_2023_p0160 crossref_primary_10_3390_machines12020092 crossref_primary_10_3390_act12020075 crossref_primary_10_1016_j_eswa_2023_120136 crossref_primary_10_1063_5_0273201 crossref_primary_10_1177_16878132241293630
Cites_doi	10.1109/TRO.2020.2964787 10.1177/0278364910388677 10.1109/ICRA.2019.8794144 10.1007/978-3-319-60916-4_17 10.1007/s10514-015-9479-3 10.1109/IROS.2016.7758092 10.1109/LRA.2019.2899434 10.1002/rob.21610 10.1145/3072959.3073602 10.1109/LRA.2018.2798285 10.1109/TRO.2018.2819658 10.1109/Humanoids43949.2019.9035046 10.1109/LRA.2018.2792536 10.1109/ICRA.2018.8460731
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2020
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2020
DBID	97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
DOI	10.1109/LRA.2020.2979660
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE/IET Electronic Library CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional
DatabaseTitleList	Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE/IET Electronic Library url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering
EISSN	2377-3766
EndPage	3706
ExternalDocumentID	10_1109_LRA_2020_2979660 9028188
Genre	orig-research
GrantInformation_xml	– fundername: Swiss National Science Foundation grantid: 166232; 188596 – fundername: European Union's Horizon 2020 research and innovation program grantid: 780883 – fundername: National Centre of Competence in Research Robotics funderid: 10.13039/501100011021 – fundername: ANYmal Research – fundername: Intel Labs
GroupedDBID	0R~ 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFS AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD IFIPE IPLJI JAVBF KQ8 M43 M~E O9- OCL RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c399t-6659501e689c3ba47fb8560422a0a2c886b46cd006b1a6044adfd2be3cb17eea3
IEDL.DBID	RIE
ISICitedReferencesCount	145
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000524331300013&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	2377-3766
IngestDate	Sun Jun 29 15:39:01 EDT 2025 Tue Nov 18 21:26:41 EST 2025 Sat Nov 29 06:03:07 EST 2025 Wed Aug 27 02:30:19 EDT 2025
IsDoiOpenAccess	false
IsOpenAccess	true
IsPeerReviewed	true
IsScholarly	true
Issue	2
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c399t-6659501e689c3ba47fb8560422a0a2c886b46cd006b1a6044adfd2be3cb17eea3
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0001-8269-6272 0000-0002-4285-4990 0000-0003-3428-8455 0000-0002-5072-7385
OpenAccessLink	http://hdl.handle.net/20.500.11850/404175
PQID	2386053824
PQPubID	4437225
PageCount	8
ParticipantIDs	crossref_primary_10_1109_LRA_2020_2979660 crossref_citationtrail_10_1109_LRA_2020_2979660 proquest_journals_2386053824 ieee_primary_9028188
PublicationCentury	2000
PublicationDate	2020-04-01
PublicationDateYYYYMMDD	2020-04-01
PublicationDate_xml	– month: 04 year: 2020 text: 2020-04-01 day: 01
PublicationDecade	2020
PublicationPlace	Piscataway
PublicationPlace_xml	– name: Piscataway
PublicationTitle	IEEE robotics and automation letters
PublicationTitleAbbrev	LRA
PublicationYear	2020
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 schulman (ref17) 2015 ref14 ref11 ref10 ref2 ref1 sutton (ref15) 2018 ref18 ref8 ref7 ref9 ref4 ref3 ref6 ref5 schulman (ref16) 2017 schulman (ref19) 0 heess (ref12) 2017
References_xml	– ident: ref14 doi: 10.1109/TRO.2020.2964787 – ident: ref6 doi: 10.1177/0278364910388677 – year: 2017 ident: ref12 article-title: Emergence of locomotion behaviours in rich environments – year: 2015 ident: ref17 article-title: High-dimensional continuous control using generalized advantage estimation – year: 2017 ident: ref16 article-title: Proximal policy optimization algorithms – ident: ref11 doi: 10.1109/ICRA.2019.8794144 – year: 2018 ident: ref15 publication-title: Introduction to Reinforcement Learning – ident: ref7 doi: 10.1007/978-3-319-60916-4_17 – ident: ref2 doi: 10.1007/s10514-015-9479-3 – ident: ref1 doi: 10.1109/IROS.2016.7758092 – ident: ref10 doi: 10.1109/LRA.2019.2899434 – ident: ref5 doi: 10.1002/rob.21610 – ident: ref13 doi: 10.1145/3072959.3073602 – ident: ref3 doi: 10.1109/LRA.2018.2798285 – start-page: 1889 year: 0 ident: ref19 article-title: Trust region policy optimization publication-title: Proc Int Conf Mach Learn – ident: ref8 doi: 10.1109/TRO.2018.2819658 – ident: ref9 doi: 10.1109/Humanoids43949.2019.9035046 – ident: ref18 doi: 10.1109/LRA.2018.2792536 – ident: ref4 doi: 10.1109/ICRA.2018.8460731
SSID	ssj0001527395
Score	2.5685241
Snippet	This letter addresses the problem of legged locomotion in non-flat terrain. As legged robots such as quadrupeds are to be deployed in terrains with geometries...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	3699
SubjectTerms	Computer simulation Deep learning deep learning in robotics and automation Legged locomotion Legged robots Locomotion Machine learning Markov processes motion and path planning Motion planning Optimization Path planning Physical simulation Policies Robot dynamics Terrain
Title	DeepGait: Planning and Control of Quadrupedal Gaits Using Deep Reinforcement Learning
URI	https://ieeexplore.ieee.org/document/9028188 https://www.proquest.com/docview/2386053824
Volume	5
WOSCitedRecordID	wos000524331300013&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE/IET Electronic Library customDbUrl: eissn: 2377-3766 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001527395 issn: 2377-3766 databaseCode: RIE dateStart: 20160101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources customDbUrl: eissn: 2377-3766 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001527395 issn: 2377-3766 databaseCode: M~E dateStart: 20160101 isFulltext: true titleUrlDefault: https://road.issn.org providerName: ISSN International Centre
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB7a4kEPvqpYrSUHL4Lb7mYfyXortdWDFi0WeluyyawUpC19ePS3m2S3raAI3pZlvrDMJJuZyeQbgCsZYOYqETmeinWAkvLIEVGIjs8xNHxUfoTCNptg_T4fjeLnEtxs7sIgoi0-w6Z5tGf5aipXJlXWMkwjHudlKDPG8rta23yKYRKLw_VJpBu3HgdtHf9Rt0ljVnBQbnce20rlx__Xbiq9g_99ziHsF84jaefWPoISTo5h7xulYBWGd4izezFe3pJ1PyIiJop08pJ0Ms3Iy0qo-WqGSg9lJBfE1g0QgyQDtFSq0mYNScG--nYCw173tfPgFK0THKk9jqUTGZpA18OIx9JPRcCylGvfJqBUuIJKzqM0iKTSSy71hH4fCJUpmqIvU48hCv8UKpPpBM-ACO2hhDFVngYG1IRormIazcLUz7R8DVprtSay4BU37S3eExtfuHGiDZEYQySFIWpwvUHMck6NP2SrRvEbuULnNaivLZcUi26RaO9DB2c-p8H576gL2DVj54U3dags5yu8hB35sRwv5g0oP312G3ZWfQHt3cmO
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3dS8MwED_mFNQHv6Y4nZoHXwTr2jRtU99kOifOoWMD30qaXGUg29iHf79J2k1BEXwr5a4td0lzv8vldwDnkmHmKhE6noo1QEl56IgwQMfnGBg-Kj9EYZtNRJ0Of32Nn0twuTwLg4i2-AyvzKXdy1cjOTepsrphGvE4X4HVgDHq5ae1vjIqhkssDhZ7kW5cb3dvNAKk7hWNo4KF8mvtsc1UfvyB7bLS3P7fB-3AVhE-kpvc37tQwuEebH4jFaxA_xZxfC8Gs2uy6EhExFCRRl6UTkYZeZkLNZmPUelHGckpsZUDxGiSLloyVWnzhqTgX33bh37zrtdoOUXzBEfqmGPmhIYo0PUw5LH0U8GiLOU6umGUCldQyXmYslAqPelST-j7TKhM0RR9mXoRovAPoDwcDfEQiNAxShBT5WlFRg1Ic1WktaMg9TMtX4X6wqyJLJjFTYOL98QiDDdOtCMS44ikcEQVLpYa45xV4w_ZijH8Uq6weRVqC88lxbSbJjr-0PDM55Qd_a51Buut3lM7aT90Ho9hw7wnL8OpQXk2meMJrMmP2WA6ObVj6xPN88uk
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=DeepGait%3A+Planning+and+Control+of+Quadrupedal+Gaits+Using+Deep+Reinforcement+Learning&rft.jtitle=IEEE+robotics+and+automation+letters&rft.au=Tsounis%2C+Vassilios&rft.au=Alge%2C+Mitja&rft.au=Lee%2C+Joonho&rft.au=Farshidian%2C+Farbod&rft.date=2020-04-01&rft.pub=IEEE&rft.eissn=2377-3766&rft.volume=5&rft.issue=2&rft.spage=3699&rft.epage=3706&rft_id=info:doi/10.1109%2FLRA.2020.2979660&rft.externalDocID=9028188
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2377-3766&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2377-3766&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2377-3766&client=summon