Interactive Language: Talking to Robots in Real Time

We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajector...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE robotics and automation letters s. 1 - 8
Hlavní autoři: Lynch, Corey, Wahid, Ayzaan, Tompson, Jonathan, Ding, Tianli, Betker, James, Baruch, Robert, Armstrong, Travis, Florence, Pete
Médium: Journal Article
Jazyk:angličtina
Vydáno: IEEE 2024
Témata:
Behavioral sciences
Data Sets for Robot Learning
Engineering for Robotic Systems
Imitation Learning
Natural languages
Real-time systems
Robot kinematics
Robots
Stars
Task analysis
ISSN:2377-3766, 2377-3766
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Abstract We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuolinguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. " make a smiley face out of blocks ". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io .
AbstractList We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuolinguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. " make a smiley face out of blocks ". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io .
Author Armstrong, Travis
Tompson, Jonathan
Lynch, Corey
Baruch, Robert
Wahid, Ayzaan
Ding, Tianli
Betker, James
Florence, Pete
Author_xml – sequence: 1
  givenname: Corey
  orcidid: 0000-0002-2092-6690
  surname: Lynch
  fullname: Lynch, Corey
  organization: Robotics, Google, USA
– sequence: 2
  givenname: Ayzaan
  surname: Wahid
  fullname: Wahid, Ayzaan
  organization: Robotics, Google, USA
– sequence: 3
  givenname: Jonathan
  surname: Tompson
  fullname: Tompson, Jonathan
  organization: Robotics, Google, USA
– sequence: 4
  givenname: Tianli
  surname: Ding
  fullname: Ding, Tianli
  organization: Robotics, Google, USA
– sequence: 5
  givenname: James
  surname: Betker
  fullname: Betker, James
  organization: Robotics, Google, USA
– sequence: 6
  givenname: Robert
  surname: Baruch
  fullname: Baruch, Robert
  organization: Robotics, Google, USA
– sequence: 7
  givenname: Travis
  surname: Armstrong
  fullname: Armstrong, Travis
  organization: Robotics, Google, USA
– sequence: 8
  givenname: Pete
  orcidid: 0000-0002-7148-5645
  surname: Florence
  fullname: Florence, Pete
  organization: Robotics, Google, USA
BookMark eNp9j01Lw0AURQepYK3du3AxfyDxzZvJTOKuFD8KASHEdZgkL2U0nUgyCv57W9pFceHq3sU9F841m_nBE2O3AmIhILvPi1WMgDKWmCWYJBdsjtKYSBqtZ2f9ii2n6R0ARIJGZsmcqY0PNNomuG_iufXbL7ulB17a_sP5LQ8DL4Z6CBN3nhdke166Hd2wy872Ey1PuWBvT4_l-iXKX58361UeNcJkIeraNFFSQFsbQi0ok6nFDoVKFZCtlaS2U9RAStCaDLWWVmErQTS6rckouWD6-NuMwzSN1FWNCza4wYfRur4SUB30q71-ddCvTvp7EP6An6Pb2fHnP-TuiDgiOpuLFFEr-QuXV2Xe
CODEN IRALC6
CitedBy_id crossref_primary_10_1007_s00146_023_01670_9
crossref_primary_10_3390_su16156678
crossref_primary_10_1007_s11548_025_03351_y
crossref_primary_10_1007_s12555_024_0438_7
crossref_primary_10_1002_advs_202402705
crossref_primary_10_1038_s41598_025_01045_8
crossref_primary_10_3390_electronics13193956
crossref_primary_10_1007_s40031_025_01228_x
crossref_primary_10_1007_s11548_024_03120_3
crossref_primary_10_1007_s41809_024_00152_8
ContentType Journal Article
DBID 97E
RIA
RIE
AAYXX
CITATION
DOI 10.1109/LRA.2023.3295255
DatabaseName IEEE All-Society Periodicals Package (ASPP) 2005–Present
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
DatabaseTitle CrossRef
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISSN 2377-3766
EndPage 8
ExternalDocumentID 10_1109_LRA_2023_3295255
10182264
Genre orig-research
GrantInformation_xml – fundername: Google
GroupedDBID 0R~
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACGFS
AGQYO
AHBIQ
AKJIK
AKQYR
ALMA_UNASSIGNED_HOLDINGS
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
EBS
IFIPE
IPLJI
JAVBF
KQ8
M43
M~E
O9-
OCL
RIA
RIE
AAYXX
AGSQL
CITATION
EJD
ID FETCH-LOGICAL-c179t-fd854310db7e261e938a2f214840eab43edf4ec08e0d792663a42d301c6dbe743
IEDL.DBID RIE
ISSN 2377-3766
IngestDate Sat Nov 29 06:03:29 EST 2025
Tue Nov 18 21:59:18 EST 2025
Wed Aug 27 02:13:42 EDT 2025
IsPeerReviewed true
IsScholarly true
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
https://doi.org/10.15223/policy-029
https://doi.org/10.15223/policy-037
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c179t-fd854310db7e261e938a2f214840eab43edf4ec08e0d792663a42d301c6dbe743
ORCID 0000-0002-7148-5645
0000-0002-2092-6690
PageCount 8
ParticipantIDs crossref_citationtrail_10_1109_LRA_2023_3295255
crossref_primary_10_1109_LRA_2023_3295255
ieee_primary_10182264
PublicationCentury 2000
PublicationDate 2024-00-00
PublicationDateYYYYMMDD 2024-01-01
PublicationDate_xml – year: 2024
  text: 2024-00-00
PublicationDecade 2020
PublicationTitle IEEE robotics and automation letters
PublicationTitleAbbrev LRA
PublicationYear 2024
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0001527395
Score 2.6254916
Snippet We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset,...
SourceID crossref
ieee
SourceType Enrichment Source
Index Database
Publisher
StartPage 1
SubjectTerms Behavioral sciences
Data Sets for Robot Learning
Engineering for Robotic Systems
Imitation Learning
Natural languages
Real-time systems
Robot kinematics
Robots
Stars
Task analysis
Title Interactive Language: Talking to Robots in Real Time
URI https://ieeexplore.ieee.org/document/10182264
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE Electronic Library (IEL)
  customDbUrl:
  eissn: 2377-3766
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0001527395
  issn: 2377-3766
  databaseCode: RIE
  dateStart: 20160101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
– providerCode: PRVHPJ
  databaseName: ROAD: Directory of Open Access Scholarly Resources
  customDbUrl:
  eissn: 2377-3766
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0001527395
  issn: 2377-3766
  databaseCode: M~E
  dateStart: 20160101
  isFulltext: true
  titleUrlDefault: https://road.issn.org
  providerName: ISSN International Centre
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NS8NAEB1s8aAHPyvWj7IHLx7SxmST3fVWpMVDLVIq9Bb2YwqF0kibevS3u7tJNRcFbyHsQnjJZmbe7nsDcMeZSlIjtG9oEtiMWAacxmmgJLJIcSYw9kLhERuP-WwmXiuxutfCIKI_fIZdd-n38k2ut44q6zl3KSf8bECDMVaKtX4IFWclJpLdVmQoeqNJv-u6g3fjSCSRE_PVQk-tl4oPJcPjfz7ECRxVOSPply_5FPZwdQaHNSfBc6Ce2ZP-50VGFQf5SKZy6ahwUuRkkqu82JDFikxsbkic9KMFb8PB9Ok5qBoiBNqumyKYG-6k66GzRLaVD4qYy2ge2YqGhigVjdHMKeqQY2iYsKE3ljQydgnr1Ci0ucIFNFf5Ci-BPKDihqZUsAdNFU1kIo121V-cCM00b0Nvh1WmK7dw17RimfmqIRSZRTdz6GYVum24_57xXjpl_DG25YCtjSsxvfrl_jUc2Om0pD5uoFmst3gL-_qjWGzWHWi8fA46_mv4Au4ZsCc
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NS8NAEB20CurBz4r1cw9ePKRNk02y662IpWIsUip4C_sxhUJJpE39_e5uUu1FwVsImxBespmZt_veANyyREax5so1NPFMRiw8RsPYkwKTQLKEY-iEwmkyHLL3d_5ai9WdFgYR3eYzbNtDt5avC7W0VFnHuktZ4ecmbEWUBt1KrvVDqVgzMR6tFiN93klHvbbtD94OAx4FVs63FnzWuqm4YNI_-OdjHMJ-nTWSXvWaj2AD82PYW_MSPAHquD3hfl8krVnIezIWM0uGk7Igo0IW5YJMczIy2SGx4o8mvPUfxw8Dr26J4Ckzc0pvopkVr_vWFNnUPshDJoJJYGoa6qOQNEQ9oah8hr5OuAm-oaCBNpNYxVqiyRZOoZEXOZ4B6aJkmsaUJ11FJY1EJLSy9V8YcZUo1oLOCqtM1X7htm3FLHN1g88zg25m0c1qdFtw933FR-WV8cfYpgV2bVyF6fkv529gZzB-SbP0afh8AbvmVrQiQi6hUc6XeAXb6rOcLubX7pv4AhtRsj0
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Interactive+Language%3A+Talking+to+Robots+in+Real+Time&rft.jtitle=IEEE+robotics+and+automation+letters&rft.au=Lynch%2C+Corey&rft.au=Wahid%2C+Ayzaan&rft.au=Tompson%2C+Jonathan&rft.au=Ding%2C+Tianli&rft.date=2024&rft.issn=2377-3766&rft.eissn=2377-3766&rft.spage=1&rft.epage=8&rft_id=info:doi/10.1109%2FLRA.2023.3295255&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_LRA_2023_3295255
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2377-3766&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2377-3766&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2377-3766&client=summon