Interactive Language: Talking to Robots in Real Time

We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajector...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE robotics and automation letters s. 1 - 8
Hlavní autori: Lynch, Corey, Wahid, Ayzaan, Tompson, Jonathan, Ding, Tianli, Betker, James, Baruch, Robert, Armstrong, Travis, Florence, Pete
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: IEEE 2024
Predmet:
Behavioral sciences
Data Sets for Robot Learning
Engineering for Robotic Systems
Imitation Learning
Natural languages
Real-time systems
Robot kinematics
Robots
Stars
Task analysis
ISSN:2377-3766, 2377-3766
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Abstract We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuolinguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. " make a smiley face out of blocks ". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io .
AbstractList We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuolinguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. " make a smiley face out of blocks ". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io .
Author Armstrong, Travis
Tompson, Jonathan
Lynch, Corey
Baruch, Robert
Wahid, Ayzaan
Ding, Tianli
Betker, James
Florence, Pete
Author_xml – sequence: 1
  givenname: Corey
  orcidid: 0000-0002-2092-6690
  surname: Lynch
  fullname: Lynch, Corey
  organization: Robotics, Google, USA
– sequence: 2
  givenname: Ayzaan
  surname: Wahid
  fullname: Wahid, Ayzaan
  organization: Robotics, Google, USA
– sequence: 3
  givenname: Jonathan
  surname: Tompson
  fullname: Tompson, Jonathan
  organization: Robotics, Google, USA
– sequence: 4
  givenname: Tianli
  surname: Ding
  fullname: Ding, Tianli
  organization: Robotics, Google, USA
– sequence: 5
  givenname: James
  surname: Betker
  fullname: Betker, James
  organization: Robotics, Google, USA
– sequence: 6
  givenname: Robert
  surname: Baruch
  fullname: Baruch, Robert
  organization: Robotics, Google, USA
– sequence: 7
  givenname: Travis
  surname: Armstrong
  fullname: Armstrong, Travis
  organization: Robotics, Google, USA
– sequence: 8
  givenname: Pete
  orcidid: 0000-0002-7148-5645
  surname: Florence
  fullname: Florence, Pete
  organization: Robotics, Google, USA
BookMark eNp9j01Lw0AURQepYK3du3AxfyDxzZvJTOKuFD8KASHEdZgkL2U0nUgyCv57W9pFceHq3sU9F841m_nBE2O3AmIhILvPi1WMgDKWmCWYJBdsjtKYSBqtZ2f9ii2n6R0ARIJGZsmcqY0PNNomuG_iufXbL7ulB17a_sP5LQ8DL4Z6CBN3nhdke166Hd2wy872Ey1PuWBvT4_l-iXKX58361UeNcJkIeraNFFSQFsbQi0ok6nFDoVKFZCtlaS2U9RAStCaDLWWVmErQTS6rckouWD6-NuMwzSN1FWNCza4wYfRur4SUB30q71-ddCvTvp7EP6An6Pb2fHnP-TuiDgiOpuLFFEr-QuXV2Xe
CODEN IRALC6
CitedBy_id crossref_primary_10_1007_s00146_023_01670_9
crossref_primary_10_3390_su16156678
crossref_primary_10_1007_s11548_025_03351_y
crossref_primary_10_1007_s12555_024_0438_7
crossref_primary_10_1002_advs_202402705
crossref_primary_10_1038_s41598_025_01045_8
crossref_primary_10_3390_electronics13193956
crossref_primary_10_1007_s40031_025_01228_x
crossref_primary_10_1007_s11548_024_03120_3
crossref_primary_10_1007_s41809_024_00152_8
ContentType Journal Article
DBID 97E
RIA
RIE
AAYXX
CITATION
DOI 10.1109/LRA.2023.3295255
DatabaseName IEEE All-Society Periodicals Package (ASPP) 2005–Present
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Xplore
CrossRef
DatabaseTitle CrossRef
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Xplore
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISSN 2377-3766
EndPage 8
ExternalDocumentID 10_1109_LRA_2023_3295255
10182264
Genre orig-research
GrantInformation_xml – fundername: Google
GroupedDBID 0R~
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACGFS
AGQYO
AHBIQ
AKJIK
AKQYR
ALMA_UNASSIGNED_HOLDINGS
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
EBS
IFIPE
IPLJI
JAVBF
KQ8
M43
M~E
O9-
OCL
RIA
RIE
AAYXX
AGSQL
CITATION
EJD
ID FETCH-LOGICAL-c179t-fd854310db7e261e938a2f214840eab43edf4ec08e0d792663a42d301c6dbe743
IEDL.DBID RIE
ISSN 2377-3766
IngestDate Sat Nov 29 06:03:29 EST 2025
Tue Nov 18 21:59:18 EST 2025
Wed Aug 27 02:13:42 EDT 2025
IsPeerReviewed true
IsScholarly true
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
https://doi.org/10.15223/policy-029
https://doi.org/10.15223/policy-037
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c179t-fd854310db7e261e938a2f214840eab43edf4ec08e0d792663a42d301c6dbe743
ORCID 0000-0002-7148-5645
0000-0002-2092-6690
PageCount 8
ParticipantIDs crossref_citationtrail_10_1109_LRA_2023_3295255
crossref_primary_10_1109_LRA_2023_3295255
ieee_primary_10182264
PublicationCentury 2000
PublicationDate 2024-00-00
PublicationDateYYYYMMDD 2024-01-01
PublicationDate_xml – year: 2024
  text: 2024-00-00
PublicationDecade 2020
PublicationTitle IEEE robotics and automation letters
PublicationTitleAbbrev LRA
PublicationYear 2024
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0001527395
Score 2.6259453
Snippet We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset,...
SourceID crossref
ieee
SourceType Enrichment Source
Index Database
Publisher
StartPage 1
SubjectTerms Behavioral sciences
Data Sets for Robot Learning
Engineering for Robotic Systems
Imitation Learning
Natural languages
Real-time systems
Robot kinematics
Robots
Stars
Task analysis
Title Interactive Language: Talking to Robots in Real Time
URI https://ieeexplore.ieee.org/document/10182264
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE Xplore
  customDbUrl:
  eissn: 2377-3766
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0001527395
  issn: 2377-3766
  databaseCode: RIE
  dateStart: 20160101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
– providerCode: PRVHPJ
  databaseName: ROAD: Directory of Open Access Scholarly Resources
  customDbUrl:
  eissn: 2377-3766
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0001527395
  issn: 2377-3766
  databaseCode: M~E
  dateStart: 20160101
  isFulltext: true
  titleUrlDefault: https://road.issn.org
  providerName: ISSN International Centre
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFH644UEP_pw4f4wcvHjo1qXpmngbsuFhDhlTditN8gqD0crWefRvN0k77UXBWygJlK9NXt6XfN8DuKMh1xwH3OMiVR6T1LR4apIVnlAuQ01TV5LlbRJNp3yxEC-VWN1pYRDRXT7Drm26s3ydq62lynrWXcoKPxvQiKKoFGv9ECrWSkyEu6NIX_Qms2HXVgfvBlSE1Ir5aqGnVkvFhZLx8T9f4gSOqj0jGZYf-RT2MDuDw5qT4Dkwx-wlbvEik4qDfCDzZGWpcFLkZJbLvNiQZUZmZm9IrPSjBa_j0fzxyasKInjKzJvCSzW30nXfWiKbzAdFYBBNqclomI-JZAHqlKHyOfo6Eib0Bgmj2kxhNdASzV7hAppZnuElEEEVR5kGXGmzVuq-QIPoQAeyr2XIaL8NvR1Wsarcwm3RilXssgZfxAbd2KIbV-i24f57xHvplPFH35YFttavxPTql-fXcGCGs5L6uIFmsd7iLeyrj2K5WXeg8fw56ri_4QsxxbBp
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFH7oFNSDPyfOnzl48dCtS9M18TbEMbEOGVN2K23yCoPRytb595uknfai4C2UtJSvTV7el3zfA7ilPlcce9zhIpUOS6hu8VQnKzymPPEVTW1JlvcwGI34dCpeK7G61cIgoj18hm3TtHv5KpcrQ5V1jLuUEX5uwpbPGO2Wcq0fSsWYiQl_vRnpik447rdNffC2R4VPjZyvFnxq1VRsMBkc_PM1DmG_WjWSfvmZj2ADs2PYq3kJngCz3F5spy8SVizkPZnEc0OGkyIn4zzJiyWZZWSsV4fEiD-a8DZ4nDwMnaokgiP1yCmcVHEjXneNKbLOfVB4GtOU6pyGuRgnzEOVMpQuR1cFQgdfL2ZU6UEseypBvVo4hUaWZ3gGRFDJMUk9LpWeLVVXYBAEPeUlXZX4GtwWdNZYRbLyCzdlK-aRzRtcEWl0I4NuVKHbgrvvOz5Kr4w_-jYNsLV-Jabnv1y_gZ3h5CWMwqfR8wXs6kexkgi5hEaxWOEVbMvPYrZcXNt_4gtghrJ_
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Interactive+Language%3A+Talking+to+Robots+in+Real+Time&rft.jtitle=IEEE+robotics+and+automation+letters&rft.au=Lynch%2C+Corey&rft.au=Wahid%2C+Ayzaan&rft.au=Tompson%2C+Jonathan&rft.au=Ding%2C+Tianli&rft.date=2024&rft.issn=2377-3766&rft.eissn=2377-3766&rft.spage=1&rft.epage=8&rft_id=info:doi/10.1109%2FLRA.2023.3295255&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_LRA_2023_3295255
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2377-3766&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2377-3766&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2377-3766&client=summon