Interactive Language: Talking to Robots in Real Time

We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajector...

Full description

Saved in:
Bibliographic Details
Published in:IEEE robotics and automation letters pp. 1 - 8
Main Authors: Lynch, Corey, Wahid, Ayzaan, Tompson, Jonathan, Ding, Tianli, Betker, James, Baruch, Robert, Armstrong, Travis, Florence, Pete
Format: Journal Article
Language:English
Published: IEEE 2024
Subjects:
Behavioral sciences
Data Sets for Robot Learning
Engineering for Robotic Systems
Imitation Learning
Natural languages
Real-time systems
Robot kinematics
Robots
Stars
Task analysis
ISSN:2377-3766, 2377-3766
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuolinguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. " make a smiley face out of blocks ". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io .
AbstractList We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuolinguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. " make a smiley face out of blocks ". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io .
Author Armstrong, Travis
Tompson, Jonathan
Lynch, Corey
Baruch, Robert
Wahid, Ayzaan
Ding, Tianli
Betker, James
Florence, Pete
Author_xml – sequence: 1
  givenname: Corey
  orcidid: 0000-0002-2092-6690
  surname: Lynch
  fullname: Lynch, Corey
  organization: Robotics, Google, USA
– sequence: 2
  givenname: Ayzaan
  surname: Wahid
  fullname: Wahid, Ayzaan
  organization: Robotics, Google, USA
– sequence: 3
  givenname: Jonathan
  surname: Tompson
  fullname: Tompson, Jonathan
  organization: Robotics, Google, USA
– sequence: 4
  givenname: Tianli
  surname: Ding
  fullname: Ding, Tianli
  organization: Robotics, Google, USA
– sequence: 5
  givenname: James
  surname: Betker
  fullname: Betker, James
  organization: Robotics, Google, USA
– sequence: 6
  givenname: Robert
  surname: Baruch
  fullname: Baruch, Robert
  organization: Robotics, Google, USA
– sequence: 7
  givenname: Travis
  surname: Armstrong
  fullname: Armstrong, Travis
  organization: Robotics, Google, USA
– sequence: 8
  givenname: Pete
  orcidid: 0000-0002-7148-5645
  surname: Florence
  fullname: Florence, Pete
  organization: Robotics, Google, USA
BookMark eNp9j01Lw0AURQepYK3du3AxfyDxzZvJTOKuFD8KASHEdZgkL2U0nUgyCv57W9pFceHq3sU9F841m_nBE2O3AmIhILvPi1WMgDKWmCWYJBdsjtKYSBqtZ2f9ii2n6R0ARIJGZsmcqY0PNNomuG_iufXbL7ulB17a_sP5LQ8DL4Z6CBN3nhdke166Hd2wy872Ey1PuWBvT4_l-iXKX58361UeNcJkIeraNFFSQFsbQi0ok6nFDoVKFZCtlaS2U9RAStCaDLWWVmErQTS6rckouWD6-NuMwzSN1FWNCza4wYfRur4SUB30q71-ddCvTvp7EP6An6Pb2fHnP-TuiDgiOpuLFFEr-QuXV2Xe
CODEN IRALC6
CitedBy_id crossref_primary_10_1007_s00146_023_01670_9
crossref_primary_10_3390_su16156678
crossref_primary_10_1007_s11548_025_03351_y
crossref_primary_10_1007_s12555_024_0438_7
crossref_primary_10_1002_advs_202402705
crossref_primary_10_1038_s41598_025_01045_8
crossref_primary_10_3390_electronics13193956
crossref_primary_10_1007_s40031_025_01228_x
crossref_primary_10_1007_s11548_024_03120_3
crossref_primary_10_1007_s41809_024_00152_8
ContentType Journal Article
DBID 97E
RIA
RIE
AAYXX
CITATION
DOI 10.1109/LRA.2023.3295255
DatabaseName IEEE All-Society Periodicals Package (ASPP) 2005–Present
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Xplore
CrossRef
DatabaseTitle CrossRef
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Xplore: IEL
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISSN 2377-3766
EndPage 8
ExternalDocumentID 10_1109_LRA_2023_3295255
10182264
Genre orig-research
GrantInformation_xml – fundername: Google
GroupedDBID 0R~
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACGFS
AGQYO
AHBIQ
AKJIK
AKQYR
ALMA_UNASSIGNED_HOLDINGS
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
EBS
IFIPE
IPLJI
JAVBF
KQ8
M43
M~E
O9-
OCL
RIA
RIE
AAYXX
AGSQL
CITATION
EJD
ID FETCH-LOGICAL-c179t-fd854310db7e261e938a2f214840eab43edf4ec08e0d792663a42d301c6dbe743
IEDL.DBID RIE
ISSN 2377-3766
IngestDate Sat Nov 29 06:03:29 EST 2025
Tue Nov 18 21:59:18 EST 2025
Wed Aug 27 02:13:42 EDT 2025
IsPeerReviewed true
IsScholarly true
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
https://doi.org/10.15223/policy-029
https://doi.org/10.15223/policy-037
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c179t-fd854310db7e261e938a2f214840eab43edf4ec08e0d792663a42d301c6dbe743
ORCID 0000-0002-7148-5645
0000-0002-2092-6690
PageCount 8
ParticipantIDs crossref_citationtrail_10_1109_LRA_2023_3295255
crossref_primary_10_1109_LRA_2023_3295255
ieee_primary_10182264
PublicationCentury 2000
PublicationDate 2024-00-00
PublicationDateYYYYMMDD 2024-01-01
PublicationDate_xml – year: 2024
  text: 2024-00-00
PublicationDecade 2020
PublicationTitle IEEE robotics and automation letters
PublicationTitleAbbrev LRA
PublicationYear 2024
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0001527395
Score 2.6254916
Snippet We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset,...
SourceID crossref
ieee
SourceType Enrichment Source
Index Database
Publisher
StartPage 1
SubjectTerms Behavioral sciences
Data Sets for Robot Learning
Engineering for Robotic Systems
Imitation Learning
Natural languages
Real-time systems
Robot kinematics
Robots
Stars
Task analysis
Title Interactive Language: Talking to Robots in Real Time
URI https://ieeexplore.ieee.org/document/10182264
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE Xplore: IEL
  customDbUrl:
  eissn: 2377-3766
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0001527395
  issn: 2377-3766
  databaseCode: RIE
  dateStart: 20160101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
– providerCode: PRVHPJ
  databaseName: ROAD: Directory of Open Access Scholarly Resources
  customDbUrl:
  eissn: 2377-3766
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0001527395
  issn: 2377-3766
  databaseCode: M~E
  dateStart: 20160101
  isFulltext: true
  titleUrlDefault: https://road.issn.org
  providerName: ISSN International Centre
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFH644UEP_pw4f4wcvHjoFtN0abwN2fAwh4wJu5U0eYXBWGXrPPq3m6Sd9qLgrZQEytem770v-b4HcKcYjRVqE4Q8ZQEXOgyU7svA8DhSUqU0Qu6bTYjJJJ7P5WslVvdaGET0h8-w6y79Xr7J9dZRZT3nLuWEnw1oCCFKsdYPoeKsxGS024qksjeeDrquO3g3ZDJiTsxXCz21Xio-lIyO__kQJ3BU5YxkUL7kU9jD1Rkc1pwEz4F7Zk_5nxcZVxzkI5mppaPCSZGTaZ7mxYYsVmRqc0PipB8teBsNZ0_PQdUQIdB23RRBZmInXafOEtlWPijDWLGM2YqGU1QpD9FkHDWNkRohbegNFWfGLmHdNynaXOECmqt8hZdApIiYCp0O3hZE4kFLZqhGWwtJzPo8TdvQ22GV6Mot3DWtWCa-aqAysegmDt2kQrcN998z3kunjD_GthywtXElple_3L-GAzudl9THDTSL9RZvYV9_FIvNugONl89hx38NXwbqsGA
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFH7oFNSDPyfOnzl48dAtS9O18TbEMbEOGRO8lTR5hcFoZev8-03STndR8FZKWsrXpu-9L_m-B3ArGY0kKu35PGUeD5XvSdUTnuZRIIVMaYDcNZsIR6Po_V281mJ1p4VBRLf5DNv20K3l60ItLVXWse5SVvi5CVsB56xbybV-KBVrJiaC1WIkFZ143G_b_uBtn4mAWTnfWvBZ66bigsng4J-PcQj7ddZI-tVrPoINzI9hb81L8AS44_ak-32RuGYh78lEziwZTsqCjIu0KBdkmpOxyQ6JFX804W3wOHkYenVLBE-ZmVN6mY6seJ1aU2RT-6DwI8kyZmoaTlGm3EedcVQ0QqpDYYKvLznTZhKrnk7RZAun0MiLHM-AiDBg0rdKeFMShV0lmKYKTTUkMOvxNG1BZ4VVomq_cNu2Ypa4uoGKxKCbWHSTGt0W3H1f8VF5ZfwxtmmBXRtXYXr-y_kb2BlOXuIkfho9X8CuuRWviJBLaJTzJV7Btvosp4v5tfsmvgA2kbJ2
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Interactive+Language%3A+Talking+to+Robots+in+Real+Time&rft.jtitle=IEEE+robotics+and+automation+letters&rft.au=Lynch%2C+Corey&rft.au=Wahid%2C+Ayzaan&rft.au=Tompson%2C+Jonathan&rft.au=Ding%2C+Tianli&rft.date=2024&rft.pub=IEEE&rft.eissn=2377-3766&rft.spage=1&rft.epage=8&rft_id=info:doi/10.1109%2FLRA.2023.3295255&rft.externalDocID=10182264
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2377-3766&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2377-3766&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2377-3766&client=summon