Interactive Language: Talking to Robots in Real Time

We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajector...

Full description

Saved in:
Bibliographic Details
Published in:IEEE robotics and automation letters pp. 1 - 8
Main Authors: Lynch, Corey, Wahid, Ayzaan, Tompson, Jonathan, Ding, Tianli, Betker, James, Baruch, Robert, Armstrong, Travis, Florence, Pete
Format: Journal Article
Language:English
Published: IEEE 2024
Subjects:
Behavioral sciences
Data Sets for Robot Learning
Engineering for Robotic Systems
Imitation Learning
Natural languages
Real-time systems
Robot kinematics
Robots
Stars
Task analysis
ISSN:2377-3766, 2377-3766
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuolinguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. " make a smiley face out of blocks ". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io .
AbstractList We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuolinguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. " make a smiley face out of blocks ". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io .
Author Armstrong, Travis
Tompson, Jonathan
Lynch, Corey
Baruch, Robert
Wahid, Ayzaan
Ding, Tianli
Betker, James
Florence, Pete
Author_xml – sequence: 1
  givenname: Corey
  orcidid: 0000-0002-2092-6690
  surname: Lynch
  fullname: Lynch, Corey
  organization: Robotics, Google, USA
– sequence: 2
  givenname: Ayzaan
  surname: Wahid
  fullname: Wahid, Ayzaan
  organization: Robotics, Google, USA
– sequence: 3
  givenname: Jonathan
  surname: Tompson
  fullname: Tompson, Jonathan
  organization: Robotics, Google, USA
– sequence: 4
  givenname: Tianli
  surname: Ding
  fullname: Ding, Tianli
  organization: Robotics, Google, USA
– sequence: 5
  givenname: James
  surname: Betker
  fullname: Betker, James
  organization: Robotics, Google, USA
– sequence: 6
  givenname: Robert
  surname: Baruch
  fullname: Baruch, Robert
  organization: Robotics, Google, USA
– sequence: 7
  givenname: Travis
  surname: Armstrong
  fullname: Armstrong, Travis
  organization: Robotics, Google, USA
– sequence: 8
  givenname: Pete
  orcidid: 0000-0002-7148-5645
  surname: Florence
  fullname: Florence, Pete
  organization: Robotics, Google, USA
BookMark eNp9j01Lw0AURQepYK3du3AxfyDxzZvJTOKuFD8KASHEdZgkL2U0nUgyCv57W9pFceHq3sU9F841m_nBE2O3AmIhILvPi1WMgDKWmCWYJBdsjtKYSBqtZ2f9ii2n6R0ARIJGZsmcqY0PNNomuG_iufXbL7ulB17a_sP5LQ8DL4Z6CBN3nhdke166Hd2wy872Ey1PuWBvT4_l-iXKX58361UeNcJkIeraNFFSQFsbQi0ok6nFDoVKFZCtlaS2U9RAStCaDLWWVmErQTS6rckouWD6-NuMwzSN1FWNCza4wYfRur4SUB30q71-ddCvTvp7EP6An6Pb2fHnP-TuiDgiOpuLFFEr-QuXV2Xe
CODEN IRALC6
CitedBy_id crossref_primary_10_1007_s00146_023_01670_9
crossref_primary_10_3390_su16156678
crossref_primary_10_1007_s11548_025_03351_y
crossref_primary_10_1007_s12555_024_0438_7
crossref_primary_10_1002_advs_202402705
crossref_primary_10_1038_s41598_025_01045_8
crossref_primary_10_3390_electronics13193956
crossref_primary_10_1007_s40031_025_01228_x
crossref_primary_10_1007_s11548_024_03120_3
crossref_primary_10_1007_s41809_024_00152_8
ContentType Journal Article
DBID 97E
RIA
RIE
AAYXX
CITATION
DOI 10.1109/LRA.2023.3295255
DatabaseName IEEE Xplore (IEEE)
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE/IET Electronic Library
CrossRef
DatabaseTitle CrossRef
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Xplore
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISSN 2377-3766
EndPage 8
ExternalDocumentID 10_1109_LRA_2023_3295255
10182264
Genre orig-research
GrantInformation_xml – fundername: Google
GroupedDBID 0R~
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACGFS
AGQYO
AHBIQ
AKJIK
AKQYR
ALMA_UNASSIGNED_HOLDINGS
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
EBS
IFIPE
IPLJI
JAVBF
KQ8
M43
M~E
O9-
OCL
RIA
RIE
AAYXX
AGSQL
CITATION
EJD
ID FETCH-LOGICAL-c179t-fd854310db7e261e938a2f214840eab43edf4ec08e0d792663a42d301c6dbe743
IEDL.DBID RIE
ISSN 2377-3766
IngestDate Sat Nov 29 06:03:29 EST 2025
Tue Nov 18 21:59:18 EST 2025
Wed Aug 27 02:13:42 EDT 2025
IsPeerReviewed true
IsScholarly true
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
https://doi.org/10.15223/policy-029
https://doi.org/10.15223/policy-037
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c179t-fd854310db7e261e938a2f214840eab43edf4ec08e0d792663a42d301c6dbe743
ORCID 0000-0002-7148-5645
0000-0002-2092-6690
PageCount 8
ParticipantIDs crossref_citationtrail_10_1109_LRA_2023_3295255
crossref_primary_10_1109_LRA_2023_3295255
ieee_primary_10182264
PublicationCentury 2000
PublicationDate 2024-00-00
PublicationDateYYYYMMDD 2024-01-01
PublicationDate_xml – year: 2024
  text: 2024-00-00
PublicationDecade 2020
PublicationTitle IEEE robotics and automation letters
PublicationTitleAbbrev LRA
PublicationYear 2024
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0001527395
Score 2.6254916
Snippet We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset,...
SourceID crossref
ieee
SourceType Enrichment Source
Index Database
Publisher
StartPage 1
SubjectTerms Behavioral sciences
Data Sets for Robot Learning
Engineering for Robotic Systems
Imitation Learning
Natural languages
Real-time systems
Robot kinematics
Robots
Stars
Task analysis
Title Interactive Language: Talking to Robots in Real Time
URI https://ieeexplore.ieee.org/document/10182264
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE Xplore
  customDbUrl:
  eissn: 2377-3766
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0001527395
  issn: 2377-3766
  databaseCode: RIE
  dateStart: 20160101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
– providerCode: PRVHPJ
  databaseName: ROAD: Directory of Open Access Scholarly Resources
  customDbUrl:
  eissn: 2377-3766
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0001527395
  issn: 2377-3766
  databaseCode: M~E
  dateStart: 20160101
  isFulltext: true
  titleUrlDefault: https://road.issn.org
  providerName: ISSN International Centre
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NS8NAEB1s8aAHPyvWj7IHLx7SxmTbZL0VafFQi5QKvYX9mEChJNKmHv3tzm5SzUXBWwi7ECbZnTdv894A3MVpPwo5Co_QBRUoOkhpH9QDLxUxJXxllDLOMn8STafxYiFeK7G608Igovv5DLv20p3lm1xvLVXWs-5SVvjZgEYURaVY64dQsVZior87ivRFbzIbdm138G4YiH5gxXy11FPrpeJSyfj4nw9xAkcVZmTD8iWfwh5mZ3BYcxI8B-6YPek2LzapOMhHNpcrS4WzImezXOXFhi0zNiNsyKz0owVv49H86dmrGiJ4mtZN4aUmttJ131oiU-WDIoxlkAZU0XAfpeIhmpSj9mP0TSQo9YaSB4aWsB4YhYQVLqCZ5RleAlNURhB4s7pSWsPaV1yjMJEMURJCkQ9t6O1ilejKLdw2rVglrmrwRULRTWx0kyq6bbj_nvFeOmX8MbZlA1sbV8b06pf713BA03lJfdxAs1hv8Rb29Uex3Kw70Hj5HHXc1_AFBO-wFw
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NS8NAEB20CurBz4r1cw9ePKRNk02T9VbEUjEWKRW8hezuBAolkTb19zu7SbUXBW8hbEJ4ye68mc17A3AbZUHocxQOsQtKUJSX0Tqoek4mIgr4UkuprWV-HI5G0fu7eK3F6lYLg4j25zNsm0O7l68LtTSlso5xlzLCz03YCjj3upVc66ekYszERLDajHRFJx7326Y_eNv3ROAZOd9a8FnrpmKDyeDgn49xCPs1a2T96jUfwQbmx7C35iV4AtzW9lK7fLG4rkLes0k6M8VwVhZsXMiiXLBpzsbEDpkRfzThbfA4eRg6dUsER9HMKZ1MR0a87hpTZMp9UPhR6mUe5TTcxVRyH3XGUbkRujoUFHz9lHuaJrHqaYnEFk6hkRc5ngGTlEgQfTPKUprFypVcodBh6mNKHCXttqCzwipRtV-4aVsxS2ze4IqE0E0MukmNbgvuvq_4qLwy_hjbNMCujaswPf_l_A3sDCcvcRI_jZ4vYJduxatCyCU0yvkSr2BbfZbTxfzafhNfMfKyLQ
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Interactive+Language%3A+Talking+to+Robots+in+Real+Time&rft.jtitle=IEEE+robotics+and+automation+letters&rft.au=Lynch%2C+Corey&rft.au=Wahid%2C+Ayzaan&rft.au=Tompson%2C+Jonathan&rft.au=Ding%2C+Tianli&rft.date=2024&rft.pub=IEEE&rft.eissn=2377-3766&rft.spage=1&rft.epage=8&rft_id=info:doi/10.1109%2FLRA.2023.3295255&rft.externalDocID=10182264
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2377-3766&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2377-3766&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2377-3766&client=summon