Interactive Language: Talking to Robots in Real Time

We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajector...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE robotics and automation letters s. 1 - 8
Hlavní autoři: Lynch, Corey, Wahid, Ayzaan, Tompson, Jonathan, Ding, Tianli, Betker, James, Baruch, Robert, Armstrong, Travis, Florence, Pete
Médium: Journal Article
Jazyk:angličtina
Vydáno: IEEE 2024
Témata:
Behavioral sciences
Data Sets for Robot Learning
Engineering for Robotic Systems
Imitation Learning
Natural languages
Real-time systems
Robot kinematics
Robots
Stars
Task analysis
ISSN:2377-3766, 2377-3766
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuolinguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. " make a smiley face out of blocks ". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io .
ISSN:2377-3766
2377-3766
DOI:10.1109/LRA.2023.3295255