Text2Motion: from natural language instructions to feasible plans

We propose Text2Motion, a language-based planning framework enabling robots to solve sequential manipulation tasks that require long-horizon reasoning. Given a natural language instruction, our framework constructs both a task- and motion-level plan that is verified to reach inferred symbolic goals....

Full description

Saved in:
Bibliographic Details
Published in:Autonomous robots Vol. 47; no. 8; pp. 1345 - 1365
Main Authors: Lin, Kevin, Agia, Christopher, Migimatsu, Toki, Pavone, Marco, Bohg, Jeannette
Format: Journal Article
Language:English
Published: New York Springer US 01.12.2023
Springer Nature B.V
Subjects:
Artificial Intelligence
Computer Imaging
Control
Engineering
Feasibility
Language instruction
Large language models
Mechatronics
Natural language
Pattern Recognition and Graphics
Planning
Reasoning
Robotics
Robotics and Automation
Skills
Task planning (robotics)
Vision
Large language models
Long-horizon planning
Robot manipulation
ISSN:0929-5593, 1573-7527
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We propose Text2Motion, a language-based planning framework enabling robots to solve sequential manipulation tasks that require long-horizon reasoning. Given a natural language instruction, our framework constructs both a task- and motion-level plan that is verified to reach inferred symbolic goals. Text2Motion uses feasibility heuristics encoded in Q-functions of a library of skills to guide task planning with Large Language Models. Whereas previous language-based planners only consider the feasibility of individual skills, Text2Motion actively resolves geometric dependencies spanning skill sequences by performing geometric feasibility planning during its search. We evaluate our method on a suite of problems that require long-horizon reasoning, interpretation of abstract goals, and handling of partial affordance perception. Our experiments show that Text2Motion can solve these challenging problems with a success rate of 82%, while prior state-of-the-art language-based planning methods only achieve 13%. Text2Motion thus provides promising generalization characteristics to semantically diverse sequential manipulation tasks with geometric dependencies between skills. Qualitative results are made available at https://sites.google.com/stanford.edu/text2motion .
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0929-5593
1573-7527
DOI:10.1007/s10514-023-10131-7