MARCER: Multimodal Augmented Reality for Composing and Executing Robot Tasks

In this work, we combine the strengths of humans and robots by developing MARCER, a novel interactive and multimodal end-user robot programming system. MARCER utilizes a Large Language Model to translate users' natural language task descriptions and environmental context into Action Plans for r...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:2025 20th ACM/IEEE International Conference on Human-Robot Interaction (HRI) S. 529 - 539
Hauptverfasser: Ikeda, Bryce, Gramopadhye, Maitrey, Nekervis, LillyAnn, Szafir, Daniel
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: IEEE 04.03.2025
Schlagworte:
Augmented reality
End-user Robot Programming
Human-Robot Collaboration
Human-robot interaction
Large language models
Natural language processing
Real-time systems
Refining
Robot programming
Translation
Usability
Visualization
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:In this work, we combine the strengths of humans and robots by developing MARCER, a novel interactive and multimodal end-user robot programming system. MARCER utilizes a Large Language Model to translate users' natural language task descriptions and environmental context into Action Plans for robot execution, based on a trigger-action programming paradigm that facilitates authoring reactive robot behaviors. MARCER also affords interaction via augmented reality to help users parameterize and validate robot programs and provide real-time, visual previews and feedback directly in the context of the robot's operating environment. We present the design, implementation, and evaluation of MARCER to explore the usability of such systems and demonstrate how trigger-action programming, Large Language Models, and augmented reality hold deep-seated synergies that, when combined, empower users to program general-purpose robots to perform everyday tasks.
DOI:10.1109/HRI61500.2025.10974232