MARCER: Multimodal Augmented Reality for Composing and Executing Robot Tasks

In this work, we combine the strengths of humans and robots by developing MARCER, a novel interactive and multimodal end-user robot programming system. MARCER utilizes a Large Language Model to translate users' natural language task descriptions and environmental context into Action Plans for r...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2025 20th ACM/IEEE International Conference on Human-Robot Interaction (HRI) s. 529 - 539
Hlavní autoři: Ikeda, Bryce, Gramopadhye, Maitrey, Nekervis, LillyAnn, Szafir, Daniel
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 04.03.2025
Témata:
Augmented reality
End-user Robot Programming
Human-Robot Collaboration
Human-robot interaction
Large language models
Natural language processing
Real-time systems
Refining
Robot programming
Translation
Usability
Visualization
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Abstract In this work, we combine the strengths of humans and robots by developing MARCER, a novel interactive and multimodal end-user robot programming system. MARCER utilizes a Large Language Model to translate users' natural language task descriptions and environmental context into Action Plans for robot execution, based on a trigger-action programming paradigm that facilitates authoring reactive robot behaviors. MARCER also affords interaction via augmented reality to help users parameterize and validate robot programs and provide real-time, visual previews and feedback directly in the context of the robot's operating environment. We present the design, implementation, and evaluation of MARCER to explore the usability of such systems and demonstrate how trigger-action programming, Large Language Models, and augmented reality hold deep-seated synergies that, when combined, empower users to program general-purpose robots to perform everyday tasks.
AbstractList In this work, we combine the strengths of humans and robots by developing MARCER, a novel interactive and multimodal end-user robot programming system. MARCER utilizes a Large Language Model to translate users' natural language task descriptions and environmental context into Action Plans for robot execution, based on a trigger-action programming paradigm that facilitates authoring reactive robot behaviors. MARCER also affords interaction via augmented reality to help users parameterize and validate robot programs and provide real-time, visual previews and feedback directly in the context of the robot's operating environment. We present the design, implementation, and evaluation of MARCER to explore the usability of such systems and demonstrate how trigger-action programming, Large Language Models, and augmented reality hold deep-seated synergies that, when combined, empower users to program general-purpose robots to perform everyday tasks.
Author Szafir, Daniel
Gramopadhye, Maitrey
Nekervis, LillyAnn
Ikeda, Bryce
Author_xml – sequence: 1
  givenname: Bryce
  surname: Ikeda
  fullname: Ikeda, Bryce
  email: bikeda@cs.unc.edu
  organization: University of North Carolina at Chapel Hill,Department of Computer Science,Chapel Hill,NC,USA
– sequence: 2
  givenname: Maitrey
  surname: Gramopadhye
  fullname: Gramopadhye, Maitrey
  email: maitrey@cs.unc.edu
  organization: University of North Carolina at Chapel Hill,Department of Computer Science,Chapel Hill,NC,USA
– sequence: 3
  givenname: LillyAnn
  surname: Nekervis
  fullname: Nekervis, LillyAnn
  email: lnekervis@unc.edu
  organization: University of North Carolina at Chapel Hill,Department of Computer Science,Chapel Hill,NC,USA
– sequence: 4
  givenname: Daniel
  surname: Szafir
  fullname: Szafir, Daniel
  email: daniel.szafir@cs.unc.edu
  organization: University of North Carolina at Chapel Hill,Department of Computer Science,Chapel Hill,NC,USA
BookMark eNo1j99KwzAcRiPohc69gUheoDN_miXxrpTqBh1CmdcjTX4ZwbYZbQru7Z04rw7fufjgPKDbIQ6A0DMlK0qJftk02zUVhKwYYWJ1MTJnnN2gpZZacUG4VJrTe1Tviqasmle8m7sU-uhMh4v52MOQwOEGTBfSGfs44jL2pziF4YjN4HD1DXZOv6uJbUx4b6av6RHdedNNsLxygT7fqn25yeqP921Z1FmgUqVsLSlTbesIb70QRkCrBafWKkedFUCcppxZZ73KQVNjBJHMS3HxXPtcOr5AT3-_AQAOpzH0Zjwf_iP5D6lrS6Q
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/HRI61500.2025.10974232
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Xplore POP ALL
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 9798350378931
EndPage 539
ExternalDocumentID 10974232
Genre orig-research
GrantInformation_xml – fundername: NSF
  grantid: 2222953
  funderid: 10.13039/100000001
GroupedDBID 6IE
6IL
CBEJK
RIE
RIL
ID FETCH-LOGICAL-i178t-67128bbd03bf55a5eb9531cc8d1dc5e0d9132cdcf84e91aa5072f750d939f47d3
IEDL.DBID RIE
ISICitedReferencesCount 0
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001492540600055&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Thu May 29 05:57:37 EDT 2025
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i178t-67128bbd03bf55a5eb9531cc8d1dc5e0d9132cdcf84e91aa5072f750d939f47d3
PageCount 11
ParticipantIDs ieee_primary_10974232
PublicationCentury 2000
PublicationDate 2025-March-4
PublicationDateYYYYMMDD 2025-03-04
PublicationDate_xml – month: 03
  year: 2025
  text: 2025-March-4
  day: 04
PublicationDecade 2020
PublicationTitle 2025 20th ACM/IEEE International Conference on Human-Robot Interaction (HRI)
PublicationTitleAbbrev HRI
PublicationYear 2025
Publisher IEEE
Publisher_xml – name: IEEE
Score 1.9010043
Snippet In this work, we combine the strengths of humans and robots by developing MARCER, a novel interactive and multimodal end-user robot programming system. MARCER...
SourceID ieee
SourceType Publisher
StartPage 529
SubjectTerms Augmented reality
End-user Robot Programming
Human-Robot Collaboration
Human-robot interaction
Large language models
Natural language processing
Real-time systems
Refining
Robot programming
Translation
Usability
Visualization
Title MARCER: Multimodal Augmented Reality for Composing and Executing Robot Tasks
URI https://ieeexplore.ieee.org/document/10974232
WOSCitedRecordID wos001492540600055&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB5s8eBJxYpvcvC6fWw2m8RbKS0VtJSlQm8lj1kp6q60u-LPN0lbxYMHbyEJBCZMZr7MfDMAt4JSw7V7_bSlcZRIlJFQNI2YMJIqoTRPTGg2wScTMZ_L6ZasHrgwiBiSz7DthyGWb0tT-6-yjo-W-sBiAxqcpxuy1pb165Y64-zelzfvOtQXs_Zu86-2KcFqjA7_ed4RtH74d2T6bVmOYQ-LE3h47GeDYXZHAmf2rbTqlfTr51BU05IMg0NNnA9KvI6X_g-AqMKS4Sea2ic3k6zUZUVmav2ybsHTaDgbjKNtL4Ro2eOiilLuDInWtkt1zphiqKXTHmOE7VnDsGulg5XGmlwkKHtKOTcvzp03YCWVecItPYVmURZ4BkQjszmmcUIdIk0l1w4iawdVkXPElNFzaHlRLN435S4WOylc_DF_CQde4CExK7mCZrWq8Rr2zUe1XK9uwiV9AX6Nk6w
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA5aBT2pWPFtDl63j81mk3grpaXFtpSlQm8lj1kp1l1pd8WfbxK3igcP3kISCEyYzHyZ-WYQuueEaKbs66cMCYNIgAi4JHFAuRZEcqlYpH2zCTaZ8PlcTCuyuufCAIBPPoOGG_pYvsl16b7Kmi5a6gKLu2jPtc6q6FoV79cuNgfJ0BU4b1ncF9LGdvuvxinebvSP_nniMar_MPDw9Nu2nKAdyE7RaNxJur3kAXvW7Gtu5Ap3ymdfVtPgBLxLja0Xip2W5-4XAMvM4N4H6NKlN-MkV3mBZ3Lzsqmjp35v1h0EVTeEYNlmvAhiZk2JUqZFVEqppKCE1R-tuWkbTaFlhAWW2uiURyDaUlpHL0ytP2AEEWnEDDlDtSzP4BxhBdSkEIcRsZg0FkxZkKwsWAXGAGJKLlDdiWLx9lXwYrGVwuUf83foYDAbjxaj4eTxCh064fs0rega1Yp1CTdoX78Xy8361l_YJ2GplvU
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2025+20th+ACM%2FIEEE+International+Conference+on+Human-Robot+Interaction+%28HRI%29&rft.atitle=MARCER%3A+Multimodal+Augmented+Reality+for+Composing+and+Executing+Robot+Tasks&rft.au=Ikeda%2C+Bryce&rft.au=Gramopadhye%2C+Maitrey&rft.au=Nekervis%2C+LillyAnn&rft.au=Szafir%2C+Daniel&rft.date=2025-03-04&rft.pub=IEEE&rft.spage=529&rft.epage=539&rft_id=info:doi/10.1109%2FHRI61500.2025.10974232&rft.externalDocID=10974232