An Offline-Merge-Online Robot Teaching Method Based on Natural Human-Robot Interaction and Visual-Aid Algorithm

This article proposes an offline-merge-online robot teaching method (OMORTM). Specifically, a virtual-real fusion interactive interface (VRFII) is first developed by projecting a virtual robot into the real scene with an augmented-reality (AR) device, aiming to implement offline teaching. Second, a...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE/ASME transactions on mechatronics Jg. 27; H. 5; S. 2752 - 2763
Hauptverfasser: Du, Guanglong, Yao, Gengcheng, Li, Chunquan, Liu, Peter
Format: Journal Article
Sprache:Englisch
Veröffentlicht: New York IEEE 01.10.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Schlagworte:
Algorithms
Augmented reality
CAI
Computer assisted instruction
Education
Gesture and speech teaching algorithm
Human engineering
offline-merge-online robot teaching
Robot dynamics
Robot kinematics
Robot sensing systems
Robots
Service robots
Task analysis
Teaching methods
Trajectory
Virtual reality
virtual-real fusion interactive interface (VRFII)
visual-aid algorithm (VAA)
Workpieces
ISSN:1083-4435, 1941-014X
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Abstract This article proposes an offline-merge-online robot teaching method (OMORTM). Specifically, a virtual-real fusion interactive interface (VRFII) is first developed by projecting a virtual robot into the real scene with an augmented-reality (AR) device, aiming to implement offline teaching. Second, a visual-aid algorithm (VAA) is proposed to improve offline teaching accuracy. Third, a gesture and speech teaching fusion algorithm (GSTA) with the fingertip tactile force feedback is developed to obtain the natural teaching pattern and improve the interactive accuracy of teaching the real or virtual robot. More specifically, through the VRFII, the operator can use the GSTA and the VAA to teach the virtual robot naturally and safely, and then the real robot reproduces the motion of the virtual robot. Therefore, OMORTM enables the teaching results to be quickly verified while ensuring the operator's safety and avoiding damage to the robot or workpiece. A series of experiments were conducted to validate the practicality and effectiveness of OMORTM. The results show that by effectively combining the offline and online, OMORTMprovides accurate robotic teaching processes, suitable for nonprofessionals.
AbstractList This article proposes an offline-merge-online robot teaching method (OMORTM). Specifically, a virtual-real fusion interactive interface (VRFII) is first developed by projecting a virtual robot into the real scene with an augmented-reality (AR) device, aiming to implement offline teaching. Second, a visual-aid algorithm (VAA) is proposed to improve offline teaching accuracy. Third, a gesture and speech teaching fusion algorithm (GSTA) with the fingertip tactile force feedback is developed to obtain the natural teaching pattern and improve the interactive accuracy of teaching the real or virtual robot. More specifically, through the VRFII, the operator can use the GSTA and the VAA to teach the virtual robot naturally and safely, and then the real robot reproduces the motion of the virtual robot. Therefore, OMORTM enables the teaching results to be quickly verified while ensuring the operator's safety and avoiding damage to the robot or workpiece. A series of experiments were conducted to validate the practicality and effectiveness of OMORTM. The results show that by effectively combining the offline and online, OMORTMprovides accurate robotic teaching processes, suitable for nonprofessionals.
Author Li, Chunquan
Du, Guanglong
Liu, Peter
Yao, Gengcheng
Author_xml – sequence: 1
  givenname: Guanglong
  orcidid: 0000-0001-9425-843X
  surname: Du
  fullname: Du, Guanglong
  email: csgldu@scut.edu.cn
  organization: School of Computer Science and Engineering, South China University of Technology, Guangzhou, China
– sequence: 2
  givenname: Gengcheng
  orcidid: 0000-0002-0282-6231
  surname: Yao
  fullname: Yao, Gengcheng
  email: 1040189529@qq.com
  organization: School of Computer Science and Engineering, South China University of Technology, Guangzhou, China
– sequence: 3
  givenname: Chunquan
  orcidid: 0000-0002-5493-6379
  surname: Li
  fullname: Li, Chunquan
  email: lichunquan@ncu.edu.cn
  organization: School of Information Engineering, Nanchang University, Nanchang, China
– sequence: 4
  givenname: Peter
  orcidid: 0000-0002-8703-6967
  surname: Liu
  fullname: Liu, Peter
  organization: Department of Systems and Computer Engineering, Carleton University, Ottawa, ON, Canada
BookMark eNp9kEFPwjAUxxuDiYB-Ab008VzsW7utOyJBIQFJDBpvS-k6KBktdt3Bb-8mxIMHT33N-__ey_sNUM86qxG6BToCoNnDejmdzEYRjWDEAKI0ii5QHzIOhAL_6LU1FYxwzuIrNKjrPaWUA4U-cmOLV2VZGavJUvutJivbffCr27iA11qqnbFbvNRh5wr8KGtdYGfxiwyNlxWeNQdpySk8t0F7qYJp-9IW-N3UjazI2BR4XG2dN2F3uEaXpaxqfXN-h-jtabqezMhi9TyfjBdERVkcSAIpB81KtYGN4CxJMgFClqoQLOEyoaAyYIImkBU04UIVacTKNBWUxVwwKdkQ3Z_mHr37bHQd8r1rvG1X5q2dpHMk4jYlTinlXV17XebKBNkdELw0VQ407_TmP3rzTm9-1tui0R_06M1B-q__obsTZLTWv0AWZ0KkjH0DtVyGhQ
CODEN IATEFW
CitedBy_id crossref_primary_10_1007_s00521_025_11190_w
crossref_primary_10_1016_j_sasc_2025_200222
crossref_primary_10_1016_j_inffus_2024_102910
crossref_primary_10_1038_s41467_022_32702_5
crossref_primary_10_1177_14727978251376187
crossref_primary_10_3390_app13158836
crossref_primary_10_1109_ACCESS_2025_3595386
Cites_doi 10.1177/0954405414534642
10.1007/s40436-014-0087-9
10.1145/1065167.1065210
10.1109/TMECH.2002.1011250
10.4028/www.scientific.net/AMM.330.648
10.1109/TCDS.2018.2875052
10.1016/j.procir.2018.02.028
10.1109/ROBIO.2014.7090476
10.1109/TIE.2014.2301728
10.1109/TMECH.2020.2980434
10.1109/TPAMI.1987.4767965
10.1109/TIE.2018.2821649
10.1109/TIE.2018.2823667
10.3390/s120608301
10.1109/TASE.2015.2392095
10.5772/50978
10.1108/01439911011018911
10.1016/j.robot.2010.01.004
10.1016/j.rcim.2013.09.003
10.1016/j.rcim.2011.09.003
10.1145/2604969
10.1109/TMECH.2019.2945135
10.1109/TNNLS.2018.2872595
10.4028/www.scientific.net/AMR.463-464.1654
10.1109/COASE.2017.8256120
10.1109/TMECH.2020.2999532
10.1109/IROS.2018.8593700
10.1007/s12555-012-0091-4
10.1016/j.surfcoat.2012.03.038
10.31274/etd-180810-6008
10.1109/TIE.2003.814765
10.1109/TMECH.2020.2975960
10.1109/TRO.2017.2765334
10.2307/2684684
10.1109/TMECH.2013.2240312
10.1016/j.mechatronics.2018.03.006
10.1080/0951192x.2015.1033753
10.1109/34.87344
ContentType Journal Article
Copyright Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022
Copyright_xml – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022
DBID 97E
RIA
RIE
AAYXX
CITATION
7SC
7SP
7TB
8FD
FR3
JQ2
L7M
L~C
L~D
DOI 10.1109/TMECH.2021.3112722
DatabaseName IEEE Xplore (IEEE)
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
Computer and Information Systems Abstracts
Electronics & Communications Abstracts
Mechanical & Transportation Engineering Abstracts
Technology Research Database
Engineering Research Database
ProQuest Computer Science Collection
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts – Academic
Computer and Information Systems Abstracts Professional
DatabaseTitle CrossRef
Technology Research Database
Computer and Information Systems Abstracts – Academic
Mechanical & Transportation Engineering Abstracts
Electronics & Communications Abstracts
ProQuest Computer Science Collection
Computer and Information Systems Abstracts
Engineering Research Database
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts Professional
DatabaseTitleList Technology Research Database
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
Education
EISSN 1941-014X
EndPage 2763
ExternalDocumentID 10_1109_TMECH_2021_3112722
9598873
Genre orig-research
GrantInformation_xml – fundername: National Natural Science Foundation of China
  grantid: 6217022520; 61973126; 61863028; 81660299; 61503177
  funderid: 10.13039/501100001809
– fundername: Innovation Team of the Modern Agriculture Industry Technology System in Guangdong Province
  grantid: 2019KJ139
– fundername: Key-Area Research and Development Program of Guangdong Province
  grantid: 2020B010166006
– fundername: Science and Technology Department of Jiangxi Province of China
  grantid: 2020ABC03A39; 20161ACB21007; 20171BBE50071; 20171BAB202033
– fundername: CIE-Tencent Robotics X Rhino-Bird Focused Research Program
GroupedDBID -~X
0R~
29I
4.4
5GY
5VS
6IK
97E
9M8
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACGFS
ACIWK
ACKIV
AETIX
AGQYO
AGSQL
AHBIQ
AI.
AIBXA
AKJIK
AKQYR
ALLEH
ALMA_UNASSIGNED_HOLDINGS
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CS3
EBS
EJD
F5P
H~9
IFIPE
IFJZH
IPLJI
JAVBF
LAI
M43
OCL
RIA
RIE
RNS
TN5
VH1
AAYXX
CITATION
7SC
7SP
7TB
8FD
FR3
JQ2
L7M
L~C
L~D
ID FETCH-LOGICAL-c295t-61741e3fcb1b843669818afcd8364a601c91380619d0648cd723f778035483aa3
IEDL.DBID RIE
ISICitedReferencesCount 6
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000732641800001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 1083-4435
IngestDate Mon Jun 30 02:46:23 EDT 2025
Sat Nov 29 06:33:13 EST 2025
Tue Nov 18 22:37:42 EST 2025
Wed Aug 27 02:18:43 EDT 2025
IsPeerReviewed true
IsScholarly true
Issue 5
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
https://doi.org/10.15223/policy-029
https://doi.org/10.15223/policy-037
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c295t-61741e3fcb1b843669818afcd8364a601c91380619d0648cd723f778035483aa3
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ORCID 0000-0002-5493-6379
0000-0002-8703-6967
0000-0001-9425-843X
0000-0002-0282-6231
PQID 2726112785
PQPubID 85420
PageCount 12
ParticipantIDs crossref_citationtrail_10_1109_TMECH_2021_3112722
crossref_primary_10_1109_TMECH_2021_3112722
proquest_journals_2726112785
ieee_primary_9598873
PublicationCentury 2000
PublicationDate 2022-Oct.
2022-10-00
20221001
PublicationDateYYYYMMDD 2022-10-01
PublicationDate_xml – month: 10
  year: 2022
  text: 2022-Oct.
PublicationDecade 2020
PublicationPlace New York
PublicationPlace_xml – name: New York
PublicationTitle IEEE/ASME transactions on mechatronics
PublicationTitleAbbrev TMECH
PublicationYear 2022
Publisher IEEE
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml – name: IEEE
– name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References ref13
ref35
ref12
ref34
ref15
ref37
ref14
ref36
(ref26) 14, 2018.
ref31
ref30
ref11
ref33
ref10
ref32
ref2
ref1
ref39
ref16
ref38
ref19
ref18
ref24
ref23
ref25
ref20
ref41
ref22
ref21
ref28
ref29
ref8
ref7
ref9
ref4
ref3
ref6
ref5
Jen (ref17) 2008; 15
ref40
References_xml – ident: ref24
  doi: 10.1177/0954405414534642
– ident: ref39
  doi: 10.1007/s40436-014-0087-9
– ident: ref35
  doi: 10.1145/1065167.1065210
– ident: ref28
  doi: 10.1109/TMECH.2002.1011250
– ident: ref2
  doi: 10.4028/www.scientific.net/AMM.330.648
– ident: ref10
  doi: 10.1109/TCDS.2018.2875052
– ident: ref36
  doi: 10.1016/j.procir.2018.02.028
– ident: ref16
  doi: 10.1109/ROBIO.2014.7090476
– ident: ref30
  doi: 10.1109/TIE.2014.2301728
– ident: ref31
  doi: 10.1109/TMECH.2020.2980434
– ident: ref34
  doi: 10.1109/TPAMI.1987.4767965
– ident: ref8
  doi: 10.1109/TIE.2018.2821649
– ident: ref14
  doi: 10.1109/TIE.2018.2823667
– ident: ref5
  doi: 10.3390/s120608301
– ident: ref4
  doi: 10.1109/TASE.2015.2392095
– ident: ref15
  doi: 10.5772/50978
– volume: 15
  start-page: 317
  issue: 3
  year: 2008
  ident: ref17
  article-title: VR-based robot programming and simulation system for an industrial robot
  publication-title: Int. J. Ind. Eng.
– year: 14, 2018.
  ident: ref26
  article-title: Recommendations for Designing Multi Targets
– ident: ref13
  doi: 10.1108/01439911011018911
– ident: ref18
  doi: 10.1016/j.robot.2010.01.004
– ident: ref37
  doi: 10.1016/j.rcim.2013.09.003
– ident: ref25
  doi: 10.1016/j.rcim.2011.09.003
– ident: ref41
  doi: 10.1145/2604969
– ident: ref32
  doi: 10.1109/TMECH.2019.2945135
– ident: ref9
  doi: 10.1109/TNNLS.2018.2872595
– ident: ref19
  doi: 10.4028/www.scientific.net/AMR.463-464.1654
– ident: ref23
  doi: 10.1109/COASE.2017.8256120
– ident: ref12
  doi: 10.1109/TMECH.2020.2999532
– ident: ref40
  doi: 10.1109/IROS.2018.8593700
– ident: ref3
  doi: 10.1007/s12555-012-0091-4
– ident: ref21
  doi: 10.1016/j.surfcoat.2012.03.038
– ident: ref22
  doi: 10.31274/etd-180810-6008
– ident: ref6
  doi: 10.1109/TIE.2003.814765
– ident: ref29
  doi: 10.1109/TMECH.2020.2975960
– ident: ref7
  doi: 10.1109/TRO.2017.2765334
– ident: ref38
  doi: 10.2307/2684684
– ident: ref11
  doi: 10.1109/TMECH.2013.2240312
– ident: ref1
  doi: 10.1016/j.mechatronics.2018.03.006
– ident: ref20
  doi: 10.1080/0951192x.2015.1033753
– ident: ref33
  doi: 10.1109/34.87344
SSID ssj0004101
Score 2.4005046
Snippet This article proposes an offline-merge-online robot teaching method (OMORTM). Specifically, a virtual-real fusion interactive interface (VRFII) is first...
SourceID proquest
crossref
ieee
SourceType Aggregation Database
Enrichment Source
Index Database
Publisher
StartPage 2752
SubjectTerms Algorithms
Augmented reality
CAI
Computer assisted instruction
Education
Gesture and speech teaching algorithm
Human engineering
offline-merge-online robot teaching
Robot dynamics
Robot kinematics
Robot sensing systems
Robots
Service robots
Task analysis
Teaching methods
Trajectory
Virtual reality
virtual-real fusion interactive interface (VRFII)
visual-aid algorithm (VAA)
Workpieces
Title An Offline-Merge-Online Robot Teaching Method Based on Natural Human-Robot Interaction and Visual-Aid Algorithm
URI https://ieeexplore.ieee.org/document/9598873
https://www.proquest.com/docview/2726112785
Volume 27
WOSCitedRecordID wos000732641800001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE Electronic Library (IEL)
  customDbUrl:
  eissn: 1941-014X
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0004101
  issn: 1083-4435
  databaseCode: RIE
  dateStart: 19960101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LTxsxEB5B1EN74NmqgYB84AYua3tfPgYEyiUpQinitvL60UYKu1Wy4fcz9m5Cq1ZI3PYwI6121p5v7PnmAzhLEUIbi2WJ0y6iiMBziqhIURnugNIy09wFsYlsMskfH-XdFlxsuDDW2tB8Zr_5x3CXb2q98kdllzKRuCbENmxnWdpytV45kCxIHTOEFDRGDLAmyETycjq-uR5hKcgZVqiMZ5z_lYSCqso_W3HIL7e773uzPdjpcCQZtoHfhy1bHXgJ5q5d4wA-_TFp8BDqYUW-O-cxJR17wiVtZ4yS-7qsGzLteirJOAhKkyvMbYbUFZmoMJeDhLN-2hqHQ8SWD0FUZcjDbLlSczqcGTKc_6wXs-bX02f4cXszvR7RTmuBai6TxjMFY2aF0yUr81ikqcRMrpw2uUhjhVWblkzkmPylQRCTa5Nx4bIsjwSWPEIp8QV6VV3Zr0B0UjqTRNZpGcex1WWpI60Ys2UaoRvvA1t__EJ3g8i9Hsa8CAVJJIsQsMIHrOgC1ofzjc_vdgzHm9aHPkQbyy46fRisY1x0K3VZoH3qvfLk6P9ex_CRe8pDaOAbQK9ZrOwJfNDPzWy5OA0_4Qs6wtes
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1Lb9NAEB6VgkQ5QOlDpC_2wA2Weh9-7DFUrVLRpAgF1Ju13gdECjZKHH4_s2sngEBIvfkwI1ke7843u_PNB_AqQwhtHZYl3viEIgIvKKIiTVW8A8qq3HAfxSbyyaS4u1MftuDNhgvjnIvNZ-5teIx3-bYxq3BUdq5ShWtCPICHqZQ86dhav1iQLIodMwQVVCIKWFNkEnU-HV9ejLAY5AxrVMZzzv9IQ1FX5a_NOGaYq2f3e7ddeNojSTLsQv8ctly9F0SY-4aNPXjy26zBfWiGNbn1PqBKOg6US9pNGSUfm6ppybTvqiTjKClN3mF2s6SpyUTHyRwknvbTzjgeI3aMCKJrSz7Plis9p8OZJcP5l2Yxa79-O4BPV5fTixHt1Rao4SptA1dQMie8qVhVSJFlCnO59sYWIpMa6zajmCgw_SuLMKYwNufC53mRCCx6hNbiELbrpnYvgJi08jZNnDdKSulMVZnEaMZclSXoxgfA1h-_NP0o8qCIMS9jSZKoMgasDAEr-4AN4PXG53s3iOO_1vshRBvLPjoDOFnHuOzX6rJE-yx4FenRv71ewuPRdHxT3lxP3h_DDg8EiNjOdwLb7WLlTuGR-dHOlouz-EP-BJvG2vM
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+Offline-Merge-Online+Robot+Teaching+Method+Based+on+Natural+Human-Robot+Interaction+and+Visual-Aid+Algorithm&rft.jtitle=IEEE%2FASME+transactions+on+mechatronics&rft.au=Du%2C+Guanglong&rft.au=Yao%2C+Gengcheng&rft.au=Li%2C+Chunquan&rft.au=Liu%2C+Peter&rft.date=2022-10-01&rft.pub=IEEE&rft.issn=1083-4435&rft.volume=27&rft.issue=5&rft.spage=2752&rft.epage=2763&rft_id=info:doi/10.1109%2FTMECH.2021.3112722&rft.externalDocID=9598873
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1083-4435&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1083-4435&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1083-4435&client=summon