Off-Line Programming Framework for Sorting Task Based on Human-Demonstration
Sorting tasks are typical applications in the product and industrial domains. When facing new settings, such as different types of objects and their positions, the system has to be reprogrammed by experienced engineers, which decreases production efficiency and increases labour and downtime costs. I...
Saved in:
| Published in: | IEEE transactions on automation science and engineering Vol. 22; pp. 2199 - 2212 |
|---|---|
| Main Authors: | , , , |
| Format: | Journal Article |
| Language: | English |
| Published: |
IEEE
01.01.2025
|
| Subjects: | |
| ISSN: | 1545-5955, 1558-3783 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Sorting tasks are typical applications in the product and industrial domains. When facing new settings, such as different types of objects and their positions, the system has to be reprogrammed by experienced engineers, which decreases production efficiency and increases labour and downtime costs. In this paper, an off-line programming framework for sorting tasks based on human demonstration has been developed, taking advantage of machine vision and programming-by-demonstration concepts. Only one eye-to-hand depth camera has been used to record both the demonstrated trajectory and the corresponding relationship of the sorting task. A feature generalization algorithm has also been proposed to ensure that the system is able to reproduce the sorting task using both the key features of the trajectory and the sorting features. Finally, tracking, object and target matching, and generalization experiments have been conducted using the JAKA Zu7 robot with different types of fruit models and target boxes. The results show that the proposed framework is able to record the sorting trajectory at a frequency of 10 Hz, acquire and maintain the sorting relationship after new demonstration, and automatically reproduce the sorting task in new settings. Note to Practitioners-This paper is motivated by a goal to accomplish intuitive and user-friendly off-line programming for sorting task. The programming framework should be usable by non-experts and be able to complete programming tasks with a few demonstrations when faced with new scenarios, such as different types of objects, different correspondence relationships of the object, and different positions of the object. By observing the human demonstrated sorting tasks, the above-mentioned features can be effectively extracted, so as to complete the convenient teaching. The underlying principle of this paper is to let the robot observe human demonstrated sorting task by a depth camera, tracking the relative trajectory between the object and box as well as recording their matching information. The key features of the trajectory are extracted and further generalized combing the matching information to automatically reproduce the executable path in different settings. The proposed framework allows the users without prior programming knowledge to program the sorting task in new setting by simply conducting the pick and place process with their hands. |
|---|---|
| AbstractList | Sorting tasks are typical applications in the product and industrial domains. When facing new settings, such as different types of objects and their positions, the system has to be reprogrammed by experienced engineers, which decreases production efficiency and increases labour and downtime costs. In this paper, an off-line programming framework for sorting tasks based on human demonstration has been developed, taking advantage of machine vision and programming-by-demonstration concepts. Only one eye-to-hand depth camera has been used to record both the demonstrated trajectory and the corresponding relationship of the sorting task. A feature generalization algorithm has also been proposed to ensure that the system is able to reproduce the sorting task using both the key features of the trajectory and the sorting features. Finally, tracking, object and target matching, and generalization experiments have been conducted using the JAKA Zu7 robot with different types of fruit models and target boxes. The results show that the proposed framework is able to record the sorting trajectory at a frequency of 10 Hz, acquire and maintain the sorting relationship after new demonstration, and automatically reproduce the sorting task in new settings. Note to Practitioners-This paper is motivated by a goal to accomplish intuitive and user-friendly off-line programming for sorting task. The programming framework should be usable by non-experts and be able to complete programming tasks with a few demonstrations when faced with new scenarios, such as different types of objects, different correspondence relationships of the object, and different positions of the object. By observing the human demonstrated sorting tasks, the above-mentioned features can be effectively extracted, so as to complete the convenient teaching. The underlying principle of this paper is to let the robot observe human demonstrated sorting task by a depth camera, tracking the relative trajectory between the object and box as well as recording their matching information. The key features of the trajectory are extracted and further generalized combing the matching information to automatically reproduce the executable path in different settings. The proposed framework allows the users without prior programming knowledge to program the sorting task in new setting by simply conducting the pick and place process with their hands. |
| Author | Wu, Jianhua Ding, Cheng Du, Wei Xiong, Zhenhua |
| Author_xml | – sequence: 1 givenname: Wei orcidid: 0000-0001-7289-5972 surname: Du fullname: Du, Wei email: 01802021006@sjtu.edu.cn organization: State Key Laboratory of Mechanical System and Vibration, Institute of Robotics, and the School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China – sequence: 2 givenname: Cheng orcidid: 0000-0002-4361-266X surname: Ding fullname: Ding, Cheng email: shyreckdc@gmail.com organization: State Key Laboratory of Mechanical System and Vibration, Institute of Robotics, and the School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China – sequence: 3 givenname: Jianhua orcidid: 0000-0002-2410-5013 surname: Wu fullname: Wu, Jianhua email: wujh@sjtu.edu.cn organization: State Key Laboratory of Mechanical System and Vibration, Institute of Robotics, and the School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China – sequence: 4 givenname: Zhenhua orcidid: 0000-0001-6046-3101 surname: Xiong fullname: Xiong, Zhenhua email: mexiong@sjtu.edu.cn organization: State Key Laboratory of Mechanical System and Vibration, Institute of Robotics, and the School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China |
| BookMark | eNp9kNFKwzAUhoNMcJs-gOBFXiAzaZomuZxzc0JhwuZ1SdNkxK2JJBXx7W3dLsQLr87P4f8Oh28CRj54A8AtwTNCsLzfzbfLWYazfEYpLzjJLsCYMCYQ5YKOhpwzxCRjV2CS0hvum0LiMSg31qLSeQNfYthH1bbO7-GqD-YzxAO0IcJtiN2w3al0gA8qmQYGD9cfrfLo0bTBpy6qzgV_DS6tOiZzc55T8Lpa7hZrVG6enhfzEumsKDpEdJMpWoiaCa1rUmjLMJbK5JQqzjNGCLHSaE1ryQrBueIWy6YWTV8lIq_pFPDTXR1DStHYSrvu54P-EXesCK4GKdUgpRqkVGcpPUn-kO_RtSp-_cvcnRhnjPnVz7kQuaTfcAJvmQ |
| CODEN | ITASC7 |
| CitedBy_id | crossref_primary_10_3389_frobt_2024_1340334 |
| Cites_doi | 10.1109/TASE.2022.3217468 10.1109/ROBIO54168.2021.9739365 10.3390/s22072678 10.1049/iet-cta.2019.1316 10.1109/TASE.2020.3045880 10.1016/j.rcim.2020.102109 10.1016/j.rcim.2022.102444 10.1016/j.rcim.2020.101996 10.1109/TII.2019.2947539 10.1109/TSMC.2022.3225381 10.1007/978-3-319-97589-4_37 10.1109/TASE.2014.2361346 10.1016/j.robot.2017.11.012 10.1016/j.rcim.2023.102539 10.1109/TASE.2022.3141460 10.1177/0020294018819552 10.1146/annurev-control-100819-063206 10.1088/1361-6501/acb075 10.1109/ROBIO49542.2019.8961481 10.1016/j.rcim.2018.12.007 10.1016/j.mechatronics.2021.102609 10.1109/70.34770 10.1007/s40436-020-00303-4 10.1109/TASE.2020.3045655 |
| ContentType | Journal Article |
| DBID | 97E RIA RIE AAYXX CITATION |
| DOI | 10.1109/TASE.2024.3376712 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1558-3783 |
| EndPage | 2212 |
| ExternalDocumentID | 10_1109_TASE_2024_3376712 10478849 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: State Key Laboratory of Mechanical System and Vibration grantid: MSVZD202205 funderid: 10.13039/501100011415 |
| GroupedDBID | -~X 0R~ 29I 4.4 5GY 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACGFS ACIWK AENEX AETIX AGQYO AGSQL AHBIQ AIBXA AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD F5P HZ~ H~9 IFIPE IPLJI JAVBF LAI M43 O9- OCL PQQKQ RIA RIE RNS AAYXX CITATION |
| ID | FETCH-LOGICAL-c266t-1cd2a368b58ccb16cf5009ae433a7725111f9ecc3b956877a7f09db8d16c184b3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 1 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001193700500001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1545-5955 |
| IngestDate | Sat Nov 29 04:12:51 EST 2025 Tue Nov 18 21:37:43 EST 2025 Wed Aug 27 01:53:40 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c266t-1cd2a368b58ccb16cf5009ae433a7725111f9ecc3b956877a7f09db8d16c184b3 |
| ORCID | 0000-0001-7289-5972 0000-0001-6046-3101 0000-0002-4361-266X 0000-0002-2410-5013 |
| PageCount | 14 |
| ParticipantIDs | ieee_primary_10478849 crossref_citationtrail_10_1109_TASE_2024_3376712 crossref_primary_10_1109_TASE_2024_3376712 |
| PublicationCentury | 2000 |
| PublicationDate | 2025-01-01 |
| PublicationDateYYYYMMDD | 2025-01-01 |
| PublicationDate_xml | – month: 01 year: 2025 text: 2025-01-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationTitle | IEEE transactions on automation science and engineering |
| PublicationTitleAbbrev | TASE |
| PublicationYear | 2025 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| References | ref13 ref12 ref15 ref14 ref11 ref1 ref17 ref16 ref19 ref18 ref24 ref23 Khojastehnazhand (ref2) 2010; 4 ref26 ref25 ref20 ref22 ref21 ref8 ref7 ref9 ref4 ref3 Deom (ref10) ref6 ref5 |
| References_xml | – ident: ref23 doi: 10.1109/TASE.2022.3217468 – ident: ref13 doi: 10.1109/ROBIO54168.2021.9739365 – ident: ref7 doi: 10.3390/s22072678 – ident: ref21 doi: 10.1049/iet-cta.2019.1316 – ident: ref9 doi: 10.1109/TASE.2020.3045880 – ident: ref15 doi: 10.1016/j.rcim.2020.102109 – ident: ref22 doi: 10.1016/j.rcim.2022.102444 – ident: ref18 doi: 10.1016/j.rcim.2020.101996 – ident: ref3 doi: 10.1109/TII.2019.2947539 – ident: ref17 doi: 10.1109/TSMC.2022.3225381 – ident: ref11 doi: 10.1007/978-3-319-97589-4_37 – ident: ref4 doi: 10.1109/TASE.2014.2361346 – ident: ref24 doi: 10.1016/j.robot.2017.11.012 – volume: 4 start-page: 122 issue: 4 year: 2010 ident: ref2 article-title: Development of a lemon sorting system based on color and size publication-title: Afr. J. Plant Sci. – ident: ref6 doi: 10.1016/j.rcim.2023.102539 – volume-title: Proc. Robotix Acad. Conf. Ind. Robot. (RACIR) ident: ref10 article-title: Programming by demonstration using fiducial markers – ident: ref25 doi: 10.1109/TASE.2022.3141460 – ident: ref5 doi: 10.1177/0020294018819552 – ident: ref1 doi: 10.1146/annurev-control-100819-063206 – ident: ref14 doi: 10.1088/1361-6501/acb075 – ident: ref12 doi: 10.1109/ROBIO49542.2019.8961481 – ident: ref19 doi: 10.1016/j.rcim.2018.12.007 – ident: ref16 doi: 10.1016/j.mechatronics.2021.102609 – ident: ref26 doi: 10.1109/70.34770 – ident: ref8 doi: 10.1007/s40436-020-00303-4 – ident: ref20 doi: 10.1109/TASE.2020.3045655 |
| SSID | ssj0024890 |
| Score | 2.3954074 |
| Snippet | Sorting tasks are typical applications in the product and industrial domains. When facing new settings, such as different types of objects and their positions,... |
| SourceID | crossref ieee |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 2199 |
| SubjectTerms | Cameras Containers corresponding relationships Feature extraction object tracking path reproduction Programming programming by demonstration Sorting Sorting task Task analysis Trajectory |
| Title | Off-Line Programming Framework for Sorting Task Based on Human-Demonstration |
| URI | https://ieeexplore.ieee.org/document/10478849 |
| Volume | 22 |
| WOSCitedRecordID | wos001193700500001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1558-3783 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0024890 issn: 1545-5955 databaseCode: RIE dateStart: 20040101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA5aPOjBZ8X6IgdPQtrsJtlNjlVbPEgttEJvy-Ylot2VPvz9Jtmt1oOCtyXMLsuXwHwzmfkGgCtORS6JFSjRTCAqaYIkoQZxwrkweaStxWHYRDoY8MlEDOtm9dALY4wJxWem7R_DXb4u1dKnyjpBSsZ9fRNspmlSNWt9C-vxkFDxlAAxwVh9hRlh0Rl3Rz0XCsa0Tbx4SRT_cEJrU1WCU-nv_fN39sFuzR5ht9ruA7BhikOws6YpeAQeHq1FLsI0cFiVXk3dMuyvarCgI6lwVHrtgGc4zuev8Mb5MQ3LAoZ8ProzU08Zq4PRBE_93vj2HtUjE5BynnaBIqXjnCRcMq6UjBJlmSNRuaGE5I5HO3YVWeF2jUjfJpimeWqx0JJrZ-piPUmOQaMoC3MCIFdUCWwxYzylhmrBhCQxERobZ52QFsArDDNV64n7sRZvWYgrsMg87JmHPathb4Hrr1feKzGNv4ybHvI1wwrt01_Wz8B27GfzhvTIOWgsZktzAbbUx-JlPrsMZ-UT8SW61g |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA6-QD34rPg2B09C2uwm2U2OVVsq1lpohd6WzUtE24q2_n6T7FbrQcHbEmaX5UtgvpnMfAPAOacil8QKlGgmEJU0QZJQgzjhXJg80tbiMGwi7XT4YCC6ZbN66IUxxoTiM1P1j-EuX4_V1KfKakFKxn19ESwzSmNctGt9S-vxkFLxpAAxwVh5iRlhUevXew0XDMa0Srx8SRT_cENzc1WCW2lu_vOHtsBGyR9hvdjwbbBgRjtgfU5VcBe0761FLsY0sFsUXw3dMmzOqrCgo6mwN_bqAY-wn78_w0vnyTQcj2DI6KNrM_SksTgaFfDQbPSvWqgcmoCU87UTFCkd5yThknGlZJQoyxyNyg0lJHdM2vGryAq3b0T6RsE0zVOLhZZcO1MX7UmyB5ZG45HZB5ArqgS2mDGeUkO1YEKSmAiNjbNOyAHAMwwzVSqK-8EWL1mILLDIPOyZhz0rYT8AF1-vvBZyGn8ZVzzkc4YF2oe_rJ-B1Vb_rp21bzq3R2At9pN6Q7LkGCxN3qbmBKyoj8nT-9tpODef2L--HQ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Off-Line+Programming+Framework+for+Sorting+Task+Based+on+Human-Demonstration&rft.jtitle=IEEE+transactions+on+automation+science+and+engineering&rft.au=Du%2C+Wei&rft.au=Ding%2C+Cheng&rft.au=Wu%2C+Jianhua&rft.au=Xiong%2C+Zhenhua&rft.date=2025-01-01&rft.pub=IEEE&rft.issn=1545-5955&rft.volume=22&rft.spage=2199&rft.epage=2212&rft_id=info:doi/10.1109%2FTASE.2024.3376712&rft.externalDocID=10478849 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1545-5955&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1545-5955&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1545-5955&client=summon |