Improving Robotic Fruit Harvesting Within Cluttered Environments Through 3D Shape Completion
The world population is increasing and will, by 2050, nearly double its demand for food, feed, fuel, and fiber. Besides environmental challenges, labor shortage also poses crucial challenges to the agricultural production system. Automation of manual tasks in crop production can potentially increase...
Uloženo v:
| Vydáno v: | IEEE robotics and automation letters Ročník 9; číslo 8; s. 7357 - 7364 |
|---|---|
| Hlavní autoři: | , , , , , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
Piscataway
IEEE
01.08.2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Témata: | |
| ISSN: | 2377-3766, 2377-3766 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | The world population is increasing and will, by 2050, nearly double its demand for food, feed, fuel, and fiber. Besides environmental challenges, labor shortage also poses crucial challenges to the agricultural production system. Automation of manual tasks in crop production can potentially increase efficiency but also lead to a change in agricultural practices for more effective usage of available land. In this letter, we address the problem of robotic fruit harvesting in challenging real-world scenarios such as vertical farms, where robotic sensing and acting need to cope with a cluttered environment. Robotic fruit harvesting is typically done by directly detecting a grasp point in the sensor reading, which can lie on the fruit itself or on its peduncle depending on crop harvesting requirements. However, grasp point detection is not always possible as the ideal grasp point may be hidden behind leaves or other fruits. Our approach exploits shape completion techniques allowing us to estimate the complete 3D shape of a target fruit together with its pose even under strong occlusions. In this way, we can estimate a grasp point even when the fruit is only partially visible. We evaluate our approach on a real robotic manipulator operating in a vertical farm growing different fruit species and employing different harvesting tools. Our experiments show that, on average, our proposed pipeline increases the success rate by 18.5 percentage points, in terms of end-effector positioning, compared to the most competitive baseline among the ones reported in this work, that does not rely on shape completion. |
|---|---|
| AbstractList | The world population is increasing and will, by 2050, nearly double its demand for food, feed, fuel, and fiber. Besides environmental challenges, labor shortage also poses crucial challenges to the agricultural production system. Automation of manual tasks in crop production can potentially increase efficiency but also lead to a change in agricultural practices for more effective usage of available land. In this letter, we address the problem of robotic fruit harvesting in challenging real-world scenarios such as vertical farms, where robotic sensing and acting need to cope with a cluttered environment. Robotic fruit harvesting is typically done by directly detecting a grasp point in the sensor reading, which can lie on the fruit itself or on its peduncle depending on crop harvesting requirements. However, grasp point detection is not always possible as the ideal grasp point may be hidden behind leaves or other fruits. Our approach exploits shape completion techniques allowing us to estimate the complete 3D shape of a target fruit together with its pose even under strong occlusions. In this way, we can estimate a grasp point even when the fruit is only partially visible. We evaluate our approach on a real robotic manipulator operating in a vertical farm growing different fruit species and employing different harvesting tools. Our experiments show that, on average, our proposed pipeline increases the success rate by 18.5 percentage points, in terms of end-effector positioning, compared to the most competitive baseline among the ones reported in this work, that does not rely on shape completion. |
| Author | Behley, Jens Magistri, Federico Pan, Yue Stachniss, Cyrill Bartels, Jake Lehnert, Christopher |
| Author_xml | – sequence: 1 givenname: Federico orcidid: 0000-0003-2815-5760 surname: Magistri fullname: Magistri, Federico organization: Center for Robotics, University of Bonn, Bonn, Germany – sequence: 2 givenname: Yue orcidid: 0000-0002-5208-7481 surname: Pan fullname: Pan, Yue organization: Center for Robotics, University of Bonn, Bonn, Germany – sequence: 3 givenname: Jake orcidid: 0009-0008-0286-7990 surname: Bartels fullname: Bartels, Jake email: n9747656@qut.edu.au organization: Queensland University of Technology, Brisbane, QLD, Australia – sequence: 4 givenname: Jens orcidid: 0000-0001-6483-0319 surname: Behley fullname: Behley, Jens organization: Center for Robotics, University of Bonn, Bonn, Germany – sequence: 5 givenname: Cyrill orcidid: 0000-0003-1173-6972 surname: Stachniss fullname: Stachniss, Cyrill organization: Center for Robotics, University of Bonn, Bonn, Germany – sequence: 6 givenname: Christopher orcidid: 0000-0002-4230-8068 surname: Lehnert fullname: Lehnert, Christopher organization: Queensland University of Technology, Brisbane, QLD, Australia |
| BookMark | eNp9UE1Lw0AUXKSCtfbuwcOC59b9SLLJsURrCwWhVrwIYZO8NFuS3bjZFPz3JrSH4sHTezAzb97MLRppowGhe0rmlJLoabNdzBlh3px7jIowvEJjxoWYcREEo4v9Bk3b9kAIoT4TPPLH6GtdN9Ycld7jrUmNUxle2k45vJL2CK0bgE_lSqVxXHXOgYUcv-ijskbXoF2Ld6U13b7E_Bm_l7IBHJu6qcApo-_QdSGrFqbnOUEfy5ddvJpt3l7X8WIzy1jE3IwWEQc_JZQxP4UsLbxcBNJjPJQsDKgPAkgGGS9YnveoFHnueUUqc5CRYB7nE_R4uttH-e76r5OD6azuLRNORMRDGtCoZ5ETK7OmbS0USWNVLe1PQkky1Jj0NSZDjcm5xl4S_JFkyskhmrNSVf8JH05CBQAXPn7IPD_kv7xwgsg |
| CODEN | IRALC6 |
| CitedBy_id | crossref_primary_10_3390_agriengineering7070208 crossref_primary_10_1016_j_asej_2025_103735 crossref_primary_10_1016_j_atech_2025_101343 crossref_primary_10_1016_j_asoc_2025_112971 crossref_primary_10_3390_plants14132080 |
| Cites_doi | 10.1109/LRA.2022.3193239 10.3390/agriculture12081240 10.1002/rob.22207 10.1109/ICRA.2016.7487394 10.1007/s43154-020-00034-1 10.1109/ICRA.2017.7989418 10.1109/ICCV.2017.322 10.3390/s20195670 10.1145/3530687 10.1109/LRA.2023.3236571 10.1109/ICRA48506.2021.9561848 10.1002/rob.22041 10.1016/j.biosystemseng.2021.06.001 10.1289/ehp.02110445 10.1109/LRA.2017.2651952 10.1109/CVPR.2016.90 10.1109/ICRA48506.2021.9561356 10.1109/ICRA46639.2022.9811358 10.1016/j.compag.2023.107878 10.1002/rob.22227 10.1109/CASE49997.2022.9926466 10.1002/rob.21715 10.1371/journal.pone.0256340 10.1109/ICRA46639.2022.9812303 10.1109/HUMANOIDS.2015.7363472 10.1109/IROS45743.2020.9341013 10.1016/j.compag.2017.11.023 10.1016/j.compag.2023.108004 10.1002/rob.21937 10.1371/journal.pone.0159781 10.1109/IROS55552.2023.10342067 10.1016/j.compag.2022.107430 10.1109/3DV53792.2021.00143 10.3389/frai.2022.830026 10.1109/CVPR.2019.00025 10.1145/37401.37422 10.1007/s12571-017-0648-4 10.15607/RSS.2020.XVI.027 10.1109/CVPR42600.2020.01224 10.3390/s22031296 10.1109/IROS55552.2023.10341855 10.1016/j.compag.2020.105736 10.1787/ed758aab-en 10.1109/IROS51168.2021.9636701 10.1002/rob.21889 10.1109/IROS47612.2022.9981304 10.1002/rob.21987 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024 |
| DBID | 97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| DOI | 10.1109/LRA.2024.3421788 |
| DatabaseName | IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Forestry |
| EISSN | 2377-3766 |
| EndPage | 7364 |
| ExternalDocumentID | 10_1109_LRA_2024_3421788 10582458 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: FOR 5351 – fundername: Deutsche Forschungsgemeinschaft funderid: 10.13039/501100001659 – fundername: Germany's Excellence Strategy grantid: 390732324; STA1051/5-1 |
| GroupedDBID | 0R~ 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFS AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD IFIPE IPLJI JAVBF KQ8 M43 M~E O9- OCL RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c292t-1f93e5b01225becbf4d76a4238a28615e7e0cec3f2ddcbfa7dd44fbadea972433 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 7 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001271562400004&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 2377-3766 |
| IngestDate | Sun Nov 09 06:04:42 EST 2025 Sat Nov 29 06:03:31 EST 2025 Tue Nov 18 22:43:58 EST 2025 Wed Aug 27 02:29:07 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 8 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c292t-1f93e5b01225becbf4d76a4238a28615e7e0cec3f2ddcbfa7dd44fbadea972433 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0003-1173-6972 0000-0002-4230-8068 0000-0003-2815-5760 0009-0008-0286-7990 0000-0001-6483-0319 0000-0002-5208-7481 |
| PQID | 3079381619 |
| PQPubID | 4437225 |
| PageCount | 8 |
| ParticipantIDs | crossref_primary_10_1109_LRA_2024_3421788 crossref_citationtrail_10_1109_LRA_2024_3421788 ieee_primary_10582458 proquest_journals_3079381619 |
| PublicationCentury | 2000 |
| PublicationDate | 2024-08-01 |
| PublicationDateYYYYMMDD | 2024-08-01 |
| PublicationDate_xml | – month: 08 year: 2024 text: 2024-08-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | Piscataway |
| PublicationPlace_xml | – name: Piscataway |
| PublicationTitle | IEEE robotics and automation letters |
| PublicationTitleAbbrev | LRA |
| PublicationYear | 2024 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref13 ref12 ref15 ref14 ref53 ref52 ref11 ref55 ref10 ref54 ref17 ref16 ref19 ref18 ref51 ref50 ref46 ref45 ref48 ref47 ref42 Coleman (ref8) 2014; 5 ref41 ref44 ref43 ref49 ref7 Kozai (ref21) 2019 (ref1) 2020 LaValle (ref23) 1998 ref3 ref6 ref5 ref40 ref35 ref34 Loudec (ref24) 2023 ref37 ref36 ref31 ref30 ref33 ref32 ref2 ref39 ref38 Constant (ref9) 2011 ref26 ref25 ref20 ref22 Loshchilov (ref27) 2019 ref28 Barthelme (ref4) 2021 ref29 |
| References_xml | – year: 1998 ident: ref23 article-title: Rapidly-exploring random trees: A new tool for path planning – ident: ref29 doi: 10.1109/LRA.2022.3193239 – volume: 5 start-page: 3 issue: 1 year: 2014 ident: ref8 article-title: Reducing the barrier to entry of complex robotic software: A moveit! case study publication-title: J. Softw. Eng. Robot. – ident: ref48 doi: 10.3390/agriculture12081240 – ident: ref37 doi: 10.1002/rob.22207 – ident: ref25 doi: 10.1109/ICRA.2016.7487394 – ident: ref20 doi: 10.1007/s43154-020-00034-1 – ident: ref33 doi: 10.1109/ICRA.2017.7989418 – ident: ref12 doi: 10.1109/ICCV.2017.322 – ident: ref16 doi: 10.3390/s20195670 – ident: ref22 doi: 10.1145/3530687 – ident: ref47 doi: 10.1109/LRA.2023.3236571 – ident: ref49 doi: 10.1109/ICRA48506.2021.9561848 – ident: ref10 doi: 10.1002/rob.22041 – ident: ref6 doi: 10.1016/j.biosystemseng.2021.06.001 – ident: ref14 doi: 10.1289/ehp.02110445 – ident: ref40 doi: 10.1109/LRA.2017.2651952 – ident: ref13 doi: 10.1109/CVPR.2016.90 – ident: ref28 doi: 10.1109/ICRA48506.2021.9561356 – volume-title: Proc. Int. Conf. Learn. Representations year: 2019 ident: ref27 article-title: Decoupled weight decay regularization – start-page: 173 volume-title: Proc. Australas. Conf. Robot. Automat. year: 2021 ident: ref4 article-title: Implementation of a vertical hydroponic farming end effector for harvesting – ident: ref31 doi: 10.1109/ICRA46639.2022.9811358 – ident: ref55 doi: 10.1016/j.compag.2023.107878 – ident: ref11 doi: 10.1002/rob.22227 – ident: ref30 doi: 10.1109/CASE49997.2022.9926466 – ident: ref43 doi: 10.1002/rob.21715 – ident: ref41 doi: 10.1371/journal.pone.0256340 – ident: ref44 doi: 10.1109/ICRA46639.2022.9812303 – ident: ref5 doi: 10.1109/HUMANOIDS.2015.7363472 – ident: ref17 doi: 10.1109/IROS45743.2020.9341013 – ident: ref52 doi: 10.1016/j.compag.2017.11.023 – ident: ref19 doi: 10.1016/j.compag.2023.108004 – start-page: 148 volume-title: Proc. Int. Conf. Comput. Vis. Syst. year: 2023 ident: ref24 article-title: Key point-based orientation estimation of strawberries for robotic fruit picking – ident: ref3 doi: 10.1002/rob.21937 – ident: ref42 doi: 10.1371/journal.pone.0159781 – ident: ref34 doi: 10.1109/IROS55552.2023.10342067 – volume-title: Plant Factory: An Indoor Vertical Farming System for Efficient Quality Food Production year: 2019 ident: ref21 – ident: ref38 doi: 10.1016/j.compag.2022.107430 – ident: ref50 doi: 10.1109/3DV53792.2021.00143 – ident: ref18 doi: 10.3389/frai.2022.830026 – ident: ref35 doi: 10.1109/CVPR.2019.00025 – ident: ref26 doi: 10.1145/37401.37422 – ident: ref15 doi: 10.1007/s12571-017-0648-4 – ident: ref45 doi: 10.15607/RSS.2020.XVI.027 – ident: ref54 doi: 10.1109/CVPR42600.2020.01224 – year: 2011 ident: ref9 article-title: Germanys immigration policy and labor shortages – ident: ref46 doi: 10.3390/s22031296 – ident: ref32 doi: 10.1109/IROS55552.2023.10341855 – ident: ref36 doi: 10.1016/j.compag.2020.105736 – ident: ref39 doi: 10.1787/ed758aab-en – ident: ref53 doi: 10.1109/IROS51168.2021.9636701 – year: 2020 ident: ref1 article-title: Laborotomato: Instance segmentation dataset – ident: ref51 doi: 10.1002/rob.21889 – ident: ref2 doi: 10.1109/IROS47612.2022.9981304 – ident: ref7 doi: 10.1002/rob.21987 |
| SSID | ssj0001527395 |
| Score | 2.3332186 |
| Snippet | The world population is increasing and will, by 2050, nearly double its demand for food, feed, fuel, and fiber. Besides environmental challenges, labor... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 7357 |
| SubjectTerms | agricultural automation Agricultural practices Agricultural production Agricultural robots Crop production End effectors Forestry Grasping Grasping (robotics) Harvesting Manipulators perception for grasping and manipulation Point cloud compression Robot arms Robot sensors Robotics Robotics and automation in agriculture and forestry Target detection Three-dimensional displays |
| Title | Improving Robotic Fruit Harvesting Within Cluttered Environments Through 3D Shape Completion |
| URI | https://ieeexplore.ieee.org/document/10582458 https://www.proquest.com/docview/3079381619 |
| Volume | 9 |
| WOSCitedRecordID | wos001271562400004&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 2377-3766 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001527395 issn: 2377-3766 databaseCode: RIE dateStart: 20160101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources customDbUrl: eissn: 2377-3766 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001527395 issn: 2377-3766 databaseCode: M~E dateStart: 20160101 isFulltext: true titleUrlDefault: https://road.issn.org providerName: ISSN International Centre |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEB6siOjBR61YH2UPXjxE091NN3sUbfFQRXz2IIRtMqGFkkhNPfrb3dmkWhAFb4F9EPJlZ2Z3Z74P4Filacd6ReHxIIw9qXjHrjnle0aFvhBcoHEynU99dXMTDgb6tipWd7UwiOiSz_CUHt1dfpLHMzoqsys8CLkMwhrUlFJlsdb3gQpRielgfhXp67P-3bndAHJ5KqQNvJ22yrfrcVoqPwyw8yq9zX--zxZsVOEjOy_x3oYlzOqwvkAqWIdVUtskCbcdePk6MmB3-TC3Y1hvOhsXjBSBiF7DNjyPi9E4YxcTp1iNCesulL6xh1LGh4lLdj8yr8jIfhBfd5414LHXfbi48io5BS_mmhdeO9UCgyHdpQUWOcrQUx1jw6nQ8NAGNqjQjzEWKU8S22pUkkiZDk2CRisuhdiF5SzPcA-YMCpVmCjU7Vim0h9KO7O1-sRpkFqEm3A2_9JRXHGNk-TFJHJ7Dl9HFpuIsIkqbJpw8jXiteTZ-KNvg7BY6FfC0ITDOZpRtRLfIkEMgKGNa_X-L8MOYI1mL7P6DmG5mM7wCFbi92L8Nm1B7fqj23K_2ieMdtHm |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEB584ePgW6zPPXjxEE13N93sUdSiWItofRyEsE0mWJCktKm_351NqgVR8BbYnSTky87MPub7AI5UmjZsVBQeD8LYk4o37JhTvmdU6AvBBRon0_nUUu12-PKi76pidVcLg4ju8Bme0KXby0_yeERLZXaEByGXQTgNs4GUvF6Wa30vqRCZmA7Gm5G-Pm3dn9kpIJcnwvZWTl3lO_g4NZUfLtjFlebKP99oFZarBJKdlYivwRRm67A0QSu4DvOkt0kibhvw-rVowO7zbm5tWHMw6hWMNIGIYMM2PPeKt17Gzt-dZjUm7HKi-I11SiEfJi7Yw5vpIyMPQozdebYJj83LzvmVVwkqeDHXvPDqqRYYdGk3LbDY0Rk91TA2oQoND21qgwr9GGOR8iSxrUYliZRp1yRotOJSiC2YyfIMt4EJo1KFiUJdj2Uq_a60d7Z-n1gNUotxDU7HXzqKK7ZxEr14j9ysw9eRxSYibKIKmxocf1n0S6aNP_puEhYT_UoYarA3RjOqxuIwEsQBGNrMVu_8YnYIC1ed21bUum7f7MIiPak847cHM8VghPswF38UveHgwP1wn6P_0_w |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Improving+Robotic+Fruit+Harvesting+Within+Cluttered+Environments+Through+3D+Shape+Completion&rft.jtitle=IEEE+robotics+and+automation+letters&rft.au=Magistri%2C+Federico&rft.au=Pan%2C+Yue&rft.au=Bartels%2C+Jake&rft.au=Behley%2C+Jens&rft.date=2024-08-01&rft.issn=2377-3766&rft.eissn=2377-3766&rft.volume=9&rft.issue=8&rft.spage=7357&rft.epage=7364&rft_id=info:doi/10.1109%2FLRA.2024.3421788&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_LRA_2024_3421788 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2377-3766&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2377-3766&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2377-3766&client=summon |