Learning Occluded Branch Depth Maps in Forest Environments Using RGB-D Images
Covering over a third of all terrestrial land area, forests are crucial environments; as ecosystems, for farming, and for human leisure. However, they are challenging to access for environmental monitoring, for agricultural uses, and for search and rescue applications. To enter, aerial robots need t...
Gespeichert in:
| Veröffentlicht in: | IEEE robotics and automation letters Jg. 9; H. 3; S. 2439 - 2446 |
|---|---|
| Hauptverfasser: | , , , , , , |
| Format: | Journal Article |
| Sprache: | Englisch |
| Veröffentlicht: |
Piscataway
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
01.03.2024
|
| Schlagworte: | |
| ISSN: | 2377-3766, 2377-3766 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | Covering over a third of all terrestrial land area, forests are crucial environments; as ecosystems, for farming, and for human leisure. However, they are challenging to access for environmental monitoring, for agricultural uses, and for search and rescue applications. To enter, aerial robots need to fly through dense vegetation, where foliage can be pushed aside, but occluded branches pose critical obstacles. Therefore, we propose pixel-wise depth regression of occluded branches using three different U-Net inspired architectures. Given RGB-D input of trees with partially occluded branches, the models estimate depth values of only the wooden parts of the tree. A large photorealistic simulation dataset comprising around 44 K images of nine different tree species is generated, on which the models are trained. Extensive evaluation and analysis of the models on this dataset is shown. To improve network generalization to real-world data, different data augmentation and transformation techniques are performed. The approaches are then also successfully demonstrated on real-world data of broadleaf trees from Swiss temperate forests and a tropical Masoala Rainforest. This work showcases the previously unexplored task of frame-by-frame pixel-based occluded branch depth reconstruction to facilitate robot traversal of forest environments. |
|---|---|
| AbstractList | Covering over a third of all terrestrial land area, forests are crucial environments; as ecosystems, for farming, and for human leisure. However, they are challenging to access for environmental monitoring, for agricultural uses, and for search and rescue applications. To enter, aerial robots need to fly through dense vegetation, where foliage can be pushed aside, but occluded branches pose critical obstacles. Therefore, we propose pixel-wise depth regression of occluded branches using three different U-Net inspired architectures. Given RGB-D input of trees with partially occluded branches, the models estimate depth values of only the wooden parts of the tree. A large photorealistic simulation dataset comprising around 44 K images of nine different tree species is generated, on which the models are trained. Extensive evaluation and analysis of the models on this dataset is shown. To improve network generalization to real-world data, different data augmentation and transformation techniques are performed. The approaches are then also successfully demonstrated on real-world data of broadleaf trees from Swiss temperate forests and a tropical Masoala Rainforest. This work showcases the previously unexplored task of frame-by-frame pixel-based occluded branch depth reconstruction to facilitate robot traversal of forest environments. |
| Author | Schnider, Yannick von Bassewitz, Jan-Philipp Mintchev, Stefano Aucone, Emanuele Geckeler, Christian Simeon, Andri Zhu, Yunying |
| Author_xml | – sequence: 1 givenname: Christian orcidid: 0000-0001-7680-7730 surname: Geckeler fullname: Geckeler, Christian organization: Environmental Robotics Laboratory, Department of Environmental Systems Science, ETH Zurich, Zurich, Switzerland – sequence: 2 givenname: Emanuele orcidid: 0000-0003-2956-0047 surname: Aucone fullname: Aucone, Emanuele organization: Environmental Robotics Laboratory, Department of Environmental Systems Science, ETH Zurich, Zurich, Switzerland – sequence: 3 givenname: Yannick orcidid: 0009-0007-7755-6436 surname: Schnider fullname: Schnider, Yannick organization: Environmental Robotics Laboratory, Department of Environmental Systems Science, ETH Zurich, Zurich, Switzerland – sequence: 4 givenname: Andri orcidid: 0009-0008-9262-2304 surname: Simeon fullname: Simeon, Andri organization: Environmental Robotics Laboratory, Department of Environmental Systems Science, ETH Zurich, Zurich, Switzerland – sequence: 5 givenname: Jan-Philipp orcidid: 0009-0002-1652-5592 surname: von Bassewitz fullname: von Bassewitz, Jan-Philipp organization: Environmental Robotics Laboratory, Department of Environmental Systems Science, ETH Zurich, Zurich, Switzerland – sequence: 6 givenname: Yunying orcidid: 0009-0008-8198-0982 surname: Zhu fullname: Zhu, Yunying organization: Environmental Robotics Laboratory, Department of Environmental Systems Science, ETH Zurich, Zurich, Switzerland – sequence: 7 givenname: Stefano orcidid: 0000-0001-6272-0212 surname: Mintchev fullname: Mintchev, Stefano organization: Environmental Robotics Laboratory, Department of Environmental Systems Science, ETH Zurich, Zurich, Switzerland |
| BookMark | eNp1kEtrAjEURkOxUGvddxnoemxe81r6bmFEkLoOmSSjEU2mSSz033dEF6XQ1b2L73z3ch5BzzqrAXjGaIQxKl-rzXhEEGEjStM0o-QO9AnN84TmWdb7tT-AYQgHhBBOSU7LtA9WlRbeGruDaymPZ6UVnHhh5R7OdBv3cCXaAI2FC-d1iHBuv4x39qRtDHAbLtxmOUlm8P0kdjo8gftGHIMe3uYAbBfzj-lbUq2X79NxlUiKaUxqQaRSmDGmS4E1EUQx2VCE66ZuGJOSNZnO8wIXEuWKKF3UgmmcYsS6eCnpALxce1vvPs_dY_zgzt52JzkpCaIFKzPSpbJrSnoXgtcNlyaKaJyNXpgjx4hf7PHOHr_Y4zd7HYj-gK03J-G__0d-AIIUclo |
| CitedBy_id | crossref_primary_10_1109_TMECH_2024_3410167 crossref_primary_10_1109_TRO_2024_3508140 crossref_primary_10_1109_ACCESS_2025_3604842 crossref_primary_10_1109_LRA_2025_3583603 crossref_primary_10_3390_rs17040717 |
| Cites_doi | 10.1139/juvs-2020-0005 10.1109/ICRA.2019.8793576 10.1126/scirobotics.abg5810 10.1109/lra.2024.3355632/mm1 10.1109/LRA.2020.2974710 10.1109/ICRA48506.2021.9561675 10.4060/ca8642en 10.1109/3DV.2016.32 10.1109/LRA.2022.3154047 10.1109/ICCV.2015.304 10.1109/TPAMI.2020.3013679 10.1109/ICIP.2019.8803146 10.1109/URAI.2019.8768489 10.1007/978-3-319-24574-4_28 10.1126/scirobotics.abm5954 10.1109/LRA.2017.2775699 10.1126/scirobotics.add5762 10.1007/978-3-642-33715-4_54 10.1126/scirobotics.abg1188 10.1007/978-3-030-01252-6_9 10.1007/s11119-021-09813-y 10.1145/3123266.3129396 10.1002/fee.2420 10.1002/aisy.202200087 10.1016/j.compag.2020.105952 10.1109/ICUAS54217.2022.9836115 10.1109/ICRA48891.2023.10160650 10.1016/j.displa.2021.102080 10.1007/s10531-017-1453-2 10.1038/srep29987 10.1007/978-3-319-10584-0_23 10.3389/frai.2022.830026 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024 |
| DBID | AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| DOI | 10.1109/LRA.2024.3355632 |
| DatabaseName | CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Technology Research Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 2377-3766 |
| EndPage | 2446 |
| ExternalDocumentID | 10_1109_LRA_2024_3355632 |
| GroupedDBID | 0R~ 97E AAJGR AASAJ AAWTH AAYXX ABQJQ ABVLG ACGFS AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CITATION EBS EJD IFIPE IPLJI JAVBF KQ8 M43 M~E O9- OCL RIA RIE 7SC 7SP 8FD AARMG ABAZT JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c313t-ba2cdd1444e9a1e2a2d4cf301bfbf44cc4f6e77818c07d2de8ba4e15104a1e9c3 |
| ISICitedReferencesCount | 6 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001167554600002&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 2377-3766 |
| IngestDate | Sun Jun 29 16:54:02 EDT 2025 Sat Nov 29 06:03:29 EST 2025 Tue Nov 18 21:33:22 EST 2025 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 3 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-c313t-ba2cdd1444e9a1e2a2d4cf301bfbf44cc4f6e77818c07d2de8ba4e15104a1e9c3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0001-6272-0212 0009-0008-8198-0982 0000-0003-2956-0047 0009-0007-7755-6436 0009-0002-1652-5592 0000-0001-7680-7730 0009-0008-9262-2304 |
| OpenAccessLink | http://hdl.handle.net/20.500.11850/659145 |
| PQID | 2920384962 |
| PQPubID | 4437225 |
| PageCount | 8 |
| ParticipantIDs | proquest_journals_2920384962 crossref_citationtrail_10_1109_LRA_2024_3355632 crossref_primary_10_1109_LRA_2024_3355632 |
| PublicationCentury | 2000 |
| PublicationDate | 2024-03-01 |
| PublicationDateYYYYMMDD | 2024-03-01 |
| PublicationDate_xml | – month: 03 year: 2024 text: 2024-03-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | Piscataway |
| PublicationPlace_xml | – name: Piscataway |
| PublicationTitle | IEEE robotics and automation letters |
| PublicationYear | 2024 |
| Publisher | The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref35 ref12 ref14 ref36 ref31 ref30 ref33 Chen (ref34) (ref38) 2024 ref10 ref32 ref2 ref1 ref17 ref39 ref16 ref19 Eigen (ref23) 2014; 27 ref18 Aucone (ref15) 2023 (ref37) 2024 ref24 Csurka (ref43) 2021 ref26 ref25 ref20 ref41 ref22 ref44 ref21 ref28 ref27 ref8 ref7 Hamaza (ref11) 2020 ref9 ref4 Jiang (ref29) 2018 ref3 ref6 ref5 Kingma (ref42) 2017 ref40 (ref13) 2023 |
| References_xml | – ident: ref8 doi: 10.1139/juvs-2020-0005 – ident: ref22 doi: 10.1109/ICRA.2019.8793576 – ident: ref32 doi: 10.1126/scirobotics.abg5810 – ident: ref1 doi: 10.1109/lra.2024.3355632/mm1 – ident: ref17 doi: 10.1109/LRA.2020.2974710 – year: 2017 ident: ref42 article-title: Adam: A method for stochastic optimization – ident: ref35 doi: 10.1109/ICRA48506.2021.9561675 – ident: ref2 doi: 10.4060/ca8642en – ident: ref25 doi: 10.1109/3DV.2016.32 – ident: ref7 doi: 10.1109/LRA.2022.3154047 – ident: ref31 doi: 10.1109/ICRA48506.2021.9561675 – ident: ref24 doi: 10.1109/ICCV.2015.304 – ident: ref44 doi: 10.1109/TPAMI.2020.3013679 – year: 2020 ident: ref11 article-title: Sensor delivery in forests with aerial robots: A new paradigm for environmental monitoring publication-title: IEEE IROS Workshop Perception, Planning Mobility Forestry Robot. – ident: ref28 doi: 10.1109/ICIP.2019.8803146 – ident: ref40 doi: 10.1109/URAI.2019.8768489 – ident: ref36 doi: 10.1007/978-3-319-24574-4_28 – ident: ref6 doi: 10.1126/scirobotics.abm5954 – ident: ref18 doi: 10.1007/978-3-319-24574-4_28 – year: 2024 ident: ref37 article-title: SpeedTree – ident: ref16 doi: 10.1109/LRA.2017.2775699 – year: 2021 ident: ref43 article-title: Unsupervised domain adaptation for semantic image segmentation: A comprehensive survey – ident: ref10 doi: 10.1126/scirobotics.add5762 – ident: ref41 doi: 10.1007/978-3-642-33715-4_54 – ident: ref14 doi: 10.1126/scirobotics.abg1188 – ident: ref26 doi: 10.1007/978-3-030-01252-6_9 – year: 2018 ident: ref29 article-title: Rednet: Residual encoder-decoder network for indoor RGB-D semantic segmentation – ident: ref12 doi: 10.1007/s11119-021-09813-y – ident: ref39 doi: 10.1145/3123266.3129396 – ident: ref3 doi: 10.1002/fee.2420 – ident: ref9 doi: 10.1002/aisy.202200087 – ident: ref21 doi: 10.1016/j.compag.2020.105952 – ident: ref33 doi: 10.1109/ICUAS54217.2022.9836115 – year: 2023 ident: ref13 article-title: Tevel – ident: ref19 doi: 10.1109/ICRA48891.2023.10160650 – ident: ref27 doi: 10.1016/j.displa.2021.102080 – ident: ref4 doi: 10.1007/s10531-017-1453-2 – ident: ref5 doi: 10.1038/srep29987 – ident: ref30 doi: 10.1007/978-3-319-10584-0_23 – volume: 27 start-page: 2366 volume-title: Proc. Adv. Neural Inf. Process. Syst. year: 2014 ident: ref23 article-title: Depth map prediction from a single image using a multi-scale deep network – year: 2024 ident: ref38 article-title: Unreal engine – year: 2023 ident: ref15 article-title: Synergistic morphology and feedback control for traversal of unknown compliant obstacles with aerial robots synergistic morphology and feedback control for traversal of unknown compliant obstacles with aerial robots – ident: ref20 doi: 10.3389/frai.2022.830026 – start-page: 2021 ident: ref34 article-title: TransUNet: Transformers make strong encoders for medical image segmentation |
| SSID | ssj0001527395 |
| Score | 2.2893317 |
| Snippet | Covering over a third of all terrestrial land area, forests are crucial environments; as ecosystems, for farming, and for human leisure. However, they are... |
| SourceID | proquest crossref |
| SourceType | Aggregation Database Enrichment Source Index Database |
| StartPage | 2439 |
| SubjectTerms | Data augmentation Datasets Environmental monitoring Foliage Forests Image reconstruction Pixels Rainforests Robots Trees |
| Title | Learning Occluded Branch Depth Maps in Forest Environments Using RGB-D Images |
| URI | https://www.proquest.com/docview/2920384962 |
| Volume | 9 |
| WOSCitedRecordID | wos001167554600002&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 2377-3766 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001527395 issn: 2377-3766 databaseCode: RIE dateStart: 20160101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources (ISSN International Center) customDbUrl: eissn: 2377-3766 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001527395 issn: 2377-3766 databaseCode: M~E dateStart: 20160101 isFulltext: true titleUrlDefault: https://road.issn.org providerName: ISSN International Centre |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1Lj9MwELbKwgEOiKf2BfIBIaEq3dRx4vq4C9kFqe2i3a5UTpHjOGqlNi1tutoTf4-_xdiJ3RSkFRy4RFUefXi-jmfGnu9D6F3aDYQUing5D4VeZux5QkXSy5QvCcxokDlXYhNsOOyNx_xrq_XT9sLczlhR9O7u-PK_mhrOgbF16-w_mNu9KZyA12B0OILZ4fhXhu_bYsellLNNBgHlmRbPmIBrWZaT9kAszRZYrcm5Lttxs9Gt2j9wdXHmfWp_mYOnWTdjV50XtleLdOGYncWmXFStj-2ZaQtyAfqFAu9gmwwntSNx4NpAEm4KqfFcFBs1c-C61oSyWfXYN62mJF0j0fV0rmr96yJbTZvVCkK327U6tt9tZxdEbMR-HDFC7LR_1o6Q0XlLE3PDT7U1EmXcJAmYcZNR06fzBnSDpn-mFXXSnxOH4V3tX5129LfuBIEmTiPbSdJuDBheJuc3_X4yisej98vvnpYv08v8tZbLA_SQsJBr9zr40Sj2aZo7Htplcp-f_P5Ju2HRblRgQp3RM_S0zlHwaYWt56ilihfoSYO58iUaWJRhizJcoQwblGGNMjwtcIUy3EQZNijDBmW4QtkrdHMejz5-9mplDk8G3aD0UkFklkEuThUXXUUEyajMYa5I8zSnVEqaR4oxCAalzzKSqV4qqILg0qdwO5fBa7RXANL2EWaQkfjdXFK4RgmLeMgyJqI8FCpQIVUH6MQOTCJr2nqtnjJLTPrq8wSGMtFDmdRDeYA-uCeWFWXLPfce27FO6v_wOtECbkGP8ogc3n_5CD3eYvwY7ZWrjXqDHsnbcrpevTUQ-AW8mpd5 |
| linkProvider | ISSN International Centre |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Learning+Occluded+Branch+Depth+Maps+in+Forest+Environments+Using+RGB-D+Images&rft.jtitle=IEEE+robotics+and+automation+letters&rft.au=Geckeler%2C+Christian&rft.au=Aucone%2C+Emanuele&rft.au=Schnider%2C+Yannick&rft.au=Simeon%2C+Andri&rft.date=2024-03-01&rft.pub=The+Institute+of+Electrical+and+Electronics+Engineers%2C+Inc.+%28IEEE%29&rft.eissn=2377-3766&rft.volume=9&rft.issue=3&rft.spage=2439&rft_id=info:doi/10.1109%2FLRA.2024.3355632&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2377-3766&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2377-3766&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2377-3766&client=summon |