Real-time Instance-Aware Segmentation and Semantic Mapping on Edge Devices
Perceiving the environment semantically in real-time is challenging for unmanned aerial vehicles (UAVs) with limited computational resources. In this paper, a real-time instance-aware segmentation and semantic mapping method on small edge devices is proposed. Taking RGB-D image as input, the present...
Uložené v:
| Vydané v: | IEEE transactions on instrumentation and measurement Ročník 72; s. 1 |
|---|---|
| Hlavní autori: | , , , |
| Médium: | Journal Article |
| Jazyk: | English |
| Vydavateľské údaje: |
New York
IEEE
01.01.2023
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Predmet: | |
| ISSN: | 0018-9456, 1557-9662 |
| On-line prístup: | Získať plný text |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | Perceiving the environment semantically in real-time is challenging for unmanned aerial vehicles (UAVs) with limited computational resources. In this paper, a real-time instance-aware segmentation and semantic mapping method on small edge devices is proposed. Taking RGB-D image as input, the presented instance segmentation pipeline is able to run at the speed of 38 FPS on AGX Xavier. To achieve this, we take a lightweight object detection model as backbone and reformulate the mask generation problem as threshold regression in depth by a novel designed truncation network. After that, a probability grid map is constructed to integrate the categories of voxels and object-level entities. Objects parameterized by pose, extent, category, and point cloud are tracked and fused across frames by data association. Finally, autonomous exploration experiments of UAV are conducted to demonstrate the effectiveness of the proposed method in both simulation and real-world. |
|---|---|
| AbstractList | Perceiving the environment semantically in real-time is challenging for unmanned aerial vehicles (UAVs) with limited computational resources. In this paper, a real-time instance-aware segmentation and semantic mapping method on small edge devices is proposed. Taking RGB-D image as input, the presented instance segmentation pipeline is able to run at the speed of 38 FPS on AGX Xavier. To achieve this, we take a lightweight object detection model as backbone and reformulate the mask generation problem as threshold regression in depth by a novel designed truncation network. After that, a probability grid map is constructed to integrate the categories of voxels and object-level entities. Objects parameterized by pose, extent, category, and point cloud are tracked and fused across frames by data association. Finally, autonomous exploration experiments of UAV are conducted to demonstrate the effectiveness of the proposed method in both simulation and real-world. Perceiving the environment semantically in real-time is challenging for unmanned aerial vehicles (UAVs) with limited computational resources. In this article, a real-time instance-aware segmentation and semantic mapping method on small edge devices is proposed. Taking red, green, blue, and the depth (RGB-D) image as input, the presented instance segmentation pipeline is able to run at the speed of 38 frames/s on AGX Xavier. To achieve this, we take a lightweight object detection model as the backbone and reformulate the mask generation problem as threshold regression in depth by a novel designed truncation network. After that, a probability grid map is constructed to integrate the categories of voxels and object-level entities. Objects parameterized by pose, extent, category, and point cloud are tracked and fused across frames by data association. Finally, autonomous exploration experiments of UAVs are conducted to demonstrate the effectiveness of the proposed method in both simulation and real-world. |
| Author | Shen, Hongming Lu, Junjie Zhang, Xuewei Tian, Bailing |
| Author_xml | – sequence: 1 givenname: Junjie orcidid: 0000-0002-0993-9765 surname: Lu fullname: Lu, Junjie organization: School of Electrical and Information Egineering, Tianjin University, Tianjin, China – sequence: 2 givenname: Bailing orcidid: 0000-0003-1004-8350 surname: Tian fullname: Tian, Bailing organization: School of Electrical and Information Egineering, Tianjin University, Tianjin, China – sequence: 3 givenname: Hongming orcidid: 0000-0003-0141-1904 surname: Shen fullname: Shen, Hongming organization: School of Electrical and Information Egineering, Tianjin University, Tianjin, China – sequence: 4 givenname: Xuewei orcidid: 0000-0003-4512-1335 surname: Zhang fullname: Zhang, Xuewei organization: School of Electrical and Information Egineering, Tianjin University, Tianjin, China |
| BookMark | eNp9kE1LAzEQhoNUsK3eBS8Lnrcm2Xw0x1KrVloEreeQTWdLSjdbk1Tx37tliwcPngZe5pmXeQao5xsPCF0TPCIEq7vVfDmimNJRQSnjhJ6hPuFc5koI2kN9jMk4V4yLCzSIcYsxloLJPnp-BbPLk6shm_uYjLeQT75MgOwNNjX4ZJJrfGb8ug1q45Oz2dLs985vsjafrTeQ3cOnsxAv0XlldhGuTnOI3h9mq-lTvnh5nE8ni9xSSlJuAJdcCEt5aSvMpKCVwrYNlGBMMiO5LCyUtlSsYGNcGm6YLFlluQVi1rgYotvu7j40HweISW-bQ_BtpaZSCEG4lON2C3dbNjQxBqj0PrjahG9NsD4a060xfTSmT8ZaRPxBrOv-T8G43X_gTQc6APjtUUoUApPiB6dBeVU |
| CODEN | IEIMAO |
| CitedBy_id | crossref_primary_10_1109_TRO_2023_3273180 |
| Cites_doi | 10.1109/TIM.2019.2930158 10.1109/CVPR.2019.00086 10.15607/RSS.2014.X.007 10.1109/TRO.2017.2705103 10.1109/LRA.2019.2923960 10.1109/IROS40897.2019.8968199 10.1109/ICRA.2017.7989538 10.1109/IROS.2017.8206392 10.1109/TII.2020.2981367 10.1109/IROS.2018.8593993 10.1109/ICCV.2019.00925 10.1109/TIE.2019.2892698 10.1109/IROS45743.2020.9341176 10.1109/LRA.2021.3097242 10.1109/CVPR.2019.00455 10.1109/3DV.2018.00015 10.1109/ICRA.2017.7989518 10.1109/CVPR.2013.178 10.1109/CVPR.2018.00472 10.1109/LRA.2021.3051563 10.1109/CVPR.2017.50 10.1145/504729.504754 10.1109/TSMC.1979.4310076 10.1109/ISMAR.2018.00024 10.1109/ICCV.2017.322 10.1109/ICCV.2015.259 10.1109/ICCV48922.2021.00369 10.1109/ICRA40945.2020.9196885 10.1109/TIM.2018.2834085 10.1109/TRO.2018.2853729 10.1109/TIM.2021.3102739 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023 |
| DBID | 97E RIA RIE AAYXX CITATION 7SP 7U5 8FD L7M |
| DOI | 10.1109/TIM.2022.3224512 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE/IET Electronic Library CrossRef Electronics & Communications Abstracts Solid State and Superconductivity Abstracts Technology Research Database Advanced Technologies Database with Aerospace |
| DatabaseTitle | CrossRef Solid State and Superconductivity Abstracts Technology Research Database Advanced Technologies Database with Aerospace Electronics & Communications Abstracts |
| DatabaseTitleList | Solid State and Superconductivity Abstracts |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE/IET Electronic Library (IEL) (UW System Shared) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Physics |
| EISSN | 1557-9662 |
| EndPage | 1 |
| ExternalDocumentID | 10_1109_TIM_2022_3224512 9963601 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: National Natural Science Foundation of China grantid: 62273249; 62022060; 61903349; 62073234; 62003236 funderid: 10.13039/501100001809 – fundername: Integration of Industry, Education and Research Innovation Fund of China University grantid: 2021ZYA08009 |
| GroupedDBID | -~X 0R~ 29I 4.4 5GY 6IK 85S 97E AAJGR AASAJ AAWTH ABQJQ ABVLG ACGFO ACIWK ACNCT AENEX AGQYO AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS F5P HZ~ IFIPE IPLJI JAVBF LAI M43 O9- OCL P2P RIA RIE RNS TN5 TWZ 5VS 8WZ A6W AAYXX AETIX AGSQL AI. AIBXA ALLEH CITATION EJD H~9 IAAWW IBMZZ ICLAB IDIHD IFJZH VH1 VJK 7SP 7U5 8FD AARMG ABAZT L7M |
| ID | FETCH-LOGICAL-c221t-ae0b566c25bcf04762f90c66c964474a7573cebcb943480ba5a47b4fc5ce1ad03 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 4 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000922147200044&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 0018-9456 |
| IngestDate | Mon Jun 30 10:14:06 EDT 2025 Sat Nov 29 04:38:29 EST 2025 Tue Nov 18 22:21:56 EST 2025 Tue Nov 25 14:44:28 EST 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c221t-ae0b566c25bcf04762f90c66c964474a7573cebcb943480ba5a47b4fc5ce1ad03 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0002-0993-9765 0000-0003-4512-1335 0000-0003-0141-1904 0000-0003-1004-8350 |
| PQID | 2766615778 |
| PQPubID | 85462 |
| PageCount | 1 |
| ParticipantIDs | crossref_primary_10_1109_TIM_2022_3224512 crossref_citationtrail_10_1109_TIM_2022_3224512 ieee_primary_9963601 proquest_journals_2766615778 |
| PublicationCentury | 2000 |
| PublicationDate | 2023-01-01 |
| PublicationDateYYYYMMDD | 2023-01-01 |
| PublicationDate_xml | – month: 01 year: 2023 text: 2023-01-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | New York |
| PublicationPlace_xml | – name: New York |
| PublicationTitle | IEEE transactions on instrumentation and measurement |
| PublicationTitleAbbrev | TIM |
| PublicationYear | 2023 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref13 ref12 ref34 ref14 Redmon (ref15) 2018 ref31 Qi (ref27); 30 ref30 ref11 ref33 ref10 ref32 ref2 ref1 ref17 ref16 ref19 ref18 ref24 ref23 ref26 ref25 ref20 ref21 ref28 ref29 ref8 ref7 ref9 ref4 ref3 ref6 ref5 Wang (ref22); 33 |
| References_xml | – ident: ref21 doi: 10.1109/TIM.2019.2930158 – ident: ref25 doi: 10.1109/CVPR.2019.00086 – ident: ref1 doi: 10.15607/RSS.2014.X.007 – ident: ref4 doi: 10.1109/TRO.2017.2705103 – year: 2018 ident: ref15 article-title: YOLOv3: An incremental improvement publication-title: arXiv:1804.02767 – ident: ref10 doi: 10.1109/LRA.2019.2923960 – ident: ref5 doi: 10.1109/IROS40897.2019.8968199 – ident: ref7 doi: 10.1109/ICRA.2017.7989538 – ident: ref11 doi: 10.1109/IROS.2017.8206392 – ident: ref30 doi: 10.1109/TII.2020.2981367 – ident: ref8 doi: 10.1109/IROS.2018.8593993 – ident: ref23 doi: 10.1109/ICCV.2019.00925 – ident: ref29 doi: 10.1109/TIE.2019.2892698 – ident: ref2 doi: 10.1109/IROS45743.2020.9341176 – ident: ref34 doi: 10.1109/LRA.2021.3097242 – ident: ref17 doi: 10.1109/CVPR.2019.00455 – ident: ref33 doi: 10.1109/3DV.2018.00015 – ident: ref12 doi: 10.1109/ICRA.2017.7989518 – volume: 33 start-page: 17721 volume-title: Proc. Adv. Neural Inf. Process. Syst. ident: ref22 article-title: SOLOv2: Dynamic and fast instance segmentation – ident: ref20 doi: 10.1109/CVPR.2013.178 – ident: ref24 doi: 10.1109/CVPR.2018.00472 – ident: ref6 doi: 10.1109/LRA.2021.3051563 – ident: ref26 doi: 10.1109/CVPR.2017.50 – ident: ref32 doi: 10.1145/504729.504754 – ident: ref31 doi: 10.1109/TSMC.1979.4310076 – ident: ref13 doi: 10.1109/ISMAR.2018.00024 – ident: ref14 doi: 10.1109/ICCV.2017.322 – ident: ref16 doi: 10.1109/ICCV.2015.259 – ident: ref18 doi: 10.1109/ICCV48922.2021.00369 – ident: ref9 doi: 10.1109/ICRA40945.2020.9196885 – ident: ref19 doi: 10.1109/TIM.2018.2834085 – volume: 30 start-page: 1 volume-title: Proc. Adv. Neural Inf. Process. Syst. ident: ref27 article-title: PointNet: Deep hierarchical feature learning on point sets in a metric space – ident: ref3 doi: 10.1109/TRO.2018.2853729 – ident: ref28 doi: 10.1109/TIM.2021.3102739 |
| SSID | ssj0007647 |
| Score | 2.3874478 |
| Snippet | Perceiving the environment semantically in real-time is challenging for unmanned aerial vehicles (UAVs) with limited computational resources. In this paper, a... Perceiving the environment semantically in real-time is challenging for unmanned aerial vehicles (UAVs) with limited computational resources. In this article,... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 1 |
| SubjectTerms | autonomous exploration Feature extraction Image edge detection Image segmentation instance segmentation Mapping Object recognition Real time Real-time systems Semantic mapping Semantics Simultaneous localization and mapping Statistical analysis Three-dimensional displays Unmanned aerial vehicles |
| Title | Real-time Instance-Aware Segmentation and Semantic Mapping on Edge Devices |
| URI | https://ieeexplore.ieee.org/document/9963601 https://www.proquest.com/docview/2766615778 |
| Volume | 72 |
| WOSCitedRecordID | wos000922147200044&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE/IET Electronic Library (IEL) (UW System Shared) customDbUrl: eissn: 1557-9662 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0007647 issn: 0018-9456 databaseCode: RIE dateStart: 19630101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3dbxMxDLe2CaTtgUG7aYWB7mEvSKS9pMnl8jhBK4rWCUGZ9nZKcm5Vab2ifsC_j5O7VqAhJN5OVqKc7MT-OY5tgCvu8lIbnjNLA5h0qmQGM8Ukt6l12igRs_jvbvTtbX5_bz4fwLt9Lgwixsdn2A2fMZZfLv02XJX1CJv3s5Csdah1Vudq7bWuzmRdH5PWNYQKdiHJ1PQmozE5gkJ0afNKxcUfJij2VHmkiKN1GZ7-3389h2cNikyua7G_gAOsWnDyW23BFjyNbzv9ug2fvhAYZKGJfDKKYNAju_5pV5h8xdmiyT2qEluVRFgQp-c-GdtQt2GWEH1QzjD5gFGjnMG34WDy_iNrWigwLwTfMIupI8DmhXJ-mkrSfFOTeiIYwkFaWq1036PzLpSJy1NnlZXayalXHrkt0_45HFXLCi8gEaXmSB4bISwuNYYAKi1RkhWcosr6ogO9HVcL39QXD20uHoroZ6SmIDkUQQ5FI4cOvN3P-F7X1vjH2Hbg-35cw_IOXO4EVzSHb10ITT4ZV1rnL_8-6xUch67x9U3KJRxtVlt8DU_8j818vXoT99Uvu3nH8w |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3dTxsxDLcQ2wQ87IMPUT62e9jLJEKTNGkuj2gD0a2tpq0g3k5Jzq2Q4EBt2f79OblrtWkIibeTlSgnO7F_jmMb4KPweWmsyJmjAUx5XTKLXc2UcNx5Y7VMWfyXfTMc5ldX9vsKHC1zYRAxPT7D4_iZYvnlXXiIV2VtwuadbkzWeqGVkrzO1lrqXdNVdYVMWtkSLlgEJbltj3oDcgWlPKbtq7SQ_xih1FXlP1Wc7MvZm-f92Vt43eDI7KQW_DtYwWoTNv6qLrgJr9LrzjDbgq8_CA6y2EY-6yU4GJCd_HZTzH7i5LbJPqoyV5VEuCVeX4ds4GLlhklG9NNygtkXTDplGy7OTkefz1nTRIEFKcWcOeSeIFuQ2ocxV6T7xpYHIlhCQkY5o00noA8-ForLuXfaKePVOOiAwpW8swOr1V2Fu5DJ0ggkn40wllAGYwiVlijJDo5RdzuyBe0FV4vQVBiPjS5uiuRpcFuQHIooh6KRQws-LWfc19U1nhi7Ffm-HNewvAUHC8EVzfGbFdKQVya0Mfne47M-wNr5aNAv-r3ht31Yjz3k63uVA1idTx_wEF6GX_Pr2fR92mN_AE_4yzo |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Real-Time+Instance-Aware+Segmentation+and+Semantic+Mapping+on+Edge+Devices&rft.jtitle=IEEE+transactions+on+instrumentation+and+measurement&rft.au=Lu%2C+Junjie&rft.au=Tian%2C+Bailing&rft.au=Shen%2C+Hongming&rft.au=Zhang%2C+Xuewei&rft.date=2023-01-01&rft.pub=The+Institute+of+Electrical+and+Electronics+Engineers%2C+Inc.+%28IEEE%29&rft.issn=0018-9456&rft.eissn=1557-9662&rft.volume=72&rft.spage=1&rft_id=info:doi/10.1109%2FTIM.2022.3224512&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9456&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9456&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9456&client=summon |