FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection

Accurate detection of obstacles in 3D is an essential task for autonomous driving and intelligent transportation. In this work, we propose a general multimodal fusion framework FusionPainting to fuse the 2D RGB image and 3D point clouds at a semantic level for boosting the 3D object detection task....

Full description

Saved in:
Bibliographic Details
Published in:2021 IEEE International Intelligent Transportation Systems Conference (ITSC) pp. 3047 - 3054
Main Authors: Xu, Shaoqing, Zhou, Dingfu, Fang, Jin, Yin, Junbo, Bin, Zhou, Zhang, Liangjun
Format: Conference Proceeding
Language:English
Published: IEEE 19.09.2021
Subjects:
Benchmark testing
Detectors
Image segmentation
Object detection
Semantics
Three-dimensional displays
Transportation
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Accurate detection of obstacles in 3D is an essential task for autonomous driving and intelligent transportation. In this work, we propose a general multimodal fusion framework FusionPainting to fuse the 2D RGB image and 3D point clouds at a semantic level for boosting the 3D object detection task. Especially, the FusionPainting framework consists of three main modules: a multi-modal semantic segmentation module, an adaptive attention-based semantic fusion module, and a 3D object detector. First, semantic information is obtained for 2D image and 3D Lidar point clouds based on 2D and 3D segmentation approaches. Then the segmentation results from different sensors are adaptively fused based on the proposed attention-based semantic fusion module. Finally, the point clouds painted with the fused semantic label are sent to the 3D detector for obtaining the 3D objection results. The effectiveness of the proposed framework has been verified on the large-scale nuScenes detection benchmark by comparing with three different baselines. The experimental results show that the fusion strategy can significantly improve the detection performance compared to the methods using only point clouds, and the methods using point clouds only painted with 2D segmentation information. Furthermore, the proposed approach outperforms other state-of-the-art methods on the nuScenes testing benchmark. Code will be available at https://github.com/Shaoqing26/FusionPainting/.
AbstractList Accurate detection of obstacles in 3D is an essential task for autonomous driving and intelligent transportation. In this work, we propose a general multimodal fusion framework FusionPainting to fuse the 2D RGB image and 3D point clouds at a semantic level for boosting the 3D object detection task. Especially, the FusionPainting framework consists of three main modules: a multi-modal semantic segmentation module, an adaptive attention-based semantic fusion module, and a 3D object detector. First, semantic information is obtained for 2D image and 3D Lidar point clouds based on 2D and 3D segmentation approaches. Then the segmentation results from different sensors are adaptively fused based on the proposed attention-based semantic fusion module. Finally, the point clouds painted with the fused semantic label are sent to the 3D detector for obtaining the 3D objection results. The effectiveness of the proposed framework has been verified on the large-scale nuScenes detection benchmark by comparing with three different baselines. The experimental results show that the fusion strategy can significantly improve the detection performance compared to the methods using only point clouds, and the methods using point clouds only painted with 2D segmentation information. Furthermore, the proposed approach outperforms other state-of-the-art methods on the nuScenes testing benchmark. Code will be available at https://github.com/Shaoqing26/FusionPainting/.
Author Xu, Shaoqing
Fang, Jin
Zhang, Liangjun
Yin, Junbo
Zhou, Dingfu
Bin, Zhou
Author_xml – sequence: 1
  givenname: Shaoqing
  surname: Xu
  fullname: Xu, Shaoqing
  email: xushaoqing@baidu.com
  organization: Beihang University,Beijing,China,100083
– sequence: 2
  givenname: Dingfu
  surname: Zhou
  fullname: Zhou, Dingfu
  email: zhoudingfu@baidu.com
  organization: Baidu Research,Robotics and Autonomous Driving Laboratory
– sequence: 3
  givenname: Jin
  surname: Fang
  fullname: Fang, Jin
  email: fangjin@baidu.com
  organization: Baidu Research,Robotics and Autonomous Driving Laboratory
– sequence: 4
  givenname: Junbo
  surname: Yin
  fullname: Yin, Junbo
  email: xsq0226@buaa.edu.cn
  organization: Baidu Research,Robotics and Autonomous Driving Laboratory
– sequence: 5
  givenname: Zhou
  surname: Bin
  fullname: Bin, Zhou
  email: binzhou@buaa.edu.cn
  organization: Beihang University,Beijing,China,100083
– sequence: 6
  givenname: Liangjun
  surname: Zhang
  fullname: Zhang, Liangjun
  email: liangjunzhang@baidu.com
  organization: Baidu Research,Robotics and Autonomous Driving Laboratory
BookMark eNotj11LwzAYhSPohZv-AkHyB1rz5qNJvCudm4PJFOf1SJq3Guna0WWK_97CdvXAcw4HzoRcdn2HhNwDywGYfVhu3itprDY5ZxxyqwppFVyQCWhuwILk8pq8zY-H2HevLnYpdp-P9OXYprjrg2vpKaK_MX3RMrh9ij9Iy5RwrI6-6QcqZnTtv7FOdIZpxOhvyFXj2gPenjklH_OnTfWcrdaLZVWussiZSJnxRSjQ1yhCHWplvWEmGGON1lazBhmiAQVeyUKidVwHCU3RyCA5QBBSTMndaTci4nY_xJ0b_rbnl-IfMLtNKA
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/ITSC48978.2021.9564951
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 1728191424
9781728191423
EndPage 3054
ExternalDocumentID 9564951
Genre orig-research
GroupedDBID 6IE
6IH
CBEJK
RIE
RIO
ID FETCH-LOGICAL-i203t-8b6d6ebce3dcdc59b808d889877970fe0ee8151b5464e9a27d41f6f4d4211d343
IEDL.DBID RIE
ISICitedReferencesCount 165
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000841862503009&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Thu Jun 29 18:37:24 EDT 2023
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i203t-8b6d6ebce3dcdc59b808d889877970fe0ee8151b5464e9a27d41f6f4d4211d343
PageCount 8
ParticipantIDs ieee_primary_9564951
PublicationCentury 2000
PublicationDate 2021-Sept.-19
PublicationDateYYYYMMDD 2021-09-19
PublicationDate_xml – month: 09
  year: 2021
  text: 2021-Sept.-19
  day: 19
PublicationDecade 2020
PublicationTitle 2021 IEEE International Intelligent Transportation Systems Conference (ITSC)
PublicationTitleAbbrev ITSC
PublicationYear 2021
Publisher IEEE
Publisher_xml – name: IEEE
Score 2.3247578
Snippet Accurate detection of obstacles in 3D is an essential task for autonomous driving and intelligent transportation. In this work, we propose a general multimodal...
SourceID ieee
SourceType Publisher
StartPage 3047
SubjectTerms Benchmark testing
Detectors
Image segmentation
Object detection
Semantics
Three-dimensional displays
Transportation
Title FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection
URI https://ieeexplore.ieee.org/document/9564951
WOSCitedRecordID wos000841862503009&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA61ePCk0opvcvDottlNdpN4K61FL7Vihd5KHhMo6La0W3-_SXapCF68hTwITCAz3-T7MgjdKSK1NZok0nHqAQq3iWYOEqcEtWB0mhoSi03wyUTM53LaQvd7LQwARPIZ9EIzvuXbldmFVFnfx_I-nvdY54BzXmu1GtFvSmT_efY2ZMKjIo_6srTXTP5VNSU6jfHx_7Y7Qd0f9R2e7v3KKWpB2UGv413Ia009kA9E5QcclbOfK6s-cD2EQ0oVD6xahxsMD6qqZjJiH5ZiOsIvOqRc8AiqyL4qu-h9_DgbPiVNOYRkmRFaJUIXtgBtgFpjTS61IMIKIQXnkhMHBEB4_61zVjCQKuOWpa5wzDIP8ixl9Ay1y1UJ5wjnlDnvmDShSrPMpSpnfrmPFAzPhAc0F6gTzLFY1z9eLBpLXP7dfYWOgsUDiyKV16hdbXZwgw7NV7Xcbm7jMX0D5NyWNA
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NSwMxEA1FBT2ptOK3OXh022ST3STeSmtpsdaKFXorm2QCBd2WuvX3m2yXiuDFW8gHgQlk5k3eyyB0mxGlrdEkUk4wD1CEjTR3ELlMMgtGU2pIWWxCjEZyOlXjGrrbamEAoCSfQTM0y7d8uzDrkCpr-Vjex_Me6-wmnMd0o9aqZL-UqNZg8trh0uMij_ti2qym_6qbUrqN3uH_NjxCjR_9HR5vPcsxqkFeRy-9dchsjT2UD1Tle1xqZz8WNnvHmyEckqq4bbNluMNwuyg2XEbsA1PMuvhZh6QL7kJR8q_yBnrrPUw6_agqiBDNY8KKSOrUpqANMGusSZSWRFoplRRCCeKAAEjvwXXCUw4qi4Xl1KWOW28sahlnJ2gnX-RwinDCuPOuSROWaR47miXcL_exghGx9JDmDNWDOWbLzZ8Xs8oS539336D9_uRpOBsORo8X6CBYP3AqqLpEO8VqDVdoz3wV88_VdXlk3-T9mXs
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2021+IEEE+International+Intelligent+Transportation+Systems+Conference+%28ITSC%29&rft.atitle=FusionPainting%3A+Multimodal+Fusion+with+Adaptive+Attention+for+3D+Object+Detection&rft.au=Xu%2C+Shaoqing&rft.au=Zhou%2C+Dingfu&rft.au=Fang%2C+Jin&rft.au=Yin%2C+Junbo&rft.date=2021-09-19&rft.pub=IEEE&rft.spage=3047&rft.epage=3054&rft_id=info:doi/10.1109%2FITSC48978.2021.9564951&rft.externalDocID=9564951