Multimodal Token Fusion for Vision Transformers

Many adaptations of transformers have emerged to address the single-modal vision tasks, where self-attention modules are stacked to handle input sources like images. Intuitively, feeding multiple modalities of data to vision transformers could improve the performance, yet the innermodal attentive we...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 12176 - 12185
Main Authors: Wang, Yikai, Chen, Xinghao, Cao, Lele, Huang, Wenbing, Sun, Fuchun, Wang, Yunhe
Format: Conference Proceeding
Language:English
Published: IEEE 01.06.2022
Subjects:
categorization
Computer architecture
Deep learning architectures and techniques; Recognition: detection
grouping and shape analysis; Vision + X
Image segmentation
Object detection
Point cloud compression
retrieval; Segmentation
Semantics
Shape
Three-dimensional displays
ISSN:1063-6919
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Many adaptations of transformers have emerged to address the single-modal vision tasks, where self-attention modules are stacked to handle input sources like images. Intuitively, feeding multiple modalities of data to vision transformers could improve the performance, yet the innermodal attentive weights may be diluted, which could thus greatly undermine the final performance. In this paper, we propose a multimodal token fusion method (TokenFusion), tailored for transformer-based vision tasks. To effectively fuse multiple modalities, TokenFusion dynamically detects uninformative tokens and substitute these tokens with projected and aggregated inter-modal features. Residual positional alignment is also adopted to enable explicit utilization of the inter-modal alignments after fusion. The design of TokenFusion allows the transformer to learn correlations among multimodal features, while the single-modal transformer architecture remains largely intact. Extensive experiments are conducted on a variety of homogeneous and heterogeneous modalities and demonstrate that TokenFusion surpasses state-of-the-art methods in three typical vision tasks: multimodal image-to-image translation, RGB-depth semantic segmentation, and 3D object detection with point cloud and images. Code will be released 1 1 https://github.com/huawei-noah/noah-research 2 2 https://gitee.com/mindspore/models/tree/master/research/cv/TokenFusion.
ISSN:1063-6919
DOI:10.1109/CVPR52688.2022.01187