Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation

Vision Transformers (ViTs) have emerged with superior performance on computer vision tasks compared to the convolutional neural network (CNN)-based models. However, ViTs mainly designed for image classification will generate single-scale low-resolution representations, which makes dense prediction t...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) s. 12084 - 12093
Hlavní autoři: Gu, Jiaqi, Kwon, Hyoukjun, Wang, Dilin, Ye, Wei, Li, Meng, Chen, Yu-Hsin, Lai, Liangzhen, Chandra, Vikas, Pan, David Z.
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 01.01.2022
Témata:
Architecture
Computer architecture
Computer vision
Deep learning architectures and techniques; Efficient learning and inferences; Representation learning; Segmentation
grouping and shape analysis
Object detection
Redundancy
Representation learning
Semantics
ISSN:1063-6919
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Vision Transformers (ViTs) have emerged with superior performance on computer vision tasks compared to the convolutional neural network (CNN)-based models. However, ViTs mainly designed for image classification will generate single-scale low-resolution representations, which makes dense prediction tasks such as semantic segmentation challenging for ViTs. Therefore, we propose HRViT, which enhances ViTs to learn semantically-rich and spatially-precise multi-scale representations by integrating high-resolution multi-branch architectures with ViTs. We balance the model performance and efficiency of HRViT by various branch-block co-optimization techniques. Specifically, we explore heterogeneous branch designs, reduce the redundancy in linear layers, and augment the attention block with enhanced expressiveness. Those approaches enabled HRViT to push the Pareto frontier of performance and efficiency on semantic segmentation to a new level, as our evaluation results on ADE20K and Cityscapes show. HRViT achieves 50.20% mIoU on ADE20K and 83.16% mIoU on Cityscapes, surpassing state-of-the-art MiT and CSWin backbones with an average of +1.78 mIoU improvement, 28% parameter saving, and 21% FLOPs reduction, demonstrating the potential of HRViT as a strong vision backbone for semantic segmentation. Our code is publicly available 1 1 https://github.com/facebookresearch/HRViT.
ISSN:1063-6919
DOI:10.1109/CVPR52688.2022.01178