Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation

Vision Transformers (ViTs) have emerged with superior performance on computer vision tasks compared to the convolutional neural network (CNN)-based models. However, ViTs mainly designed for image classification will generate single-scale low-resolution representations, which makes dense prediction t...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) s. 12084 - 12093
Hlavní autori: Gu, Jiaqi, Kwon, Hyoukjun, Wang, Dilin, Ye, Wei, Li, Meng, Chen, Yu-Hsin, Lai, Liangzhen, Chandra, Vikas, Pan, David Z.
Médium: Konferenčný príspevok..
Jazyk:English
Vydavateľské údaje: IEEE 01.01.2022
Predmet:
Architecture
Computer architecture
Computer vision
Deep learning architectures and techniques; Efficient learning and inferences; Representation learning; Segmentation
grouping and shape analysis
Object detection
Redundancy
Representation learning
Semantics
ISSN:1063-6919
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Vision Transformers (ViTs) have emerged with superior performance on computer vision tasks compared to the convolutional neural network (CNN)-based models. However, ViTs mainly designed for image classification will generate single-scale low-resolution representations, which makes dense prediction tasks such as semantic segmentation challenging for ViTs. Therefore, we propose HRViT, which enhances ViTs to learn semantically-rich and spatially-precise multi-scale representations by integrating high-resolution multi-branch architectures with ViTs. We balance the model performance and efficiency of HRViT by various branch-block co-optimization techniques. Specifically, we explore heterogeneous branch designs, reduce the redundancy in linear layers, and augment the attention block with enhanced expressiveness. Those approaches enabled HRViT to push the Pareto frontier of performance and efficiency on semantic segmentation to a new level, as our evaluation results on ADE20K and Cityscapes show. HRViT achieves 50.20% mIoU on ADE20K and 83.16% mIoU on Cityscapes, surpassing state-of-the-art MiT and CSWin backbones with an average of +1.78 mIoU improvement, 28% parameter saving, and 21% FLOPs reduction, demonstrating the potential of HRViT as a strong vision backbone for semantic segmentation. Our code is publicly available 1 1 https://github.com/facebookresearch/HRViT.
ISSN:1063-6919
DOI:10.1109/CVPR52688.2022.01178