CLIP-Adapter: Better Vision-Language Models with Feature Adapters

Large-scale contrastive vision-language pretraining has shown significant progress in visual representation learning. Unlike traditional visual systems trained by a fixed set of discrete labels, a new paradigm was introduced in Radford et al. (International conference on machine learning, PMLR, 2021...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:International journal of computer vision Ročník 132; číslo 2; s. 581 - 595
Hlavní autori: Gao, Peng, Geng, Shijie, Zhang, Renrui, Ma, Teli, Fang, Rongyao, Zhang, Yongfeng, Li, Hongsheng, Qiao, Yu
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: New York Springer US 01.02.2024
Springer
Springer Nature B.V
Predmet:
Ablation
Adapters
Artificial Intelligence
Classification
Computational linguistics
Computer Imaging
Computer Science
Context
Datasets
Design
Design optimization
Image Processing and Computer Vision
Innovations
Language
Language processing
Machine learning
Natural language
Natural language interfaces
Optimization
Pattern Recognition
Pattern Recognition and Graphics
Vision
Visual tasks
Vision-language model
Few-shot learning
Open-vocabulary
Feature adapter
ISSN:0920-5691, 1573-1405
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Large-scale contrastive vision-language pretraining has shown significant progress in visual representation learning. Unlike traditional visual systems trained by a fixed set of discrete labels, a new paradigm was introduced in Radford et al. (International conference on machine learning, PMLR, 2021) to directly learn to align images with raw texts in an open-vocabulary setting. On downstream tasks, a carefully chosen text prompt is employed to make zero-shot predictions. To avoid non-trivial prompt engineering, context optimization (Zhou et al. in Int J Comput Vis 130(9):2337–2348, 2022) has been proposed to learn continuous vectors as task-specific prompts with few-shot training examples. In this paper, we show that there is an alternative path to achieve better vision-language models other than prompt tuning. While prompt tuning is for the textual inputs, we propose CLIP-Adapter to conduct fine-tuning with feature adapters on either visual or language branch. Specifically, CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending with the original pretrained features. As a consequence, CLIP-Adapter is able to outperform context optimization while maintaining a simple design. Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0920-5691
1573-1405
DOI:10.1007/s11263-023-01891-x