CLUE: Contrastive language-guided learning for referring video object segmentation

Referring video object segmentation (R-VOS), the task of separating the object described by a natural language query from the video frames, has become increasingly critical with recent advances in multi-modal understanding. Existing approaches are mainly visual-dominant in both representation-learni...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Pattern recognition letters Ročník 178; s. 115 - 121
Hlavní autori: Gao, Qiqi, Zhong, Wanjun, Li, Jie, Zhao, Tiejun
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Elsevier B.V 01.02.2024
Predmet:
Contrastive learning
Deep learning
Multi-modal
Video object segmentation
Video object segmentation
Deep learning
Multi-modal
Contrastive learning
ISSN:0167-8655, 1872-7344
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Referring video object segmentation (R-VOS), the task of separating the object described by a natural language query from the video frames, has become increasingly critical with recent advances in multi-modal understanding. Existing approaches are mainly visual-dominant in both representation-learning and decision-making process, and are less sensitive to fine-grained clues in text description. To address this, we propose a language-guided contrastive learning and data augmentation framework to enhance the model sensitivity to the fine-grained textual clues (i.e., color, location, subject) in the text that relate heavily to the video information. By substituting key information of the original sentences and paraphrasing them with a text-based generation model, our approach conducts contrastive learning through automatically building diverse and fluent contrastive samples. We further enhance the multi-modal alignment with a sparse attention mechanism, which can find the most relevant video information by optimal transport. Experiments on a large-scale R-VOS benchmark show that our method significantly improves strong Transformer-based baselines, and further analysis demonstrates the better ability of our model in identifying textual semantics. •A language-guided contrastive learning and data augmentation method for R-VOS.•A sparse attention method to enhance multi-modal alignment.•An improvement over R-VOS baselines with better identification of textual semantics.
ISSN:0167-8655
1872-7344
DOI:10.1016/j.patrec.2023.12.017