CLUE: Contrastive language-guided learning for referring video object segmentation

Referring video object segmentation (R-VOS), the task of separating the object described by a natural language query from the video frames, has become increasingly critical with recent advances in multi-modal understanding. Existing approaches are mainly visual-dominant in both representation-learni...

Full description

Saved in:
Bibliographic Details
Published in:Pattern recognition letters Vol. 178; pp. 115 - 121
Main Authors: Gao, Qiqi, Zhong, Wanjun, Li, Jie, Zhao, Tiejun
Format: Journal Article
Language:English
Published: Elsevier B.V 01.02.2024
Subjects:
Contrastive learning
Deep learning
Multi-modal
Video object segmentation
Video object segmentation
Deep learning
Multi-modal
Contrastive learning
ISSN:0167-8655, 1872-7344
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Referring video object segmentation (R-VOS), the task of separating the object described by a natural language query from the video frames, has become increasingly critical with recent advances in multi-modal understanding. Existing approaches are mainly visual-dominant in both representation-learning and decision-making process, and are less sensitive to fine-grained clues in text description. To address this, we propose a language-guided contrastive learning and data augmentation framework to enhance the model sensitivity to the fine-grained textual clues (i.e., color, location, subject) in the text that relate heavily to the video information. By substituting key information of the original sentences and paraphrasing them with a text-based generation model, our approach conducts contrastive learning through automatically building diverse and fluent contrastive samples. We further enhance the multi-modal alignment with a sparse attention mechanism, which can find the most relevant video information by optimal transport. Experiments on a large-scale R-VOS benchmark show that our method significantly improves strong Transformer-based baselines, and further analysis demonstrates the better ability of our model in identifying textual semantics. •A language-guided contrastive learning and data augmentation method for R-VOS.•A sparse attention method to enhance multi-modal alignment.•An improvement over R-VOS baselines with better identification of textual semantics.
ISSN:0167-8655
1872-7344
DOI:10.1016/j.patrec.2023.12.017