CLUE: Contrastive language-guided learning for referring video object segmentation - Gao, Qiqi; Zhong, Wanjun; Li, Jie; Zhao, Tiejun (Pattern recognition letters, 2024)
Referring video object segmentation (R-VOS), the task of separating the object described by a natural language query from the video frames, has become increasingly critical with recent advances in multi-modal understanding. Existing approaches are mainly visual-dominant in both representation-learning and decision-making process, and are less sensitive to fine-grained clues in text description. To address this, we propose a language-guided contrastive learning and data augmentation framework to enhance the model sensitivity to the fine-grained textual clues (i.e., color, location, subject) in the text that relate heavily to the video information. By substituting key information of the original sentences and paraphrasing them with a text-based generation model, our approach conducts contrastive learning through automatically building diverse and fluent contrastive samples. We further enhance the multi-modal alignment with a sparse attention mechanism, which can find the most relevant video information by optimal transport. Experiments on a large-scale R-VOS benchmark show that our method significantly improves strong Transformer-based baselines, and further analysis demonstrates the better ability of our model in identifying textual semantics. •A language-guided contrastive learning and data augmentation method for R-VOS.•A sparse attention method to enhance multi-modal alignment.•An improvement over R-VOS baselines with better identification of textual semantics.