Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation

This paper presents Video K-Net, a simple, strong, and unified framework for fully end-to-end video panoptic seg-mentation. The method is built upon K-Net, a method that unifies image segmentation via a group of learnable ker-nels. We observe that these learnable kernels from K-Net, which encode obj...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 18825 - 18835
Main Authors: Li, Xiangtai, Zhang, Wenwei, Pang, Jiangmiao, Chen, Kai, Cheng, Guangliang, Tong, Yunhai, Loy, Chen Change
Format: Conference Proceeding
Language:English
Published: IEEE 01.06.2022
Subjects:
Fuses
grouping and shape analysis; Video analysis and understanding; Vision applications and systems
Image analysis
Image segmentation
Machine vision
Pipelines
Scene analysis and understanding; Segmentation
Semantics
Shape
ISSN:1063-6919
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper presents Video K-Net, a simple, strong, and unified framework for fully end-to-end video panoptic seg-mentation. The method is built upon K-Net, a method that unifies image segmentation via a group of learnable ker-nels. We observe that these learnable kernels from K-Net, which encode object appearances and contexts, can naturally associate identical instances across video frames. Motivated by this observation, Video K-Net learns to simultaneously segment and track "things" and "stuff" in a video with simple kernel-based appearance modeling and cross-temporal kernel interaction. Despite the simplicity, it achieves state-of-the-art video panoptic segmentation results on Citscapes-VPS and KITTI-STEP without bells and whistles. In particular on KITTI-STEP, the simple method can boost almost 12% relative improvements over previous methods. We also validate its generalization on video semantic segmentation, where we boost various baselines by 2% on the VSPW dataset. Moreover, we extend K-Net into clip-level video framework for video instance segmentation where we obtain 40.5% for ResNet50 backbone and 51.5% mAP for Swin-base on YouTube-2019 validation set. We hope this simple yet effective method can serve as a new flexible baseline in video segmentation. 1 1 Both code and models are released at here.
ISSN:1063-6919
DOI:10.1109/CVPR52688.2022.01828