Configurable Embodied Data Generation for Class-Agnostic RGB-D Video Segmentation

This letter presents a method for generating large-scale datasets to improve class-agnostic video segmentation across robots with different form factors. Specifically, we consider the question of whether video segmentation models trained on generic segmentation data could be more effective for parti...

Full description

Saved in:
Bibliographic Details
Published in:IEEE robotics and automation letters Vol. 9; no. 12; pp. 11409 - 11416
Main Authors: Opipari, Anthony, Krishnan, Aravindhan K, Gayaka, Shreekant, Sun, Min, Kuo, Cheng-Hao, Sen, Arnie, Jenkins, Odest Chadwicke
Format: Journal Article
Language:English
Published: Piscataway IEEE 01.12.2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Benchmark testing
Cameras
Data collection
data sets for robotic vision
Datasets
Form factors
Image segmentation
Motion segmentation
Object detection
RGB-D perception
Robot vision systems
Robots
segmentation and categorization
Semantics
Sensor placement
Three-dimensional displays
Trajectory
Video data
ISSN:2377-3766, 2377-3766
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This letter presents a method for generating large-scale datasets to improve class-agnostic video segmentation across robots with different form factors. Specifically, we consider the question of whether video segmentation models trained on generic segmentation data could be more effective for particular robot platforms if robot embodiment is factored into the data generation process. To answer this question, a pipeline is formulated for using 3D reconstructions (e.g. from HM3DSem (Yadav et al., 2023)) to generate segmented videos that are configurable based on a robot's embodiment (e.g. sensor type, sensor placement, and illumination source). A resulting massive RGB-D video panoptic segmentation dataset (MVPd) is introduced for extensive benchmarking with foundation and video segmentation models, as well as to support embodiment-focused research in video segmentation. Our experimental findings demonstrate that using MVPd for finetuning can lead to performance improvements when transferring foundation models to certain robot embodiments, such as specific camera placements. These experiments also show that using 3D modalities (depth images and camera pose) can lead to improvements in video segmentation accuracy and consistency.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2377-3766
2377-3766
DOI:10.1109/LRA.2024.3486213