SwinMTL: A Shared Architecture for Simultaneous Depth Estimation and Semantic Segmentation from Monocular Camera Images

This research paper presents an innovative multitask learning framework that allows concurrent depth estimation and semantic segmentation using a single camera. The proposed approach is based on a shared encoder-decoder architecture, which integrates various techniques to improve the accuracy of the...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings of the ... IEEE/RSJ International Conference on Intelligent Robots and Systems pp. 4957 - 4964
Main Authors: Taghavi, Pardis, Langari, Reza, Pandey, Gaurav
Format: Conference Proceeding
Language:English
Published: IEEE 14.10.2024
Subjects:
Accuracy
Cameras
Computational efficiency
Computer architecture
Depth measurement
Multitasking
Semantic segmentation
Source coding
Streaming media
Training
ISSN:2153-0866
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This research paper presents an innovative multitask learning framework that allows concurrent depth estimation and semantic segmentation using a single camera. The proposed approach is based on a shared encoder-decoder architecture, which integrates various techniques to improve the accuracy of the depth estimation and semantic segmentation task without compromising computational efficiency. Additionally, the paper incorporates an adversarial training component, employing a Wasserstein GAN framework with a critic network, to refine model's predictions. The framework is thoroughly evaluated on two datasets - the outdoor Cityscapes dataset and the indoor NYU Depth V2 dataset - and it outperforms existing state-of-the-art methods in both segmentation and depth estimation tasks. We also conducted ablation studies to analyze the contributions of different components, including pre-training strategies, the inclusion of critics, the use of logarithmic depth scaling, and advanced image augmentations, to provide a better understanding of the proposed framework. The accompanying source code is accessible at https://github.com/PardisTaghavi/SwinMTL.
ISSN:2153-0866
DOI:10.1109/IROS58592.2024.10802239