Secrets of Event-Based Optical Flow, Depth and Ego-Motion Estimation by Contrast Maximization

Event cameras respond to scene dynamics and provide signals naturally suitable for motion estimation with advantages, such as high dynamic range. The emerging field of event-based vision motivates a revisit of fundamental computer vision tasks related to motion, such as optical flow and depth estima...

Full description

Saved in:
Bibliographic Details
Published in:IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 46; no. 12; pp. 7742 - 7759
Main Authors: Shiba, Shintaro, Klose, Yannick, Aoki, Yoshimitsu, Gallego, Guillermo
Format: Journal Article
Language:English
Published: United States IEEE 01.12.2024
Institute of Electrical and Electronics Engineers (IEEE)
Institute of Electrical and Electronics Engineers
Subjects:
3D reconstruction
[INFO.INFO-CV]Computer Science [cs]/Computer Vision and Pattern Recognition [cs.CV]
[INFO.INFO-RB]Computer Science [cs]/Robotics [cs.RO]
Adaptive optics
Asynchronous sensors
Biomedical optical imaging
Camera motion estimation
Cameras
Computer Science
Computer Vision and Pattern Recognition
Estimation
Event camera
High dynamic range
Optical flow
Optical imaging
Optical sensors
Optical variables control
Robotics
3D reconstruction
High dynamic range
Camera motion estimation
Event camera
Optical flow
Asynchronous sensors
ISSN:0162-8828, 1939-3539, 1939-3539, 2160-9292
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Event cameras respond to scene dynamics and provide signals naturally suitable for motion estimation with advantages, such as high dynamic range. The emerging field of event-based vision motivates a revisit of fundamental computer vision tasks related to motion, such as optical flow and depth estimation. However, state-of-the-art event-based optical flow methods tend to originate in frame-based deep-learning methods, which require several adaptations (data conversion, loss function, etc.) as they have very different properties. We develop a principled method to extend the Contrast Maximization framework to estimate dense optical flow, depth, and ego-motion from events alone. The proposed method sensibly models the space-time properties of event data and tackles the event alignment problem. It designs the objective function to prevent overfitting, deals better with occlusions, and improves convergence using a multi-scale approach. With these key elements, our method ranks first among unsupervised methods on the MVSEC benchmark and is competitive on the DSEC benchmark. Moreover, it allows us to simultaneously estimate dense depth and ego-motion, exposes the limitations of current flow benchmarks, and produces remarkable results when it is transferred to unsupervised learning settings. Along with various downstream applications shown, we hope the proposed method becomes a cornerstone on event-based motion-related tasks.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:0162-8828
1939-3539
1939-3539
2160-9292
DOI:10.1109/TPAMI.2024.3396116