NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions

Humans constantly interact with objects in daily life tasks. Capturing such processes and subsequently conducting visual inferences from a fixed viewpoint suffers from occlusions, shape and texture ambiguities, motions, etc. To mitigate the problem, it is essential to build a training dataset that c...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 8834 - 8845
Main Authors: Zhang, Juze, Luo, Haimin, Yang, Hongdi, Xu, Xinru, Wu, Qianyang, Shi, Ye, Yu, Jingyi, Xu, Lan, Wang, Jingya
Format: Conference Proceeding
Language:English
Published: IEEE 01.06.2023
Subjects:
3D from multi-view and sensors
Computer vision
Geometry
Pipelines
Rendering (computer graphics)
Shape
Training
Visualization
ISSN:1063-6919
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Humans constantly interact with objects in daily life tasks. Capturing such processes and subsequently conducting visual inferences from a fixed viewpoint suffers from occlusions, shape and texture ambiguities, motions, etc. To mitigate the problem, it is essential to build a training dataset that captures free-viewpoint interactions. We construct a dense multi-view dome to acquire a complex human object interaction dataset, named HODome, that consists of ~71 M frames on 10 subjects interacting with 23 objects. To process the HODome dataset, we develop NeuralDome, a layer-wise neural processing pipeline tailored for multi-view video inputs to conduct accurate tracking, geometry reconstruction and free-view rendering, for both human subjects and objects. Extensive experiments on the HODome dataset demonstrate the effectiveness of NeuralDome on a variety of inference, modeling, and rendering tasks. Both the dataset and the NeuralDome tools will be disseminated to the community for further development, which can be found at https://juzezhang.github.io/NeuralDome
ISSN:1063-6919
DOI:10.1109/CVPR52729.2023.00853