NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions

Humans constantly interact with objects in daily life tasks. Capturing such processes and subsequently conducting visual inferences from a fixed viewpoint suffers from occlusions, shape and texture ambiguities, motions, etc. To mitigate the problem, it is essential to build a training dataset that c...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 8834 - 8845
Main Authors: Zhang, Juze, Luo, Haimin, Yang, Hongdi, Xu, Xinru, Wu, Qianyang, Shi, Ye, Yu, Jingyi, Xu, Lan, Wang, Jingya
Format: Conference Proceeding
Language:English
Published: IEEE 01.06.2023
Subjects:
3D from multi-view and sensors
Computer vision
Geometry
Pipelines
Rendering (computer graphics)
Shape
Training
Visualization
ISSN:1063-6919
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Humans constantly interact with objects in daily life tasks. Capturing such processes and subsequently conducting visual inferences from a fixed viewpoint suffers from occlusions, shape and texture ambiguities, motions, etc. To mitigate the problem, it is essential to build a training dataset that captures free-viewpoint interactions. We construct a dense multi-view dome to acquire a complex human object interaction dataset, named HODome, that consists of ~71 M frames on 10 subjects interacting with 23 objects. To process the HODome dataset, we develop NeuralDome, a layer-wise neural processing pipeline tailored for multi-view video inputs to conduct accurate tracking, geometry reconstruction and free-view rendering, for both human subjects and objects. Extensive experiments on the HODome dataset demonstrate the effectiveness of NeuralDome on a variety of inference, modeling, and rendering tasks. Both the dataset and the NeuralDome tools will be disseminated to the community for further development, which can be found at https://juzezhang.github.io/NeuralDome
AbstractList Humans constantly interact with objects in daily life tasks. Capturing such processes and subsequently conducting visual inferences from a fixed viewpoint suffers from occlusions, shape and texture ambiguities, motions, etc. To mitigate the problem, it is essential to build a training dataset that captures free-viewpoint interactions. We construct a dense multi-view dome to acquire a complex human object interaction dataset, named HODome, that consists of ~71 M frames on 10 subjects interacting with 23 objects. To process the HODome dataset, we develop NeuralDome, a layer-wise neural processing pipeline tailored for multi-view video inputs to conduct accurate tracking, geometry reconstruction and free-view rendering, for both human subjects and objects. Extensive experiments on the HODome dataset demonstrate the effectiveness of NeuralDome on a variety of inference, modeling, and rendering tasks. Both the dataset and the NeuralDome tools will be disseminated to the community for further development, which can be found at https://juzezhang.github.io/NeuralDome
Author Yang, Hongdi
Wu, Qianyang
Yu, Jingyi
Xu, Xinru
Wang, Jingya
Shi, Ye
Luo, Haimin
Zhang, Juze
Xu, Lan
Author_xml – sequence: 1
  givenname: Juze
  surname: Zhang
  fullname: Zhang, Juze
  email: zhangjz@shanghaitech.edu.cn
  organization: ShanghaiTech University
– sequence: 2
  givenname: Haimin
  surname: Luo
  fullname: Luo, Haimin
  email: luohm@shanghaitech.edu.cn
  organization: ShanghaiTech University
– sequence: 3
  givenname: Hongdi
  surname: Yang
  fullname: Yang, Hongdi
  email: yanghd@shanghaitech.edu.cn
  organization: ShanghaiTech University
– sequence: 4
  givenname: Xinru
  surname: Xu
  fullname: Xu, Xinru
  email: xuxr2022@shanghaitech.edu.cn
  organization: ShanghaiTech University
– sequence: 5
  givenname: Qianyang
  surname: Wu
  fullname: Wu, Qianyang
  email: wuqy2022@shanghaitech.edu.cn
  organization: ShanghaiTech University
– sequence: 6
  givenname: Ye
  surname: Shi
  fullname: Shi, Ye
  email: shiye@shanghaitech.edu.cn
  organization: ShanghaiTech University
– sequence: 7
  givenname: Jingyi
  surname: Yu
  fullname: Yu, Jingyi
  email: yujingyi@shanghaitech.edu.cn
  organization: ShanghaiTech University
– sequence: 8
  givenname: Lan
  surname: Xu
  fullname: Xu, Lan
  email: xulan1@shanghaitech.edu.cn
  organization: ShanghaiTech University
– sequence: 9
  givenname: Jingya
  surname: Wang
  fullname: Wang, Jingya
  email: wangjingya@shanghaitech.edu.cn
  organization: ShanghaiTech University
BookMark eNotjF1PwjAUQKvRRET-AQ_9A8PbXrrd-kZQhISvGOWVdN2dKRkd2UaM_14NPp1zXs69uIl1ZCGGCkZKgX2c7rZvRmfajjRoHAGQwSsxsJklNICgtKVr0VOQYpJaZe_EoG0PAIBaqdRST6zXfG5c9Vwf-UlO5KXkqi64CvFTbsPpT1jWUa7OVReSXeAvOT8fXUw2-YF9Jxex48b5LtSxfRC3pataHvyzLz5mL-_TebLcvC6mk2USNIy7xPiysNqBN8aMkYBy8ujyHHxh8rHTpJXPNOUFZoxIviSvS0tlShkWFhz2xfDyDcy8PzXh6JrvvYLfuzYWfwDr2lE6
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/CVPR52729.2023.00853
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Applied Sciences
EISBN 9798350301298
EISSN 1063-6919
EndPage 8845
ExternalDocumentID 10204259
Genre orig-research
GroupedDBID 6IE
6IH
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-i204t-5cfd92a0c55543808b8c3abb0cd5b4a2821c728bd37e338cf8c2f98f6873d90a3
IEDL.DBID RIE
ISICitedReferencesCount 16
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001062522101013&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:56:29 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i204t-5cfd92a0c55543808b8c3abb0cd5b4a2821c728bd37e338cf8c2f98f6873d90a3
PageCount 12
ParticipantIDs ieee_primary_10204259
PublicationCentury 2000
PublicationDate 2023-June
PublicationDateYYYYMMDD 2023-06-01
PublicationDate_xml – month: 06
  year: 2023
  text: 2023-June
PublicationDecade 2020
PublicationTitle Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online)
PublicationTitleAbbrev CVPR
PublicationYear 2023
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0003211698
Score 2.4464958
Snippet Humans constantly interact with objects in daily life tasks. Capturing such processes and subsequently conducting visual inferences from a fixed viewpoint...
SourceID ieee
SourceType Publisher
StartPage 8834
SubjectTerms 3D from multi-view and sensors
Computer vision
Geometry
Pipelines
Rendering (computer graphics)
Shape
Training
Visualization
Title NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions
URI https://ieeexplore.ieee.org/document/10204259
WOSCitedRecordID wos001062522101013&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELZoxcBUHkW85YHVJW_bbKiiYkAlQlB1q_w4SxlIq774-_icUFgY2C6WokQ-W777fN99hNxarGDKjGJK2JxlmnsLkpg5Hz4Ll-lEBf2UyTMfj8V0KsuWrB64MAAQis9ggGa4y7dzs0GozO_wBNeY7JAO50VD1toBKqlPZQopWnpcHMm74aR8zRMfPQ5QIxwbmaIE8i8RlXCGjHr__Poh6f-w8Wi5O2eOyB7Ux6TXho-03ZyrEzLGRhtY-_YB9_SBNk8Uxc6Qck7LaoEG0HlNA-2WTSr4pAHFZy8a8Rga8MGG6rDqk_fR49vwibVyCazyP7ZmuXFWJioyuQ8RUhEJLUyqtI6MzXWmfG4VG54IbVMOPjE1TpjESeEKwVMrI5Wekm49r-GMUJdb8K8LiITNUsOlsaZQMcRWSOUTuHPSx_mZLZqOGLPvqbn4Y_ySHKALmhKrK9JdLzdwTfbNdl2tljfBj18S4J8n
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwED5BQYKpPIp444HVJXGSxmZDFaiIEiJUqm6VHxcpA23VB_x9bCcUFga2i6Uokc-W7z7fdx_AtXEVTLGWVHKT0Fil1kIW0sKGz7yIFZNeP2XYT7OMj0Yir8nqnguDiL74DNvO9Hf5ZqpXDiqzO5y5NSY2YSuJYxZUdK01pBLZZKYjeE2QCwNx0x3mrwmz8WPbqYS7VqZOBPmXjIo_RR6a__z-HrR--HgkX580-7CBkwNo1gEkqbfn4hAy12rDVb-94y25I9UTcXJnjnRO8nLmDCTTCfHEWzos8ZN4HJ--KIfIEI8QVmSHRQveHu4H3R6tBRNoaX9sSRNdGMFkoBMbJEQ84IrrSCoVaJOoWNrsKtQp48pEKdrUVBdcs0LwosPTyIhARkfQmEwneAykSAza1zkG3MSRToU2uiNDDA0X0qZwJ9By8zOeVT0xxt9Tc_rH-BXs9AbP_XH_MXs6g13njqrg6hway_kKL2BbfyzLxfzS-_QLitKibg
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=NeuralDome%3A+A+Neural+Modeling+Pipeline+on+Multi-View+Human-Object+Interactions&rft.au=Zhang%2C+Juze&rft.au=Luo%2C+Haimin&rft.au=Yang%2C+Hongdi&rft.au=Xu%2C+Xinru&rft.date=2023-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=8834&rft.epage=8845&rft_id=info:doi/10.1109%2FCVPR52729.2023.00853&rft.externalDocID=10204259