BEHAVE: Dataset and Method for Tracking Human Object Interactions

Modelling interactions between humans and objects in natural environments is central to many applications including gaming, virtual and mixed reality, as well as human behavior analysis and human-robot collaboration. This challenging operation scenario requires generalization to vast number of objec...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) s. 15914 - 15925
Hlavní autoři: Bhatnagar, Bharat Lal, Xie, Xianghui, Petrov, Ilya A., Sminchisescu, Cristian, Theobalt, Christian, Pons-Moll, Gerard
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 01.01.2022
Témata:
3D from multi-view and sensors; 3D from single images; Pose estimation and tracking; Vision + graphics
Codes
Mixed reality
Neural networks
Pose estimation
Predictive models
Solid modeling
Three-dimensional displays
ISSN:1063-6919
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Abstract Modelling interactions between humans and objects in natural environments is central to many applications including gaming, virtual and mixed reality, as well as human behavior analysis and human-robot collaboration. This challenging operation scenario requires generalization to vast number of objects, scenes, and human actions. Unfortunately, there exist no such dataset. Moreover, this data needs to be acquired in diverse natural environments, which rules out 4D scanners and marker based capture systems. We present BEHAVE dataset, the first full body human-object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them. We record ~15k frames at 5 locations with 8 subjects performing a wide range of interactions with 20 common objects. We use this data to learn a model that can jointly track humans and objects in natural environments with an easy-to-use portable multi-camera setup. Our key insight is to predict correspondences from the human and the object to a statistical body model to obtain human-object contacts during interactions. Our approach can record and track not just the humans and objects but also their interactions, modeled as surface contacts, in 3D. Our code and data can be found at: http://virtualhumans.mpi-inf.mpg.de/behave.
AbstractList Modelling interactions between humans and objects in natural environments is central to many applications including gaming, virtual and mixed reality, as well as human behavior analysis and human-robot collaboration. This challenging operation scenario requires generalization to vast number of objects, scenes, and human actions. Unfortunately, there exist no such dataset. Moreover, this data needs to be acquired in diverse natural environments, which rules out 4D scanners and marker based capture systems. We present BEHAVE dataset, the first full body human-object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them. We record ~15k frames at 5 locations with 8 subjects performing a wide range of interactions with 20 common objects. We use this data to learn a model that can jointly track humans and objects in natural environments with an easy-to-use portable multi-camera setup. Our key insight is to predict correspondences from the human and the object to a statistical body model to obtain human-object contacts during interactions. Our approach can record and track not just the humans and objects but also their interactions, modeled as surface contacts, in 3D. Our code and data can be found at: http://virtualhumans.mpi-inf.mpg.de/behave.
Author Bhatnagar, Bharat Lal
Xie, Xianghui
Theobalt, Christian
Petrov, Ilya A.
Sminchisescu, Cristian
Pons-Moll, Gerard
Author_xml – sequence: 1
  givenname: Bharat Lal
  surname: Bhatnagar
  fullname: Bhatnagar, Bharat Lal
  email: bbhatnag@mpi-inf.mpg.de
  organization: University of Tubingen,Germany
– sequence: 2
  givenname: Xianghui
  surname: Xie
  fullname: Xie, Xianghui
  email: xxie@mpi-inf.mpg.de
  organization: Max Planck Institute for Informatics, Saarland Informatics Campus,Germany
– sequence: 3
  givenname: Ilya A.
  surname: Petrov
  fullname: Petrov, Ilya A.
  email: i.petrov@uni-tuebingen.de
  organization: University of Tubingen,Germany
– sequence: 4
  givenname: Cristian
  surname: Sminchisescu
  fullname: Sminchisescu, Cristian
  email: sminchisescu@google.com
  organization: Google Research
– sequence: 5
  givenname: Christian
  surname: Theobalt
  fullname: Theobalt, Christian
  email: theobalt@mpi-inf.mpg.de
  organization: Max Planck Institute for Informatics, Saarland Informatics Campus,Germany
– sequence: 6
  givenname: Gerard
  surname: Pons-Moll
  fullname: Pons-Moll, Gerard
  email: gerard.pons-moll@uni-tuebingen.de
  organization: University of Tubingen,Germany
BookMark eNotjkFOwzAQAA0Cibb0BXDwBxJ2bWcdcysl0EpFRaj0WjnJBlKogxJz4PdUgtMcRhrNWJyFLrAQ1wgpIrib-fb5JVOU56kCpVLAzNgTMUaizJAzpE_FCIF0Qg7dhZgOwx4AtEIkl4_E7K5YzLbFrbz30Q8cpQ-1fOL43tWy6Xq56X310YY3ufg--CDX5Z6rKJch8lHEtgvDpThv_OfA039OxOtDsZkvktX6cTmfrZJWkYsJI2vyx0ciashjXnGDyOANe1PWmq2vSee6tBWDBaOJoNRoSltT41SmJ-Lqr9sy8-6rbw--_9m53DoAq38ByKJK2Q
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/CVPR52688.2022.01547
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE/IET Electronic Library
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Applied Sciences
EISBN 1665469463
9781665469463
EISSN 1063-6919
EndPage 15925
ExternalDocumentID 9879007
Genre orig-research
GrantInformation_xml – fundername: Deutsche Forschungsgemeinschaft
  grantid: 409792180
  funderid: 10.13039/501100001659
GroupedDBID 6IE
6IH
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-i269t-e1e36a202666f6a18cef11e0a4ea4bd3e7ad6383b7ce07043660b314b7d6f9253
IEDL.DBID RIE
ISICitedReferencesCount 101
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000870783001070&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:15:11 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i269t-e1e36a202666f6a18cef11e0a4ea4bd3e7ad6383b7ce07043660b314b7d6f9253
PageCount 12
ParticipantIDs ieee_primary_9879007
PublicationCentury 2000
PublicationDate 2022-01-01
PublicationDateYYYYMMDD 2022-01-01
PublicationDate_xml – month: 01
  year: 2022
  text: 2022-01-01
  day: 01
PublicationDecade 2020
PublicationTitle Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online)
PublicationTitleAbbrev CVPR
PublicationYear 2022
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0003211698
Score 2.6059506
Snippet Modelling interactions between humans and objects in natural environments is central to many applications including gaming, virtual and mixed reality, as well...
SourceID ieee
SourceType Publisher
StartPage 15914
SubjectTerms 3D from multi-view and sensors; 3D from single images; Pose estimation and tracking; Vision + graphics
Codes
Mixed reality
Neural networks
Pose estimation
Predictive models
Solid modeling
Three-dimensional displays
Title BEHAVE: Dataset and Method for Tracking Human Object Interactions
URI https://ieeexplore.ieee.org/document/9879007
WOSCitedRecordID wos000870783001070&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEJ0g8eAJFYzf6cGjC9t2abfeECFcRGKUcCPt7jThshBY_P223Q3GxIu3TQ_bdJrJzHudNwPwYJhxaYOMI0mNByhooxR9BUCS6VT0Fae6GjYhp9N0sVCzBjwetDCIGIrPsOs_w1t-vs72nirrOXysgnT8SEpRabUOfAp3SEaotFbH0Vj1hvPZu29m4gu4GOv6ZOH3DJUQQsat_21-Cp0fLR6ZHaLMGTSwOIdWnTyS2jV3bRg8jyaD-eiJvOjSRaaS6CInr2E-NHGJKXFBKfO0OAm0PXkznoAhgRCstA27DnyORx_DSVTPR4hWTKgyQopcaHc4B0Gs0DTN0FKKsU5QJybnKHXu3IsbmaHz7IQLERtOEyNzYRXr8wtoFusCL4HQnNqEWmusdJCJMmMRmbJGun_5hjNX0PYWWW6qFhjL2hjXfy_fwIk3ecVU3EKz3O7xDo6zr3K1296He_sGeyuXnA
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEG2ImugJFYzf9uDRhe0H7dYbIgQjIDFIuJF2d5pwAQOLv9-2u8GYePG26WGbTjOZea_zZhC6N9S4tEHGkSTGAxSwUQK-AoCnOhEtxYguhk3I0SiZzdS4gh52WhgACMVn0PCf4S0_W6VbT5U1HT5WQTq-3-KcxoVaa8eoMIdlhEpKfRyJVbMzHb_7dia-hIvShk8Xfk9RCUGkV_3f9seo_qPGw-NdnDlBFVieomqZPuLSOTc11H7q9tvT7iN-1rmLTTnWywwPw4Ro7FJT7MJS6olxHIh7_GY8BYMDJVioGzZ19NHrTjr9qJyQEC2oUHkEBJjQ7nAOhFihSZKCJQRizUFzkzGQOnMOxoxMwfk2Z0LEhhFuZCasoi12hvaWqyWcI0wyYjmx1ljpQBOhxgJQZY10__ItZy5QzVtk_lk0wZiXxrj8e_kOHfYnw8F88DJ6vUJH3vwFb3GN9vL1Fm7QQfqVLzbr23CH35ATmuM
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=BEHAVE%3A+Dataset+and+Method+for+Tracking+Human+Object+Interactions&rft.au=Bhatnagar%2C+Bharat+Lal&rft.au=Xie%2C+Xianghui&rft.au=Petrov%2C+Ilya+A.&rft.au=Sminchisescu%2C+Cristian&rft.date=2022-01-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=15914&rft.epage=15925&rft_id=info:doi/10.1109%2FCVPR52688.2022.01547&rft.externalDocID=9879007