BEHAVE: Dataset and Method for Tracking Human Object Interactions

Modelling interactions between humans and objects in natural environments is central to many applications including gaming, virtual and mixed reality, as well as human behavior analysis and human-robot collaboration. This challenging operation scenario requires generalization to vast number of objec...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) s. 15914 - 15925
Hlavní autori: Bhatnagar, Bharat Lal, Xie, Xianghui, Petrov, Ilya A., Sminchisescu, Cristian, Theobalt, Christian, Pons-Moll, Gerard
Médium: Konferenčný príspevok..
Jazyk:English
Vydavateľské údaje: IEEE 01.01.2022
Predmet:
3D from multi-view and sensors; 3D from single images; Pose estimation and tracking; Vision + graphics
Codes
Mixed reality
Neural networks
Pose estimation
Predictive models
Solid modeling
Three-dimensional displays
ISSN:1063-6919
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Abstract Modelling interactions between humans and objects in natural environments is central to many applications including gaming, virtual and mixed reality, as well as human behavior analysis and human-robot collaboration. This challenging operation scenario requires generalization to vast number of objects, scenes, and human actions. Unfortunately, there exist no such dataset. Moreover, this data needs to be acquired in diverse natural environments, which rules out 4D scanners and marker based capture systems. We present BEHAVE dataset, the first full body human-object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them. We record ~15k frames at 5 locations with 8 subjects performing a wide range of interactions with 20 common objects. We use this data to learn a model that can jointly track humans and objects in natural environments with an easy-to-use portable multi-camera setup. Our key insight is to predict correspondences from the human and the object to a statistical body model to obtain human-object contacts during interactions. Our approach can record and track not just the humans and objects but also their interactions, modeled as surface contacts, in 3D. Our code and data can be found at: http://virtualhumans.mpi-inf.mpg.de/behave.
AbstractList Modelling interactions between humans and objects in natural environments is central to many applications including gaming, virtual and mixed reality, as well as human behavior analysis and human-robot collaboration. This challenging operation scenario requires generalization to vast number of objects, scenes, and human actions. Unfortunately, there exist no such dataset. Moreover, this data needs to be acquired in diverse natural environments, which rules out 4D scanners and marker based capture systems. We present BEHAVE dataset, the first full body human-object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them. We record ~15k frames at 5 locations with 8 subjects performing a wide range of interactions with 20 common objects. We use this data to learn a model that can jointly track humans and objects in natural environments with an easy-to-use portable multi-camera setup. Our key insight is to predict correspondences from the human and the object to a statistical body model to obtain human-object contacts during interactions. Our approach can record and track not just the humans and objects but also their interactions, modeled as surface contacts, in 3D. Our code and data can be found at: http://virtualhumans.mpi-inf.mpg.de/behave.
Author Bhatnagar, Bharat Lal
Xie, Xianghui
Theobalt, Christian
Petrov, Ilya A.
Sminchisescu, Cristian
Pons-Moll, Gerard
Author_xml – sequence: 1
  givenname: Bharat Lal
  surname: Bhatnagar
  fullname: Bhatnagar, Bharat Lal
  email: bbhatnag@mpi-inf.mpg.de
  organization: University of Tubingen,Germany
– sequence: 2
  givenname: Xianghui
  surname: Xie
  fullname: Xie, Xianghui
  email: xxie@mpi-inf.mpg.de
  organization: Max Planck Institute for Informatics, Saarland Informatics Campus,Germany
– sequence: 3
  givenname: Ilya A.
  surname: Petrov
  fullname: Petrov, Ilya A.
  email: i.petrov@uni-tuebingen.de
  organization: University of Tubingen,Germany
– sequence: 4
  givenname: Cristian
  surname: Sminchisescu
  fullname: Sminchisescu, Cristian
  email: sminchisescu@google.com
  organization: Google Research
– sequence: 5
  givenname: Christian
  surname: Theobalt
  fullname: Theobalt, Christian
  email: theobalt@mpi-inf.mpg.de
  organization: Max Planck Institute for Informatics, Saarland Informatics Campus,Germany
– sequence: 6
  givenname: Gerard
  surname: Pons-Moll
  fullname: Pons-Moll, Gerard
  email: gerard.pons-moll@uni-tuebingen.de
  organization: University of Tubingen,Germany
BookMark eNotjkFOwzAQAA0Cibb0BXDwBxJ2bWcdcysl0EpFRaj0WjnJBlKogxJz4PdUgtMcRhrNWJyFLrAQ1wgpIrib-fb5JVOU56kCpVLAzNgTMUaizJAzpE_FCIF0Qg7dhZgOwx4AtEIkl4_E7K5YzLbFrbz30Q8cpQ-1fOL43tWy6Xq56X310YY3ufg--CDX5Z6rKJch8lHEtgvDpThv_OfA039OxOtDsZkvktX6cTmfrZJWkYsJI2vyx0ciashjXnGDyOANe1PWmq2vSee6tBWDBaOJoNRoSltT41SmJ-Lqr9sy8-6rbw--_9m53DoAq38ByKJK2Q
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/CVPR52688.2022.01547
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Applied Sciences
EISBN 1665469463
9781665469463
EISSN 1063-6919
EndPage 15925
ExternalDocumentID 9879007
Genre orig-research
GrantInformation_xml – fundername: Deutsche Forschungsgemeinschaft
  grantid: 409792180
  funderid: 10.13039/501100001659
GroupedDBID 6IE
6IH
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-i269t-e1e36a202666f6a18cef11e0a4ea4bd3e7ad6383b7ce07043660b314b7d6f9253
IEDL.DBID RIE
ISICitedReferencesCount 101
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000870783001070&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:15:11 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i269t-e1e36a202666f6a18cef11e0a4ea4bd3e7ad6383b7ce07043660b314b7d6f9253
PageCount 12
ParticipantIDs ieee_primary_9879007
PublicationCentury 2000
PublicationDate 2022-01-01
PublicationDateYYYYMMDD 2022-01-01
PublicationDate_xml – month: 01
  year: 2022
  text: 2022-01-01
  day: 01
PublicationDecade 2020
PublicationTitle Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online)
PublicationTitleAbbrev CVPR
PublicationYear 2022
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0003211698
Score 2.6059506
Snippet Modelling interactions between humans and objects in natural environments is central to many applications including gaming, virtual and mixed reality, as well...
SourceID ieee
SourceType Publisher
StartPage 15914
SubjectTerms 3D from multi-view and sensors; 3D from single images; Pose estimation and tracking; Vision + graphics
Codes
Mixed reality
Neural networks
Pose estimation
Predictive models
Solid modeling
Three-dimensional displays
Title BEHAVE: Dataset and Method for Tracking Human Object Interactions
URI https://ieeexplore.ieee.org/document/9879007
WOSCitedRecordID wos000870783001070&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEG2QePCECsbv9ODRwnZb2q03RAgHRWKUcCP9mE24LAYWf79td4Mx8eKt6aXJayZv5rVvBqE7bnKdJo6RzNqU-PoLiHaSEkgkOCcdJHFm5PxZTqfZYqFmDXS_98IAQPx8Bt2wjG_5bm13QSrr-fpYRev4gZSi8mrt9RTmKxmhstodRxPVG85nb6GZSfjAlabdkCz8nqESKWTc-t_hx6jz48XDsz3LnKAGFKeoVSePuA7NbRsNHkeTwXz0gJ906ZmpxLpw-CXOh8Y-McWelGyQxXGU7fGrCQIMjoJg5W3YdtDHePQ-nJB6PgJZpUKVBCgw4ZH2HCtyoWlmIacUEs1Bc-MYSO18eDEjrYc-9JoXiWGUG-lErtI-O0PNYl3AOcJgfWxbYXz2wbjloEzfl44izR3nfS3lBWoHRJafVQuMZQ3G5d_bV-goQF4pFdeoWW52cIMO7Ve52m5u4719A6gemBE
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEG2ImugJFYzf9uDRhd222-56Q4RgBCQGCTfSj9mEy2Jg8ffbdjcYEy_eml6avGbyZl77ZhC6ZyqTJDQ0SLQmga2_IJBGRAGEAowRBkI_M3I2FONxMp-nkxp62HlhAMB_PoOWW_q3fLPSWyeVtW19nHrr-H7MGAlLt9ZOUaG2luFpUvnjojBtd2eTd9fOxH3hIqTl0oXfU1Q8ifTr_zv-GDV_3Hh4suOZE1SD_BTVq_QRV8G5aaDOU2_QmfUe8bMsLDcVWOYGj_yEaGxTU2xpSTthHHvhHr8pJ8FgLwmW7oZNE330e9PuIKgmJARLwtMigAgot1hbluUZl1GiIYsiCCUDyZShIKSxAUaV0BZ8122eh4pGTAnDs5TE9Azt5asczhEGbaNbc2XzD8o0g1TFtnjkJDOMxVKIC9RwiCw-yyYYiwqMy7-379DhYDoaLoYv49crdOTgL3WLa7RXrLdwgw70V7HcrG_9HX4DxOCbWA
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=BEHAVE%3A+Dataset+and+Method+for+Tracking+Human+Object+Interactions&rft.au=Bhatnagar%2C+Bharat+Lal&rft.au=Xie%2C+Xianghui&rft.au=Petrov%2C+Ilya+A.&rft.au=Sminchisescu%2C+Cristian&rft.date=2022-01-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=15914&rft.epage=15925&rft_id=info:doi/10.1109%2FCVPR52688.2022.01547&rft.externalDocID=9879007