RigNeRF: Fully Controllable Neural 3D Portraits

Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes bey...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 20332 - 20341
Main Authors: Athar, ShahRukh, Xu, Zexiang, Sunkavalli, Kalyan, Shechtman, Eli, Shu, Zhixin
Format: Conference Proceeding
Language:English
Published: IEEE 01.06.2022
Subjects:
Deformable models
Face and gestures; 3D from multi-view and sensors; Scene analysis and understanding
Face recognition
Image analysis
Rendering (computer graphics)
Solid modeling
Three-dimensional displays
Training
ISSN:1063-6919
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls.
AbstractList Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls.
Author Xu, Zexiang
Sunkavalli, Kalyan
Athar, ShahRukh
Shu, Zhixin
Shechtman, Eli
Author_xml – sequence: 1
  givenname: ShahRukh
  surname: Athar
  fullname: Athar, ShahRukh
  email: sathar@cs.stonybrook.edu
  organization: Stony Brook University
– sequence: 2
  givenname: Zexiang
  surname: Xu
  fullname: Xu, Zexiang
  email: zexu@adobe.com
  organization: Adobe Research
– sequence: 3
  givenname: Kalyan
  surname: Sunkavalli
  fullname: Sunkavalli, Kalyan
  email: sunkaval@adobe.com
  organization: Adobe Research
– sequence: 4
  givenname: Eli
  surname: Shechtman
  fullname: Shechtman, Eli
  email: elishe@adobe.com
  organization: Adobe Research
– sequence: 5
  givenname: Zhixin
  surname: Shu
  fullname: Shu, Zhixin
  email: zshu@adobe.com
  organization: Adobe Research
BookMark eNotj81Kw0AURkdRsK19Al3kBZLe-Z9xJ9GoUGoJ6rZMJjcyMiYySRd9ewO6-Di7w_mW5KIfeiTklkJBKdhN-bGvJVPGFAwYK4Bazc7IkiolhbJC8XOyoKB4riy1V2Q9jl8AwBmlypoF2dThc4d1dZdVxxhPWTn0UxpidE3EbIfH5GLGH7L9kKbkwjRek8vOxRHX_1yR9-rxrXzOt69PL-X9Ng8M-JRzx6joQM4FUnipLZdInbPgBKKXrXdOd43xuvUGPTK0um2E18IbNa_hK3Lz5w2IePhJ4dul08EaM59k_BdJPEWi
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/CVPR52688.2022.01972
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Xplore
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEL
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Applied Sciences
EISBN 1665469463
9781665469463
EISSN 1063-6919
EndPage 20341
ExternalDocumentID 9880222
Genre orig-research
GroupedDBID 6IE
6IH
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-i203t-3a214f0569454c57935e1aa90a4eec5dcaa7fb8c7dc8ece2e97db4c74c864c8b3
IEDL.DBID RIE
ISICitedReferencesCount 90
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000870783006017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:15:10 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i203t-3a214f0569454c57935e1aa90a4eec5dcaa7fb8c7dc8ece2e97db4c74c864c8b3
PageCount 10
ParticipantIDs ieee_primary_9880222
PublicationCentury 2000
PublicationDate 2022-June
PublicationDateYYYYMMDD 2022-06-01
PublicationDate_xml – month: 06
  year: 2022
  text: 2022-June
PublicationDecade 2020
PublicationTitle Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online)
PublicationTitleAbbrev CVPR
PublicationYear 2022
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0003211698
Score 2.5837767
Snippet Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard...
SourceID ieee
SourceType Publisher
StartPage 20332
SubjectTerms Deformable models
Face and gestures; 3D from multi-view and sensors; Scene analysis and understanding
Face recognition
Image analysis
Rendering (computer graphics)
Solid modeling
Three-dimensional displays
Training
Title RigNeRF: Fully Controllable Neural 3D Portraits
URI https://ieeexplore.ieee.org/document/9880222
WOSCitedRecordID wos000870783006017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA61ePBUtRXf5ODRtNlNdpN4rRYPUkrR0lvJJrOyUFppt4L_3szuUhG8eAiEXMIkTOaR75sh5C4FqYIhjpjIrWAyjiQzPo1ZbrwIDgZPdZXMmb2o8VjP52bSIvd7LgwAVOAz6OO0-sv3a7fDVNnAaCSGhgf3QKm05mrt8ykiRDKp0Q07LuJmMJxNpljMBAFccdzn2GDrVw-VyoSMOv_b_Jj0frh4dLK3MiekBatT0mmcR9qo5rZLBtPifQzT0QPFqPKLDmsM-hKpURRLcNglFY8UkaMbW5TbHnkbPb0On1nTDYEVMRclEzYcZB78FSMT6ZKgVwlE1hpuJYBLvLNW5Zl2yjsNDmIwymfSKel0GkYmzkh7tV7BOaFcSiuwlJ5XsXQ-M8Zj70_lhDDec3VBuij_4qMueLFoRL_8e_mKHOEB1_ipa9IuNzu4IYfusyy2m9vqlr4BDz2Q5g
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3NS8MwFA9jCnqauonf5uDRbmmSNo3X6Zg4yxhz7DbS5FUKY5OuE_zvTdoyEbx4CIRcwkt4eR_5_d5D6C4ELqwh9j2WKuZx6nNPmpB6qTTMOhgkjMpkzmwk4jiaz-W4ge53XBgAKMFn0HXT8i_frPXWpcp6MnLEUPvg7gWcU1KxtXYZFWZjmVBGNT_OJ7LXn40nrpyJg3BR2iWuxdavLiqlERm0_rf9Eer8sPHweGdnjlEDVieoVbuPuFbOTRv1Jtl7DJPBA3Zx5RfuVyj0pSNHYVeEQy0xe8QOO5qrrNh00NvgadofenU_BC-jhBUeU_YoU-uxSB5wHVjNCsBXShLFAXRgtFIiTSItjI5AAwUpTMK14DoK7UjYKWqu1is4Q5hwrpgrpmcE5dokUhrX_VNoxqQxRJyjtpN_8VGVvFjUol_8vXyLDobT19Fi9By_XKJDd9gVmuoKNYt8C9doX38W2Sa_KW_sG8vGlC0
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=RigNeRF%3A+Fully+Controllable+Neural+3D+Portraits&rft.au=Athar%2C+ShahRukh&rft.au=Xu%2C+Zexiang&rft.au=Sunkavalli%2C+Kalyan&rft.au=Shechtman%2C+Eli&rft.date=2022-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=20332&rft.epage=20341&rft_id=info:doi/10.1109%2FCVPR52688.2022.01972&rft.externalDocID=9880222