RigNeRF: Fully Controllable Neural 3D Portraits

Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes bey...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 20332 - 20341
Main Authors: Athar, ShahRukh, Xu, Zexiang, Sunkavalli, Kalyan, Shechtman, Eli, Shu, Zhixin
Format: Conference Proceeding
Language:English
Published: IEEE 01.06.2022
Subjects:
Deformable models
Face and gestures; 3D from multi-view and sensors; Scene analysis and understanding
Face recognition
Image analysis
Rendering (computer graphics)
Solid modeling
Three-dimensional displays
Training
ISSN:1063-6919
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls.
AbstractList Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls.
Author Xu, Zexiang
Sunkavalli, Kalyan
Athar, ShahRukh
Shu, Zhixin
Shechtman, Eli
Author_xml – sequence: 1
  givenname: ShahRukh
  surname: Athar
  fullname: Athar, ShahRukh
  email: sathar@cs.stonybrook.edu
  organization: Stony Brook University
– sequence: 2
  givenname: Zexiang
  surname: Xu
  fullname: Xu, Zexiang
  email: zexu@adobe.com
  organization: Adobe Research
– sequence: 3
  givenname: Kalyan
  surname: Sunkavalli
  fullname: Sunkavalli, Kalyan
  email: sunkaval@adobe.com
  organization: Adobe Research
– sequence: 4
  givenname: Eli
  surname: Shechtman
  fullname: Shechtman, Eli
  email: elishe@adobe.com
  organization: Adobe Research
– sequence: 5
  givenname: Zhixin
  surname: Shu
  fullname: Shu, Zhixin
  email: zshu@adobe.com
  organization: Adobe Research
BookMark eNotj81Kw0AURkdRsK19Al3kBZLe-Z9xJ9GoUGoJ6rZMJjcyMiYySRd9ewO6-Di7w_mW5KIfeiTklkJBKdhN-bGvJVPGFAwYK4Bazc7IkiolhbJC8XOyoKB4riy1V2Q9jl8AwBmlypoF2dThc4d1dZdVxxhPWTn0UxpidE3EbIfH5GLGH7L9kKbkwjRek8vOxRHX_1yR9-rxrXzOt69PL-X9Ng8M-JRzx6joQM4FUnipLZdInbPgBKKXrXdOd43xuvUGPTK0um2E18IbNa_hK3Lz5w2IePhJ4dul08EaM59k_BdJPEWi
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/CVPR52688.2022.01972
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE/IET Electronic Library
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Applied Sciences
EISBN 1665469463
9781665469463
EISSN 1063-6919
EndPage 20341
ExternalDocumentID 9880222
Genre orig-research
GroupedDBID 6IE
6IH
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-i203t-3a214f0569454c57935e1aa90a4eec5dcaa7fb8c7dc8ece2e97db4c74c864c8b3
IEDL.DBID RIE
ISICitedReferencesCount 90
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000870783006017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:15:10 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i203t-3a214f0569454c57935e1aa90a4eec5dcaa7fb8c7dc8ece2e97db4c74c864c8b3
PageCount 10
ParticipantIDs ieee_primary_9880222
PublicationCentury 2000
PublicationDate 2022-June
PublicationDateYYYYMMDD 2022-06-01
PublicationDate_xml – month: 06
  year: 2022
  text: 2022-June
PublicationDecade 2020
PublicationTitle Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online)
PublicationTitleAbbrev CVPR
PublicationYear 2022
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0003211698
Score 2.5837767
Snippet Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard...
SourceID ieee
SourceType Publisher
StartPage 20332
SubjectTerms Deformable models
Face and gestures; 3D from multi-view and sensors; Scene analysis and understanding
Face recognition
Image analysis
Rendering (computer graphics)
Solid modeling
Three-dimensional displays
Training
Title RigNeRF: Fully Controllable Neural 3D Portraits
URI https://ieeexplore.ieee.org/document/9880222
WOSCitedRecordID wos000870783006017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NSwMxEB1q8eCpait-k4NHt93dZDeJ12rxIKUULb2VZHZWFkor7Vbw35vsLhXBi4fAEEJCEpKZSd6bAbhLQ50bHZpAE6nA6WsV2MQ4iZvIaomoqiwKsxc5Hqv5XE9acL_nwhBRBT6jvherv_xsjTv_VDbQyhND3YV7IGVac7X27ynceTKpVg07Lgr1YDibTH0wEw_giuN-6BNs_cqhUqmQUed_gx9D74eLxyZ7LXMCLVqdQqcxHllzNLddGEyL9zFNRw_Me5VfbFhj0JeeGsV8CA6zZPyReeToxhTltgdvo6fX4XPQZEMIijjkZcBNHInc2StaJAITd64SioxfZ0GESYbGyNwqlBkqQopJy8wKlAJV6orlZ9BerVd0Diwncl6xRp7pXCiRW3KOg-teWddMaLyArp__4qMOeLFopn75d_UVHPkFrvFT19AuNzu6gUP8LIvt5rbapW9TbpKu
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB5KFfRUtRXf5uDRbfeR7SZeq6ViXUqppbeSzM7KQmml3Qr-e5PdpSJ48RAYQkhIQjIzyffNANx1XZkq6SpHEgnH6Gvh6FAZKVCelhGiKLIoTIdRHIvZTI5qcL_jwhBRAT6jthWLv_xkhVv7VNaRwhJDzYW7F3LuuyVba_eiEhhfpitFxY_zXNnpTUdjG87EQrh8v-3aFFu_sqgUSqTf-N_wR9D6YeOx0U7PHEONlifQqMxHVh3OTRM64-w9pnH_gVm_8ov1ShT6wpKjmA3CoRYseGQWO7pWWb5pwVv_adIbOFU-BCfz3SB3AuV7PDUWi-Qhx9CcrJA8ZVeaE2GYoFJRqgVGCQpC8klGieYYcRRdU3RwCvXlaklnwFIi4xdLDBKZcsFTTcZ1MN0LbZpxiefQtPOff5QhL-bV1C_-rr6Fg8HkdTgfPscvl3BoF7tEU11BPV9v6Rr28TPPNuubYse-ASdUlfU
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=RigNeRF%3A+Fully+Controllable+Neural+3D+Portraits&rft.au=Athar%2C+ShahRukh&rft.au=Xu%2C+Zexiang&rft.au=Sunkavalli%2C+Kalyan&rft.au=Shechtman%2C+Eli&rft.date=2022-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=20332&rft.epage=20341&rft_id=info:doi/10.1109%2FCVPR52688.2022.01972&rft.externalDocID=9880222