RigNeRF: Fully Controllable Neural 3D Portraits

Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes bey...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 20332 - 20341
Main Authors: Athar, ShahRukh, Xu, Zexiang, Sunkavalli, Kalyan, Shechtman, Eli, Shu, Zhixin
Format: Conference Proceeding
Language:English
Published: IEEE 01.06.2022
Subjects:
Deformable models
Face and gestures; 3D from multi-view and sensors; Scene analysis and understanding
Face recognition
Image analysis
Rendering (computer graphics)
Solid modeling
Three-dimensional displays
Training
ISSN:1063-6919
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls.
AbstractList Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls.
Author Xu, Zexiang
Sunkavalli, Kalyan
Athar, ShahRukh
Shu, Zhixin
Shechtman, Eli
Author_xml – sequence: 1
  givenname: ShahRukh
  surname: Athar
  fullname: Athar, ShahRukh
  email: sathar@cs.stonybrook.edu
  organization: Stony Brook University
– sequence: 2
  givenname: Zexiang
  surname: Xu
  fullname: Xu, Zexiang
  email: zexu@adobe.com
  organization: Adobe Research
– sequence: 3
  givenname: Kalyan
  surname: Sunkavalli
  fullname: Sunkavalli, Kalyan
  email: sunkaval@adobe.com
  organization: Adobe Research
– sequence: 4
  givenname: Eli
  surname: Shechtman
  fullname: Shechtman, Eli
  email: elishe@adobe.com
  organization: Adobe Research
– sequence: 5
  givenname: Zhixin
  surname: Shu
  fullname: Shu, Zhixin
  email: zshu@adobe.com
  organization: Adobe Research
BookMark eNotj81Kw0AURkdRsK19Al3kBZLe-Z9xJ9GoUGoJ6rZMJjcyMiYySRd9ewO6-Di7w_mW5KIfeiTklkJBKdhN-bGvJVPGFAwYK4Bazc7IkiolhbJC8XOyoKB4riy1V2Q9jl8AwBmlypoF2dThc4d1dZdVxxhPWTn0UxpidE3EbIfH5GLGH7L9kKbkwjRek8vOxRHX_1yR9-rxrXzOt69PL-X9Ng8M-JRzx6joQM4FUnipLZdInbPgBKKXrXdOd43xuvUGPTK0um2E18IbNa_hK3Lz5w2IePhJ4dul08EaM59k_BdJPEWi
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/CVPR52688.2022.01972
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Applied Sciences
EISBN 1665469463
9781665469463
EISSN 1063-6919
EndPage 20341
ExternalDocumentID 9880222
Genre orig-research
GroupedDBID 6IE
6IH
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-i203t-3a214f0569454c57935e1aa90a4eec5dcaa7fb8c7dc8ece2e97db4c74c864c8b3
IEDL.DBID RIE
ISICitedReferencesCount 90
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000870783006017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:15:10 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i203t-3a214f0569454c57935e1aa90a4eec5dcaa7fb8c7dc8ece2e97db4c74c864c8b3
PageCount 10
ParticipantIDs ieee_primary_9880222
PublicationCentury 2000
PublicationDate 2022-June
PublicationDateYYYYMMDD 2022-06-01
PublicationDate_xml – month: 06
  year: 2022
  text: 2022-June
PublicationDecade 2020
PublicationTitle Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online)
PublicationTitleAbbrev CVPR
PublicationYear 2022
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0003211698
Score 2.5838366
Snippet Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard...
SourceID ieee
SourceType Publisher
StartPage 20332
SubjectTerms Deformable models
Face and gestures; 3D from multi-view and sensors; Scene analysis and understanding
Face recognition
Image analysis
Rendering (computer graphics)
Solid modeling
Three-dimensional displays
Training
Title RigNeRF: Fully Controllable Neural 3D Portraits
URI https://ieeexplore.ieee.org/document/9880222
WOSCitedRecordID wos000870783006017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NSwMxEA21ePBUtRW_ycGj2-4m2U3itVo8lVJUeivZyUQWSivdreC_N9ldKoIXD4GQQ8IkTGYmeW-GkDvOABhYEblMmkikLo9yBzZihtuQb8zHJLYuNiGnU7VY6FmH3O-5MIhYg89wGLr1X77dwC48lY20CsRQf-EeSJk1XK39ewr3kUymVcuOS2I9Gr_N5iGZSQBwMTaMQ4GtXzVUahMy6f1v8WMy-OHi0dneypyQDq5PSa91HmmrmmWfjObF-xTnkwcaosovOm4w6KtAjaIhBYdZUf5IA3J0a4qqHJDXydPL-DlqqyFEBYt5FXHDEuG8v6JFKiD1epViYoyOjUCE1IIx0uUKpAWFgAy1tLkAKUBlvuX8jHTXmzWeE5oolYETFhlDb9BtnliXKesnj52PR8QF6Qf5lx9NwotlK_rl38NX5ChscIOfuibdarvDG3IIn1VRbm_rU_oGhPGTEA
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3fS8MwEA5DBX2auom_zYOPdmvTtE18nY6Js4wxZW8jvVykMDbZOsH_3qQtE8EXHwIhDwmXcLm75PvuCLkNGQADzT0TJ8rjkcm8zID2mAq1yzdmYxJdFptI0lRMp3LUIHdbLgwiluAz7Lhu-Zevl7BxT2VdKRwx1F64uxHnzK_YWtsXldDGMrEUNT8u8GW39zYau3QmDsLFWMd3JbZ-VVEpjUi_-b_lD0n7h41HR1s7c0QauDgmzdp9pLVyrlukO87fUxz376mLK79or0Khzx05irokHGpOwwfqsKMrlRfrNnntP056A6-uh-DlzA8LL1Qs4MZ6LJJHHCKrWREGSklfcUSINCiVmExAokEgIEOZ6IxDwkHEtmXhCdlZLBd4SmggRAyGa2QMrUnXWaBNLLSd3Dc2IuFnpOXkn31UKS9mtejnfw_fkP3B5GU4Gz6lzxfkwG12haa6JDvFaoNXZA8-i3y9ui5P7Btet5ZX
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=RigNeRF%3A+Fully+Controllable+Neural+3D+Portraits&rft.au=Athar%2C+ShahRukh&rft.au=Xu%2C+Zexiang&rft.au=Sunkavalli%2C+Kalyan&rft.au=Shechtman%2C+Eli&rft.date=2022-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=20332&rft.epage=20341&rft_id=info:doi/10.1109%2FCVPR52688.2022.01972&rft.externalDocID=9880222