RigNeRF: Fully Controllable Neural 3D Portraits
Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes bey...
Uloženo v:
| Vydáno v: | Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) s. 20332 - 20341 |
|---|---|
| Hlavní autoři: | , , , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
IEEE
01.06.2022
|
| Témata: | |
| ISSN: | 1063-6919 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls. |
|---|---|
| AbstractList | Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls. |
| Author | Xu, Zexiang Sunkavalli, Kalyan Athar, ShahRukh Shu, Zhixin Shechtman, Eli |
| Author_xml | – sequence: 1 givenname: ShahRukh surname: Athar fullname: Athar, ShahRukh email: sathar@cs.stonybrook.edu organization: Stony Brook University – sequence: 2 givenname: Zexiang surname: Xu fullname: Xu, Zexiang email: zexu@adobe.com organization: Adobe Research – sequence: 3 givenname: Kalyan surname: Sunkavalli fullname: Sunkavalli, Kalyan email: sunkaval@adobe.com organization: Adobe Research – sequence: 4 givenname: Eli surname: Shechtman fullname: Shechtman, Eli email: elishe@adobe.com organization: Adobe Research – sequence: 5 givenname: Zhixin surname: Shu fullname: Shu, Zhixin email: zshu@adobe.com organization: Adobe Research |
| BookMark | eNotj81Kw0AURkdRsK19Al3kBZLe-Z9xJ9GoUGoJ6rZMJjcyMiYySRd9ewO6-Di7w_mW5KIfeiTklkJBKdhN-bGvJVPGFAwYK4Bazc7IkiolhbJC8XOyoKB4riy1V2Q9jl8AwBmlypoF2dThc4d1dZdVxxhPWTn0UxpidE3EbIfH5GLGH7L9kKbkwjRek8vOxRHX_1yR9-rxrXzOt69PL-X9Ng8M-JRzx6joQM4FUnipLZdInbPgBKKXrXdOd43xuvUGPTK0um2E18IbNa_hK3Lz5w2IePhJ4dul08EaM59k_BdJPEWi |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/CVPR52688.2022.01972 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Applied Sciences |
| EISBN | 1665469463 9781665469463 |
| EISSN | 1063-6919 |
| EndPage | 20341 |
| ExternalDocumentID | 9880222 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IH 6IL 6IN AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP OCL RIE RIL RIO |
| ID | FETCH-LOGICAL-i203t-3a214f0569454c57935e1aa90a4eec5dcaa7fb8c7dc8ece2e97db4c74c864c8b3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 90 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000870783006017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:15:10 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i203t-3a214f0569454c57935e1aa90a4eec5dcaa7fb8c7dc8ece2e97db4c74c864c8b3 |
| PageCount | 10 |
| ParticipantIDs | ieee_primary_9880222 |
| PublicationCentury | 2000 |
| PublicationDate | 2022-June |
| PublicationDateYYYYMMDD | 2022-06-01 |
| PublicationDate_xml | – month: 06 year: 2022 text: 2022-June |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) |
| PublicationTitleAbbrev | CVPR |
| PublicationYear | 2022 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0003211698 |
| Score | 2.5838366 |
| Snippet | Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 20332 |
| SubjectTerms | Deformable models Face and gestures; 3D from multi-view and sensors; Scene analysis and understanding Face recognition Image analysis Rendering (computer graphics) Solid modeling Three-dimensional displays Training |
| Title | RigNeRF: Fully Controllable Neural 3D Portraits |
| URI | https://ieeexplore.ieee.org/document/9880222 |
| WOSCitedRecordID | wos000870783006017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFH7M4cHT1E38TQ4e7ZamSZN4nQ5PYwyV3cZr-iqFscnaCf73Nm2ZCF48BEIuIQnvR5Lv-x7AneJpLKzAwGuRBDKRlR-0WRxgiCoSqA3HrC42oadTs1jYWQfu91wYIqrBZzT03fovP924nX8qG1njiaGVwz3QOm64Wvv3lKi6ycTWtOy4kNvR-G0292ImHsAlxJD7Alu_aqjUIWTS-9_kxzD44eKx2T7KnECH1qfQa5NH1ppm0YfRPH-f0nzywPyt8ouNGwz6ylOjmJfgwBWLHplHjm4xL4sBvE6eXsbPQVsNIcgFj8ogQhHKrMpXrFTSqcquFIWIlqMkcip1iDpLjNOpM-RIkNVpIp2WzsRVS6Iz6K43azoHlgjlpLJWkEwlRQ6zkGckvVScI5TuAvp-_cuPRvBi2S798u_hKzjyG9zgp66hW253dAOH7rPMi-1tfUrfaZSSVg |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFA5DBT1N3cTf5uDRbmn60jZep2PiLGNM2W28pq9SGJtsneB_b9OWieDFQyDkEpLwfiT5vu8xdqtE4kst0bFaJA7EUPhBnfoOuqg8iUEoMC2LTQRRFE6netRgd1suDBGV4DPq2G75l58szcY-lXV1aImhhcPdVQBSVGyt7YuKV9xlfB3W_DhX6G7vbTS2ciYWwiVlR9gSW7-qqJRBpN_83_SHrP3DxuOjbZw5Yg1aHLNmnT7y2jjXLdYdZ-8Rjfv33N4rv3ivQqHPLTmKWxEOnHPvgVvs6AqzfN1mr_3HSW_g1PUQnEwKL3c8lC6kRcaiQYFRhWUpchG1QCAyKjGIQRqHJkhMSIYk6SCJwQRgQr9osXfCdhbLBZ0yHktlQGktCRIgz2DqipTAisUZQjBnrGXXP_uoJC9m9dLP_x6-YfuDyctwNnyKni_Ygd3sCk11yXby1Yau2J75zLP16ro8sW84dpWd |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=RigNeRF%3A+Fully+Controllable+Neural+3D+Portraits&rft.au=Athar%2C+ShahRukh&rft.au=Xu%2C+Zexiang&rft.au=Sunkavalli%2C+Kalyan&rft.au=Shechtman%2C+Eli&rft.date=2022-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=20332&rft.epage=20341&rft_id=info:doi/10.1109%2FCVPR52688.2022.01972&rft.externalDocID=9880222 |