RigNeRF: Fully Controllable Neural 3D Portraits
Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes bey...
Uložené v:
| Vydané v: | Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) s. 20332 - 20341 |
|---|---|
| Hlavní autori: | , , , , |
| Médium: | Konferenčný príspevok.. |
| Jazyk: | English |
| Vydavateľské údaje: |
IEEE
01.06.2022
|
| Predmet: | |
| ISSN: | 1063-6919 |
| On-line prístup: | Získať plný text |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls. |
|---|---|
| AbstractList | Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls. |
| Author | Xu, Zexiang Sunkavalli, Kalyan Athar, ShahRukh Shu, Zhixin Shechtman, Eli |
| Author_xml | – sequence: 1 givenname: ShahRukh surname: Athar fullname: Athar, ShahRukh email: sathar@cs.stonybrook.edu organization: Stony Brook University – sequence: 2 givenname: Zexiang surname: Xu fullname: Xu, Zexiang email: zexu@adobe.com organization: Adobe Research – sequence: 3 givenname: Kalyan surname: Sunkavalli fullname: Sunkavalli, Kalyan email: sunkaval@adobe.com organization: Adobe Research – sequence: 4 givenname: Eli surname: Shechtman fullname: Shechtman, Eli email: elishe@adobe.com organization: Adobe Research – sequence: 5 givenname: Zhixin surname: Shu fullname: Shu, Zhixin email: zshu@adobe.com organization: Adobe Research |
| BookMark | eNotj81Kw0AURkdRsK19Al3kBZLe-Z9xJ9GoUGoJ6rZMJjcyMiYySRd9ewO6-Di7w_mW5KIfeiTklkJBKdhN-bGvJVPGFAwYK4Bazc7IkiolhbJC8XOyoKB4riy1V2Q9jl8AwBmlypoF2dThc4d1dZdVxxhPWTn0UxpidE3EbIfH5GLGH7L9kKbkwjRek8vOxRHX_1yR9-rxrXzOt69PL-X9Ng8M-JRzx6joQM4FUnipLZdInbPgBKKXrXdOd43xuvUGPTK0um2E18IbNa_hK3Lz5w2IePhJ4dul08EaM59k_BdJPEWi |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/CVPR52688.2022.01972 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE/IET Electronic Library IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE/IET Electronic Library url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Applied Sciences |
| EISBN | 1665469463 9781665469463 |
| EISSN | 1063-6919 |
| EndPage | 20341 |
| ExternalDocumentID | 9880222 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IH 6IL 6IN AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP OCL RIE RIL RIO |
| ID | FETCH-LOGICAL-i203t-3a214f0569454c57935e1aa90a4eec5dcaa7fb8c7dc8ece2e97db4c74c864c8b3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 90 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000870783006017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:15:10 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i203t-3a214f0569454c57935e1aa90a4eec5dcaa7fb8c7dc8ece2e97db4c74c864c8b3 |
| PageCount | 10 |
| ParticipantIDs | ieee_primary_9880222 |
| PublicationCentury | 2000 |
| PublicationDate | 2022-June |
| PublicationDateYYYYMMDD | 2022-06-01 |
| PublicationDate_xml | – month: 06 year: 2022 text: 2022-June |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) |
| PublicationTitleAbbrev | CVPR |
| PublicationYear | 2022 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0003211698 |
| Score | 2.5837767 |
| Snippet | Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 20332 |
| SubjectTerms | Deformable models Face and gestures; 3D from multi-view and sensors; Scene analysis and understanding Face recognition Image analysis Rendering (computer graphics) Solid modeling Three-dimensional displays Training |
| Title | RigNeRF: Fully Controllable Neural 3D Portraits |
| URI | https://ieeexplore.ieee.org/document/9880222 |
| WOSCitedRecordID | wos000870783006017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA61ePBUtRWtD3LwaNrsJpuH12rxIKUUld5KdnZWFkor7Vbw35vsLhXBi4dAyCFhEiYzk3zfDCG3CeYq9X4wy7z9YdJxzkwukIG39UlkQEFW5Zl91pOJmc_ttEXu9lwYRKzAZzgI3eovP1vDLjyVDa0JxFB_4R5orWqu1v49RfhIRlnTsOMiboejt-ksJDMJAK44HvBQYOtXDZXKhIw7_1v8mPR-uHh0urcyJ6SFq1PSaZxH2qjmtkuGs-J9grPxPQ1R5Rcd1Rj0ZaBG0ZCCwy2peKABObpxRbntkdfx48voiTXVEFgRc1Ey4eJI5t5fsTKRkHi9SjByznInESHJwDmdpwZ0BgYBY7Q6SyVoCUb5looz0l6tV3hOKJfCORdmMqlUKXjJwGidWuFyFefqgnSD_IuPOuHFohG9__fwJTkKG1zjp65Iu9zs8JocwmdZbDc31Sl9Ax60kkI |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3fS8MwED6GCvo0dRN_mwcf7ZY2SZv4Oh0TZxljyt5Gml6lMDbZOsH_3qQtE8EXHwIhDwmXcLm75PvuAG4FZmFi_WAvtfbH45pST2YMPWNtvfClCU1a5pkdRnEsp1M1asDdlguDiCX4DDuuW_7lp0uzcU9lXSUdMdReuLuC84BWbK3tiwqzsUyoZM2P86nq9t5GY5fOxEG4gqBDXYmtX1VUSiPSb_5v-UNo_7DxyGhrZ46ggYtjaNbuI6mVc92C7jh_j3HcvycurvwivQqFPnfkKOKScOg5YQ_EYUdXOi_WbXjtP056A6-uh-DlAWWFx3Tg88x6LIoLboTVLIG-1opqjmhEarSOskSaKDUSDQaoojThJuJGhrYl7AR2FssFngKhnGmt3Uwy4WFirGRGRlGimM7CIAvPoOXkn31UKS9mtejnfw_fwP5g8jKcDZ_i5ws4cJtdoakuYadYbfAK9sxnka9X1-WJfQPtdZWJ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=RigNeRF%3A+Fully+Controllable+Neural+3D+Portraits&rft.au=Athar%2C+ShahRukh&rft.au=Xu%2C+Zexiang&rft.au=Sunkavalli%2C+Kalyan&rft.au=Shechtman%2C+Eli&rft.date=2022-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=20332&rft.epage=20341&rft_id=info:doi/10.1109%2FCVPR52688.2022.01972&rft.externalDocID=9880222 |