Neural 3D Video Synthesis from Multi-view Video
We propose a novel approach for 3D video synthesis that is able to represent multi-view video recordings of a dynamic real-world scene in a compact, yet expressive representation that enables high-quality view synthesis and motion interpolation. Our approach takes the high quality and compactness of...
Uloženo v:
| Vydáno v: | Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) s. 5511 - 5521 |
|---|---|
| Hlavní autoři: | , , , , , , , , , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
IEEE
01.06.2022
|
| Témata: | |
| ISSN: | 1063-6919 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | We propose a novel approach for 3D video synthesis that is able to represent multi-view video recordings of a dynamic real-world scene in a compact, yet expressive representation that enables high-quality view synthesis and motion interpolation. Our approach takes the high quality and compactness of static neural radiance fields in a new direction: to a model-free, dynamic setting. At the core of our approach is a novel time-conditioned neural radiance field that represents scene dynamics using a set of compact latent codes. We are able to significantly boost the training speed and perceptual quality of the generated imagery by a novel hierarchical training scheme in combination with ray importance sampling. Our learned representation is highly compact and able to represent a 10 second 30 FPS multi-view video recording by 18 cameras with a model size of only 28MB. We demonstrate that our method can render high-fidelity wide-angle novel views at over 1K resolution, even for complex and dynamic scenes. We perform an extensive qualitative and quantitative evaluation that shows that our approach outperforms the state of the art. Project website: https://neural-3d-video.github.io/. |
|---|---|
| AbstractList | We propose a novel approach for 3D video synthesis that is able to represent multi-view video recordings of a dynamic real-world scene in a compact, yet expressive representation that enables high-quality view synthesis and motion interpolation. Our approach takes the high quality and compactness of static neural radiance fields in a new direction: to a model-free, dynamic setting. At the core of our approach is a novel time-conditioned neural radiance field that represents scene dynamics using a set of compact latent codes. We are able to significantly boost the training speed and perceptual quality of the generated imagery by a novel hierarchical training scheme in combination with ray importance sampling. Our learned representation is highly compact and able to represent a 10 second 30 FPS multi-view video recording by 18 cameras with a model size of only 28MB. We demonstrate that our method can render high-fidelity wide-angle novel views at over 1K resolution, even for complex and dynamic scenes. We perform an extensive qualitative and quantitative evaluation that shows that our approach outperforms the state of the art. Project website: https://neural-3d-video.github.io/. |
| Author | Zollhoefer, Michael Schmidt, Tanner Goesele, Michael Lovegrove, Steven Green, Simon Li, Tianye Slavcheva, Mira Lv, Zhaoyang Lassner, Christoph Newcombe, Richard Kim, Changil |
| Author_xml | – sequence: 1 givenname: Tianye surname: Li fullname: Li, Tianye organization: University of Southern,California – sequence: 2 givenname: Mira surname: Slavcheva fullname: Slavcheva, Mira organization: Reality Labs Research – sequence: 3 givenname: Michael surname: Zollhoefer fullname: Zollhoefer, Michael organization: Reality Labs Research – sequence: 4 givenname: Simon surname: Green fullname: Green, Simon organization: Reality Labs Research – sequence: 5 givenname: Christoph surname: Lassner fullname: Lassner, Christoph organization: Reality Labs Research – sequence: 6 givenname: Changil surname: Kim fullname: Kim, Changil organization: Meta – sequence: 7 givenname: Tanner surname: Schmidt fullname: Schmidt, Tanner organization: Reality Labs Research – sequence: 8 givenname: Steven surname: Lovegrove fullname: Lovegrove, Steven organization: Reality Labs Research – sequence: 9 givenname: Michael surname: Goesele fullname: Goesele, Michael organization: Reality Labs Research – sequence: 10 givenname: Richard surname: Newcombe fullname: Newcombe, Richard organization: Reality Labs Research – sequence: 11 givenname: Zhaoyang surname: Lv fullname: Lv, Zhaoyang organization: Reality Labs Research |
| BookMark | eNotjstKw0AUQEdRsK39Al3kB5LeO4-bmaXEJ9QHProtk8wNjqSJJKnSv1epq7M4cDhTcdR2LQtxjpAhglsUq6dnI8naTIKUGYDR-kBMkchocprUoZggkErJoTsR82H4AAAlEcnZiVg88Lb3TaIuk1UM3CUvu3Z85yEOSd13m-R-24wx_Yr8vfen4rj2zcDzf87E2_XVa3GbLh9v7oqLZRolqDH1VgWJdU7GVMFom5s8eJJVYHKevf2Do1pqJKxUCWVlqvL3CZT3eZClmomzfTcy8_qzjxvf79bO5tZZp34AQ5lFMg |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/CVPR52688.2022.00544 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Applied Sciences |
| EISBN | 1665469463 9781665469463 |
| EISSN | 1063-6919 |
| EndPage | 5521 |
| ExternalDocumentID | 9878989 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IH 6IL 6IN AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP OCL RIE RIL RIO |
| ID | FETCH-LOGICAL-i203t-a83d21f7655cd548757da62cde69aea8e69a96f24161c3b0bc5cb21103aa7d2b3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 177 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000867754205075&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:15:10 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i203t-a83d21f7655cd548757da62cde69aea8e69a96f24161c3b0bc5cb21103aa7d2b3 |
| PageCount | 11 |
| ParticipantIDs | ieee_primary_9878989 |
| PublicationCentury | 2000 |
| PublicationDate | 2022-June |
| PublicationDateYYYYMMDD | 2022-06-01 |
| PublicationDate_xml | – month: 06 year: 2022 text: 2022-June |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) |
| PublicationTitleAbbrev | CVPR |
| PublicationYear | 2022 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0003211698 |
| Score | 2.660716 |
| Snippet | We propose a novel approach for 3D video synthesis that is able to represent multi-view video recordings of a dynamic real-world scene in a compact, yet... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 5511 |
| SubjectTerms | 3D from multi-view and sensors; Image and video synthesis and generation Cameras Dynamics Heuristic algorithms Interpolation Monte Carlo methods Three-dimensional displays Training |
| Title | Neural 3D Video Synthesis from Multi-view Video |
| URI | https://ieeexplore.ieee.org/document/9878989 |
| WOSCitedRecordID | wos000867754205075&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB5q8eCpaiu-ycGjsbvJbh7navEgpfgovZW8CguylW4r-O_N7C4VwYunhOQQkhBmvsl83wDcKBmsYSmnJsJlmkUPgUYrl1BlteI21UHUap-zJzmZqPlcTztwu-PChBDq5LNwh936L9-v3BZDZcOIj7Ha4R7sSSkartYunsIjkhFatey4NNHD0Wz6jGImmMDFUJYzz7JfNVRqEzLu_W_xQxj8cPHIdGdljqATymPotc4jaZ9m1YchymyYd8LvyazwYUVevsro3FVFRZBCQmqmLcWPgGZ-AG_jh9fRI22rIdCCJXxDjeKepUsp8tz5GmdIbwRzPghtglHYaLFkiFgct4l1ubMI77gx0jPLT6BbrspwCoQ5pZa51doIn2krbXSzvGTB5UkQmVZn0Mf9Lz4awYtFu_Xzv4cv4AAPuMmfuoTuZr0NV7DvPjdFtb6ub-kbyp2Rdg |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB5qFfRUtYpvc_Bo7G6ym8e5WirWUrSW3kpehQXZSrcV_Pcmu0tF8OIpISGEJISZL5nvG4AbwZ1WJKZYebiME-8hYG_lIiy0FFTH0rFS7XMy4MOhmE7lqAG3Gy6Mc64MPnN3oVr-5duFWYenso7HxyHb4RZsp0lCooqttXlRoR7LMClqflwcyU53MnoJciYhhIsEYU4_7FcWldKI9Fr_m34fjn7YeGi0sTMH0HD5IbRq9xHVl7NoQycIbah3RO_RJLNugV6_cu_eFVmBAokElVxbHL4Cqv4jeOs9jLt9XOdDwBmJ6AorQS2J55ylqbEl0uBWMWKsY1I5JUIh2ZwEzGKojrRJjQ4AjyrFLdH0GJr5IncngIgRYp5qKRWzidRce0fLcuJMGjmWSHEK7bD-2UcleTGrl372d_M17PbHz4PZ4HH4dA57YbOraKoLaK6Wa3cJO-ZzlRXLq_LEvgGOk5S9 |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=Neural+3D+Video+Synthesis+from+Multi-view+Video&rft.au=Li%2C+Tianye&rft.au=Slavcheva%2C+Mira&rft.au=Zollhoefer%2C+Michael&rft.au=Green%2C+Simon&rft.date=2022-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=5511&rft.epage=5521&rft_id=info:doi/10.1109%2FCVPR52688.2022.00544&rft.externalDocID=9878989 |