D‐NPC: Dynamic Neural Point Clouds for Non‐Rigid View Synthesis from Monocular Video

Dynamic reconstruction and spatiotemporal novel‐view synthesis of non‐rigidly deforming scenes recently gained increased attention. While existing work achieves impressive quality and performance on multi‐view or teleporting camera setups, most methods fail to efficiently and faithfully recover moti...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Computer graphics forum Ročník 44; číslo 2
Hlavní autoři: Kappel, Moritz, Hahlbohm, Florian, Scholz, Timon, Castillo, Susana, Theobalt, Christian, Eisemann, Martin, Golyanik, Vladislav, Magnor, Marcus
Médium: Journal Article
Jazyk:angličtina
Vydáno: Oxford Blackwell Publishing Ltd 01.05.2025
Témata:
CCS Concepts
Computing methodologies → Image‐based rendering
Image quality
Image reconstruction
Optimization
Point‐based models
Rasterization
Reconstruction
Scene analysis
Synthesis
Teleportation
ISSN:0167-7055, 1467-8659
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Dynamic reconstruction and spatiotemporal novel‐view synthesis of non‐rigidly deforming scenes recently gained increased attention. While existing work achieves impressive quality and performance on multi‐view or teleporting camera setups, most methods fail to efficiently and faithfully recover motion and appearance from casual monocular captures. This paper contributes to the field by introducing a new method for dynamic novel view synthesis from monocular video, such as casual smartphone captures. Our approach represents the scene as a dynamic neural point cloud, an implicit time‐conditioned point distribution that encodes local geometry and appearance in separate hash‐encoded neural feature grids for static and dynamic regions. By sampling a discrete point cloud from our model, we can efficiently render high‐quality novel views using a fast differentiable rasterizer and neural rendering network. Similar to recent work, we leverage advances in neural scene analysis by incorporating data‐driven priors like monocular depth estimation and object segmentation to resolve motion and depth ambiguities originating from the monocular captures. In addition to guiding the optimization process, we show that these priors can be exploited to explicitly initialize our scene representation to drastically improve optimization speed and final image quality. As evidenced by our experimental evaluation, our dynamic point cloud model not only enables fast optimization and real‐time frame rates for interactive applications, but also achieves competitive image quality on monocular benchmark sequences. Our code and data are available online https://moritzkappel.github.io/projects/dnpc/.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0167-7055
1467-8659
DOI:10.1111/cgf.70038