SPIDER: A framework for processing, editing and presenting immersive high-resolution spherical indoor scenes

Today’s Extended Reality (XR) applications that call for specific Diminished Reality (DR) strategies to hide specific classes of objects are increasingly using 360° cameras, which can capture entire areas in a single picture. In this work, we present an interactive-based image processing, editing an...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Graphical models Ročník 128; s. 101182
Hlavní autoři: Tukur, M., Pintore, G., Gobbetti, E., Schneider, J., Agus, M.
Médium: Journal Article
Jazyk:angličtina
Vydáno: Elsevier Inc 01.07.2023
Elsevier
Témata:
Deep Learning
Extended Reality
Indoor Environments
Spherical images
Super resolution
Reconstruction
Spherical images
Computing methodologies: Image-based rendering
Indoor Environments
Extended Reality
Applied computing: Architecture (buildings)
Deep Learning
Super resolution
ISSN:1524-0703
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Today’s Extended Reality (XR) applications that call for specific Diminished Reality (DR) strategies to hide specific classes of objects are increasingly using 360° cameras, which can capture entire areas in a single picture. In this work, we present an interactive-based image processing, editing and rendering system named SPIDER, that takes a spherical 360° indoor scene as input. The system is composed of a novel integrated deep learning architecture for extracting geometric and semantic information of full and empty rooms, based on gated and dilated convolutions, followed by a super-resolution module for improving the resolution of the color and depth signals. The obtained high resolution representations allow users to perform interactive exploration and basic editing operations on the reconstructed indoor scene, namely: (i) rendering of the scene in various modalities (point cloud, polygonal, wireframe) (ii) refurnishing (transferring portions of rooms) (iii) deferred shading through the usage of precomputed normal maps. These kinds of scene editing and manipulations can be used for assessing the inference from deep learning models and enable several Mixed Reality applications in areas such as furniture retails, interior designs, and real estates. Moreover, it can also be useful in data augmentation, arts, designs, and paintings. We report on the performance improvement of the various processing components on public domain spherical image indoor datasets. [Display omitted]
ISSN:1524-0703
DOI:10.1016/j.gmod.2023.101182