Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation

Standard registration algorithms need to be independently applied to each surface to register, following careful pre-processing and hand-tuning. Recently, learning-based approaches have emerged that reduce the registration of new scans to running inference with a previously-trained model. The potent...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:International journal of computer vision Ročník 129; číslo 9; s. 2680 - 2713
Hlavní autoři: Bahri, Mehdi, O’ Sullivan, Eimear, Gong, Shunwang, Liu, Feng, Liu, Xiaoming, Bronstein, Michael M., Zafeiriou, Stefanos
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York Springer US 01.09.2021
Springer
Springer Nature B.V
Témata:
Activities and Shape in 3D
Algorithms
Artificial Intelligence
Biometry
Coders
Computer Imaging
Computer Science
Data mining
Datasets
Deep learning
Encoders-Decoders
Finite element method
Image Processing and Computer Vision
Inference
Machine learning
Morphing
Motion
Optimization
Parameter robustness
Pattern Recognition
Pattern Recognition and Graphics
Registration
Robustness
Special Issue on Human Pose
Vision
Generative modeling
Non linear morphable models
Face modeling
Point cloud
Surface registration
Graph neural network
ISSN:0920-5691, 1573-1405
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Standard registration algorithms need to be independently applied to each surface to register, following careful pre-processing and hand-tuning. Recently, learning-based approaches have emerged that reduce the registration of new scans to running inference with a previously-trained model. The potential benefits are multifold: inference is typically orders of magnitude faster than solving a new instance of a difficult optimization problem, deep learning models can be made robust to noise and corruption, and the trained model may be re-used for other tasks, e.g. through transfer learning. In this paper, we cast the registration task as a surface-to-surface translation problem, and design a model to reliably capture the latent geometric information directly from raw 3D face scans. We introduce Shape-My-Face (SMF), a powerful encoder-decoder architecture based on an improved point cloud encoder, a novel visual attention mechanism, graph convolutional decoders with skip connections, and a specialized mouth model that we smoothly integrate with the mesh convolutions. Compared to the previous state-of-the-art learning algorithms for non-rigid registration of face scans, SMF only requires the raw data to be rigidly aligned (with scaling) with a pre-defined face template. Additionally, our model provides topologically-sound meshes with minimal supervision, offers faster training time, has orders of magnitude fewer trainable parameters, is more robust to noise, and can generalize to previously unseen datasets. We extensively evaluate the quality of our registrations on diverse data. We demonstrate the robustness and generalizability of our model with in-the-wild face scans across different modalities, sensor types, and resolutions. Finally, we show that, by learning to register scans, SMF produces a hybrid linear and non-linear morphable model. Manipulation of the latent space of SMF allows for shape generation, and morphing applications such as expression transfer in-the-wild. We train SMF on a dataset of human faces comprising 9 large-scale databases on commodity hardware.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0920-5691
1573-1405
DOI:10.1007/s11263-021-01494-4