High-Fidelity Monocular Face Reconstruction Based on an Unsupervised Model-Based Face Autoencoder

In this work, we propose a novel model-based deep convolutional autoencoder that addresses the highly challenging problem of reconstructing a 3D human face from a single in-the-wild color image. To this end, we combine a convolutional encoder network with an expert-designed generative model that ser...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE transactions on pattern analysis and machine intelligence Vol. 42; no. 2; pp. 357 - 370
Main Authors:	Tewari, Ayush, Zollhofer, Michael, Bernard, Florian, Garrido, Pablo, Kim, Hyeongwoo, Perez, Patrick, Theobalt, Christian
Format:	Journal Article
Language:	English
Published:	United States IEEE 01.02.2020 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:	Accuracy Coders Color imagery Decoding Deep Learning Face Face - anatomy & histology Face - diagnostic imaging Female Humans Image reconstruction Imaging, Three-Dimensional - methods Lighting Male Neural Networks, Computer Shading Shape Three-dimensional displays Training Unsupervised Machine Learning
ISSN:	0162-8828, 1939-3539, 2160-9292, 1939-3539
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	In this work, we propose a novel model-based deep convolutional autoencoder that addresses the highly challenging problem of reconstructing a 3D human face from a single in-the-wild color image. To this end, we combine a convolutional encoder network with an expert-designed generative model that serves as decoder. The core innovation is the differentiable parametric decoder that encapsulates image formation analytically based on a generative model. Our decoder takes as input a code vector with exactly defined semantic meaning that encodes detailed face pose, shape, expression, skin reflectance, and scene illumination. Due to this new way of combining CNN-based with model-based face reconstruction, the CNN-based encoder learns to extract semantically meaningful parameters from a single monocular input image. For the first time, a CNN encoder and an expert-designed generative model can be trained end-to-end in an unsupervised manner, which renders training on very large (unlabeled) real world datasets feasible. The obtained reconstructions compare favorably to current state-of-the-art approaches in terms of quality and richness of representation. This work is an extended version of [1] , where we additionally present a stochastic vertex sampling technique for faster training of our networks, and moreover, we propose and evaluate analysis-by-synthesis and shape-from-shading refinement approaches to achieve a high-fidelity reconstruction.
Bibliography:	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 content type line 23
ISSN:	0162-8828 1939-3539 2160-9292 1939-3539
DOI:	10.1109/TPAMI.2018.2876842