Realistic Speech-Driven Facial Animation with GANs

Speech-driven facial animation is the process that automatically synthesizes talking characters based on speech signals. The majority of work in this domain creates a mapping from audio features to visual features. This approach often requires post-processing using computer graphics techniques to pr...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:International journal of computer vision Ročník 128; číslo 5; s. 1398 - 1413
Hlavní autoři: Vougioukas, Konstantinos, Petridis, Stavros, Pantic, Maja
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York Springer US 01.05.2020
Springer
Springer Nature B.V
Témata:
Ablation
Animation
Artificial Intelligence
Computer graphics
Computer Imaging
Computer Science
Discriminators
Graphics software
Image Processing and Computer Vision
Image reconstruction
Logistics
Mapping
Pattern Recognition
Pattern Recognition and Graphics
Post-production processing
Sharpness
Signal processing
Special Issue on Generating Realistic Visual Data of Human Behavior
Speech
Synchronism
Vision
Visual discrimination
Face generation
Generative modelling
Speech-driven animation
ISSN:0920-5691, 1573-1405
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Speech-driven facial animation is the process that automatically synthesizes talking characters based on speech signals. The majority of work in this domain creates a mapping from audio features to visual features. This approach often requires post-processing using computer graphics techniques to produce realistic albeit subject dependent results. We present an end-to-end system that generates videos of a talking head, using only a still image of a person and an audio clip containing speech, without relying on handcrafted intermediate features. Our method generates videos which have (a) lip movements that are in sync with the audio and (b) natural facial expressions such as blinks and eyebrow movements. Our temporal GAN uses 3 discriminators focused on achieving detailed frames, audio-visual synchronization, and realistic expressions. We quantify the contribution of each component in our model using an ablation study and we provide insights into the latent representation of the model. The generated videos are evaluated based on sharpness, reconstruction quality, lip-reading accuracy, synchronization as well as their ability to generate natural blinks.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0920-5691
1573-1405
DOI:10.1007/s11263-019-01251-8