Makeup Lamps: Live Augmentation of Human Faces via Projection

We propose the first system for live dynamic augmentation of human faces. Using projector‐based illumination, we alter the appearance of human performers during novel performances. The key challenge of live augmentation is latency — an image is generated according to a specific pose, but is displaye...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Computer graphics forum Ročník 36; číslo 2; s. 311 - 323
Hlavní autoři: Bermano, Amit H., Billeter, Markus, Iwai, Daisuke, Grundhöfer, Anselm
Médium: Journal Article
Jazyk:angličtina
Vydáno: Oxford Blackwell Publishing Ltd 01.05.2017
Témata:
Adaptive filters
Augmentation
CAPTURE
Categories and Subject Descriptors (according to ACM CCS)
Computer graphics
Deformation
Face
FACIAL EXPRESSION
Forecasting
FRAMES
H.5.1 [HCI]: Multimedia Information Systems—Artificial, augmented, and virtual realities
Human performance
I.3.7 [Computer Graphics]: Three‐Dimensional Graphics and Realism—Animation
Illumination
Kalman filters
Real time
ISSN:0167-7055, 1467-8659, 1467-8659
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:We propose the first system for live dynamic augmentation of human faces. Using projector‐based illumination, we alter the appearance of human performers during novel performances. The key challenge of live augmentation is latency — an image is generated according to a specific pose, but is displayed on a different facial configuration by the time it is projected. Therefore, our system aims at reducing latency during every step of the process, from capture, through processing, to projection. Using infrared illumination, an optically and computationally aligned high‐speed camera detects facial orientation as well as expression. The estimated expression blendshapes are mapped onto a lower dimensional space, and the facial motion and non‐rigid deformation are estimated, smoothed and predicted through adaptive Kalman filtering. Finally, the desired appearance is generated interpolating precomputed offset textures according to time, global position, and expression. We have evaluated our system through an optimized CPU and GPU prototype, and demonstrated successful low latency augmentation for different performers and performances with varying facial play and motion speed. In contrast to existing methods, the presented system is the first method which fully supports dynamic facial projection mapping without the requirement of any physical tracking markers and incorporates facial expressions.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0167-7055
1467-8659
1467-8659
DOI:10.1111/cgf.13128