FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-Pose, and Facial Expression Features

The task of face reenactment is to transfer the head motion and facial expressions from a driving video to the appearance of a source image, which may be of a different person (cross-reenactment). Most existing methods are CNN-based and estimate optical flow from the source image to the current driv...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 7716 - 7726
Main Authors: Rochow, Andre, Schwarz, Max, Behnke, Sven
Format: Conference Proceeding
Language:English
Published: IEEE 16.06.2024
Subjects:
Animation
Face recognition
Face Reenactment
Facial Animation
Rendering (computer graphics)
Shape
Training
Transformers
Vectors
ISSN:1063-6919
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract The task of face reenactment is to transfer the head motion and facial expressions from a driving video to the appearance of a source image, which may be of a different person (cross-reenactment). Most existing methods are CNN-based and estimate optical flow from the source image to the current driving frame, which is then inpainted and refined to produce the output animation. We propose a transformer-based encoder for computing a set-latent representation of the source image(s). We then predict the output color of a query pixel using a transformer-based decoder, which is conditioned with keypoints and a facial expression vector extracted from the driving frame. Latent representations of the source person are learned in a self-supervised manner that factorize their appearance, head pose, and facial expressions. Thus, they are perfectly suited for cross-reenactment. In contrast to most related work, our method naturally extends to multiple source images and can thus adapt to person-specific facial dynamics. We also propose data augmentation and regularization schemes that are necessary to prevent overfitting and support generalizability of the learned representations. We evaluated our approach in a randomized user study. The results indicate superior performance compared to the state-of-the-art in terms of motion transfer quality and temporal consistency. 1 1 Code & Video: https://andrerochow.github.io/fsrt
AbstractList The task of face reenactment is to transfer the head motion and facial expressions from a driving video to the appearance of a source image, which may be of a different person (cross-reenactment). Most existing methods are CNN-based and estimate optical flow from the source image to the current driving frame, which is then inpainted and refined to produce the output animation. We propose a transformer-based encoder for computing a set-latent representation of the source image(s). We then predict the output color of a query pixel using a transformer-based decoder, which is conditioned with keypoints and a facial expression vector extracted from the driving frame. Latent representations of the source person are learned in a self-supervised manner that factorize their appearance, head pose, and facial expressions. Thus, they are perfectly suited for cross-reenactment. In contrast to most related work, our method naturally extends to multiple source images and can thus adapt to person-specific facial dynamics. We also propose data augmentation and regularization schemes that are necessary to prevent overfitting and support generalizability of the learned representations. We evaluated our approach in a randomized user study. The results indicate superior performance compared to the state-of-the-art in terms of motion transfer quality and temporal consistency. 1 1 Code & Video: https://andrerochow.github.io/fsrt
Author Schwarz, Max
Behnke, Sven
Rochow, Andre
Author_xml – sequence: 1
  givenname: Andre
  surname: Rochow
  fullname: Rochow, Andre
  email: rochow@ais.uni-bonn.de
  organization: University of Bonn
– sequence: 2
  givenname: Max
  surname: Schwarz
  fullname: Schwarz, Max
  email: schwarz@ais.uni-bonn.de
  organization: University of Bonn
– sequence: 3
  givenname: Sven
  surname: Behnke
  fullname: Behnke, Sven
  email: behnke@cs.uni-bonn.de
  organization: University of Bonn
BookMark eNo1kN1Kw0AQhVdRUGvfoBf7AKbuTzab9a6UxgoFS1u9LZPdCUSaTdhEUJ_Ax3aDejGcmeGbc2BuyIVvPRIy42zOOTP3y9ftTgkt5Vwwkc4Z01KfkanRJpeKSSUZy87JNWeZTDLDzRWZ9v0bY0wKzjOTX5PvYr87PNACbA0nurfoke6wC9ijH2CoW08PAXxftaHBQKOM7MigBzs0kaJVaJtxO7Sh_kJHF12HEI8s3tE1gku2bR9b8O4_Z_UxJvSje4EwvMfhllxWcOpx-qcT8lKsDst1snl-fFouNknNdTYkvExzIVHHyrUEk2KaGzDWpsI6UEaI0maa2RJL4SrunFCpy1PplLOl4iAnZPbrWyPisQt1A-HzGD-klFRa_gCWs2az
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/CVPR52733.2024.00737
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Applied Sciences
EISBN 9798350353006
EISSN 1063-6919
EndPage 7726
ExternalDocumentID 10655357
Genre orig-research
GroupedDBID 6IE
6IH
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-i176t-1b4823e723e873a94e489a9cc42cda5922bc670cbeb2df1dd254d843d5dcb51a3
IEDL.DBID RIE
ISICitedReferencesCount 12
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001322555908012&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:00:51 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i176t-1b4823e723e873a94e489a9cc42cda5922bc670cbeb2df1dd254d843d5dcb51a3
PageCount 11
ParticipantIDs ieee_primary_10655357
PublicationCentury 2000
PublicationDate 2024-June-16
PublicationDateYYYYMMDD 2024-06-16
PublicationDate_xml – month: 06
  year: 2024
  text: 2024-June-16
  day: 16
PublicationDecade 2020
PublicationTitle Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online)
PublicationTitleAbbrev CVPR
PublicationYear 2024
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0003211698
Score 2.3341343
Snippet The task of face reenactment is to transfer the head motion and facial expressions from a driving video to the appearance of a source image, which may be of a...
SourceID ieee
SourceType Publisher
StartPage 7716
SubjectTerms Animation
Face recognition
Face Reenactment
Facial Animation
Rendering (computer graphics)
Shape
Training
Transformers
Vectors
Title FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-Pose, and Facial Expression Features
URI https://ieeexplore.ieee.org/document/10655357
WOSCitedRecordID wos001322555908012&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LT8MwDI5g4sBpPIZ4KweOFNo6aRKu06od0FSNMe02pUkq7dKhPRDiF_CzcbJuiAMHDlVTy6ql1G5sx59DyF2VmhLdBIis4TpiUMpIA5dRZQGYThNtQh5y_CwGAzmZqKIBqwcsjHMuFJ-5Bz8Me_l2btY-VYYWnnEOXOyTfSGyDVhrl1ABDGUyJRt4XBKrx-64GPr-YoBhYBqaZMPvQ1TCGpK3_yn9iHR-0Hi02K0zx2TP1Sek3biPtDHO5Sn5yl-Goyeaa58FRzr-xegwFLo2-KKajrZuqltQvHlez-NqbUK1OfVoE0_1rUM-8e0oBU3Ba8Y97aM6RMV8iUNd262c3kdTSltT706u8aFDXvPeqNuPmoMWolkislWUlEym4AReUoBWzDGptDKGpcZqrtK0NJmITYlhuK0SazGqtJKB5daUPNFwRlr1vHbnhNrYiExLyyDWyGCkBKc5MKiU4AzcBen4mZ2-bXppTLeTevkH_Yoc-o_ni7OS7Jq0Vou1uyEH5n01Wy5ugwZ8A9-Is5Q
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LT8MwDI5gIMFpPIZ4kwNHCm2dtCnXadUQY6pGmbhNaZJKu3RoD4T4BfxsnKwb4sCBQ9XUsmoptRvb8ecQcl2GqkA3ATytuPQYFMKTwIVXagAmw0Aql4cc9uJ-X7y-JlkNVndYGGOMKz4zt3bo9vL1RC1sqgwtPOIceLxJtjjDwGcJ11qnVACDmSgRNUAu8JO79jAb2A5jgIFg6Npkw-9jVNwqkjb_KX-PtH7weDRbrzT7ZMNUB6RZO5C0Ns_ZIflKnwf5PU2lzYMjHf9jdOBKXWuEUUXzlaNqphRvltfymEoqV29OLd7EUm3zkE98O0pBY7C6cUO7qBBeNpnhUFZ6JafzURfTVtQ6lAt8aJGXtJO3u1591II3DuJo7gUFEyGYGC8Rg0yYYSKRiVIsVFryJAwLFcW-KjAQ12WgNcaVWjDQXKuCBxKOSKOaVOaYUO2rOJJCM_AlMighwEgODMok5gzMCWnZmR29LbtpjFaTevoH_YrsdPOn3qj30H88I7v2Q9pSrSA6J435dGEuyLZ6n49n00unDd9gRrbb
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=FSRT%3A+Facial+Scene+Representation+Transformer+for+Face+Reenactment+from+Factorized+Appearance%2C+Head-Pose%2C+and+Facial+Expression+Features&rft.au=Rochow%2C+Andre&rft.au=Schwarz%2C+Max&rft.au=Behnke%2C+Sven&rft.date=2024-06-16&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=7716&rft.epage=7726&rft_id=info:doi/10.1109%2FCVPR52733.2024.00737&rft.externalDocID=10655357