SMFS‐GAN: Style‐Guided Multi‐class Freehand Sketch‐to‐Image Synthesis

Freehand sketch‐to‐image (S2I) is a challenging task due to the individualized lines and the random shape of freehand sketches. The multi‐class freehand sketch‐to‐image synthesis task, in turn, presents new challenges for this research area. This task requires not only the consideration of the probl...

Full description

Saved in:
Bibliographic Details
Published in:Computer graphics forum Vol. 43; no. 6
Main Authors: Cheng, Zhenwei, Wu, Lei, Li, Xiang, Meng, Xiangxu
Format: Journal Article
Language:English
Published: Oxford Blackwell Publishing Ltd 01.09.2024
Subjects:
Controllability
Datasets
image and video processing
Image contrast
image generation
Sketches
Synthesis
Texture
ISSN:0167-7055, 1467-8659
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Freehand sketch‐to‐image (S2I) is a challenging task due to the individualized lines and the random shape of freehand sketches. The multi‐class freehand sketch‐to‐image synthesis task, in turn, presents new challenges for this research area. This task requires not only the consideration of the problems posed by freehand sketches but also the analysis of multi‐class domain differences in the conditions of a single model. However, existing methods often have difficulty learning domain differences between multiple classes, and cannot generate controllable and appropriate textures while maintaining shape stability. In this paper, we propose a style‐guided multi‐class freehand sketch‐to‐image synthesis model, SMFS‐GAN, which can be trained using only unpaired data. To this end, we introduce a contrast‐based style encoder that optimizes the network's perception of domain disparities by explicitly modelling the differences between classes and thus extracting style information across domains. Further, to optimize the fine‐grained texture of the generated results and the shape consistency with freehand sketches, we propose a local texture refinement discriminator and a Shape Constraint Module, respectively. In addition, to address the imbalance of data classes in the QMUL‐Sketch dataset, we add 6K images by drawing manually and obtain QMUL‐Sketch+ dataset. Extensive experiments on SketchyCOCO Object dataset, QMUL‐Sketch+ dataset and Pseudosketches dataset demonstrate the effectiveness as well as the superiority of our proposed method. We propose SMFS‐GAN, a style‐guided multiclass freehand sketch‐to‐image synthesis model. We optimize image generation from both style and shape perspectives, enabling the model to generate high‐quality images with controllable style and stable shape from multiclass freehand sketches and style reference images.
Bibliography:Corresponding author
i_lily@sdu.edu.cn
ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0167-7055
1467-8659
DOI:10.1111/cgf.15190