Automatic Front-end Code Generation from image Via Multi-Head Attention

Code generation from Graphical User Interface (GUI) screenshots is a challenging task in machine learning. Existing methods (e.g., Pix2code) can handle simple datasets well but struggle with complex datasets requiring hundreds of code tokens. This paper proposes a novel method for generating front-e...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (International Conference on Computer Engineering and Applications. Online) pp. 869 - 872
Main Authors: Zhang, Zhihang, Ding, Ye, Huang, Chenlin
Format: Conference Proceeding
Language:English
Published: IEEE 07.04.2023
Subjects:
Codes
Computational modeling
Computer architecture
Graphical user interfaces
Image recognition
Machine learning
Neural networks
Object recognition
Scene understanding
Task analysis
User interface programming
ISSN:2159-1288
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Code generation from Graphical User Interface (GUI) screenshots is a challenging task in machine learning. Existing methods (e.g., Pix2code) can handle simple datasets well but struggle with complex datasets requiring hundreds of code tokens. This paper proposes a novel method for generating front-end code based on multi-head attention. Our method uses a special technique called multi-head attention to analyze a GUI screenshot's feature vector, generate the code tokens, and link the analysis and generation processes. This architecture gives our method a significant advantage over similar models in terms of effectiveness. We conduct experiments on two types of datasets: Pix2code datasets and our own datasets. The experimental results demonstrate that our method achieves the best performance among existing methods.
ISSN:2159-1288
DOI:10.1109/ICCEA58433.2023.10135462