Automatic Front-end Code Generation from image Via Multi-Head Attention

Code generation from Graphical User Interface (GUI) screenshots is a challenging task in machine learning. Existing methods (e.g., Pix2code) can handle simple datasets well but struggle with complex datasets requiring hundreds of code tokens. This paper proposes a novel method for generating front-e...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (International Conference on Computer Engineering and Applications. Online) pp. 869 - 872
Main Authors: Zhang, Zhihang, Ding, Ye, Huang, Chenlin
Format: Conference Proceeding
Language:English
Published: IEEE 07.04.2023
Subjects:
Codes
Computational modeling
Computer architecture
Graphical user interfaces
Image recognition
Machine learning
Neural networks
Object recognition
Scene understanding
Task analysis
User interface programming
ISSN:2159-1288
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Code generation from Graphical User Interface (GUI) screenshots is a challenging task in machine learning. Existing methods (e.g., Pix2code) can handle simple datasets well but struggle with complex datasets requiring hundreds of code tokens. This paper proposes a novel method for generating front-end code based on multi-head attention. Our method uses a special technique called multi-head attention to analyze a GUI screenshot's feature vector, generate the code tokens, and link the analysis and generation processes. This architecture gives our method a significant advantage over similar models in terms of effectiveness. We conduct experiments on two types of datasets: Pix2code datasets and our own datasets. The experimental results demonstrate that our method achieves the best performance among existing methods.
AbstractList Code generation from Graphical User Interface (GUI) screenshots is a challenging task in machine learning. Existing methods (e.g., Pix2code) can handle simple datasets well but struggle with complex datasets requiring hundreds of code tokens. This paper proposes a novel method for generating front-end code based on multi-head attention. Our method uses a special technique called multi-head attention to analyze a GUI screenshot's feature vector, generate the code tokens, and link the analysis and generation processes. This architecture gives our method a significant advantage over similar models in terms of effectiveness. We conduct experiments on two types of datasets: Pix2code datasets and our own datasets. The experimental results demonstrate that our method achieves the best performance among existing methods.
Author Ding, Ye
Huang, Chenlin
Zhang, Zhihang
Author_xml – sequence: 1
  givenname: Zhihang
  surname: Zhang
  fullname: Zhang, Zhihang
  organization: School of Cyberspace Security, Dongguan University of Technology,Dongguan,China
– sequence: 2
  givenname: Ye
  surname: Ding
  fullname: Ding, Ye
  email: dingye@dgut.edu.cn
  organization: School of Cyberspace Security, Dongguan University of Technology,Dongguan,China
– sequence: 3
  givenname: Chenlin
  surname: Huang
  fullname: Huang, Chenlin
  organization: Academy of Computer Science, National University of Defense Technology,Changsha,China
BookMark eNo1j8tOwzAURA0CiVL6Byz8Aym-ftzYyyjqSypiA2wrp7lGQY2NEnfB3xMEXc1iRkdz7tlNTJEY4yCWAMI97ep6VRmrlVpKIdUSBCijUV6xhSudVUYoXRptr9lMgnEFSGvv2GIcP4UQSgKgwxnbVOecep-7I18PKeaCYsvr1BLfUKRhKlLkYUg973r_Qfy98_z5fMpdsSXf8ipnir-bB3Yb_GmkxX_O2dt69Vpvi_3LZldX-6IDcLlAJ83RBIcYNHkpLVKJeGyDsQ2hbADRtzKgCkoEQB8mKQ2-KbUqQ6O1mrPHP25HRIevYXo1fB8u7uoHO7lPEg
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/ICCEA58433.2023.10135462
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Xplore POP ALL
IEEE Xplore All Conference Proceedings
IEEE/IET Electronic Library (IEL) (UW System Shared)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE/IET Electronic Library (IEL) (UW System Shared)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9798350347548
EISSN 2159-1288
EndPage 872
ExternalDocumentID 10135462
Genre orig-research
GroupedDBID 6IE
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
OCL
RIE
RIL
ID FETCH-LOGICAL-i119t-6925c5f966f4ea2286e766cdf58be62b166ad2f63f30f16af01341ab7437fb443
IEDL.DBID RIE
IngestDate Wed Aug 27 02:19:52 EDT 2025
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i119t-6925c5f966f4ea2286e766cdf58be62b166ad2f63f30f16af01341ab7437fb443
PageCount 4
ParticipantIDs ieee_primary_10135462
PublicationCentury 2000
PublicationDate 2023-April-7
PublicationDateYYYYMMDD 2023-04-07
PublicationDate_xml – month: 04
  year: 2023
  text: 2023-April-7
  day: 07
PublicationDecade 2020
PublicationTitle Proceedings (International Conference on Computer Engineering and Applications. Online)
PublicationTitleAbbrev ICCEA
PublicationYear 2023
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0003211696
Score 1.8277982
Snippet Code generation from Graphical User Interface (GUI) screenshots is a challenging task in machine learning. Existing methods (e.g., Pix2code) can handle simple...
SourceID ieee
SourceType Publisher
StartPage 869
SubjectTerms Codes
Computational modeling
Computer architecture
Graphical user interfaces
Image recognition
Machine learning
Neural networks
Object recognition
Scene understanding
Task analysis
User interface programming
Title Automatic Front-end Code Generation from image Via Multi-Head Attention
URI https://ieeexplore.ieee.org/document/10135462
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELagYmDiVcRbHlhd4kfseKyiliKhqgOgblUcn6UMJKik_H5spyliYGCzLFmy7uQ7f_Z93yF0r6mhkFiPVFNdeICSaGKsYiThhnMDzFEb1fWf1XyeLZd6sSWrRy4MAMTiMxiFYfzLt025CU9l_oRTnooQcfeVkh1Za_egwj2UkVr21TqJfnjK88nYJ1jOR6FJ-Khf_quRSswj06N_7uAYDX8YeXixyzUnaA_qU3TUt2TA2xN6hh7Hm7aJKqx4GqQJCNQW540F3OlLBzfgQCnB1buPJPitKnDk4JKZdzYet21X_jhEr9PJSz4j214JpKJUt0Rqlpap8-DFCSgYyyQoKUvr0syAZIZKWVjmJHc8cVQWLglKboXxFwjljBD8HA3qpoYLbz4nfDrzMENnpfDLM8c4t7qUnBqRAbtEw2CY1Ucnh7HqbXL1x_w1Ogzmj-Uu6gYN2vUGbtFB-dVWn-u76MRvQVua-A
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELZQQYKpPIp444E1JX7EiccqamlFqToU1K2K47OUgQSVlN-P7TRFDAxsliVb1p3s82ff9x1CD5IoAqG2SDWSmQUooQyUjmkQMsWYAmqI9ur603g2S5ZLOd-S1T0XBgB88hn0XdP_5esq37inMrvDCYu4O3H3I85p2NC1dk8qzIIZIUWbrxPKx0maDgc2xDLWd2XC--0Ev0qp-Egy6v5zDceo98PJw_NdtDlBe1Ceom5blAFv9-gZehps6srrsOKREycIoNQ4rTTgRmHaOQI7Ugku3u1Zgt-KDHsWbjC27saDum4SIHvodTRcpONgWy0hKAiRdSAkjfLIWPhiOGSUJgJiIXJtokSBoIoIkWlqBDMsNERkJnRabpmyV4jYKM7ZOeqUVQkX1nyG24BmgYZMcm6HJ4YypmUuGFE8AXqJes4wq49GEGPV2uTqj_57dDhevExX08ns-RodOVf45Jf4BnXq9QZu0UH-VRef6zvv0G-D0p4_
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%28International+Conference+on+Computer+Engineering+and+Applications.+Online%29&rft.atitle=Automatic+Front-end+Code+Generation+from+image+Via+Multi-Head+Attention&rft.au=Zhang%2C+Zhihang&rft.au=Ding%2C+Ye&rft.au=Huang%2C+Chenlin&rft.date=2023-04-07&rft.pub=IEEE&rft.eissn=2159-1288&rft.spage=869&rft.epage=872&rft_id=info:doi/10.1109%2FICCEA58433.2023.10135462&rft.externalDocID=10135462