Natural Language is Not Enough: Benchmarking Multi-Modal Generative AI for Verilog Generation

Natural language interfaces have exhibited considerable potential in the automation of Verilog generation derived from high-level specifications through the utilization of large language models, garnering significant attention. Nevertheless, this paper elucidates that visual representations contribu...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Digest of technical papers - IEEE/ACM International Conference on Computer-Aided Design S. 1 - 9
Hauptverfasser: Chang, Kaiyan, Chen, Zhirong, Zhou, Yunhao, Zhu, Wenlong, Wang, Kun, Xu, Haobo, Li, Cangyuan, Wang, Mengdi, Liang, Shengwen, Li, Huawei, Han, Yinhe, Wang, Ying
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: ACM 27.10.2024
Schlagworte:
Benchmark testing
Database languages
Design automation
Design methodology
Generative AI
Hardware
Hardware design languages
Large language models
Natural language processing
Visualization
ISSN:1558-2434
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Abstract Natural language interfaces have exhibited considerable potential in the automation of Verilog generation derived from high-level specifications through the utilization of large language models, garnering significant attention. Nevertheless, this paper elucidates that visual representations contribute essential contextual information critical to design intent for hardware architectures possessing spatial complexity, potentially surpassing the efficacy of natural-language-only inputs. Expanding upon this premise, our paper introduces an open-source benchmark1 for multi-modal generative models tailored for Verilog synthesis from visual-linguistic inputs, addressing both singu-lar and complex modules. Additionally, we introduce an open-source visual and natural language Verilog query language framework to facilitate efficient and user-friendly multi-modal queries. To evaluate the performance of the proposed multi-modal hardware generative AI in Verilog generation tasks, we compare it with a popular method that relies solely on natural language. Our results demonstrate a sig-nificant accuracy improvement in the multi-modal generated Verilog compared to queries based solely on natural language. We hope to reveal a new approach to hardware design in the large- hardware-design-model era, thereby fostering a more diversified and productive approach to hardware design.
AbstractList Natural language interfaces have exhibited considerable potential in the automation of Verilog generation derived from high-level specifications through the utilization of large language models, garnering significant attention. Nevertheless, this paper elucidates that visual representations contribute essential contextual information critical to design intent for hardware architectures possessing spatial complexity, potentially surpassing the efficacy of natural-language-only inputs. Expanding upon this premise, our paper introduces an open-source benchmark1 for multi-modal generative models tailored for Verilog synthesis from visual-linguistic inputs, addressing both singu-lar and complex modules. Additionally, we introduce an open-source visual and natural language Verilog query language framework to facilitate efficient and user-friendly multi-modal queries. To evaluate the performance of the proposed multi-modal hardware generative AI in Verilog generation tasks, we compare it with a popular method that relies solely on natural language. Our results demonstrate a sig-nificant accuracy improvement in the multi-modal generated Verilog compared to queries based solely on natural language. We hope to reveal a new approach to hardware design in the large- hardware-design-model era, thereby fostering a more diversified and productive approach to hardware design.
Author Liang, Shengwen
Chen, Zhirong
Xu, Haobo
Zhou, Yunhao
Wang, Ying
Han, Yinhe
Wang, Mengdi
Wang, Kun
Li, Cangyuan
Li, Huawei
Chang, Kaiyan
Zhu, Wenlong
Author_xml – sequence: 1
  givenname: Kaiyan
  surname: Chang
  fullname: Chang, Kaiyan
  email: changkaiyan@live.com
  organization: Institute of Computing Technology, Chinese Academy of Sciences,SKLP,Beijing,China
– sequence: 2
  givenname: Zhirong
  surname: Chen
  fullname: Chen, Zhirong
  organization: Institute of Computing Technology, Chinese Academy of Sciences,SKLP,Beijing,China
– sequence: 3
  givenname: Yunhao
  surname: Zhou
  fullname: Zhou, Yunhao
  organization: Institute of Computing Technology, Chinese Academy of Sciences,CICS,Beijing,China
– sequence: 4
  givenname: Wenlong
  surname: Zhu
  fullname: Zhu, Wenlong
  organization: Institute of Computing Technology, Chinese Academy of Sciences,SKLP,Beijing,China
– sequence: 5
  givenname: Kun
  surname: Wang
  fullname: Wang, Kun
  organization: Institute of Computing Technology, Chinese Academy of Sciences,SKLP,Beijing,China
– sequence: 6
  givenname: Haobo
  surname: Xu
  fullname: Xu, Haobo
  organization: Institute of Computing Technology, Chinese Academy of Sciences,SKLP,Beijing,China
– sequence: 7
  givenname: Cangyuan
  surname: Li
  fullname: Li, Cangyuan
  organization: Institute of Computing Technology, Chinese Academy of Sciences,SKLP,Beijing,China
– sequence: 8
  givenname: Mengdi
  surname: Wang
  fullname: Wang, Mengdi
  organization: Institute of Computing Technology, Chinese Academy of Sciences,SKLP,Beijing,China
– sequence: 9
  givenname: Shengwen
  surname: Liang
  fullname: Liang, Shengwen
  organization: Institute of Computing Technology, Chinese Academy of Sciences,SKLP,Beijing,China
– sequence: 10
  givenname: Huawei
  surname: Li
  fullname: Li, Huawei
  organization: Institute of Computing Technology, Chinese Academy of Sciences,SKLP,Beijing,China
– sequence: 11
  givenname: Yinhe
  surname: Han
  fullname: Han, Yinhe
  organization: Institute of Computing Technology, Chinese Academy of Sciences,SKLP,Beijing,China
– sequence: 12
  givenname: Ying
  surname: Wang
  fullname: Wang, Ying
  email: wangying2009@ict.ac.cn
  organization: Institute of Computing Technology, Chinese Academy of Sciences,SKLP,Beijing,China
BookMark eNo9jrtOwzAUhg0CiVI6szD4BVJ8PzFbqUqp1JYF2FDlxCepRbBRLki8Pa1ATN_wX_RdkrOYIhJyzdmUc6VvpQGjpZkeacCekIkFmyvGgDMAeUpGXOs8E0qqCzLpulAww_ShoPIRedu6fmhdQ9cu1oOrkYaOblNPFzEN9f6O3mMs9x-ufQ-xppuh6UO2Sf4wWGLE1vXhC-lsRavU0ldsQ5Pq_yTFK3JeuabDyR_H5OVh8Tx_zNZPy9V8ts6csLzPbOVE5cB5lIWGox1wEMwLFFCaQom8tIVSRenL3JTaQOG9syCd8FIAcDkmN7-_ARF3n204CH_vOOfCSGnkDz7sVYI
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1145/3676536.3676679
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Xplore Digital Library
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISBN 9798400710773
EISSN 1558-2434
EndPage 9
ExternalDocumentID 11126336
Genre orig-research
GrantInformation_xml – fundername: National Natural Science Foundation of China
  grantid: NSFC.62090024,62222411,62025404,92373206,62202453
  funderid: 10.13039/501100001809
GroupedDBID 6IE
6IF
6IH
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
FEDTE
IEGSK
IJVOP
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-a291t-9fa2fa7ade3b57060571720d2e27c6b428c9b44bcdc86c567bdda973a2d327713
IEDL.DBID RIE
ISICitedReferencesCount 0
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001479882200117&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Sep 03 07:09:38 EDT 2025
IsDoiOpenAccess false
IsOpenAccess true
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a291t-9fa2fa7ade3b57060571720d2e27c6b428c9b44bcdc86c567bdda973a2d327713
OpenAccessLink https://doi.org/10.1145/3676536.3676679
PageCount 9
ParticipantIDs ieee_primary_11126336
PublicationCentury 2000
PublicationDate 2024-Oct.-27
PublicationDateYYYYMMDD 2024-10-27
PublicationDate_xml – month: 10
  year: 2024
  text: 2024-Oct.-27
  day: 27
PublicationDecade 2020
PublicationTitle Digest of technical papers - IEEE/ACM International Conference on Computer-Aided Design
PublicationTitleAbbrev ICCAD
PublicationYear 2024
Publisher ACM
Publisher_xml – name: ACM
SSID ssib060584048
ssj0020286
Score 1.9273189
Snippet Natural language interfaces have exhibited considerable potential in the automation of Verilog generation derived from high-level specifications through the...
SourceID ieee
SourceType Publisher
StartPage 1
SubjectTerms Benchmark testing
Database languages
Design automation
Design methodology
Generative AI
Hardware
Hardware design languages
Large language models
Natural language processing
Visualization
Title Natural Language is Not Enough: Benchmarking Multi-Modal Generative AI for Verilog Generation
URI https://ieeexplore.ieee.org/document/11126336
WOSCitedRecordID wos001479882200117&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELZoxQALryLe8sCaNnH8SNgAtQIJog6AuqDKT9GBBLUpv5-zk7YsDEyO7MSy7PjOd-fvO4SuWeI8pYuOEsJZRJ2DPWeEikBgWi5jq1huQrIJURTZZJKPW7B6wMJYa8PlM9v3jyGWbyq99K6yQeLxLmnKO6gjBG_AWqufx4f3aEzXIQQw6jPecvkklA08MxlLed-XPFzc2iRTCbpktPfPUeyj3gaVh8drfXOAtmx5iHZ_EQoeofdCBh4N_NS6IfFsgYuqxsPSZ-O5wXfQy8enDA5yHMC30XNl4IOGftrLPnz7iOEki9-gTxCM65aq7KHX0fDl_iFqEyhEkuRJHeVOEieFNDZVzNPkMDDeSGyIJUJzBZaHzhWlShudcc24UMbIXKSSmJQIMF-PUbesSnuCsIgF0cLBfs0kdZnLHXFZYpWGExq8y05Rz8_U9KvhyJiuJunsj_pztAMrRL0WIOICdev50l6ibf1dzxbzq7CyPw7YovY
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELagIAELryLeeGBNmzh-JGyAWrWijToU1AVVfooOJKhN-f3YTtqyMDA5shPLsuM7352_7wC4J5FxlC4yiBAlATbG7jnFRGAFpqY81IKkyiebYFmWTCbpqAareyyM1tpfPtMt9-hj-aqQS-cqa0cO7xLHdBvsEIxRWMG1Vr-PC_DhEK-DCNasT2jN5hNh0nbcZCSmLVdSf3Vrk07Fa5Pu4T_HcQSaG1weHK01zjHY0vkJOPhFKXgK3jPumTTgoHZEwtkCZkUJO7nLx_MAn2wvH5_cu8ihh98Gw0LZDyoCaif94GMf2rMsfLN9WtG4binyJnjtdsbPvaBOoRBwlEZlkBqODGdc6VgQR5RDrPmGQoU0YpIKa3vIVGAspJIJlYQyoRRPWcyRihGzBuwZaORFrs8BZCFDkhm7YxOOTWJSg0wSaSHtGc2-Sy5A083U9KtiyZiuJunyj_o7sNcbDwfTQT97uQL7drWw0wmIXYNGOV_qG7Arv8vZYn7rV_kHuQCmPQ
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Digest+of+technical+papers+-+IEEE%2FACM+International+Conference+on+Computer-Aided+Design&rft.atitle=Natural+Language+is+Not+Enough%3A+Benchmarking+Multi-Modal+Generative+AI+for+Verilog+Generation&rft.au=Chang%2C+Kaiyan&rft.au=Chen%2C+Zhirong&rft.au=Zhou%2C+Yunhao&rft.au=Zhu%2C+Wenlong&rft.date=2024-10-27&rft.pub=ACM&rft.eissn=1558-2434&rft.spage=1&rft.epage=9&rft_id=info:doi/10.1145%2F3676536.3676679&rft.externalDocID=11126336