Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models
Large Language Models (LLMs) have demonstrated unprecedented capabilities in code generation. However, there remains a limited understanding of code generation errors that LLMs can produce. To bridge the gap, we conducted an in-depth analysis of code generation errors across six representative LLMs...
Gespeichert in:
| Veröffentlicht in: | Proceedings / International Conference on Software Engineering S. 2587 - 2599 |
|---|---|
| Hauptverfasser: | , , , , , , |
| Format: | Tagungsbericht |
| Sprache: | Englisch |
| Veröffentlicht: |
IEEE
26.04.2025
|
| Schlagworte: | |
| ISSN: | 1558-1225 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | Large Language Models (LLMs) have demonstrated unprecedented capabilities in code generation. However, there remains a limited understanding of code generation errors that LLMs can produce. To bridge the gap, we conducted an in-depth analysis of code generation errors across six representative LLMs on the HumanEval dataset. Specifically, we first employed open coding and thematic analysis to distill a comprehensive taxonomy of code generation errors. We analyzed two dimensions of error characteristics-semantic characteristics and syntactic characteristics. Our analysis revealed that LLMs often made non-trivial, multi-line code generation errors in various locations and with various root causes. We further analyzed the correlation between these errors and task complexity as well as test pass rate. Our findings highlighted several challenges in locating and fixing code generation errors made by LLMs. In the end, we discussed several future directions to address these challenges. |
|---|---|
| AbstractList | Large Language Models (LLMs) have demonstrated unprecedented capabilities in code generation. However, there remains a limited understanding of code generation errors that LLMs can produce. To bridge the gap, we conducted an in-depth analysis of code generation errors across six representative LLMs on the HumanEval dataset. Specifically, we first employed open coding and thematic analysis to distill a comprehensive taxonomy of code generation errors. We analyzed two dimensions of error characteristics-semantic characteristics and syntactic characteristics. Our analysis revealed that LLMs often made non-trivial, multi-line code generation errors in various locations and with various root causes. We further analyzed the correlation between these errors and task complexity as well as test pass rate. Our findings highlighted several challenges in locating and fixing code generation errors made by LLMs. In the end, we discussed several future directions to address these challenges. |
| Author | Zhang, Tianyi Huang, Yuheng Ma, Lei Wang, Zhijie Zhou, Zijie Song, Da Chen, Shengmai |
| Author_xml | – sequence: 1 givenname: Zhijie surname: Wang fullname: Wang, Zhijie email: zhijie.wang@ualberta.ca organization: University of Alberta,Edmonton,AB,Canada – sequence: 2 givenname: Zijie surname: Zhou fullname: Zhou, Zijie email: zijiez4@illinois.edu organization: University of Illinois Urbana-Champaign,Champaign,IL,USA – sequence: 3 givenname: Da surname: Song fullname: Song, Da email: dsong4@ualberta.ca organization: University of Alberta,Edmonton,AB,Canada – sequence: 4 givenname: Yuheng surname: Huang fullname: Huang, Yuheng email: yuhenghuang42@g.ecc.u-tokyo.ac.jp organization: The University of Tokyo,Tokyo,Japan – sequence: 5 givenname: Shengmai surname: Chen fullname: Chen, Shengmai email: chen3301@purdue.edu organization: Purdue University,West Lafayette,IN,USA – sequence: 6 givenname: Lei surname: Ma fullname: Ma, Lei email: ma.lei@acm.org organization: The University of Tokyo,Tokyo,Japan – sequence: 7 givenname: Tianyi surname: Zhang fullname: Zhang, Tianyi email: tianyi@purdue.edu organization: Purdue University,West Lafayette,IN,USA |
| BookMark | eNotkEFPAjEUhKvRRED-AYf-gcX3ttvSHs0GkQTiQTiTx_YV1mDXtGsM_95N9DIz-TKZw4zFXewiCzFDmCOCe1rX70utVbWYl1DqOQBauBFTt3BWKdSgjcNbMUKtbYFlqR_EOOcPADCVcyNBu-6Hks9yHz2n3FP0bTzJ_syyPlOipufU5r5tsuyCrDvPcsWRE_VtF-UypS5luaUBH69yQ-nEg8bTNw1hO7Qv-VHcB7pknv77ROxflrv6tdi8rdb186ag0kBfENqjQSRCUxmrQ8WWVaOUBU_oAwMo03BgbcBbWNhA1dFRcF5DUwIpNRGzv92WmQ9fqf2kdD0MH5XOaVS_2tdYDw |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/ICSE55347.2025.00180 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 9798331505691 |
| EISSN | 1558-1225 |
| EndPage | 2599 |
| ExternalDocumentID | 11029951 |
| Genre | orig-research |
| GroupedDBID | -~X .4S .DC 29O 5VS 6IE 6IF 6IH 6IK 6IL 6IM 6IN 8US AAJGR AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS ARCSS AVWKF BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO EDO FEDTE I-F IEGSK IJVOP IPLJI M43 OCL RIE RIL RIO |
| ID | FETCH-LOGICAL-a260t-a18b611aa164685f4e8e3c3380da1dfe0036cefe560d8078fa4b9af9d50c20a33 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 0 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001538318100202&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 01:40:13 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a260t-a18b611aa164685f4e8e3c3380da1dfe0036cefe560d8078fa4b9af9d50c20a33 |
| PageCount | 13 |
| ParticipantIDs | ieee_primary_11029951 |
| PublicationCentury | 2000 |
| PublicationDate | 2025-April-26 |
| PublicationDateYYYYMMDD | 2025-04-26 |
| PublicationDate_xml | – month: 04 year: 2025 text: 2025-April-26 day: 26 |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings / International Conference on Software Engineering |
| PublicationTitleAbbrev | ICSE |
| PublicationYear | 2025 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0006499 |
| Score | 2.3099954 |
| Snippet | Large Language Models (LLMs) have demonstrated unprecedented capabilities in code generation. However, there remains a limited understanding of code generation... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 2587 |
| SubjectTerms | Code Generation Codes Complexity theory Correlation Empirical Study Encoding Large language models Semantics Software engineering Software reliability Syntactics Taxonomy |
| Title | Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models |
| URI | https://ieeexplore.ieee.org/document/11029951 |
| WOSCitedRecordID | wos001538318100202&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07a8MwEBZt6NApfbj0jYauaiy_PZuEFtoQaALZgiydoFDiYieB_vveyU6aDh26CKPFoEP30n3fx9hDHhoZWN8KUqQUxD8issgGgkTeMdwaDKnaiU2k43E2n-eTDqzusDAA4IbP4JE-3Vu-qfSaWmUDDFXoPQkwfZimSQvW2rndBHP3Dhsn_XzwXLwN4ziMUqwBA-qbSGJ-3FNQcQFk1P_nr0-Y9wPF45NdkDllB7A8Y_2tFgPvruY5U1M3_9rw2T5ahWN6x4vfnMy8sryoDPCWcZoMw4d1XdUNf1W4XX7xFxoPx7VtZXLSS_toPDYbDafFk-jkE4TCImUllMzKREqliEIsi20EGYQaS1LfKGksEBWNBguY8xhinbcqKnNlcxP7OvBVGF6w3rJawiXjWoLNQ60w2zKR0VCquMxKo9AZYMWj0yvm0ZEtPluGjMX2tK7_2L9hx2QVepUJklvWW9VruGNHerN6b-p7Z9dvDICkqg |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3Na8IwFA_DDbaT-3Dseznsmtm0qbbnoihTEabgTdLkBQbDjlYH--_3XludO-ywSyi5FPLI-8r7_X6MPcWBlb7znCBFSkH8IyJSzhck8o7h1mJINaXYRHcyiRaLeFqD1UssDACUw2fwTJ_lW77NzIZaZW0MVeg9CTB9GCrlexVca-d4O5i91-g46cXtYfLaC8NAdbEK9KlzIon7cU9DpQwh_eY_f37KWj9gPD7dhZkzdgCrc9bcqjHw-nJeMD0rJ2ALPt_Hq3BM8Hjym5WZZ44nmQVecU6TaXgvz7O84GON2-kXH9GAOK5VM5OTYtp70WLzfm-WDEQtoCA0lilroWWUdqTUmkjEotApiCAwWJR6VkvrgMhoDDjArMcS77zTKo21i23oGd_TQXDJGqtsBVeMGwkuDozGfMsqayDVYRqlVqM7wJrHdK9Zi45s-VFxZCy3p3Xzx_4jOx7MxqPlaDh5uWUnZCF6o_E7d6yxzjdwz47M5_qtyB9KG38DyGSn8Q |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%2F+International+Conference+on+Software+Engineering&rft.atitle=Towards+Understanding+the+Characteristics+of+Code+Generation+Errors+Made+by+Large+Language+Models&rft.au=Wang%2C+Zhijie&rft.au=Zhou%2C+Zijie&rft.au=Song%2C+Da&rft.au=Huang%2C+Yuheng&rft.date=2025-04-26&rft.pub=IEEE&rft.eissn=1558-1225&rft.spage=2587&rft.epage=2599&rft_id=info:doi/10.1109%2FICSE55347.2025.00180&rft.externalDocID=11029951 |