CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code
Recent work has focused on using machine learning to automate software engineering processes, such as code completion, code migration, and generating code from natural language description. One of the challenges faced in these tasks is evaluating the quality of the predictions, which is usually done...
Uložené v:
| Vydané v: | 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) s. 341 - 342 |
|---|---|
| Hlavní autori: | , |
| Médium: | Konferenčný príspevok.. |
| Jazyk: | English |
| Vydavateľské údaje: |
IEEE
01.05.2022
|
| Predmet: | |
| On-line prístup: | Získať plný text |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | Recent work has focused on using machine learning to automate software engineering processes, such as code completion, code migration, and generating code from natural language description. One of the challenges faced in these tasks is evaluating the quality of the predictions, which is usually done by comparing the prediction to a reference solution. BLEU score has been adopted for programming languages as it can be easily computed for any programming language and even incomplete source code, while enabling fast automated evaluation. However, programming languages are more verbose and have strict syntax when compared to natural languages. This feature causes BLEU to find common n-grams in unrelated programs, which makes distinguishing similar pairs of programs from dissimilar pairs hard. This work presents CrystalBLEU, an evaluation metric based on BLEU, that mitigates the distinguishability problem. Our metric maintains the desirable properties of BLEU, such as handling partial code, applicability to all programming languages, high correlation with human judgement, and efficiency, in addition to reducing the effects of the trivially matched n-grams. We evaluate CrystalBLEU on two datasets from previous work and a new dataset of human-written code. Our results show that CrystalBLEU differentiates similar and unrelated programs better than the original BLEU score and also a variant designed specifically for source code, CodeBLEU. |
|---|---|
| AbstractList | Recent work has focused on using machine learning to automate software engineering processes, such as code completion, code migration, and generating code from natural language description. One of the challenges faced in these tasks is evaluating the quality of the predictions, which is usually done by comparing the prediction to a reference solution. BLEU score has been adopted for programming languages as it can be easily computed for any programming language and even incomplete source code, while enabling fast automated evaluation. However, programming languages are more verbose and have strict syntax when compared to natural languages. This feature causes BLEU to find common n-grams in unrelated programs, which makes distinguishing similar pairs of programs from dissimilar pairs hard. This work presents CrystalBLEU, an evaluation metric based on BLEU, that mitigates the distinguishability problem. Our metric maintains the desirable properties of BLEU, such as handling partial code, applicability to all programming languages, high correlation with human judgement, and efficiency, in addition to reducing the effects of the trivially matched n-grams. We evaluate CrystalBLEU on two datasets from previous work and a new dataset of human-written code. Our results show that CrystalBLEU differentiates similar and unrelated programs better than the original BLEU score and also a variant designed specifically for source code, CodeBLEU. |
| Author | Eghbali, Aryaz Pradel, Michael |
| Author_xml | – sequence: 1 givenname: Aryaz surname: Eghbali fullname: Eghbali, Aryaz email: aryaz.eghbali@iste.uni-stuttgart.de organization: University of Stuttgart,Stuttgart,Germany – sequence: 2 givenname: Michael surname: Pradel fullname: Pradel, Michael email: michael@binaervarianz.de organization: University of Stuttgart,Stuttgart,Germany |
| BookMark | eNotjD1PwzAUAI0EA5TODCz-Ay1-_jYbRKEgBYEEnavn-AUspQlKwpB_TyQ6nW64u2LnXd8RYzcgtgDa3CkDQhu9VUZ6q_0ZWwfnwVqjgwkeLtmuGOZxwvaxKvf3_H2gOo_Uzhy7xMumyXWmblr8lXD8HXL3xadv4h_5mFsc8jTzvuFFn-iaXTTYjrQ-ccX2T-Vn8byp3nYvxUO1QXBu2gDEKCwYTNF44UBGL4nIB2kJI9aJAknhhJSBXK0thAjYJLWQXEKpVuz2_5uX7PAz5CMO8yG4oJx26g9Hw0hz |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1145/3510454.3528648 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| EISBN | 9781665495981 1665495987 |
| EndPage | 342 |
| ExternalDocumentID | 9793747 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: European Research Council funderid: 10.13039/501100000781 |
| GroupedDBID | 6IE 6IL CBEJK RIE RIL |
| ID | FETCH-LOGICAL-a177t-11bb0615adb580712b82eee8926eabacde9e2070229e7c4619b1afd319be7da23 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 19 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000850203800079&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Thu Jan 18 11:13:20 EST 2024 |
| IsPeerReviewed | false |
| IsScholarly | false |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a177t-11bb0615adb580712b82eee8926eabacde9e2070229e7c4619b1afd319be7da23 |
| PageCount | 2 |
| ParticipantIDs | ieee_primary_9793747 |
| PublicationCentury | 2000 |
| PublicationDate | 2022-May |
| PublicationDateYYYYMMDD | 2022-05-01 |
| PublicationDate_xml | – month: 05 year: 2022 text: 2022-May |
| PublicationDecade | 2020 |
| PublicationTitle | 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) |
| PublicationTitleAbbrev | ICSE-COMPANION |
| PublicationYear | 2022 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| Score | 1.9165957 |
| Snippet | Recent work has focused on using machine learning to automate software engineering processes, such as code completion, code migration, and generating code from... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 341 |
| SubjectTerms | BLEU Codes Computer languages Correlation Evaluation Machine learning Measurement Metric Natural languages Syntactics |
| Title | CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code |
| URI | https://ieeexplore.ieee.org/document/9793747 |
| WOSCitedRecordID | wos000850203800079&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEJ2A8eBJDRi_04NHF2jt0l2PEtADEhIl4UY67WxCorsGwYR_77QQ9ODFW7fNtt1pm3kz23kDcGOtt6gKRm55B4O3ChPW42lSZFaxiW2Up8iuPzSjUTad5uMa3O5iYYgoXj6jVijGf_m-cqvgKmvngcxNmzrUjeluYrW2bD1Sp-073l461a3AV9IN-Xx-pUuJ2mJw-L9xjqD5E3YnxjuFcgw1Khvw2FusGcG9PQz7k3tuppggaS1s6UU_8j9wX_z8HJ19_J5gSCde5u9ztlkZYouqEL3KUxMmg_5r7ynZZj9IrDRmmUiJGPCG9ZhmDAQUZopnmuWqSxatYyGS4gOrVE7GaTaEUNrC85FCMp4lfQJ7ZVXSKQjdKQrpiE0htDqzhgeQ0qF3BmWAiGfQCEKYfWwILmbb7z__u_oCDlSIAYi3_i5hb7lY0RXsu6_l_HNxHVflG97ykLA |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NT8JAEJ0gmuhJDRi_3YNHC-zasq1HCYixEBIh4Ub2Y5qQaGsqmPDvnV0IevDire2m3e1sN_NmuvMewK1SVmmREXJLWtplq3RAfjwKslgJCrGlsOjZ9VM5HMbTaTKqwN22FgYR_eYzbLhD_y_fFmbpUmXNxJG5hXIHdp1yVrSu1trw9fAwat7TBxZGYcMxlrSdos8vwRTvL3qH_-vpCOo_hXdstHUpx1DBvAZPnXJFGO7tMe1OHqgZvUTSiqncsq5ngKBn0fnAp_voPkagjr3O3-cUtRLIZkXGOoXFOkx63XGnH2z0DwLFpVwEnGvtEIeyOooJCggdCxppnIg2Kq0MmREFLVkhEpQmpFBIc5VZWlQapSVbn0A1L3I8BRa2sowbpGBIqzBWkjrg3GhrpOYOJJ5BzRlh9rGmuJht3v_878s3sN8fD9JZ-jx8uYAD4SoC_B7AS6guyiVewZ75Wsw_y2s_Q9-cNZP7 |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2022+IEEE%2FACM+44th+International+Conference+on+Software+Engineering%3A+Companion+Proceedings+%28ICSE-Companion%29&rft.atitle=CrystalBLEU%3A+Precisely+and+Efficiently+Measuring+the+Similarity+of+Code&rft.au=Eghbali%2C+Aryaz&rft.au=Pradel%2C+Michael&rft.date=2022-05-01&rft.pub=IEEE&rft.spage=341&rft.epage=342&rft_id=info:doi/10.1145%2F3510454.3528648&rft.externalDocID=9793747 |