Towards More Trustworthy Deep Code Models by Enabling Out-of-Distribution Detection
Numerous machine learning (ML) models have been developed, including those for software engineering (SE) tasks, under the assumption that training and testing data come from the same distribution. However, training and testing distributions often differ, as training datasets rarely encompass the ent...
Saved in:
| Published in: | Proceedings / International Conference on Software Engineering pp. 769 - 781 |
|---|---|
| Main Authors: | , , , |
| Format: | Conference Proceeding |
| Language: | English |
| Published: |
IEEE
26.04.2025
|
| Subjects: | |
| ISSN: | 1558-1225 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Numerous machine learning (ML) models have been developed, including those for software engineering (SE) tasks, under the assumption that training and testing data come from the same distribution. However, training and testing distributions often differ, as training datasets rarely encompass the entire distribution, while testing distribution tends to shift over time. Hence, when confronted with out-of-distribution (OOD) instances that differ from the training data, a reliable and trustworthy SE ML model must be capable of detecting them to either abstain from making predictions, or potentially forward these OODs to appropriate models handling other categories or tasks. In this paper, we develop two types of SE-specific OOD detection models, unsupervised and weakly-supervised OOD detection for code. The unsupervised OOD detection approach is trained solely on in-distribution samples while the weakly-supervised approach utilizes a tiny number of OOD samples to further enhance the detection performance in various OOD scenarios. Extensive experimental results demonstrate that our proposed methods significantly outperform the baselines in detecting OOD samples from four different scenarios simultaneously and also positively impact a main code understanding task. |
|---|---|
| AbstractList | Numerous machine learning (ML) models have been developed, including those for software engineering (SE) tasks, under the assumption that training and testing data come from the same distribution. However, training and testing distributions often differ, as training datasets rarely encompass the entire distribution, while testing distribution tends to shift over time. Hence, when confronted with out-of-distribution (OOD) instances that differ from the training data, a reliable and trustworthy SE ML model must be capable of detecting them to either abstain from making predictions, or potentially forward these OODs to appropriate models handling other categories or tasks. In this paper, we develop two types of SE-specific OOD detection models, unsupervised and weakly-supervised OOD detection for code. The unsupervised OOD detection approach is trained solely on in-distribution samples while the weakly-supervised approach utilizes a tiny number of OOD samples to further enhance the detection performance in various OOD scenarios. Extensive experimental results demonstrate that our proposed methods significantly outperform the baselines in detecting OOD samples from four different scenarios simultaneously and also positively impact a main code understanding task. |
| Author | Yan, Yanfu Poshyvanyk, Denys Duong, Viet Shao, Huajie |
| Author_xml | – sequence: 1 givenname: Yanfu surname: Yan fullname: Yan, Yanfu email: yyan09@wm.edu organization: William & Mary,Department of Computer Science,Williamsburg,Virginia,USA – sequence: 2 givenname: Viet surname: Duong fullname: Duong, Viet email: vqduong@wm.edu organization: William & Mary,Department of Computer Science,Williamsburg,Virginia,USA – sequence: 3 givenname: Huajie surname: Shao fullname: Shao, Huajie email: hshao@wm.edu organization: William & Mary,Department of Computer Science,Williamsburg,Virginia,USA – sequence: 4 givenname: Denys surname: Poshyvanyk fullname: Poshyvanyk, Denys email: dposhyvanyk@wm.edu organization: William & Mary,Department of Computer Science,Williamsburg,Virginia,USA |
| BookMark | eNotkM9OAjEYxKvRREDegENfYPFru_13NAsiCYYDeCbt7re6BrekLSG8vWv0NJPM_OYwY3LXhx4JmTGYMwb2aV3tllKKUs85cDkHYFrfkKnV1gjBJEhl2S0ZMSlNwTiXD2Sc0hcAqNLaEdntw8XFJtG3EJHu4znlS4j580oXiCdahQaHqMFjov5Kl73zx67_oNtzLkJbLLqUY-fPuQv9AGSsf90juW_dMeH0Xyfk_WW5r16LzXa1rp43heMKctGW3iJoro0XwjhwUnmvOfNKgmuYEBxVq2sp0TRcq5IZp71rG6hrzduhMCGzv90OEQ-n2H27eD0Mr3BrBvwHZGNS6Q |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/ICSE55347.2025.00177 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 9798331505691 |
| EISSN | 1558-1225 |
| EndPage | 781 |
| ExternalDocumentID | 11029813 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: Cisco Systems funderid: 10.13039/100004351 – fundername: NSF grantid: CCF-2311469,CNS-2132281,CCF-2007246,CCF-1955853,CNS-2346357 funderid: 10.13039/100000001 |
| GroupedDBID | -~X .4S .DC 29O 5VS 6IE 6IF 6IH 6IK 6IL 6IM 6IN 8US AAJGR AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS ARCSS AVWKF BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO EDO FEDTE I-F IEGSK IJVOP IPLJI M43 OCL RIE RIL RIO |
| ID | FETCH-LOGICAL-a260t-f4b9e07278b338a0a56bb721b650ad1332e6f7c55e8d276418a7bafd0cc72fd13 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 0 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001538318100060&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 01:40:13 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a260t-f4b9e07278b338a0a56bb721b650ad1332e6f7c55e8d276418a7bafd0cc72fd13 |
| PageCount | 13 |
| ParticipantIDs | ieee_primary_11029813 |
| PublicationCentury | 2000 |
| PublicationDate | 2025-April-26 |
| PublicationDateYYYYMMDD | 2025-04-26 |
| PublicationDate_xml | – month: 04 year: 2025 text: 2025-April-26 day: 26 |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings / International Conference on Software Engineering |
| PublicationTitleAbbrev | ICSE |
| PublicationYear | 2025 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0006499 |
| Score | 2.2898738 |
| Snippet | Numerous machine learning (ML) models have been developed, including those for software engineering (SE) tasks, under the assumption that training and testing... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 769 |
| SubjectTerms | Code Models Codes Contrastive learning Data models Measurement OOD detection Predictive models Reliability Software engineering Testing Training Training data Trustworthy ML |
| Title | Towards More Trustworthy Deep Code Models by Enabling Out-of-Distribution Detection |
| URI | https://ieeexplore.ieee.org/document/11029813 |
| WOSCitedRecordID | wos001538318100060&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1JSwMxFA5aPHiqS8WdHLzGzmQySebcBQWthVbprWQFQWZKOxX6731Jx4oHD95CyAJ5JO99yfvyIXRXKMO10JQ4nljCPLNEGZERxbxPbGY8i2Iwb09iNJKzWTFuyOqRC-Oci8ln7j4U41u-rcw6XJV1wVXRQgaN2n0h-JastTt2OcTuDTcuTYruY28yyPOMCcCANNybpOK3gkp0IMP2P6c-Qp0fKh4e75zMMdpz5Qlqf2sx4GZrnqLJNOa_rvAzjIyngUkRs_82uO_cAvcq63DQPftYYb3Bg8CYgvHwy7omlSf98H1uo3wFHeqYn1V20OtwMO09kEYwgSiAJTXxTBcugYhEakCeKlE51xognoYwTFlAo9RxL0yeO2mp4CyVSmjlbWKMoB4anKFWWZXuHGGpZJZaTi3gGYCAEFWw3GZ5kViRau_YBeqERZovtn9izL_X5_KP-it0GOwQ3mEov0aterl2N-jAfNbvq-VttOQXPjif3Q |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1JSwMxFA5SBT3VpeJuDl5jZ8kkM-e20mJbCx2lt5IVBJkp7VTov_clTisePHgLIQvkkbz3Je_Lh9BDJhSTXEbEsEATaqkmQvGYCGptoGNlqReDeRvy8TidzbJJTVb3XBhjjE8-M4-u6N_ydanW7qqsDa4qylKnUbvvpLNqutbu4GUQvdfsuDDI2oPOtJckMeWAAiN3cxLy3xoq3oU8Nf85-TFq_ZDx8GTnZk7QnilOUXOrxoDrzXmGprnPgF3hEYyMc8el8Pl_G9w1ZoE7pTbYKZ99rLDc4J7jTMF4-GVdkdKSrvtAt9a-gg6Vz9AqWuj1qZd3-qSWTCACgElFLJWZCSAmSSVgTxGIhEkJIE9CICY04NHIMMtVkphUR5zRMBVcCqsDpXhkocE5ahRlYS4QTkUah5pFGhANgECIK2ii4yQLNA-lNfQStdwizRffv2LMt-tz9Uf9PTrs56PhfDgYP1-jI2cT9yoTsRvUqJZrc4sO1Gf1vlreeat-AcLJoyY |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%2F+International+Conference+on+Software+Engineering&rft.atitle=Towards+More+Trustworthy+Deep+Code+Models+by+Enabling+Out-of-Distribution+Detection&rft.au=Yan%2C+Yanfu&rft.au=Duong%2C+Viet&rft.au=Shao%2C+Huajie&rft.au=Poshyvanyk%2C+Denys&rft.date=2025-04-26&rft.pub=IEEE&rft.eissn=1558-1225&rft.spage=769&rft.epage=781&rft_id=info:doi/10.1109%2FICSE55347.2025.00177&rft.externalDocID=11029813 |