Assessing Robustness of ML-Based Program Analysis Tools using Metamorphic Program Transformations
Metamorphic testing is a well-established testing technique that has been successfully applied in various domains, including testing deep learning models to assess their robustness against data noise or malicious input. Currently, metamorphic testing approaches for machine learning (ML) models focus...
Uložené v:
| Vydané v: | IEEE/ACM International Conference on Automated Software Engineering : [proceedings] s. 1377 - 1381 |
|---|---|
| Hlavní autori: | , , |
| Médium: | Konferenčný príspevok.. |
| Jazyk: | English |
| Vydavateľské údaje: |
IEEE
01.11.2021
|
| Predmet: | |
| ISSN: | 2643-1572 |
| On-line prístup: | Získať plný text |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | Metamorphic testing is a well-established testing technique that has been successfully applied in various domains, including testing deep learning models to assess their robustness against data noise or malicious input. Currently, metamorphic testing approaches for machine learning (ML) models focused on image processing and object recognition tasks. Hence, these approaches cannot be applied to ML targeting program analysis tasks. In this paper, we extend metamorphic testing approaches for ML models targeting software programs. We present LAMPION, a novel testing framework that applies (semantics preserving) metamorphic transformations on the test datasets. LAMPION produces new code snippets equivalent to the original test set but different in their identifiers or syntactic structure. We evaluate LAMPION against CodeBERT, a state-of-the-art ML model for Code-To-Text tasks that creates Javadoc summaries for given Java methods. Our results show that simple transformations significantly impact the target model behavior, providing additional information on the models reasoning apart from the classic performance metric. |
|---|---|
| AbstractList | Metamorphic testing is a well-established testing technique that has been successfully applied in various domains, including testing deep learning models to assess their robustness against data noise or malicious input. Currently, metamorphic testing approaches for machine learning (ML) models focused on image processing and object recognition tasks. Hence, these approaches cannot be applied to ML targeting program analysis tasks. In this paper, we extend metamorphic testing approaches for ML models targeting software programs. We present LAMPION, a novel testing framework that applies (semantics preserving) metamorphic transformations on the test datasets. LAMPION produces new code snippets equivalent to the original test set but different in their identifiers or syntactic structure. We evaluate LAMPION against CodeBERT, a state-of-the-art ML model for Code-To-Text tasks that creates Javadoc summaries for given Java methods. Our results show that simple transformations significantly impact the target model behavior, providing additional information on the models reasoning apart from the classic performance metric. |
| Author | Applis, Leonhard Panichella, Annibale van Deursen, Arie |
| Author_xml | – sequence: 1 givenname: Leonhard surname: Applis fullname: Applis, Leonhard email: L.H.Applis@tudelft.nl organization: Technische Universiteit,Delft,Netherlands – sequence: 2 givenname: Annibale surname: Panichella fullname: Panichella, Annibale email: A.Panichella@tudelft.nl organization: Technische Universiteit,Delft,Netherlands – sequence: 3 givenname: Arie surname: van Deursen fullname: van Deursen, Arie email: Arie.vanDeursen@tudelft.nl organization: Technische Universiteit,Delft,Netherlands |
| BookMark | eNo9UN1KwzAYjaLgNvcEIuQFWvOf9rKOTYUORXs_vjTpLKzJSLqLvb1FhzfncH6-7-LM0Y0P3iH0SElOKSmfqq-1pJKJnBFG81LpQhN1heZUKSkI55pfoxlTgmdUanaHlin1hoiiEEKXaoagSslNnt_jz2BOafSTwqHD2zp7huQs_ohhH2HAlYfDOfUJNyEcEj793mzdCEOIx---_S82EXzqQhxg7INP9-i2g0NyywsvULNZN6vXrH5_eVtVdQacizFTAILxru0048ZNaCnlpDVOSSgKapThWlvTCasLq8oJpNJaWmnaKWd8gR7-3vbOud0x9gPE8-4yCP8B_NZY5A |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK ESBDL RIE RIL |
| DOI | 10.1109/ASE51524.2021.9678706 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Xplore POP ALL IEEE Xplore All Conference Proceedings IEEE Xplore Open Access Journals IEEE/IET Electronic Library (IEL) (UW System Shared) IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 1665403373 9781665403375 |
| EISSN | 2643-1572 |
| EndPage | 1381 |
| ExternalDocumentID | 9678706 |
| Genre | orig-research |
| GroupedDBID | 29I 6IE 6IF 6IH 6IK 6IL 6IM 6IN 6J9 AAJGR AAWTH ABLEC ACREN ADYOE ADZIZ AFYQB ALMA_UNASSIGNED_HOLDINGS AMTXH APO BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO ESBDL IEGSK IPLJI M43 OCL RIE RIL |
| ID | FETCH-LOGICAL-a334t-6aa423fcf723bef72d1130cbe65a881b6b377dbf4d78d6978d56775d5bca8823 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 23 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000779309000164&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:40:46 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a334t-6aa423fcf723bef72d1130cbe65a881b6b377dbf4d78d6978d56775d5bca8823 |
| OpenAccessLink | https://ieeexplore.ieee.org/document/9678706 |
| PageCount | 5 |
| ParticipantIDs | ieee_primary_9678706 |
| PublicationCentury | 2000 |
| PublicationDate | 2021-Nov. |
| PublicationDateYYYYMMDD | 2021-11-01 |
| PublicationDate_xml | – month: 11 year: 2021 text: 2021-Nov. |
| PublicationDecade | 2020 |
| PublicationTitle | IEEE/ACM International Conference on Automated Software Engineering : [proceedings] |
| PublicationTitleAbbrev | ASE |
| PublicationYear | 2021 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssib048844796 ssj0051577 |
| Score | 2.3766448 |
| Snippet | Metamorphic testing is a well-established testing technique that has been successfully applied in various domains, including testing deep learning models to... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 1377 |
| SubjectTerms | documentation-generation Java machine learning in software engineering Measurement metamorphic testing Robustness Semantics Software Statistical analysis Syntactics |
| Title | Assessing Robustness of ML-Based Program Analysis Tools using Metamorphic Program Transformations |
| URI | https://ieeexplore.ieee.org/document/9678706 |
| WOSCitedRecordID | wos000779309000164&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwED61FQNTgRbxlgdG0jaJ7UtGQK0YaFVBhm6Vn1IlaFCb8Puxk7SAxMISWY5jRb6zv7N93x3ALWWorEEb2ISqgDqbPhCh9eQcjlwwI5kRVbIJnM2SxSKdt-Buz4UxxlTOZ2bgi9Vdvs5V6Y_Khin36sXb0EbkNVdrpztODymtQsfVq7CDacSGsROO0uH969hVRf4UJQoHTUe_MqpUgDLp_u9XjqD_zcwj8z3mHEPLrE-gu0vNQJqZ2gNRX-a6NuQll-W28CsayS2ZPgcPDri078U7ZpFdVBKS5fnblpTVN1NTiPfciWCl9g2zHyauU9U-ZJNx9vgUNNkUAhHHtAi4EM50sspiFEvjnjp0-KWk4UwkznjlMkbU3m8PE83d5lIzjsg0k8q9j-JT6KzztTkDEmkTW65FpGVEZaxFSuVIp9rZdlYlDM-h5wds-VHHy1g2Y3Xxd_UlHHqZ1Py-K-gUm9Jcw4H6LFbbzU0l5C8ge6my |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwED6VggRTgRbxxgMj6cPxIxkBtSqirSrI0K3yU6oEDWoTfj92khaQWFgiy3GsyHf2d7bvuwO4JZQra7gNbERUQJxNH4ie9eQcxpmgRlIjimQTfDKJZrN4WoO7LRfGGFM4n5m2LxZ3-TpVuT8q68TMqxfbgV1KCO6WbK2N9jhNJKQIHleuww6oOa84O71u3Ll_7bsq7M9RcK9ddfUrp0oBKYPG_37mEFrf3Dw03aLOEdTM8hgam-QMqJqrTRDlda5rg15Sma8zv6ah1KLxKHhw0KV9L941C23ikqAkTd_WKC--GZtMvKdOCAu1bZj8MHKdsrYgGfSTx2FQ5VMIRBiSLGBCOOPJKstxKI176p5DMCUNoyJy5iuTIefae-7xSDO3vdSUcU41lcq9x-EJ1Jfp0pwCwtqElmmBtcREhlrERHZ1rJ11Z1VE-Rk0_YDNP8qIGfNqrM7_rr6B_WEyHs1HT5PnCzjw8inZfpdQz1a5uYI99Zkt1qvrQuBfIJ-s-Q |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=IEEE%2FACM+International+Conference+on+Automated+Software+Engineering+%3A+%5Bproceedings%5D&rft.atitle=Assessing+Robustness+of+ML-Based+Program+Analysis+Tools+using+Metamorphic+Program+Transformations&rft.au=Applis%2C+Leonhard&rft.au=Panichella%2C+Annibale&rft.au=van+Deursen%2C+Arie&rft.date=2021-11-01&rft.pub=IEEE&rft.eissn=2643-1572&rft.spage=1377&rft.epage=1381&rft_id=info:doi/10.1109%2FASE51524.2021.9678706&rft.externalDocID=9678706 |