Prioritizing Test Inputs for Deep Neural Networks via Mutation Analysis
Deep Neural Network (DNN) testing is one of the most widely-used ways to guarantee the quality of DNNs. However, labeling test inputs to check the correctness of DNN prediction is very costly, which could largely affect the efficiency of DNN testing, even the whole process of DNN development. To rel...
Uloženo v:
| Vydáno v: | Proceedings / International Conference on Software Engineering s. 397 - 409 |
|---|---|
| Hlavní autoři: | , , , , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
IEEE
01.05.2021
|
| Témata: | |
| ISBN: | 1665402962, 9781665402965 |
| ISSN: | 1558-1225 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Deep Neural Network (DNN) testing is one of the most widely-used ways to guarantee the quality of DNNs. However, labeling test inputs to check the correctness of DNN prediction is very costly, which could largely affect the efficiency of DNN testing, even the whole process of DNN development. To relieve the labeling-cost problem, we propose a novel test input prioritization approach (called PRIMA) for DNNs via intelligent mutation analysis in order to label more bug-revealing test inputs earlier for a limited time, which facilitates to improve the efficiency of DNN testing. PRIMA is based on the key insight: a test input that is able to kill many mutated models and produce different prediction results with many mutated inputs, is more likely to reveal DNN bugs, and thus it should be prioritized higher. After obtaining a number of mutation results from a series of our designed model and input mutation rules for each test input, PRIMA further incorporates learning-to-rank (a kind of supervised machine learning to solve ranking problems) to intelligently combine these mutation results for effective test input prioritization. We conducted an extensive study based on 36 popular subjects by carefully considering their diversity from five dimensions (i.e., different domains of test inputs, different DNN tasks, different network structures, different types of test inputs, and different training scenarios). Our experimental results demonstrate the effectiveness of PRIMA, significantly outperforming the state-of-the-art approaches (with the average improvement of 8.50%~131.01% in terms of prioritization effectiveness). In particular, we have applied PRIMA to the practical autonomous-vehicle testing in a large motor company, and the results on 4 real-world scene-recognition models in autonomous vehicles further confirm the practicability of PRIMA. |
|---|---|
| AbstractList | Deep Neural Network (DNN) testing is one of the most widely-used ways to guarantee the quality of DNNs. However, labeling test inputs to check the correctness of DNN prediction is very costly, which could largely affect the efficiency of DNN testing, even the whole process of DNN development. To relieve the labeling-cost problem, we propose a novel test input prioritization approach (called PRIMA) for DNNs via intelligent mutation analysis in order to label more bug-revealing test inputs earlier for a limited time, which facilitates to improve the efficiency of DNN testing. PRIMA is based on the key insight: a test input that is able to kill many mutated models and produce different prediction results with many mutated inputs, is more likely to reveal DNN bugs, and thus it should be prioritized higher. After obtaining a number of mutation results from a series of our designed model and input mutation rules for each test input, PRIMA further incorporates learning-to-rank (a kind of supervised machine learning to solve ranking problems) to intelligently combine these mutation results for effective test input prioritization. We conducted an extensive study based on 36 popular subjects by carefully considering their diversity from five dimensions (i.e., different domains of test inputs, different DNN tasks, different network structures, different types of test inputs, and different training scenarios). Our experimental results demonstrate the effectiveness of PRIMA, significantly outperforming the state-of-the-art approaches (with the average improvement of 8.50%~131.01% in terms of prioritization effectiveness). In particular, we have applied PRIMA to the practical autonomous-vehicle testing in a large motor company, and the results on 4 real-world scene-recognition models in autonomous vehicles further confirm the practicability of PRIMA. |
| Author | Wang, Zan Dong, Xuyuan Zhang, Yingyi Zhang, Wenbin You, Hanmo Chen, Junjie |
| Author_xml | – sequence: 1 givenname: Zan surname: Wang fullname: Wang, Zan email: wangzan@tju.edu.cn organization: Tianjin University, China – sequence: 2 givenname: Hanmo surname: You fullname: You, Hanmo email: youhanmo@tju.edu.cn organization: Tianjin University, China – sequence: 3 givenname: Junjie surname: Chen fullname: Chen, Junjie email: junjiechen@tju.edu.cn organization: Tianjin University, China – sequence: 4 givenname: Yingyi surname: Zhang fullname: Zhang, Yingyi email: yingyizhang@tju.edu.cn organization: Tianjin University, China – sequence: 5 givenname: Xuyuan surname: Dong fullname: Dong, Xuyuan email: dongxuyuan@tju.edu.cn organization: Tianjin University, China – sequence: 6 givenname: Wenbin surname: Zhang fullname: Zhang, Wenbin email: zhangwenbin@tju.edu.cn organization: Tianjin University, China |
| BookMark | eNotj91KwzAYQANOcJt7Ar3IC7Tmy1-Ty1HnLMwfcF6PtE0kWNuRpMr29Bb06twcDpwFmvVDbxG6BZIDEH1XlW8bzjShOSUUckIIlxdoAVIKTqiWdIbmIITKgFJxhVYx-ppwXmggks_R9jX4Ifjkz77_wHsbE67645gidkPA99Ye8bMdg-kmpJ8hfEb87Q1-GpNJfujxujfdKfp4jS6d6aJd_XOJ3h82-_Ix271sq3K9ywxVImWt1IXQjZGiKZxsbAOEKmA1SGJ5q2pqaMsca-pCMqUdOJAFV2xyaW2EALZEN39db609HIP_MuF00NPqtMN-AexkTjQ |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/ICSE43902.2021.00046 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EndPage | 409 |
| ExternalDocumentID | 9402064 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: National Natural Science Foundation of China funderid: 10.13039/501100001809 |
| GroupedDBID | -~X .4S .DC 123 23M 29O 5VS 6IE 6IF 6IH 6IK 6IL 6IM 6IN 8US AAJGR AAWTH ABLEC ADZIZ AFFNX ALMA_UNASSIGNED_HOLDINGS APO ARCSS AVWKF BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO EDO FEDTE I-F I07 IEGSK IJVOP IPLJI M43 OCL RIE RIL RIO RNS XOL |
| ID | FETCH-LOGICAL-a285t-d69759ca65c7f6cec102813b160e4d8b2a2d3f3cb76389f1f1674837f62ba5513 |
| IEDL.DBID | RIE |
| ISBN | 1665402962 9781665402965 |
| ISICitedReferencesCount | 105 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000684601800033&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1558-1225 |
| IngestDate | Wed Aug 27 02:28:03 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a285t-d69759ca65c7f6cec102813b160e4d8b2a2d3f3cb76389f1f1674837f62ba5513 |
| PageCount | 13 |
| ParticipantIDs | ieee_primary_9402064 |
| PublicationCentury | 2000 |
| PublicationDate | 2021-May |
| PublicationDateYYYYMMDD | 2021-05-01 |
| PublicationDate_xml | – month: 05 year: 2021 text: 2021-May |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings / International Conference on Software Engineering |
| PublicationTitleAbbrev | ICSE |
| PublicationYear | 2021 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssib044791064 ssj0006499 |
| Score | 2.5482733 |
| Snippet | Deep Neural Network (DNN) testing is one of the most widely-used ways to guarantee the quality of DNNs. However, labeling test inputs to check the correctness... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 397 |
| SubjectTerms | Companies Computer bugs Deep Learning Testing Deep Neural Network Label Labeling Mutation Neural networks Predictive models Test Prioritization Testing Training |
| Title | Prioritizing Test Inputs for Deep Neural Networks via Mutation Analysis |
| URI | https://ieeexplore.ieee.org/document/9402064 |
| WOSCitedRecordID | wos000684601800033&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELbaioGpQIugPOSBkdDEdux4Li10oKpEkbpVtmNLlVBa9TXw6zk7SRESC8qQh6wounNyd_F934fQQ6yoYLnLI0eEiZh2ItJU2YgZojWEJ3B7HMQmxGSSzedy2kCPRyyMtTY0n9knfxjW8vOV2ftfZX3pix3OmqgphCixWvXcYUzAnX3qX32FOQvakRAuoUqCSetBXV5pNyaSk4rrqT5PK0xdEsv-ePA-hCAdQFokCWSe_JfySgg8o_b_HvkMdX8QfHh6jE3nqGGLC9SuJRxw9UZ30Mt0s1x5XqMvGIZnECLwuIAxWwzJLH62do09fYf6hF3oF9_iw1Lht325go9rTpMu-hgNZ4PXqNJWiBTJ0l2UcylSaRRPjXDcWOMTjYTqhMeW5ZkmiuTUUaOFT2lc4jxaAYpZx4lWXhTmErWKVWGvEE4d0YJaqIOgtgJnS51RbmGTmuc0i69Rx9tlsS7pMxaVSXp_X75Bp97wZU_hLWrtNnt7h07MYbfcbu6Dz78B06ikuw |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1dS8MwFA1zCvo0dRO_zYOP1rVJmrTPc3PDbQycsLeRpAkMpBv7evDXe5N2E8EX6UM_CKXkpj33NjnnIPQYSipYZrPAEqEDpqwIFJUmYJooBfAEYQ-92YQYDpPJJB1V0NOeC2OM8YvPzLM79HP52Vxv3K-yZuqKHc4O0GHMGIkKttZu9DAm4N4u-S-_w5x590gATKiTYNg6Wpfz2g1Jykmp9rQ7j0tWXRSmzV7rvQ0w7WlaJPJynvyX94qHnk7tfw99iho_HD482qPTGaqY_BzVdiYOuHyn6-h1tJzNnbLRFzTDYwAJ3MuhzQpDOotfjFlgJ-AhP2HnV4yv8HYm8WBTzOHjnapJA3102uNWNyjdFQJJkngdZDwVcaolj7WwXBvtUo2IqoiHhmWJIpJk1FKthEtqbGQdXwHKWcuJks4W5gJV83luLhGOLVGCGqiEoLqCcKcqodzAliqe0SS8QnXXL9NFIaAxLbvk-u_LD-i4Ox70p_3e8O0GnbggFCsMb1F1vdyYO3Skt-vZannv4_8NnXmoAg |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%2F+International+Conference+on+Software+Engineering&rft.atitle=Prioritizing+Test+Inputs+for+Deep+Neural+Networks+via+Mutation+Analysis&rft.au=Wang%2C+Zan&rft.au=You%2C+Hanmo&rft.au=Chen%2C+Junjie&rft.au=Zhang%2C+Yingyi&rft.date=2021-05-01&rft.pub=IEEE&rft.isbn=9781665402965&rft.issn=1558-1225&rft.spage=397&rft.epage=409&rft_id=info:doi/10.1109%2FICSE43902.2021.00046&rft.externalDocID=9402064 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1558-1225&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1558-1225&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1558-1225&client=summon |