RobOT: Robustness-Oriented Testing for Deep Learning Systems
Recently, there has been a significant growth of interest in applying software engineering techniques for the quality assurance of deep learning (DL) systems. One popular direction is deep learning testing, where adversarial examples (a.k.a.~bugs) of DL systems are found either by fuzzing or guided...
Uložené v:
| Vydané v: | Proceedings / International Conference on Software Engineering s. 300 - 311 |
|---|---|
| Hlavní autori: | , , , , , , |
| Médium: | Konferenčný príspevok.. |
| Jazyk: | English |
| Vydavateľské údaje: |
IEEE
01.05.2021
|
| Predmet: | |
| ISBN: | 1665402962, 9781665402965 |
| ISSN: | 1558-1225 |
| On-line prístup: | Získať plný text |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | Recently, there has been a significant growth of interest in applying software engineering techniques for the quality assurance of deep learning (DL) systems. One popular direction is deep learning testing, where adversarial examples (a.k.a.~bugs) of DL systems are found either by fuzzing or guided search with the help of certain testing metrics. However, recent studies have revealed that the commonly used neuron coverage metrics by existing DL testing approaches are not correlated to model robustness. It is also not an effective measurement on the confidence of the model robustness after testing. In this work, we address this gap by proposing a novel testing framework called Robustness-Oriented Testing (RobOT). A key part of RobOT is a quantitative measurement on 1) the value of each test case in improving model robustness (often via retraining), and 2) the convergence quality of the model robustness improvement. RobOT utilizes the proposed metric to automatically generate test cases valuable for improving model robustness. The proposed metric is also a strong indicator on how well robustness improvement has converged through testing. Experiments on multiple benchmark datasets confirm the effectiveness and efficiency of RobOT in improving DL model robustness, with 67.02% increase on the adversarial robustness that is 50.65% higher than the state-of-the-art work DeepGini. |
|---|---|
| AbstractList | Recently, there has been a significant growth of interest in applying software engineering techniques for the quality assurance of deep learning (DL) systems. One popular direction is deep learning testing, where adversarial examples (a.k.a.~bugs) of DL systems are found either by fuzzing or guided search with the help of certain testing metrics. However, recent studies have revealed that the commonly used neuron coverage metrics by existing DL testing approaches are not correlated to model robustness. It is also not an effective measurement on the confidence of the model robustness after testing. In this work, we address this gap by proposing a novel testing framework called Robustness-Oriented Testing (RobOT). A key part of RobOT is a quantitative measurement on 1) the value of each test case in improving model robustness (often via retraining), and 2) the convergence quality of the model robustness improvement. RobOT utilizes the proposed metric to automatically generate test cases valuable for improving model robustness. The proposed metric is also a strong indicator on how well robustness improvement has converged through testing. Experiments on multiple benchmark datasets confirm the effectiveness and efficiency of RobOT in improving DL model robustness, with 67.02% increase on the adversarial robustness that is 50.65% higher than the state-of-the-art work DeepGini. |
| Author | Wang, Jingyi Wang, Dongxia Ma, Xingjun Sun, Jun Chen, Jialuo Cheng, Peng Sun, Youcheng |
| Author_xml | – sequence: 1 givenname: Jingyi surname: Wang fullname: Wang, Jingyi email: wangjyee@zju.edu.cn organization: Zhejiang University – sequence: 2 givenname: Jialuo surname: Chen fullname: Chen, Jialuo email: chenjialuo@zju.edu.cn organization: Zhejiang University – sequence: 3 givenname: Youcheng surname: Sun fullname: Sun, Youcheng email: youcheng.sun@qub.ac.uk organization: Queen's University Belfast – sequence: 4 givenname: Xingjun surname: Ma fullname: Ma, Xingjun email: daniel.ma@deakin.edu.au organization: Deakin University – sequence: 5 givenname: Dongxia surname: Wang fullname: Wang, Dongxia email: dxwang@zju.edu.cn organization: Zhejiang University – sequence: 6 givenname: Jun surname: Sun fullname: Sun, Jun email: junsun@smu.edu.sg organization: Singapore Management University – sequence: 7 givenname: Peng surname: Cheng fullname: Cheng, Peng email: lunarheart@zju.edu.cn organization: Zhejiang University |
| BookMark | eNotz81Kw0AYheEBK9jWXoEucgOJ3_xmRtxIrFoIBGxcl5nJFwnYSZmJi969kbp64F0cOCuyCGNAQu4pFJSCedhV-63gBljBgNECALi-IiuqlBTAjGILsqRS6pwyJm_IJqXBgRCloaDEkjx9jK5pH7OZnzQFTClv4oBhwi5rMU1D-Mr6MWYviKesRhvDX9mf04THdEuue_udcPPvmny-btvqPa-bt131XOeWaTnlWAIvtfVCaMq14EzgjEdplUKHXmunsStBOG166gC967yT3oCTyLDka3J32R0Q8XCKw9HG88HM_4Ab_gvRk0qc |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/ICSE43902.2021.00038 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EndPage | 311 |
| ExternalDocumentID | 9402039 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: National Research Foundation funderid: 10.13039/501100001321 |
| GroupedDBID | -~X .4S .DC 123 23M 29O 5VS 6IE 6IF 6IH 6IK 6IL 6IM 6IN 8US AAJGR AAWTH ABLEC ADZIZ AFFNX ALMA_UNASSIGNED_HOLDINGS APO ARCSS AVWKF BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO EDO FEDTE I-F I07 IEGSK IJVOP IPLJI M43 OCL RIE RIL RIO RNS XOL |
| ID | FETCH-LOGICAL-a285t-e70378ac4481384324e384ce5a66ebec88b8ed704b89f1b0ecbdcb5c90b5e2e73 |
| IEDL.DBID | RIE |
| ISBN | 1665402962 9781665402965 |
| ISICitedReferencesCount | 53 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000684601800025&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1558-1225 |
| IngestDate | Wed Aug 27 02:50:26 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a285t-e70378ac4481384324e384ce5a66ebec88b8ed704b89f1b0ecbdcb5c90b5e2e73 |
| PageCount | 12 |
| ParticipantIDs | ieee_primary_9402039 |
| PublicationCentury | 2000 |
| PublicationDate | 2021-May |
| PublicationDateYYYYMMDD | 2021-05-01 |
| PublicationDate_xml | – month: 05 year: 2021 text: 2021-May |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings / International Conference on Software Engineering |
| PublicationTitleAbbrev | ICSE |
| PublicationYear | 2021 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssib044791064 ssj0006499 |
| Score | 2.4896924 |
| Snippet | Recently, there has been a significant growth of interest in applying software engineering techniques for the quality assurance of deep learning (DL) systems.... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 300 |
| SubjectTerms | Benchmark testing Convergence Deep learning Measurement Robots Robustness Software engineering |
| Title | RobOT: Robustness-Oriented Testing for Deep Learning Systems |
| URI | https://ieeexplore.ieee.org/document/9402039 |
| WOSCitedRecordID | wos000684601800025&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEJ0A8eAJFYzf6cGjC9vuR1vjDSF6AaKYcCNtd5Z4AQKLv9-2FIyJF0_d7Wkz_Zh52773AO4LnpeMljKSsWEWoBRJJHPJI5oJyo2WTjXOm03w4VBMp3Jcg4cDFwYR_eUz7LhHf5ZfLM3W_SrrSgd2ElmHOud8x9Xaz5005TbxudI_7MJ56r0jbbq0KMlOWkfqck67MZM5C1pP-_cscOpoLLuvvfe-TdKepMVoxx-d_XJe8Yln0PzfJ59A-4fBR8aH3HQKNVycQXNv4UDCim7B09tSjyaPxDbbTeW2vWjklI9tHUomToBjMSe2rCXPiCsStFjnJMict-Fj0J_0XqJgqBApJrIqQru8uVDGQjKaCKfFh7YxmKk8d4MphBZY8DjVQpZUx2h0YXRmZKwzZMiTc2gslgu8AJKkudLCYiGu4rTUiSqpKUVWUqU0K7S6hJYLxmy108yYhThc_d19Dccu2ruLhDfQqNZbvIUj81V9btZ3fqC_AYlFoPc |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PT8IwGG0QTfSECsbf9uDRwdp1a2u8IQQiAtGZcCNt1xEvg8Dw77ctBWPixVO3npavP77vrX3vAXCf0STHKOcBDxU2ACWLAp5wGqCYIaokt6pxzmyCDodsMuHjCnjYcWG01u7ymW7aR3eWn83V2v4qa3ELdiK-B_ZjQjDasLW2s4cQalKfLf79PpwQ5x5pEqbBSWbaWlqX9doNMU-wV3vavseeVYdC3uq33zsmTTuaFkZNd3j2y3vFpZ5u7X8ffQwaPxw-ON5lpxNQ0cUpqG1NHKBf03Xw9DaXo_QRmma9Ku3GF4ys9rGpRGFqJTiKGTSFLXzWegG9GusMeqHzBvjodtJ2L_CWCoHALC4DbRY4ZUIZUIYiZtX4tGmUjkWS2OFkTDKd0ZBIxnMkQ61kpmSseChjjTWNzkC1mBf6HMCIJEIyg4aoCEkuI5EjlbM4R0JInElxAeo2GNPFRjVj6uNw-Xf3HTjspa-D6aA_fLkCRzbym2uF16BaLtf6Bhyor_Jztbx1g_4NH52kPg |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%2F+International+Conference+on+Software+Engineering&rft.atitle=RobOT%3A+Robustness-Oriented+Testing+for+Deep+Learning+Systems&rft.au=Wang%2C+Jingyi&rft.au=Chen%2C+Jialuo&rft.au=Sun%2C+Youcheng&rft.au=Ma%2C+Xingjun&rft.date=2021-05-01&rft.pub=IEEE&rft.isbn=9781665402965&rft.issn=1558-1225&rft.spage=300&rft.epage=311&rft_id=info:doi/10.1109%2FICSE43902.2021.00038&rft.externalDocID=9402039 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1558-1225&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1558-1225&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1558-1225&client=summon |