DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems
Deep learning (DL) defines a new data-driven programming paradigm that constructs the internal system logic of a crafted neuron network through a set of training data. We have seen wide adoption of DL in many safety-critical scenarios. However, a plethora of studies have shown that the state-of-the-...
Gespeichert in:
| Veröffentlicht in: | 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE) S. 120 - 131 |
|---|---|
| Hauptverfasser: | , , , , , , , , , , , |
| Format: | Tagungsbericht |
| Sprache: | Englisch |
| Veröffentlicht: |
ACM
01.09.2018
|
| Schlagworte: | |
| ISSN: | 2643-1572 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | Deep learning (DL) defines a new data-driven programming paradigm that constructs the internal system logic of a crafted neuron network through a set of training data. We have seen wide adoption of DL in many safety-critical scenarios. However, a plethora of studies have shown that the state-of-the-art DL systems suffer from various vulnerabilities which can lead to severe consequences when applied to real-world applications. Currently, the testing adequacy of a DL system is usually measured by the accuracy of test data. Considering the limitation of accessible high quality test data, good accuracy performance on test data can hardly provide confidence to the testing adequacy and generality of DL systems. Unlike traditional software systems that have clear and controllable logic and functionality, the lack of interpretability in a DL system makes system analysis and defect detection difficult, which could potentially hinder its real-world deployment. In this paper, we propose DeepGauge, a set of multi-granularity testing criteria for DL systems, which aims at rendering a multi-faceted portrayal of the testbed. The in-depth evaluation of our proposed testing criteria is demonstrated on two well-known datasets, five DL systems, and with four state-of-the-art adversarial attack techniques against DL. The potential usefulness of DeepGauge sheds light on the construction of more generic and robust DL systems. |
|---|---|
| AbstractList | Deep learning (DL) defines a new data-driven programming paradigm that constructs the internal system logic of a crafted neuron network through a set of training data. We have seen wide adoption of DL in many safety-critical scenarios. However, a plethora of studies have shown that the state-of-the-art DL systems suffer from various vulnerabilities which can lead to severe consequences when applied to real-world applications. Currently, the testing adequacy of a DL system is usually measured by the accuracy of test data. Considering the limitation of accessible high quality test data, good accuracy performance on test data can hardly provide confidence to the testing adequacy and generality of DL systems. Unlike traditional software systems that have clear and controllable logic and functionality, the lack of interpretability in a DL system makes system analysis and defect detection difficult, which could potentially hinder its real-world deployment. In this paper, we propose DeepGauge, a set of multi-granularity testing criteria for DL systems, which aims at rendering a multi-faceted portrayal of the testbed. The in-depth evaluation of our proposed testing criteria is demonstrated on two well-known datasets, five DL systems, and with four state-of-the-art adversarial attack techniques against DL. The potential usefulness of DeepGauge sheds light on the construction of more generic and robust DL systems. |
| Author | Li, Bo Chen, Chunyang Li, Li Juefei-Xu, Felix Sun, Jiyuan Ma, Lei Zhang, Fuyuan Liu, Yang Wang, Yadong Su, Ting Xue, Minhui Zhao, Jianjun |
| Author_xml | – sequence: 1 givenname: Lei surname: Ma fullname: Ma, Lei organization: Nanyang Technological University,Singapore – sequence: 2 givenname: Felix surname: Juefei-Xu fullname: Juefei-Xu, Felix organization: Carnegie Mellon University,USA – sequence: 3 givenname: Fuyuan surname: Zhang fullname: Zhang, Fuyuan organization: Nanyang Technological University,Singapore – sequence: 4 givenname: Jiyuan surname: Sun fullname: Sun, Jiyuan organization: Kyushu University,Japan – sequence: 5 givenname: Minhui surname: Xue fullname: Xue, Minhui organization: Nanyang Technological University,Singapore – sequence: 6 givenname: Bo surname: Li fullname: Li, Bo organization: University of Illinois,Urbana-Champaign,USA – sequence: 7 givenname: Chunyang surname: Chen fullname: Chen, Chunyang organization: Monash University,Australia – sequence: 8 givenname: Ting surname: Su fullname: Su, Ting organization: Nanyang Technological University,Singapore – sequence: 9 givenname: Li surname: Li fullname: Li, Li organization: Monash University,Australia – sequence: 10 givenname: Yang surname: Liu fullname: Liu, Yang organization: Nanyang Technological University,Singapore – sequence: 11 givenname: Jianjun surname: Zhao fullname: Zhao, Jianjun organization: Kyushu University,Japan – sequence: 12 givenname: Yadong surname: Wang fullname: Wang, Yadong organization: Harbin Institute of Technology,China |
| BookMark | eNotT7tOwzAUNQgk2tKZgcU_kHLt60fChkIJSEEMFImtcp2bKihNKzsZ-vckgunoPKUzZ1fdsSPG7gSshFD6ASWmQtnVhBLkBZuPKqDO0H5fspk0ChOhrbxhyxh_AECmFrTUM1Y8E50KN-zpkb8Pbd8kRXDd0LrQ9Ge-odg33Z7nI6PQOF4fA58avCQXusn6PMeeDvGWXdeujbT8xwX7ellv8tek_Cje8qcycdLqPkHvTeVSKy3QDkzmMyOMttqrzEhSwiFW4HdVrYzRUgkCRFP7ymuhUgCPC3b_t9sQ0fYUmoML5202XhqT-AsR_Ewk |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1145/3238147.3238202 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Xplore POP ALL IEEE Xplore All Conference Proceedings IEEE/IET Electronic Library IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 145035937X 9781450359375 |
| EISSN | 2643-1572 |
| EndPage | 131 |
| ExternalDocumentID | 9000033 |
| Genre | orig-research |
| GroupedDBID | 29I 6IE 6IF 6IH 6IK 6IL 6IM 6IN 6J9 AAJGR AAWTH ABLEC ACREN ADYOE ADZIZ AFYQB ALMA_UNASSIGNED_HOLDINGS AMTXH APO BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IPLJI M43 OCL RIE RIL |
| ID | FETCH-LOGICAL-a275t-3cc6da87270eb069c9616575c4962e41a33d0cbdf4665241e0336fcdc514800c3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 477 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000553784500015&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:12:42 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a275t-3cc6da87270eb069c9616575c4962e41a33d0cbdf4665241e0336fcdc514800c3 |
| PageCount | 12 |
| ParticipantIDs | ieee_primary_9000033 |
| PublicationCentury | 2000 |
| PublicationDate | 2018-Sept. |
| PublicationDateYYYYMMDD | 2018-09-01 |
| PublicationDate_xml | – month: 09 year: 2018 text: 2018-Sept. |
| PublicationDecade | 2010 |
| PublicationTitle | 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE) |
| PublicationTitleAbbrev | ASE |
| PublicationYear | 2018 |
| Publisher | ACM |
| Publisher_xml | – name: ACM |
| SSID | ssj0002870525 ssj0051577 |
| Score | 2.6058493 |
| Snippet | Deep learning (DL) defines a new data-driven programming paradigm that constructs the internal system logic of a crafted neuron network through a set of... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 120 |
| SubjectTerms | Deep learning Deep neural networks Rendering (computer graphics) Software engineering Software systems Software testing System analysis and design Testing Testing criteria Training data |
| Title | DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems |
| URI | https://ieeexplore.ieee.org/document/9000033 |
| WOSCitedRecordID | wos000553784500015&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwED2VioGpQIv4lgdG3NqxYyesQMtUdShSt8qxLxVLWpWW348vDS0DC1MiS5ESJ9F7urv3HsCDlEEWQjruFRquVRm4Q-V5ppMQlwqpnKvDJux4nM1m-aQFj3stDCLWw2fYp9O6lx-WfkulsgEFXAqljuDIWrvTau3rKdSwSw8-exGmrW2sfKROB4qgSds-HROqofzKUqmhZNj5302cQu-gyWOTPdqcQQurc-j8hDKw5h_twugFcTVy2wU-sVpdy0cRjmjYNPJtNiVTjWrBKOGAvj0WOSujK1hjtLpgjYd5D96Hr9PnN96kJXCX2HTDlfcmuCzyEYGFMLnPjaSuite5SVBLp1QQvgilNiaNuI3xIUzpg4-UKbJGry6gXS0rvCQdd6QRwQrjdaLLQhTOuVCGVOVUogjmCrq0L_PVzhBj3mzJ9d_LN3ASWUa2G8y6hfZmvcU7OPZfm4_P9X39Fr8BCGechA |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NT8JAEJ0gmugJFY3f7sGjC93udtt6VQEjEg6YcCPb3SnxUgiCv9-dUsGDF09tNmnSbtu8l5l57wHcCeFEFgjDrUTNlcwdNygtT1To_FImpDFl2EQ8GCTjcTqswf1GC4OI5fAZtui07OW7mV1RqaxNAZeBlDuwGykVirVaa1NRoZZdtHXa80Adx5WZj1BRWxI4qbhFx5CqKL_SVEow6TT-dxuHcLJV5bHhBm-OoIbFMTR-YhlY9Zc2ofuEOO-a1RQfWKmv5V0PSDRu6hk3G5GtRjFllHFAXx_zrJXRFayyWp2yysX8BN47z6PHHq_yErgJ42jJpbXamcQzkgCzQKc21YL6KlalOkQljJQusJnLldaRR270D6Fz66wnTZ43WnkK9WJW4BkpuT2RcHGgrQpVngWZMcblLpIpFSmcPocm7ctkvrbEmFRbcvH38i3s90Zv_Un_ZfB6CQeecyTrMa0rqC8XK7yGPfu1_Phc3JRv9Bte9Z_L |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=2018+33rd+IEEE%2FACM+International+Conference+on+Automated+Software+Engineering+%28ASE%29&rft.atitle=DeepGauge%3A+Multi-Granularity+Testing+Criteria+for+Deep+Learning+Systems&rft.au=Ma%2C+Lei&rft.au=Juefei-Xu%2C+Felix&rft.au=Zhang%2C+Fuyuan&rft.au=Sun%2C+Jiyuan&rft.date=2018-09-01&rft.pub=ACM&rft.eissn=2643-1572&rft.spage=120&rft.epage=131&rft_id=info:doi/10.1145%2F3238147.3238202&rft.externalDocID=9000033 |