Knowledge distillation vulnerability of DeiT through CNN adversarial attack

In the field of computer vision, active research is conducted to improve model performance. The successful application of transformer models in computer vision has led to the development of new models that incorporate this structure. However, the security vulnerabilities of these new models against...

Full description

Saved in:
Bibliographic Details
Published in:Neural computing & applications Vol. 37; no. 12; pp. 7721 - 7731
Main Authors: Hong, Inpyo, Choi, Chang
Format: Journal Article
Language:English
Published: London Springer London 01.04.2025
Springer Nature B.V
Subjects:
Artificial Intelligence
Black boxes
Computational Biology/Bioinformatics
Computational Science and Engineering
Computer Science
Computer vision
Data Mining and Knowledge Discovery
Image Processing and Computer Vision
Probability and Statistics in Computer Science
S.I.: From Theory to Practice: Real-World Applications of AI in Data Science
Special Issue on From Theory to Practice: Real-World Applications of AI in Data Science
Adversarial vulnerability
Adversarial attack
Knowledge distillation
Data-efficient image transformer
ISSN:0941-0643, 1433-3058
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In the field of computer vision, active research is conducted to improve model performance. The successful application of transformer models in computer vision has led to the development of new models that incorporate this structure. However, the security vulnerabilities of these new models against adversarial attacks have not yet been thoroughly examined. This study investigated the adversarial attack vulnerabilities of DeiT, a model that combines CNN and transformer models through knowledge distillation techniques. We propose that even with only the teacher model (CNN model) information, a fatal attack on DeiT is possible, defining this attack scenario as a partial-white-box environment. In addition, owing to the integration of both CNN’s local information and the transformer’s global information, DeiT is more susceptible to attacks in a black-box environment than other models. The experimental results demonstrate that when adversarial examples (AEs) generated by the teacher model are inserted into DeiT, Fast Gradient Sign Method (FGSM) causes a 46.49% decrease in accuracy, Projected Gradient Descent (PGD) results in a 65.59% decrease. Furthermore, in a black-box environment, AEs generated by ViT and ResNet-50 have detrimental effects on DeiT. Notably, both the CNN and transformer models induced fatal FGSM attacks on DeiT, resulting in vulnerabilities of 70.49% and 53.59%, respectively. These findings demonstrate the additional vulnerability of DeiT to black-box attacks. Moreover, it highlights that DeiT poses a greater risk in practical applications compared to other models. Based on these vulnerabilities, we hope knowledge distillation research with enhanced adversarial robustness will be actively conducted.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0941-0643
1433-3058
DOI:10.1007/s00521-023-09297-z