Model Compression Algorithm via Reinforcement Learning and Knowledge Distillation

Traditional model compression techniques are dependent on handcrafted features and require domain experts, with a tradeoff between model size, speed, and accuracy. This study proposes a new approach toward resolving model compression problems. Our approach combines reinforcement-learning-based autom...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Mathematics (Basel) Ročník 11; číslo 22; s. 4589
Hlavní autori: Liu, Botao, Hu, Bing-Bing, Zhao, Ming, Peng, Sheng-Lung, Chang, Jou-Ming
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Basel MDPI AG 01.11.2023
Predmet:
Accuracy
Algorithms
attention mechanism
automatic pruning
Computation
Computer peripherals
Decomposition
Distillation
Energy consumption
Knowledge
knowledge distillation
Learning
Machine learning
Methods
model compression
network efficiency
Neural networks
reinforcement learning
ISSN:2227-7390, 2227-7390
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Traditional model compression techniques are dependent on handcrafted features and require domain experts, with a tradeoff between model size, speed, and accuracy. This study proposes a new approach toward resolving model compression problems. Our approach combines reinforcement-learning-based automated pruning and knowledge distillation to improve the pruning of unimportant network layers and the efficiency of the compression process. We introduce a new state quantity that controls the size of the reward and an attention mechanism that reinforces useful features and attenuates useless features to enhance the effects of other features. The experimental results show that the proposed model is superior to other advanced pruning methods in terms of the computation time and accuracy on CIFAR-100 and ImageNet dataset, where the accuracy is approximately 3% higher than that of similar methods with shorter computation times.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2227-7390
2227-7390
DOI:10.3390/math11224589