Deep neural networks compression learning based on multiobjective evolutionary algorithms

This work addresses the problem of deep neural network compression, which is a promising technique to reduce the number of network parameters and/or speed up the network evaluation process significantly. Most of the existing methods rely on domain experts’ experience for the selection of hyperparame...

Full description

Saved in:
Bibliographic Details
Published in:Neurocomputing (Amsterdam) Vol. 378; pp. 260 - 269
Main Authors: Huang, Junhao, Sun, Weize, Huang, Lei
Format: Journal Article
Language:English
Published: Elsevier B.V 22.02.2020
Subjects:
Evolutionary optimization
Expensive optimization
Neural network compression
Pareto-optimal solutions
Expensive optimization
Pareto-optimal solutions
Evolutionary optimization
Neural network compression
ISSN:0925-2312, 1872-8286
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This work addresses the problem of deep neural network compression, which is a promising technique to reduce the number of network parameters and/or speed up the network evaluation process significantly. Most of the existing methods rely on domain experts’ experience for the selection of hyperparameters such as the rank and sparsity ratio of the weight matrix in order to get an appropriate compression result without serious performance downgrade. However, they usually suffer from heavy computational loads due to the large number of tests in revealing the best hyperparameters. In this work, we propose an efficient approach to network compression from the perspective of multiobjective evolution. The contributions in the paper are twofolds: (1) We build a multiobjective compression learning model that considers the model classification error rate and compression rate as two objectives in the optimization, which can provide a compromise of the tradeoffs between these two objectives. (2) A mechanism for approximate compressed model generation is devised in the framework of expensive multiobjective optimization, which is able to reduce the high model training costs involved in the optimization process. Experiments are carried out to confirm the superiority of the proposed algorithm.
ISSN:0925-2312
1872-8286
DOI:10.1016/j.neucom.2019.10.053