DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks

In the past decade deep neural networks (DNNs) have shown state-of-the-art performance on a wide range of complex machine learning tasks. Many of these results have been achieved while growing the size of DNNs, creating a demand for efficient compression and transmission of them. In this work we pre...

Full description

Saved in:
Bibliographic Details
Published in:IEEE journal of selected topics in signal processing Vol. 14; no. 4; pp. 700 - 714
Main Authors: Wiedemann, Simon, Kirchhoffer, Heiner, Matlage, Stefan, Haase, Paul, Marban, Arturo, Marinc, Talmaj, Neumann, David, Nguyen, Tung, Schwarz, Heiko, Wiegand, Thomas, Marpe, Detlev, Samek, Wojciech
Format: Journal Article
Language:English
Published: New York IEEE 01.05.2020
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithms
arithmetic coding
Artificial neural networks
Coding
Coding standards
Cognitive tasks
Compression algorithms
Decoding
Deep learning
efficient representation
Machine learning
Measurement
neural network compression
Neural networks
Quantization (signal)
rate-distortion quantization
Source code
Source coding
Task complexity
Video compression
ISSN:1932-4553, 1941-0484
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In the past decade deep neural networks (DNNs) have shown state-of-the-art performance on a wide range of complex machine learning tasks. Many of these results have been achieved while growing the size of DNNs, creating a demand for efficient compression and transmission of them. In this work we present DeepCABAC, a universal compression algorithm for DNNs that is based on applying Context-based Adaptive Binary Arithmetic Coder (CABAC) to the DNN parameters. CABAC was originally designed for the H.264/AVC video coding standard and became the state-of-the-art for the lossless compression part of video compression. DeepCABAC applies a novel quantization scheme that minimizes a rate-distortion function while simultaneously taking the impact of quantization to the DNN performance into account. Experimental results show that DeepCABAC consistently attains higher compression rates than previously proposed coding techniques for DNN compression. For instance, it is able to compress the VGG16 ImageNet model by x63.6 with no loss of accuracy, thus being able to represent the entire network with merely 9 MB. The source code for encoding and decoding can be found at https://github.com/fraunhoferhhi/DeepCABAC .
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1932-4553
1941-0484
DOI:10.1109/JSTSP.2020.2969554