Approximate Softmax Functions for Energy-Efficient Deep Neural Networks

Approximate computing has emerged as a new paradigm that provides power-efficient and high-performance arithmetic designs by relaxing the stringent requirement of accuracy. Nonlinear functions (such as softmax , rectified linear unit ( ReLU ), Tanh , and Sigmoid ) are extensively used in deep neural...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE transactions on very large scale integration (VLSI) systems Ročník 31; číslo 1; s. 1 - 13
Hlavní autoři: Chen, Ke, Gao, Yue, Waris, Haroon, Liu, Weiqiang, Lombardi, Fabrizio
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York IEEE 01.01.2023
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Témata:
Accuracy
Adders
Approximate computing
Artificial neural networks
Circuit design
Complexity
Deep neural network (DNN)
Dividers
Energy dissipation
field-programmable gate array (FPGA) implementation
Hardware
Hybrid structures
Linear functions
Logic gates
Lookup tables
Mathematical analysis
Neural networks
nonlinear function
Power management
softmax layer
Table lookup
Taylor series
ISSN:1063-8210, 1557-9999
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Approximate computing has emerged as a new paradigm that provides power-efficient and high-performance arithmetic designs by relaxing the stringent requirement of accuracy. Nonlinear functions (such as softmax , rectified linear unit ( ReLU ), Tanh , and Sigmoid ) are extensively used in deep neural networks (DNNs). However, they incur significant power dissipation due to the high circuit complexity. As DNNs are error-tolerant, the design of approximation-linear functions is possible and desired. In this article, the design of an approximate softmax function (AxSF) is proposed. AxSF is based on a double hybrid structure (DHS). AxSF divides the input of the softmax function into two parts for different processing methods. The most significant bits (MSBs) are processed with lookup tables (LUTs) and an exact restoring array divider (EXDr). Taylor's expansion and a logarithmic divider are used for the less significant bits (LSBs). An improved DHS (IDHS) is also proposed to reduce the hardware complexity. In IDHS, a novel Booth multiplier is utilized for the hybrid scheme to improve the partial product generation and compression, while the truncated implementation is applied to the divider unit. The proposed DHS and IDHS are compared with existing softmax designs. The results show that the proposed approximate softmax design reduces hardware by 48% and delay by 54% while retaining a high accuracy.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1063-8210
1557-9999
DOI:10.1109/TVLSI.2022.3224011