Recurrent neural network from adder’s perspective: Carry-lookahead RNN

The recurrent network architecture is a widely used model in sequence modeling, but its serial dependency hinders the computation parallelization, which makes the operation inefficient. The same problem was encountered in serial adder at the early stage of digital electronics. In this paper, we disc...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Neural networks Ročník 144; s. 297 - 306
Hlavní autori: Jiang, Haowei, Qin, Feiwei, Cao, Jin, Peng, Yong, Shao, Yanli
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Elsevier Ltd 01.12.2021
Predmet:
Carry-lookahead
Deep learning
Parallel computation
Sequence modeling
Deep learning
Sequence modeling
Parallel computation
Carry-lookahead
ISSN:0893-6080, 1879-2782, 1879-2782
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:The recurrent network architecture is a widely used model in sequence modeling, but its serial dependency hinders the computation parallelization, which makes the operation inefficient. The same problem was encountered in serial adder at the early stage of digital electronics. In this paper, we discuss the similarities between recurrent neural network (RNN) and serial adder. Inspired by carry-lookahead adder, we introduce carry-lookahead module to RNN, which makes it possible for RNN to run in parallel. Then, we design the method of parallel RNN computation, and finally Carry-lookahead RNN (CL-RNN) is proposed. CL-RNN takes advantages in parallelism and flexible receptive field. Through a comprehensive set of tests, we verify that CL-RNN can perform better than existing typical RNNs in sequence modeling tasks which are specially designed for RNNs. Code and models are available at: https://github.com/WinnieJiangHW/Carry-lookahead_RNN.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:0893-6080
1879-2782
1879-2782
DOI:10.1016/j.neunet.2021.08.032