ReTransformer: ReRAM-based Processing-in-Memory Architecture for Transformer Acceleration

Transformer has emerged as a popular deep neural network (DNN) model for Neural Language Processing (NLP) applications and demonstrated excellent performance in neural machine translation, entity recognition, etc. However, its scaled dot-product attention mechanism in auto-regressive decoder brings...

Celý popis

Uloženo v:

Podrobná bibliografie
Vydáno v:	Digest of technical papers - IEEE/ACM International Conference on Computer-Aided Design s. 1 - 9
Hlavní autoři:	Yang, Xiaoxuan, Yan, Bonan, Li, Hai, Chen, Yiran
Médium:	Konferenční příspěvek
Jazyk:	angličtina
Vydáno:	Association on Computer Machinery 02.11.2020
Témata:	Acceleration autoregressive decoder Computational modeling Computer architecture convolutional neural networks Decoding deep neural network model hardware acceleration solution learning (artificial intelligence) mathematics computing matrix decomposition matrix multiplication matrix-matrix multiplication memory architecture multi-threading natural language processing neural language processing applications neural machine translation performance evaluation Pipelines processing-in-memory recurrent neural nets recurrent neural networks ReRAM ReRAM-based PIM architecture ReRAM-based processing-in-memory architecture ReTransformer scaled dot-product attention mechanism submatrix pipeline design Transformer Virtual machine monitors
ISSN:	1558-2434
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Popis
Shrnutí:	Transformer has emerged as a popular deep neural network (DNN) model for Neural Language Processing (NLP) applications and demonstrated excellent performance in neural machine translation, entity recognition, etc. However, its scaled dot-product attention mechanism in auto-regressive decoder brings a performance bottleneck during inference. Transformer is also computationally and memory intensive and demands for a hardware acceleration solution. Although researchers have successfully applied ReRAM-based Processing-in-Memory (PIM) to accelerate convolutional neural networks (CNNs) and recurrent neural networks (RNNs), the unique computation process of the scaled dot-product attention in Transformer makes it difficult to directly apply these designs. Besides, how to handle intermediate results in Matrix-matrix Multiplication (MatMul) and how to design a pipeline at a finer granularity of Transformer remain unsolved. In this work, we propose ReTransformer - a ReRAM-based PIM architecture for Transformer acceleration. ReTransformer can not only accelerate the scaled dot-product attention of Transformer using ReRAM-based PIM but also eliminate some data dependency by avoiding writing the intermediate results using the proposed matrix decomposition technique. Moreover, we propose a new sub-matrix pipeline design for multi-head self-attention. Experimental results show that compared to GPU and Pipelayer, ReTransformer improves computing efficiency by 23.21× and 3.25×, respectively. The corresponding overall power is reduced by 1086× and 2.82×, respectively.
ISSN:	1558-2434
DOI:	10.1145/3400302.3415640