A review on the attention mechanism of deep learning

Attention has arguably become one of the most important concepts in the deep learning field. It is inspired by the biological systems of humans that tend to focus on the distinctive parts when processing large amounts of information. With the development of deep neural networks, attention mechanism...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Neurocomputing (Amsterdam) Ročník 452; s. 48 - 62
Hlavní autori: Niu, Zhaoyang, Zhong, Guoqiang, Yu, Hui
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Elsevier B.V 10.09.2021
Predmet:
Attention mechanism
Computer vision applications
Convolutional Neural Network (CNN)
Deep learning
Encoder-decoder
Natural language processing applications
Recurrent Neural Network (RNN)
Unified attention model
Deep learning
Attention mechanism
Recurrent Neural Network (RNN)
Natural language processing applications
Unified attention model
Computer vision applications
Encoder-decoder
Convolutional Neural Network (CNN)
ISSN:0925-2312, 1872-8286
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Attention has arguably become one of the most important concepts in the deep learning field. It is inspired by the biological systems of humans that tend to focus on the distinctive parts when processing large amounts of information. With the development of deep neural networks, attention mechanism has been widely used in diverse application domains. This paper aims to give an overview of the state-of-the-art attention models proposed in recent years. Toward a better general understanding of attention mechanisms, we define a unified model that is suitable for most attention structures. Each step of the attention mechanism implemented in the model is described in detail. Furthermore, we classify existing attention models according to four criteria: the softness of attention, forms of input feature, input representation, and output representation. Besides, we summarize network architectures used in conjunction with the attention mechanism and describe some typical applications of attention mechanism. Finally, we discuss the interpretability that attention brings to deep learning and present its potential future trends.
ISSN:0925-2312
1872-8286
DOI:10.1016/j.neucom.2021.03.091